Directional pickup method and device, AR glasses, equipment, medium and product

Abstract

The invention provides a directional pickup method and device, AR glasses, equipment, a medium and a product. The method comprises the following steps: acquiring a plurality of voice signals collected by a plurality of microphones on the AR glasses; performing beam forming according to the plurality of voice signals to obtain voice beams in different directions; target speakers who are speaking currently are recognized according to the voice beams in the different directions, translation results corresponding to the voice beams of the target speakers are output, and the target speakers comprise the user and / or candidate speakers who are dialogue with the user. According to the invention, the pickup accuracy of the target voice can be improved, and the accuracy of voice translation is improved.

Description

Technical Field

[0001] This application relates to the field of voice signal processing, and particularly to a directional sound pickup method, apparatus, AR glasses, device, medium and product. Background Art

[0002] In scenarios where multi-party conversations are carried out using AR glasses, such as the conversation translation scenario of AR glasses, it is not only necessary to capture the user's voice, but also to accurately pick up the voice of the speaker who is currently talking to the user. Especially in the presence of background noise interference, or when the user and the interlocutor speak simultaneously, the voices are likely to be confused, thus affecting the accuracy of the automatic speech recognition (ASR) module. In addition, since multiple speakers may be located in different directions, it is also necessary to accurately judge and select the target voice from the correct direction.

[0003] Therefore, there is an urgent need for a method that can effectively separate the target voice to improve the accuracy of voice translation. Summary of the Invention

[0004] Based on the above technical status quo, this application provides a directional sound pickup method, apparatus, AR glasses, device, medium and product, which can improve the accuracy of picking up the target voice, and further improve the accuracy of voice translation.

[0005] To achieve the above technical objectives, the present application specifically proposes the following technical solutions:

[0006] According to the first aspect of the embodiments of the present application, a directional sound pickup method is provided, which is applied to AR glasses. A plurality of microphones are provided on the AR glasses. The method includes: obtaining a plurality of voice signals collected by the plurality of microphones; performing beamforming according to the plurality of voice signals to obtain voice beams in different directions; identifying a target speaker who is currently speaking according to the voice beams in different directions, and outputting a translation result corresponding to the voice beam of the target speaker, where the target speaker includes the user and / or a candidate speaker who is currently having a conversation with the user.

[0007] In some implementation manners, the identifying a target speaker who is currently speaking according to the voice beams in different directions, and outputting a translation result corresponding to the voice beam of the target speaker includes: if it is identified according to the voice beams in different directions that a plurality of candidate speakers who are having a conversation with the user are all speaking, determining the target speaker according to the voice signal intensities in different directions.

[0008] In some implementations, determining the target speaker according to the voice signal intensities in the respective different directions includes: determining, as the target speaker, the first candidate speaker in the direction corresponding to the maximum voice signal intensity among the voice signal intensities in the respective different directions.

[0009] In some implementations, after determining, as the target speaker, the first candidate speaker in the direction corresponding to the maximum voice signal intensity among the voice signal intensities in the respective different directions, the method further includes: determining a second candidate speaker corresponding to the direction corresponding to the maximum voice signal intensity among the voice signal intensities in the respective different directions, where the second candidate speaker and the first candidate speaker correspond to different speakers respectively; and determining whether to switch the second candidate speaker to be the target speaker according to a comparison result between the voice signal intensity corresponding to the second candidate speaker and the voice signal intensity corresponding to the first candidate speaker.

[0010] In some implementations, determining whether to switch the second candidate speaker to be the target speaker according to a comparison result between the voice signal intensity corresponding to the second candidate speaker and the voice signal intensity corresponding to the first candidate speaker includes: if a difference between the voice signal intensity corresponding to the second candidate speaker and the voice signal intensity corresponding to the first candidate speaker is greater than or equal to a preset difference, determining to switch the second candidate speaker to be the target speaker.

[0011] In some implementations, the multiple microphones correspond to respective directional beams, and each of the respective directional beams corresponds to a beam coefficient; wherein, forming voice beams in respective different directions according to the multiple voice signals includes: determining voice beams of the multiple voice signals in the respective different directions according to the multiple voice signals and the beam coefficients respectively corresponding to the respective directional beams.

[0012] In some implementations, the AR glasses include an inner display screen and an outer display screen. Among them, recognizing a target speaker who is currently speaking based on each voice beam and outputting a translation result corresponding to the voice beam of the target speaker includes: if the target speaker is the user, displaying the translation result corresponding to the user's voice signal on the outer display screen; if the target speaker is a candidate speaker who is having a conversation with the user, displaying the translation result corresponding to the voice signal of the candidate speaker who is having a conversation with the user on the inner display screen; if the target speaker is the user and a candidate speaker who is having a conversation with the user, displaying the translation result corresponding to the user's voice signal on the outer display screen and displaying the translation result corresponding to the voice signal of the candidate speaker who is having a conversation with the user on the inner display screen.

[0013] According to a second aspect of the embodiments of the present application, a directional sound pickup device is provided, including: an acquisition unit configured to acquire a plurality of voice signals collected by a plurality of microphones provided on the AR glasses; a beamforming unit configured to perform beamforming on the plurality of voice signals to obtain voice beams in respective different directions; a speaker recognition and output unit configured to recognize a target speaker who is currently speaking based on the voice beams in the respective different directions and output a translation result corresponding to the voice beam of the target speaker, where the target speaker includes the user and / or a candidate speaker who is having a conversation with the user.

[0014] According to a third aspect of the embodiments of the present application, an AR glasses is provided, including a glasses frame, a plurality of microphones provided on the glasses frame and an outer display screen, and the directional sound pickup device as described in the second aspect; the plurality of microphones are configured to collect voice signals of the user and surrounding speakers to obtain a plurality of voice signals; the directional sound pickup device is configured to perform the directional sound pickup method as described in the first aspect on the plurality of voice signals to obtain a translation result of the target speaker and / or a translation result of the user; the translation result of the user is displayed through the outer display screen, and the translation result of the target speaker is displayed through an inner display screen of the AR glasses, and the inner display screen is a virtual screen.

[0015] According to a fourth aspect of the embodiments of the present application, an electronic device is provided, including a memory and a processor; the memory is connected to the processor and configured to store a program; the processor is configured to implement the directional sound pickup method as described in the first aspect by running the program in the memory.

[0016] According to a fifth aspect of the embodiments of the present application, a storage medium is provided, on which a computer program is stored, and when the computer program is run by a processor, the directional sound pickup method as described in the first aspect is implemented.

[0017] According to a sixth aspect of the embodiments of the present application, there is provided a computer program product, including computer program instructions, which when run by a processor cause the processor to execute: the directional sound pickup method as described in the first aspect.

[0018] A directional sound pickup method, apparatus, AR glasses, device, medium and product provided by the embodiments of the present application. The method acquires multiple voice signals collected by multiple microphones configured on the AR glasses, and performs beamforming on the multiple voice signals to obtain voice beams in different directions, thereby being able to effectively separate sounds in different directions and improve the pickup accuracy of the target voice. Then, according to each voice beam, the target speaker who is currently speaking is identified, and the translation result corresponding to the voice beam of the target speaker is output, which can ensure the correct voice translation result of the speaker is output, and further improve the accuracy of voice translation and output. Description of the Drawings

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained according to the provided drawings without creative efforts.

[0020] Figure 1 It is a flowchart of a directional sound pickup method provided by the embodiments of the present application;

[0021] Figure 2 It is a schematic structural diagram of an AR glasses provided by the embodiments of the present application;

[0022] Figure 3 It is a schematic diagram of a microphone array beam provided by the embodiments of the present application;

[0023] Figure 4 It is a schematic diagram of the direction pointed by the microphone array beam in a three-dimensional coordinate provided by the embodiments of the present application;

[0024] Figure 5 It is a schematic diagram of the principle of beamforming provided by the embodiments of the present application;

[0025] Figure 6 It is a schematic structural diagram of a directional sound pickup apparatus provided by the embodiments of the present application;

[0026] Figure 7 It is a schematic structural diagram of an electronic device provided by the embodiments of the present application. Detailed Embodiments

[0027] The technical solutions provided by the embodiments of the present application can be exemplarily applied to hardware devices such as processors, electronic devices, and servers (including cloud servers), or packaged as software programs to be run. When the hardware device executes the processing procedures of the technical solutions of the embodiments of the present application, or when the above software program is run, the automatic splitting of the target task and the automatic invocation of the application programming interfaces required for the task can be realized, so as to achieve the purpose of completing the target task. The embodiments of the present application only exemplarily introduce the specific processing procedures of the technical solutions of the present application, and do not limit the specific implementation forms of the technical solutions of the present application. Any technical implementation form that can execute the processing procedures of the technical solutions of the present application can be adopted by the embodiments of the present application.

[0028] Next, the technical solutions in the embodiments of the present application will be clearly and completely described with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0029] Before introducing the solutions of the present application, the related technologies will be introduced first:

[0030] AR translation glasses are a novel wearable device that combines augmented reality (AR) and instant language translation functions. They are usually designed to have an appearance similar to ordinary glasses, are comfortable to wear, and can directly display the translated text information in the user's field of vision without the user having to look at an additional screen or handheld device.

[0031] In practical applications, when a user communicates with a person speaking a different language through AR translation glasses, the built-in microphone will capture the other person's speech and use speech recognition technology and natural language processing algorithms to convert the sound into text and perform real-time translation. Subsequently, the translated text will be displayed within the user's line of sight through the display screen inside the glasses, ensuring that the user can easily read the meaning of the other person's words without affecting normal vision, providing the user with real-time translation services.

[0032] To further enhance the user experience, an AR translation glasses with a dual-screen design is proposed. The dual-screen AR glasses can not only translate the language of the interlocutor and display it on the inner screen for the wearer to view, but also have a specially designed outer screen that can synchronously display the translated version of the words spoken by the wearer, enabling the opposite interlocutor to also understand immediately. Thus, a two-way communication mechanism is provided to enhance the fluency and interactivity of cross-language communication, enabling both parties to express and receive information more clearly and promoting more effective communication.

[0033] However, in the context of multi-person conversations, when there are other surrounding sounds interfering, or when the user and the interlocutor speak simultaneously, it is easy to have the problem of sound confusion, resulting in the sound received by the automatic speech recognition (ASR) module containing interfering sounds. In addition, since the conversation participants may be located in different directions, it is also necessary to further judge and accurately pick up the voices of the interlocutors from the correct direction. Especially on AR translation glasses with dual-screen displays, since it is necessary to simultaneously recognize the voices of the user and the interlocutor and then display the translated text on the inner and outer screens respectively, if there is sound confusion, it will lead to confusion in the translated content.

[0034] In view of this, the embodiments of the present application are committed to providing a directional sound pickup method, device, AR glasses, device, medium and product. This method first processes the sound signals from various directions and uses an audio separation algorithm to decompose these mixed sounds into independent sound sources. Then, it determines which sound source is the object with whom the user is communicating and preferentially picks up the voice of this speaker for subsequent processing. After determining the target sound source, the captured voice of the speaker will be translated in real time and displayed on the inner screen of the AR glasses for the user to read; at the same time, the user's response will also be translated in real time and shown to the other party through the outer screen. Thus, it enhances the recognition accuracy of the speaker's identity in multi-person conversation scenarios and ensures the accuracy of voice translation. The technical solutions of the embodiments of the present application are not only applicable to personal communication, but also can be widely applied to business meetings, international events and other occasions that require instant translation support. They will be described in detail one by one in the following embodiments.

[0035] Exemplary method

[0036] Figure 1 It is a flowchart of a directional sound pickup method provided by an embodiment of the present application. As Figure 1 shown, the directional sound pickup method provided in this embodiment includes steps S101 - S103:

[0037] S101. Obtain multiple voice signals collected by multiple microphones.

[0038] The method of this embodiment can be applied to AR glasses with dual-screen display functions. Figure 2 It is a schematic structural diagram of an AR glasses provided by an embodiment of the present application. As Figure 2 shown, the AR glasses include a glasses frame 21, multiple microphones arranged on the glasses frame 21, and an outer display screen 22.

[0039] Among them, the multiple microphones arranged on the glasses frame 21 are distributed in a three-dimensional array form to capture voice signals from different directions.

[0040] In some embodiments, the number of microphones can be set to 10 - 15 to provide a wide and precise sound coverage. For ease of description, the following will take 11 microphones as an example and label them as Microphone 1 to Microphone 11 in sequence.

[0041] Among them, Microphone 1 is installed on the bridge part connecting the two lenses, that is, on the connecting rod 23, serving as the center point of the entire microphone array. Microphones 2 to 5 are distributed on the inner edges of the two lenses and together with Microphone 1 form a rectangular layout. Microphone 1 is located at the geometric center of the rectangle. Microphones 6 to 11 are evenly distributed on the temple arms on both sides.

[0042] Based on the above microphone array, the following three-dimensional coordinate system can be obtained:

[0043] Set the position of Microphone 1 as the origin of the three-dimensional coordinates, then the X-axis, Y-axis, and Z-axis can be defined as follows respectively:

[0044] X-axis: The direction passing through Microphone 1 and parallel to the connecting line between the center points of the two lenses, that is, the length direction of the connecting rod 23.

[0045] Y-axis: Parallel to the connecting line between Microphone 4 and Microphone 5 and perpendicular to the direction passing through Microphone 1.

[0046] Z-axis: Parallel to the connecting line between Microphones 6 to 8 and perpendicular to the direction passing through Microphone 1.

[0047] According to the above axis definitions, the three-dimensional coordinates of each microphone are shown in Table 1 below:

[0048] Table 1 Three-dimensional coordinates of each microphone

[0049] Microphone number X-axis coordinate (cm) Y-axis coordinate (cm) Z-axis coordinate (cm) 1 0 0 0 2 2 2 0 3 2 -2 0 4 -2 -2 0 5 -2 2 0 6 6 0 0 7 6 0 2 8 6 0 4 9 -6 0 0 10 -6 0 2 11 -6 0 4

[0050] In Table 1, the unit is cm.

[0051] In a multi-person conversation scenario, these multiple microphones work together to collect the voice signals of the user and multiple speakers. The following will introduce this process in detail with reference to the accompanying drawings:

[0052] Figure 3 It is a schematic diagram of a microphone array beam provided by an embodiment of the present application. As Figure 3 shown, these multiple microphones can form N directional beams. The N directional beams include a first directional beam and multiple second directional beams. The first directional beam is used to pick up the voice signal of the user, and the multiple second directional beams are used to pick up the voice signals of multiple speakers respectively. Among them, N is a positive integer greater than 1.

[0053] The first directional beam is designed to primarily pick up the voice signals of the user. It can be understood as a user beam that points from microphone 1 to the user's mouth in the XY plane, ensuring that the clearest and most direct user voice can be captured.

[0054] Multiple second directional beams are respectively directed towards the front of the user, the front left of the user, and the front right of the user; the angle between every two of the multiple second directional beams is between 20 degrees and 30 degrees, such as 25 degrees.

[0055] In this embodiment, multiple second directional beams are responsible for respectively picking up the voice signals of multiple speakers located in different orientations. These beams are configured to cover the areas in front of the user, to the front left of the user, and to the front right of the user, so as to capture sound sources from different directions.

[0056] To ensure that each second directional beam can effectively distinguish and focus on a specific speaker, the angle between adjacent beams is set to be between 20 degrees and 30 degrees, such as 25 degrees. This angle is the balance point for both spatial resolution and coverage efficiency, which not only ensures sufficient coverage of speakers in all directions but also avoids the possibility of interference between beams.

[0057] Specifically, in the XZ plane, starting from microphone 1, three directional beams that clearly point to different speakers can be formed, namely:

[0058] Directional beam 1: Points to the candidate speaker in front of the user, and is used to capture the sound signal at this position.

[0059] Directional beam 2: Points to the candidate speaker in the front left of the user, ensuring that the sound from the left side can also be accurately captured.

[0060] Directional beam 3: Points to the candidate speaker in the front right of the user. Similarly, the sound from the right side will not be missed.

[0061] These 3 beams together form a multi-angle and all-round voice capture network. It not only improves the processing ability for multiple sound sources in a complex environment but also provides a basis for subsequent advanced processing such as sound separation, enhancement, and translation.

[0062] The beams formed by the microphone array will be introduced in detail below with reference to the accompanying drawings:

[0063] Figure 4 It is a schematic diagram of the directions of the microphone array beams provided in the embodiment of this application in a three-dimensional coordinate. As Figure 4 shown, taking the XY plane as the reference plane for the pitch angle, and taking the positive half-axis of the X axis as the starting point, the clockwise direction is the negative angle direction, and the counterclockwise rotation is the positive angle direction, as the reference plane for the horizontal angle. Then the pitch angles and horizontal angles of the 4 directional beams (the user beam and the 3 directional beams) are as shown in Table 2 below:

[0064] Table 2 Elevation Angle and Horizontal Angle of Four Directional Beams

[0065] Beam number Elevation angle Azimuth angle User beam 0 -90 Directional beam 1 90 0 Directional beam 2 65 180 Directional beam 3 65 0

[0066] In Table 2, since the user beam is the direction from microphone 1 to the user's mouth, and the angle between it and the XY plane is 0 degrees, the elevation angle of the user beam is 0 degrees; and it is -90 degrees in the clockwise direction on the positive half-axis of the X-axis. Therefore, its horizontal angle is -90 degrees. It is mainly used to pick up the user's voice signal.

[0067] The angle between directional beam 1 and the XY plane is 90 degrees, so the elevation angle of directional beam 1 is 90 degrees; and the projection point of it on the positive half-axis of the X-axis corresponds to the origin, so its horizontal angle is 0 degrees. It is mainly used to pick up the voice signal of speaker 1 directly in front of the user.

[0068] The angle between directional beam 2 and the XY plane is 65 degrees, so the elevation angle of directional beam 1 is 65 degrees; and the projection point of it on the X-axis is on the negative half-axis of the X-axis, so its horizontal angle is 180 degrees. It is mainly used to pick up the voice signal of speaker 2 diagonally in front of the user on the right.

[0069] The angle between directional beam 3 and the XY plane is 65 degrees, so the elevation angle of directional beam 1 is 65 degrees; and the projection point of it on the X-axis is on the positive half-axis of the X-axis, so its horizontal angle is 0 degrees. It is mainly used to pick up the voice signal of speaker 3 diagonally in front of the user on the left.

[0070] Through the design of the microphone array beam above, this embodiment can effectively capture the voice signals of the user and their conversation partners. The first directional beam focuses on the user himself, ensuring that his voice is recorded preferentially and with high quality; while the multiple second directional beams expand the voice perception range, so that even when multiple people speak simultaneously, the voices of each speaker can be accurately distinguished and processed.

[0071] Continue to refer to Figure 1 , after step S101, the directional sound pickup method of this embodiment may further include step S102.

[0072] S102. Perform beamforming according to multiple voice signals to obtain voice beams in different directions.

[0073] In practical applications, when the user or a candidate speaker talking to the user is speaking, there may be other candidate speakers speaking simultaneously around. Among the multiple voice signals collected by the microphone array, there are not only the voice signals of the user or the current conversation partner, but also the voice signals of other speakers from different directions.

[0074] In order to effectively separate the voice beams in different directions from multiple voice signals. In some embodiments, step S102 includes: determining the voice beams of the multiple voice signals in different directions according to the multiple voice signals and the beam coefficients corresponding to each directional beam.

[0075] The following details the specific implementation process of step S102 in conjunction with the accompanying drawings:

[0076] Figure 5 This is a schematic diagram of the principle of beamforming provided by an embodiment of the present application. As Figure 5 shown, multiple microphones are denoted as M microphones. Next, the voice signals X1 to XM collected by the M microphones are subjected to a Fast Fourier Transform (FFT) through the following formula (1) to be transformed into the frequency domain:

[0077] X(m) = FFT(mic(m)); (1)

[0078] In formula (1), mic(m) represents the time-domain voice signal collected by the mth microphone; X(m) represents the frequency-domain voice signal corresponding to the time-domain voice signal collected by the mth microphone.

[0079] After obtaining the frequency-domain voice signals corresponding to each of the M microphones, a Fixed Beamforming algorithm, a Generalized Sidelobe Canceling (GSC) algorithm, and a PostFilter algorithm can be used to process them, thereby obtaining the voice beams in different directions.

[0080] Among them, the fixed beamforming algorithm can obtain the initial voice beams in different directions, and is specifically implemented using the following formula (1):

[0081]

[0082] In formula (2), bn(f) is the initial voice beam in the nth direction; wn(f,m) is the fixed beam coefficient corresponding to the mth microphone at frequency f under the nth initial voice beam; X(f,m) is the frequency-domain representation of the mth microphone at frequency f, and f is the frequency point index after a 512-point FFT, with a range of 1 - 257.

[0083] According to the above formula (1), initial voice beams in four different directions can be obtained. Specifically, for the initial voice beams in these four different directions, the corresponding beam coefficient matrices are denoted as w1, w2, w3, and w4 respectively, each being an 11*257 matrix. Among them, in these four different directions, the respective fixed beam coefficients of the m-th microphone at the f-th frequency point are w1(f, m), w2(f, m), w3(f, m), and w4(f, m). w1, w2, w3, and w4 can be determined according to the array pattern of the microphone array. The specific determination process of w1, w2, w3, and w4 can refer to the introduction of related technologies, and will not be elaborated in this embodiment.

[0084] Assume that w1 represents the beam coefficient matrix in the user beam direction. Then when beamforming is performed using the beam coefficient matrix w1, it represents enhancing the voice signal in the user beam direction and suppressing the voice signals in other directions. Similarly, when beamforming is performed using other beam coefficient matrices, it represents enhancing the voice signals in their corresponding beam directions and suppressing the voice signals in other directions.

[0085] Through w1, w2, w3, and w4, multiple voice signals collected by M microphones can form voice beams in 4 different directions, namely the voice beams in the directions of the above four directional beams.

[0086] Due to certain leakage in the fixed beam sidelobes, there may still be a small amount of voice signals from this fixed beam received in other beam directions. For example, in an example scenario where the user is having a conversation with multiple candidate speakers, when the user and candidate speaker 1 are speaking simultaneously, the above 11 microphones will synchronously collect the voice signals from the user and candidate speaker 1, forming multiple voice signals. Then, through the beamforming algorithm, 4 directional voice beams in different directions can be obtained. Although candidate speaker 2 and candidate speaker 3 are not speaking at this time, however, due to the certain leakage in the fixed beam sidelobes, there may still be a small amount of voice signals from the user and candidate speaker 1 received in these other beam directions. Therefore, after obtaining the voice beams in different directions, in order to further improve the clarity of the target sound source, the initial voice beams can be further processed through the Generalized Sidelobe Canceller (GSC) algorithm, as follows:

[0087] Continue to refer to Figure 5 , for each initial voice beam bi, it can be used as the target signal, and the remaining voice beams in b1 - bN except bi can be used as the reference signals. Through the adaptive beamforming technology, the GSC algorithm can dynamically adjust the weights, effectively suppressing the interference noise in non-target directions, thereby significantly improving the Signal-to-Noise Ratio (SNR) of the initial voice beam bi. The algorithm process of GSC is specifically shown in formula (3):

[0088]

[0089] In Equation (3), Gsc b represents the result after applying the GSC algorithm to the nth initial speech beam, that is, the nth generalized speech beam; BM(n) is the blocking matrix, whose function is to eliminate the target signal components in the reference signal to ensure that only the sounds in non-target directions are used for noise estimation; a(n) is the Adaptive Noise Control (ANC) coefficient, which is used for dynamic adjustment to optimize the noise suppression effect.

[0090] In this embodiment, by adjusting BM(n) and a(n), the noise part in the output can be minimized, thereby significantly improving the quality of the speech signal of the target speaker.

[0091] Continuing to refer to Figure 5 , in order to further reduce the residual noise and improve the speech quality, a PostFilter algorithm can be applied to the speech signal Gsc b processed by the GSC algorithm. Thus, the preliminarily purified speech beam can be processed more finely to obtain a clearer and cleaner final speech beam. The specific process is shown in Equation (4):

[0092]

[0093] In Equation (4), Yi is the ith finally optimized speech beam; Gsc bi is the ith generalized speech beam; g(n) is the gain factor of each noise reference beam, which is used to adjust the influence degree of each reference beam on the final output; c is the total noise gain factor, which is used to overall control the degree of noise suppression.

[0094] By combining the processes of the above Equations (1) to (4), N optimized speech beams can be separated from multiple speech signals.

[0095] In this embodiment, through the Beamforming algorithm, the directivity and sensitivity of the microphone array can be dynamically adjusted, thereby enhancing the speech signal in a specific direction while suppressing the interference in other directions. On this basis, an adaptive filter is introduced to optimize the frequency response of each beam in real time, which can further reduce the influence of background noise and other non-target sounds, ensuring that the speech signal in each beam is as pure and clear as possible.

[0096] Continuing to refer to Figure 1 , after step S102, the directional sound pickup method of this embodiment may further include step S103.

[0097] S103. Identify the target speaker who is currently speaking based on the voice beams in various different directions, and output the translation result corresponding to the voice beam of the target speaker. The target speaker includes the user and / or a candidate speaker who is having a conversation with the user.

[0098] In practical applications, when a user is talking to their conversation partner, there may be interference sounds from other candidate speakers in the surrounding environment, or the situation where the user and the conversation partner are speaking simultaneously. To ensure the accuracy of voice processing, it is necessary to effectively distinguish these sounds and avoid confusion. For this purpose, this embodiment needs to identify the target speaker who is currently speaking from multiple speakers. The target speaker may be the user, may also be a candidate speaker who is having a conversation with the user, or may be the situation where the user and the candidate speaker who is having a conversation are speaking simultaneously. Specifically, step S103 includes: Step a1. If it is identified based on the voice beams in various different directions that multiple candidate speakers who are having a conversation with the user are all speaking, then determine the target speaker according to the voice signal intensities in various different directions.

[0099] Among them, the specific implementation process of step a1 may include the following steps a11 and a12:

[0100] Step a1. Determine the voice signal intensity corresponding to each voice beam in various different directions.

[0101] In this step, it is necessary to calculate the voice signal intensity of each voice beam in various different directions respectively. Among them, the voice signal intensity can be the Power Spectral Density (PSD). PSD reflects the power distribution of the voice signal in the frequency domain and is an important indicator for measuring the activity degree of the voice beam. For the voice beam corresponding to the i-th directional beam (the i-th direction), its PSD can be obtained by the following formula (5):

[0102] PSDi = abs(Yi)^2 / 512 / 16000; (5)

[0103] In the formula, Yi is the voice beam of the i-th directional beam; PSDi is the power spectral density corresponding to the voice beam of the i-th directional beam.

[0104] To further improve the stability of PSD estimation, the following formula (6) can also be used to smooth PSDi:

[0105] PSDsmi = PSDsmi * (1 - a) + PSDsmi_Last * a;

[0106] In Equation (6), PSDsmi is the power spectral density corresponding to the speech beam corresponding to the i-th directional beam after smoothing; a is a coefficient close to but less than 1, such as 0.98, which is used to adjust the degree of smoothing.

[0107] Among them, according to the comparison result between the speech signal intensities in different directions and the preset speech signal intensity, if the speech signal intensity corresponding to the speech beam in a certain direction is greater than or equal to the preset speech signal intensity, it is determined that the user or candidate speaker in that direction is speaking.

[0108] Step a2: Determine the first candidate speaker in the direction corresponding to the maximum speech signal intensity among the speech signal intensities in different directions as the target speaker.

[0109] In this step, by comparing the signal intensities of the speech beams in different directions, the first candidate speaker in the direction where the maximum speech signal intensity is located is selected as the target speaker.

[0110] In the context of a multi-person conversation, even if there are voices of other speakers around, when a user has a direct conversation with a specific speaker, the speech signal of that speaker usually has a higher intensity. And in natural social interactions, people tend to unconsciously face the object who is speaking, so that the voice of the candidate speaker talking to the user can be transmitted to the user's receiving device more clearly and directly. Therefore, compared with other voices in the background, the speech signal of the candidate speaker in the current conversation usually has a greater intensity.

[0111] Based on this, in this embodiment, the target speaker in the current conversation is determined by identifying and selecting the candidate speaker in the direction with the maximum speech signal intensity. In this way, not only the influence of the physical environment on sound propagation is considered, but also the characteristics of human social behavior are combined to ensure that the user's current conversation partner can be accurately focused on, and thus a more accurate translation service experience can be provided for the user.

[0112] In some scenarios, after initially determining the first candidate speaker as the target speaker, it may occur that the speech signal intensity of other candidate speakers exceeds that of the current target speaker. At this time, it is necessary to re-evaluate and possibly update the target speaker to ensure the accuracy of conversation tracking. Specifically, it includes Step a3 and Step a4:

[0113] Step a3: Determine the second candidate speaker corresponding to the direction corresponding to the maximum speech signal intensity among the speech signal intensities in different directions. The second candidate speaker and the first candidate speaker correspond to different speakers respectively.

[0114] After initially selecting the first candidate speaker as the target speaker, the voice beams of all candidate speakers are continuously monitored to find a new candidate speaker with the strongest voice signal, i.e., the second candidate speaker. It should be noted that the second candidate speaker must represent a different individual from the initially selected first candidate speaker. This step a3 aims to identify a possible new conversation partner, but in order to avoid misjudging background noise or brief interference sounds as valid speakers, further verification is required through step a4.

[0115] Step a4: Determine whether to switch the second candidate speaker to the target speaker based on the comparison result between the voice signal strength corresponding to the second candidate speaker and the voice signal strength corresponding to the first candidate speaker.

[0116] To ensure the rationality and accuracy of the switch, after determining that the voice signal strength of the second candidate speaker is indeed greater than that of the first candidate speaker, it is also necessary to compare the difference in voice signal strength between the two. Only when the voice signal strength of the second candidate speaker is significantly higher than that of the first candidate speaker and meets the preset conditions will the switch operation of the target speaker be executed.

[0117] Specifically, step a4 includes: If the difference between the voice signal strength corresponding to the second candidate speaker and the voice signal strength corresponding to the first candidate speaker is greater than or equal to the preset difference, then determine to switch the second candidate speaker to the target speaker.

[0118] When it is detected that the voice signal strength of the second candidate speaker has increased significantly and there is an obvious difference from the voice signal strength of the current target speaker (i.e., the first candidate speaker), a further analysis process will be initiated. For example, measure whether the difference between the two exceeds a pre-set threshold, i.e., the preset difference. The selection of the preset difference can be based on the optimization results of a large amount of experimental data and actual application scenarios, aiming to balance sensitivity and accuracy and prevent false triggering of the switch due to minor fluctuations or brief interferences.

[0119] If, after calculation, the difference between the voice signal strength of the second candidate speaker and the voice signal strength of the first candidate speaker is greater than or equal to the preset difference, it indicates that the conversation focus may have shifted to the new speaker. At this time, the second candidate speaker needs to be switched to the target speaker.

[0120] To confirm that this change is not a transient phenomenon caused by environmental noise or other non-conversation-related factors. Other auxiliary information can also be combined, such as the directionality of the sound, the direction of the user's line of sight, and the context coherence of the conversation content, etc., to help more comprehensively understand the current conversation scenario, so as to make a more accurate choice. For example, if the user is facing the direction of the second candidate speaker, or the conversation content shows signs of switching from the first candidate speaker to the second candidate speaker, it will further support the decision to switch the target speaker.

[0121] Since the target speaker may include the user, may also include a certain candidate speaker, or may include both the user and a certain candidate speaker at the same time. Therefore, for the above different situations, after determining the target speaker, the translation result corresponding to the speech beam of the target speaker output in step S103 specifically includes:

[0122] In one example, if the target speaker is the user, the translation result corresponding to the user's speech signal is displayed on the outer display screen.

[0123] In another example, if the target speaker is the candidate speaker who is having a conversation with the user, the translation result corresponding to the speech signal of the candidate speaker who is having a conversation with the user is displayed on the inner display screen.

[0124] In another example, if the target speaker is the user and the candidate speaker who is having a conversation with the user, the translation result corresponding to the user's speech signal is displayed on the outer display screen, and the translation result corresponding to the speech signal of the candidate speaker who is having a conversation with the user is displayed on the inner display screen.

[0125] When the target speaker of the current conversation is determined and the translation processing of the speech signal corresponding to its speech beam is completed, the speech translation result of this speaker will be presented on the inner display screen. The inner display screen is a virtual screen, usually located on the lens of the AR glasses, so that the user can easily read without adjusting the line of sight. It is applicable to face-to-face communication scenarios, such as meetings, interviews or daily conversations. The user can directly focus on the speech content of the other party and quickly understand the information expressed in a non-native language through the inner display screen.

[0126] The display of the speech translation result on the inner display screen can adopt an easy-to-read font and format, including but not limited to:

[0127] Real-time scrolling text: As the conversation progresses, new translation results will be automatically scrolled and displayed, and the old content will gradually move out of the field of vision to ensure that the screen always presents the latest information.

[0128] Highlighted keywords: For important words or phrases, they can be highlighted through colors or other visual effects to help users quickly grasp the key points.

[0129] Background and text contrast optimization: Ensure good readability of the translation results under various ambient light conditions. For example, use a dark background with light text, or automatically adjust the brightness and contrast according to the ambient light.

[0130] Meanwhile, after the user's own speech beam is translated, the result will be presented on the external display screen. The external display screen faces other participants in the conversation. The translation result display on the external display screen may include the following features:

[0131] Synchronous display: When the user speaks, the translation result immediately appears on the external display screen, maintaining a high degree of synchronization with the spoken expression and reducing misunderstandings caused by time delays.

[0132] Simple and clear layout: Adopt a simple design style to ensure that the translated content is clear at a glance and avoid excessive decoration from affecting information transmission.

[0133] Adapt to different cultural habits: Considering the diversity of global users, the external display screen can support multiple language directions (such as from left to right, from right to left, etc.) and automatically adjust the typesetting method according to the habits of the target audience.

[0134] Personalization options: Allow users to customize the display style of the external display screen according to their personal preferences, such as font size, color theme, etc., to enhance the usage experience.

[0135] In this embodiment, by separately picking up the voice signals of speakers in multiple different directions, specifically including the user's voice signal and the voice signals of multiple candidate speakers, and cooperating with the interlocutor orientation recognition algorithm, the voice of the candidate speaker who is speaking in the front is extracted, and the two voices of the user and the candidate speaker who is speaking are respectively sent to two ASRs for speech recognition, and then translated and displayed on the internal display screen and the external display screen, so as to realize the separation and extraction of the voices of the user and the interlocutor, prevent mutual interference, avoid translation confusion and display errors, and improve the accuracy of voice translation.

[0136] Exemplary device

[0137] Corresponding to the above-mentioned directional sound pickup method, the embodiment of the present application also provides a directional sound pickup device. Figure 6 It is a schematic structural diagram of a directional sound pickup device provided by the embodiment of the present application. As Figure 6As shown in the figure, the directional sound pickup device provided by the embodiment of the present application includes: an acquisition unit 601, a beamforming unit 602, and a speaker recognition and output unit 603; the acquisition unit 601 is configured to acquire a plurality of voice signals collected by a plurality of microphones arranged on the AR glasses; the beamforming unit 602 is configured to perform beamforming according to the plurality of voice signals to obtain voice beams in different directions; the speaker recognition and output unit 603 is configured to recognize a target speaker who is currently speaking according to the voice beams in different directions, and output a translation result corresponding to the voice beam of the target speaker, where the target speaker includes the user and / or a candidate speaker who is having a conversation with the user.

[0138] In some embodiments, the speaker recognition and output unit 603 recognizes a target speaker who is currently speaking according to the voice beams in different directions, including: if it is recognized according to the voice beams in different directions that a plurality of candidate speakers who are having a conversation with the user are all speaking, then the target speaker is determined according to the voice signal intensities corresponding to the respective different directions.

[0139] In some embodiments, the speaker recognition and output unit 603 determines the target speaker according to the voice signal intensities in different directions, including: determining a first candidate speaker in the direction corresponding to the maximum voice signal intensity among the voice signal intensities in different directions as the target speaker.

[0140] In some embodiments, the speaker recognition and output unit 603 is further configured to: determine a second candidate speaker corresponding to the direction corresponding to the maximum voice signal intensity among the voice signal intensities in different directions, where the second candidate speaker and the first candidate speaker correspond to different speakers respectively; determine whether to switch the second candidate speaker to the target speaker according to the comparison result between the voice signal intensity corresponding to the second candidate speaker and the voice signal intensity corresponding to the first candidate speaker.

[0141] In some embodiments, the speaker recognition and output unit 603 determines whether to switch the second candidate speaker to the target speaker according to the comparison result between the voice signal intensity corresponding to the second candidate speaker and the voice signal intensity corresponding to the first candidate speaker, including: if the difference between the voice signal intensity corresponding to the second candidate speaker and the voice signal intensity corresponding to the first candidate speaker is greater than or equal to a preset difference, then it is determined to switch the second candidate speaker to the target speaker.

[0142] In some embodiments, the multiple microphones correspond to respective directional beams, and each of the directional beams corresponds to a beam coefficient; wherein, the beam forming unit 602 performs beam forming on the multiple voice signals to obtain voice beams in different directions, including: determining the voice beams of the multiple voice signals in different directions according to the multiple voice signals and the beam coefficients corresponding to the respective directional beams.

[0143] In some embodiments, the AR glasses include an inner display screen and an outer display screen, and the target speaker includes the user, a candidate speaker who is having a conversation with the user, or includes the user and the candidate speaker who is having a conversation with the user; wherein, the speaker recognition and output unit 603 outputs the translation result corresponding to the voice beam of the target speaker, including: if the target speaker is the user, displaying the translation result corresponding to the user's voice signal on the outer display screen; if the target speaker is a candidate speaker who is having a conversation with the user, displaying the translation result corresponding to the voice signal of the candidate speaker who is having a conversation with the user on the inner display screen; if the target speaker is the user and a candidate speaker who is having a conversation with the user, displaying the translation result corresponding to the user's voice signal on the outer display screen and displaying the translation result corresponding to the voice signal of the candidate speaker who is having a conversation with the user on the inner display screen.

[0144] The directional sound pickup device provided in this embodiment belongs to the same inventive concept as the directional sound pickup method provided in the above embodiments of the present application, can execute the directional sound pickup method provided in any of the above embodiments of the present application, and has the corresponding functional modules and beneficial effects for executing the directional sound pickup method. For the technical details not described in detail in this embodiment, reference may be made to the specific processing content of the directional sound pickup method provided in the above embodiments of the present application, which will not be elaborated here.

[0145] The functions implemented by the above acquisition unit 601, beam determination unit 602, and beam selection and output unit 603 can be respectively implemented by the same or different processors, which is not limited in the embodiments of the present application.

[0146] It should be understood that the units in the above device can be implemented in the form of a processor calling software. For example, the device includes a processor, which is connected to a memory. Instructions are stored in the memory, and the processor calls the instructions stored in the memory to implement any of the above methods or the functions of each unit of the device. The processor can be a general-purpose processor, such as a CPU or a microprocessor, etc., and the memory can be a memory inside the device or a memory outside the device. Alternatively, the units in the device can be implemented in the form of a hardware circuit. By designing the hardware circuit, the functions of some or all of the units can be implemented. The hardware circuit can be understood as one or more processors. For example, in one implementation, the hardware circuit is an ASIC, and by designing the logical relationship of the components in the circuit, the functions of some or all of the above units are implemented. Again, for example, in another implementation, the hardware circuit can be implemented through a PLD. Taking an FPGA as an example, it can include a large number of logic gate circuits, and the connection relationship between the logic gate circuits is configured through a configuration file, so as to implement the functions of some or all of the above units. All the units of the above device can be all implemented in the form of a processor calling software, or all implemented in the form of a hardware circuit, or some implemented in the form of a processor calling software, and the remaining part implemented in the form of a hardware circuit.

[0147] In the embodiments of the present application, the processor is a circuit with the ability to process signals. In one implementation, the processor can be a circuit with the ability to read and execute instructions, such as a CPU, a microprocessor, a GPU, or a DSP, etc. In another implementation, the processor can implement certain functions through the logical relationship of a hardware circuit, and the logical relationship of the hardware circuit is fixed or can be reconfigured. For example, the processor is a hardware circuit implemented by an ASIC or a PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document to implement the configuration of the hardware circuit can be understood as the process of the processor loading instructions to implement the functions of some or all of the above units. In addition, it can also be a hardware circuit designed for artificial intelligence, which can be understood as a type of ASIC, such as an NPU, a TPU, a DPU, etc.

[0148] It can be seen that each unit in the above device can be one or more processors (or processing circuits) configured to implement the above method. For example: a CPU, a GPU, an NPU, a TPU, a DPU, a microprocessor, a DSP, an ASIC, an FPGA, or a combination of at least two of these processor forms.

[0149] In addition, all or some of the units in the above device can be integrated together or can be implemented independently. In one implementation, these units are integrated together and implemented in the form of a System on Chip (SOC). The SOC may include at least one processor for implementing any of the above methods or the functions of each unit of the device. The types of the at least one processor can be different, for example, including a CPU and an FPGA, a CPU and an artificial intelligence processor, a CPU and a GPU, etc.

[0150] Exemplary electronic device

[0151] An embodiment of the present application provides an electronic device. Referring to Figure 7 as shown, the electronic device includes:

[0152] a memory 200 and a processor 210;

[0153] wherein, the memory 200 is connected to the processor 210 and is used for storing programs;

[0154] The processor 210 is used for implementing the audio playback processing method disclosed in any of the above embodiments by running the programs stored in the memory 200.

[0155] Specifically, the above electronic device may further include: a bus, a communication interface 220, an input device 230, and an output device 240.

[0156] The processor 210, the memory 200, the communication interface 220, the input device 230, and the output device 240 are interconnected through the bus. Among them:

[0157] The bus may include a path for transmitting information between various components of the computer system.

[0158] The processor 210 may be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or may be an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the present invention. It may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0159] The processor 210 may include a main processor and may also include a baseband chip, a modem, etc.

[0160] The program for implementing the technical solution of the present invention is stored in the memory 200, and the operating system and other key services can also be stored. Specifically, the program may include program codes, and the program codes include computer operation instructions. More specifically, the memory 200 may include a read-only memory (ROM), other types of static storage devices that can store static information and instructions, a random access memory (RAM), other types of dynamic storage devices that can store information and instructions, a disk memory, a flash memory, and so on.

[0161] The input device 230 may include devices for receiving data and information input by a user, such as a keyboard, a mouse, a camera, a scanner, a light pen, a voice input device, a touch screen, a pedometer, or a gravity sensor, etc.

[0162] The output device 240 may include devices for allowing information to be output to a user, such as a display screen, a printer, a speaker, etc.

[0163] The communication interface 220 may include devices of any transceiver type for communicating with other devices or communication networks, such as an Ethernet, a radio access network (RAN), a wireless local area network (WLAN), etc.

[0164] The processor 210 executes the program stored in the memory 200 and calls other devices, and can be used to implement each step of any one of the audio playback processing methods provided in the above embodiments of the present application.

[0165] An embodiment of the present application also proposes a chip, which includes a processor and a data interface. The processor reads and runs the program stored on the memory through the data interface to execute the audio playback processing method described in any of the above embodiments. The specific processing process and its beneficial effects can be referred to the embodiments of the audio playback processing method described above.

[0166] Exemplary computer program product and storage medium

[0167] In addition to the above methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions. When the computer program instructions are run by a processor, the processor is caused to execute the steps in the audio playback processing method according to various embodiments of the present application described in any of the above embodiments of this specification.

[0168] The computer program product can be written in any combination of one or more programming languages for executing the program code of the operations of the embodiments of the present application. The programming languages include object-oriented programming languages such as Java, C++, etc., and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, executed as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0169] In addition, an embodiment of the present application can also be a storage medium on which a computer program is stored. The computer program is executed by a processor to perform the steps in the audio playback processing method according to various embodiments of the present application described in any of the above embodiments of this specification, and specifically can implement the steps of the above audio playback processing method.

[0170] For the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that the present application is not limited by the described action sequence, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the present application.

[0171] It should be noted that the various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The same or similar parts among the various embodiments can be referred to each other. For the device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0172] The steps in the methods of the various embodiments of the present application can be adjusted, combined, and deleted according to actual needs. The technical features recorded in the various embodiments can be replaced or combined.

[0173] The modules and sub-modules in the devices and terminals in the various embodiments of the present application can be combined, divided, and deleted according to actual needs.

[0174] In several embodiments provided by the present application, it should be understood that the disclosed terminals, devices, and methods can be implemented in other ways. For example, the terminal embodiments described above are merely illustrative. For example, the division of modules or sub-modules is only a logical function division. In actual implementation, there may be other division methods. For example, multiple sub-modules or modules can be combined or integrated into another module, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be indirect couplings or communication connections through some interfaces, devices, or modules, and can be in electrical, mechanical, or other forms.

[0175] The modules or sub-modules described as separate components may or may not be physically separated. The components as modules or sub-modules may or may not be physical modules or sub-modules, that is, they can be located in one place, or can be distributed to multiple network modules or sub-modules. Some or all of the modules or sub-modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0176] In addition, in each embodiment of the present application, the functional modules or sub-modules can be integrated in a processing module, or each module or sub-module can exist physically alone, or two or more modules or sub-modules can be integrated in one module. The above-mentioned integrated modules or sub-modules can be implemented in the form of hardware or in the form of software functional modules or sub-modules.

[0177] Those skilled in the art can further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods for each specific application to implement the described functions, but such implementation should not be considered to exceed the scope of the present application.

[0178] The steps of the methods or algorithms described in combination with the embodiments disclosed herein can be directly implemented by hardware, software units executed by a processor, or a combination of the two. The software units can be placed in a random access memory (RAM), memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium well-known in the technical field.

[0179] Finally, it should also be noted that in this text, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements not only includes those elements but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the said element.

[0180] The foregoing description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A directional sound pickup method, characterized in that: Applied to AR glasses, the AR glasses are provided with a plurality of microphones, and the method comprises: Acquire multiple voice signals collected by the multiple microphones; Performing beamforming according to the multiple speech signals to obtain speech beams in different directions; The target speaker who is currently speaking is identified according to the speech beams in the different directions, and the translation result corresponding to the speech beam of the target speaker is output, wherein the target speaker includes the user and / or the candidate speaker who is having a conversation with the user.

2. The method according to claim 1, characterized in that The step of identifying the target speaker who is currently speaking according to the speech beams in different directions includes: If it is identified according to the speech beams in the different directions that a plurality of candidate speakers who are in conversation with the user are all speaking, the target speaker is determined according to the speech signal strengths in the different directions.

3. The method according to claim 2, characterized in that The step of determining the target speaker according to the speech signal strengths in the different directions includes: The first candidate speaker in the direction corresponding to the maximum speech signal strength among the speech signal strengths in the different directions is determined as the target speaker.

4. The method according to claim 3, characterized in that After determining the first candidate speaker in the direction corresponding to the maximum voice signal strength among the voice signal strengths in the different directions as the target speaker, the method further includes: Determine a second candidate speaker corresponding to a direction corresponding to a maximum speech signal strength among the speech signal strengths in the different directions, wherein the second candidate speaker and the first candidate speaker correspond to different speakers respectively; According to a comparison result between the speech signal strength corresponding to the second candidate speaker and the speech signal strength corresponding to the first candidate speaker, it is determined whether to switch the second candidate speaker to the target speaker.

5. The method according to claim 4, characterized in that The determining whether to switch the second candidate speaker to the target speaker according to a comparison result between the speech signal strength corresponding to the second candidate speaker and the speech signal strength corresponding to the first candidate speaker includes: If the difference between the speech signal strength corresponding to the second candidate speaker and the speech signal strength corresponding to the first candidate speaker is greater than or equal to a preset difference, it is determined to switch the second candidate speaker to the target speaker.

6. The method according to any one of claims 1 to 5, characterized in that The multiple microphones correspond to respective directional beams, and the respective directional beams correspond to respective beam coefficients; The step of performing beamforming according to the multiple speech signals to obtain speech beams in different directions includes: According to the multiple speech signals and the beam coefficients corresponding to the respective directional beams, the speech beams of the multiple speech signals in different directions are determined.

7. The method according to any one of claims 1 to 4, characterized in that The AR glasses include an inner display screen and an outer display screen, and the target speaker includes a user, a candidate speaker who is having a conversation with the user, or the user and a candidate speaker who is having a conversation with the user; The outputting of the translation result corresponding to the speech beam of the target speaker includes: If the target speaker is a user, displaying the translation result corresponding to the user's voice signal on the external display screen; If the target speaker is a candidate speaker with whom the user is having a conversation, displaying the translation result corresponding to the voice signal of the candidate speaker with whom the user is having a conversation on the internal display screen; If the target speaker is the user and the candidate speaker who is having a conversation with the user, the translation result corresponding to the user's voice signal is displayed on the outer display screen, and the translation result corresponding to the voice signal of the candidate speaker who is having a conversation with the user is displayed on the inner display screen.

8. A directional sound pickup device, characterized in that: include: An acquisition unit, used to acquire multiple voice signals collected by multiple microphones provided on the AR glasses; A beamforming unit, configured to perform beamforming according to the multiple speech signals to obtain speech beams in different directions; The speaker recognition and output unit is used to recognize the target speaker who is currently speaking according to the speech beams in different directions, and output the translation result corresponding to the speech beam of the target speaker. The target speaker includes the user and / or the candidate speaker who is having a conversation with the user.

9. An AR glasses, characterized in that: It comprises a glasses frame, a plurality of microphones and an external display screen arranged on the glasses frame, and the directional sound pickup device according to claim 8; The multiple microphones are used to collect voice signals of the user and surrounding speakers to obtain multiple voice signals; The directional sound pickup device is used to perform the method according to any one of claims 1 to 7 on the multiple voice signals to obtain the translation result of the target speaker and / or the translation result of the user; The translation result of the user is displayed on the external display screen, and the translation result of the target speaker is displayed on the internal display screen of the AR glasses, and the internal display screen is a virtual screen.

10. An electronic device, characterized in that: including memory and processor; The memory is connected to the processor and is used to store programs; The processor is configured to implement the method according to any one of claims 1 to 7 by running the program in the memory.

11. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the method according to any one of claims 1 to 7 is implemented.

12. A computer program product, characterized in that The method comprises computer program instructions, which, when executed by a processor, cause the processor to implement the method as claimed in any one of claims 1 to 7.

Images