Subtitle display method, device, equipment, head-mounted device and program product

Abstract

The present invention relates to the field of intelligent wearable technology, and provides a subtitle display method, device, equipment, head-mounted device and program product, wherein the head-mounted device includes a near-eye display screen, a main frame, a fixed component and a microphone array; the microphone array includes at least a first microphone, a second microphone and a third microphone, the first microphone is arranged on the main frame, and the second microphone and the third microphone are respectively arranged on the fixed component, and the method includes: determining the beamforming direction of the microphone array according to the environmental voice and user behavior to obtain a voice signal; dividing the voice signal according to the role of the speaker to obtain the role voice; converting the role voice into text data, and displaying the text data with subtitles with role annotations on the near-eye display screen. The present invention adjusts the beamforming direction of the microphone array in advance to improve the efficiency of voice capture, and also separates the roles of different speakers and displays subtitles to improve the accuracy and robustness of the separation.

Description

Technical Field

[0001] The present invention relates to the field of intelligent wearable technology, and in particular to a subtitle display method, device, equipment, head-mounted device and program product. Background Art

[0002] Augmented reality technology has a wide range of applications in all walks of life. Among them, the use of AR (Augmented Reality) head-mounted display devices combined with voice recognition and intelligent translation technology for real-time subtitle display has broad commercial prospects. Compared with the use of traditional devices such as smart screens and handheld terminals, it has the advantages of more intuitive viewing of subtitles, more natural face-to-face communication, and easy to carry.

[0003] In some solutions, AR head-mounted display devices may adopt speech separation technology. However, the performance of current speech separation technology will drop sharply in complex environments such as noise and reverberation. The speech capture capability is relatively large and cannot be well adapted to speech separation in various application scenarios. It cannot meet the needs of users. In addition, during the conversation, the wearer's own speech will be mixed with the subtitles formed by others' speech, which is inconvenient to read. Summary of the invention

[0004] The present invention provides a subtitle display method, device, equipment, head-mounted device and program product, which are used to solve the problems of low voice separation applicability and confusing subtitle display of AR head-mounted display devices in the prior art.

[0005] The present invention provides a subtitle display method, which is applied to a head-mounted device. The head-mounted device includes a near-eye display screen, a main frame, a fixing component and a microphone array; the fixing component is used to fix the near-eye display screen arranged on the main frame in front of the user's eyes, the microphone array includes at least a first microphone, a second microphone and a third microphone, the first microphone is arranged on the main frame, and the second microphone and the third microphone are respectively arranged on the fixing component. The subtitle display method includes: determining a beamforming direction of the microphone array according to environmental voice and user behavior to obtain a voice signal according to the microphone array; dividing the voice signal into roles according to speakers to obtain role voices; converting the role voices into text data, and displaying the text data on the near-eye display screen with subtitles labeled with roles.

[0006] According to a subtitle display method provided by the present invention, the beamforming direction of the microphone array is determined according to the environmental speech and the user behavior, so as to obtain the speech signal according to the microphone array, including: determining the preliminary beamforming direction of the microphone array according to the environmental speech; adjusting the parameters of the beamforming algorithm in real time according to the user behavior; fine-tuning the preliminary beamforming direction according to the parameters to obtain the beamforming direction of the microphone array; and obtaining the speech signal according to the microphone array with the determined beamforming direction.

[0007] According to a subtitle display method provided by the present invention, voice signals are divided into roles according to speakers to obtain role voices, including: dividing voice signals into roles according to speakers to obtain a user's own voice signal and an other's voice signal for other speakers except the user himself.

[0008] A subtitle display method provided by the present invention also includes: determining and executing a control instruction in response to a voice signal from the user.

[0009] According to a subtitle display method provided by the present invention, the fixing component includes a first supporting component and a second supporting component, the first supporting component is arranged on one side of the main frame, and the second supporting component is arranged on the other side of the main frame; the second microphone and the third microphone are arranged on the first supporting component or the second supporting component.

[0010] A subtitle display method provided by the present invention obtains a speech signal according to a microphone array, including: taking the original signal of a first microphone, the original signal of a second microphone and the original signal of a third microphone as columns of a preliminary matrix, and calculating the covariance matrix of the preliminary matrix; determining the eigenvalues ​​and eigenvectors of the covariance matrix, and selecting the eigenvectors corresponding to the eigenvalues ​​exceeding a preset value as principal components to separate the speaker's speech; projecting the original signal of the first microphone, the original signal of the second microphone and the original signal of the third microphone onto the principal components respectively for fusion, and obtaining new signals as rows of a new matrix to obtain a target matrix; determining independent components of the signal in the target matrix, wherein each independent component represents a speaker in the original signal; and selecting the preset independent components as output signals to obtain an enhanced speech signal.

[0011] The present invention also provides a subtitle display device, which is deployed on a head-mounted device. The head-mounted device includes a near-eye display screen, a main frame, a fixing component and a microphone array; the fixing component is used to fix the near-eye display screen arranged on the main frame in front of the user's eyes, the microphone array includes at least a first microphone, a second microphone and a third microphone, the first microphone is arranged on the main frame, and the second microphone and the third microphone are respectively arranged on the fixing component. The subtitle display device includes: a voice signal acquisition module, which is used to determine the beamforming direction of the microphone array according to the environmental voice and user behavior, so as to obtain the voice signal according to the microphone array; a role voice division module, which is used to divide the voice signal into roles according to the speaker to obtain the role voice; a subtitle display module, which is used to convert the role voice into text data, and display the text data with subtitles with role annotations on the near-eye display screen.

[0012] The present invention also provides an electronic device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the above-mentioned subtitle display methods is implemented.

[0013] The present invention also provides a head-mounted device, including a near-eye display screen, a main frame, a fixing component, a microphone array and a controller; the fixing component is used to fix the near-eye display screen arranged on the main frame in front of the user's eyes, the microphone array includes at least a first microphone, a second microphone and a third microphone, the first microphone is arranged on the main frame, the second microphone and the third microphone are respectively arranged on the fixing component, and the controller is used to execute the above-mentioned subtitle display method.

[0014] The present invention also provides a computer program product, including a computer program, and when the computer program is executed by a processor, any of the above-mentioned subtitle display methods is implemented.

[0015] The present invention provides a subtitle display method, device, equipment, head-mounted device and program product, the head-mounted device includes a near-eye display screen, a main frame, a fixed component and a microphone array; the fixed component is used to fix the near-eye display screen set on the main frame in front of the user's eyes, the microphone array includes at least a first microphone, a second microphone and a third microphone, the first microphone is set on the main frame, and the second microphone and the third microphone are respectively set on the fixed component, and the subtitle display method includes: according to the environmental voice and user behavior, determine the beamforming direction of the microphone array to obtain the voice signal according to the microphone array; divide the voice signal according to the speaker to obtain the role voice; convert the role voice into text data, and display the text data on the near-eye display screen with role-annotated subtitles. In the above manner, the present invention uses the user behavior pattern to predict the direction of the voice source, adjusts the beamforming direction of the microphone array in advance, so as to improve the efficiency of voice capture; and can also separate the roles of different speakers and display subtitles, so as to improve the accuracy and robustness of separation. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 It is a flowchart of a subtitle display method provided by an embodiment of the present invention.

[0018] Figure 2 4 is a schematic diagram of wearing a head mounted device provided by an embodiment of the present invention.

[0019] Figure 3 4 is a schematic diagram of the structure of a head mounted device provided in an embodiment of the present invention.

[0020] Figure 4 It is a structural schematic diagram of a subtitle display device provided by an embodiment of the present invention.

[0021] Figure 5 It is a schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] In the description of the embodiments of the present invention, it should be noted that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the embodiments of the present invention and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore cannot be understood as limitations on the embodiments of the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance.

[0024] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiment of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0025] It should be noted that all actions of acquiring signals, information or data in the present invention are performed in compliance with the corresponding data protection laws and policies of the country where the device is located and with the authorization given by the owner of the corresponding device.

[0026] The present invention provides a subtitle display method, which is applied to a head-mounted device. The present invention uses sound source localization and sound extraction technology to separate the voices of different speakers, and after separation, different display rendering methods are used to display or hide subtitles on a near-eye display screen of the head-mounted device, so as to achieve better subtitle reading and understanding effects.

[0027] Optionally, the head-mounted device includes but is not limited to AR (Augmented Reality) glasses, VR (Virtual Reality) glasses, MR (Mixed Reality) glasses, smart helmets and other devices.

[0028] Among them, AR technology enhances the user's sensory experience by superimposing virtual information (such as images, videos, 3D models, etc.) on the view of the real world. VR technology creates a completely virtual environment, and users enter this environment through head-mounted devices (such as helmets), isolated from the real world, and experience an immersive virtual world. MR technology combines elements of VR and AR, allowing virtual objects to interact and coexist with real-world environments and objects. MR not only enhances reality, but also extends reality, allowing users to interact with elements of the virtual and real worlds at the same time.

[0029] See also Figure 1 , Figure 1 is a flow chart of a subtitle display method provided by an embodiment of the present invention. In this embodiment, the subtitle display method may include steps S110 to S130, each of which is specifically as follows: S110: Determine a beamforming direction of the microphone array according to the ambient voice and the user behavior, so as to obtain a voice signal according to the microphone array.

[0030] S120: Divide the voice signal into roles according to the speaker to obtain role voices.

[0031] S130: Convert the character's voice into text data, and display the text data on a near-eye display screen with subtitles labeled with the character.

[0032] See also Figure 2-Figure 3 , Figure 2 is a schematic diagram of wearing a head mounted device provided by an embodiment of the present invention, Figure 3 4 is a schematic diagram of the structure of a head mounted device provided in an embodiment of the present invention.

[0033] The head-mounted device 200 includes a near-eye display screen 230, a main frame 210, a fixing component 220 and a microphone array; the fixing component 220 is used to fix the near-eye display screen arranged on the main frame in front of the user's eyes, and the microphone array includes at least a first microphone MIC1, a second microphone MIC2 and a third microphone MIC3, the first microphone MIC1 is arranged on the main frame 210, and the second microphone MIC2 and the third microphone MIC3 are respectively arranged on the fixing component 220.

[0034] Optionally, the first microphone MIC1 is disposed in the middle of the main frame 210 , and the distance between the first microphone MIC1 and the user's mouth ranges from 65 mm to 85 mm.

[0035] Optionally, the fixing assembly 220 includes a first supporting component 221 and a second supporting component 222 , wherein the first supporting component 221 is disposed on one side of the main frame 210 , and the second supporting component 222 is disposed on the other side of the main frame 210 .

[0036] The second microphone MIC2 and the third microphone MIC3 may be disposed on the first supporting member 221 or the second supporting member 222. For example, the second microphone MIC2 is disposed on the first supporting member 221, and the third microphone MIC3 is disposed on the second supporting member 222.

[0037] Optionally, the distance between the second microphone MIC2 and the third microphone MIC3 ranges from 140 mm to 160 mm; the distance between the second microphone MIC2 and the user's mouth ranges from 120 mm to 140 mm; the distance between the third microphone MIC3 and the user's mouth ranges from 120 mm to 140 mm.

[0038] Optionally, a distance between two points of the second microphone MIC2 and the first microphone MIC1 ranges from 110 mm to 120 mm; and a front-to-back projection distance A between the second microphone MIC2 and the first microphone MIC1 ranges from 80 mm to 90 mm.

[0039] Optionally, a distance between two points of the third microphone MIC3 and the first microphone MIC1 ranges from 110 mm to 120 mm; and a front-to-back projection distance between the third microphone MIC3 and the first microphone MIC1 ranges from 80 mm to 90 mm.

[0040] Furthermore, the sound outlet of the first microphone MIC1 may be arranged forward, and the opening diameter of the first microphone MIC1 may be arranged between 0.8 mm and 1.2 mm.

[0041] Furthermore, the sound outlet holes of the second microphone MIC2 and the third microphone MIC3 may be arranged downward, and the opening diameters of the second microphone MIC2 and the third microphone MIC3 are arranged between 0.8 mm and 1.2 mm.

[0042] In this embodiment, the parameters of the beamforming algorithm can be adjusted in real time according to the dynamic changes of the ambient voice and the speaker's position. For example, an adaptive filter can be used to dynamically adjust the directionality of the beamforming according to the changes in the statistical characteristics of the noise to improve the speech separation performance in a complex environment. Therefore, this embodiment is an adaptive beamforming based on user behavior prediction, which uses user behavior patterns (such as head movement, glasses wearing angle, etc.) to predict the direction of the voice source and adjust the beamforming direction of the microphone array in advance to improve the efficiency of voice capture.

[0043] In some embodiments, the step of determining the beamforming direction of the microphone array according to the ambient voice and the user behavior to obtain the voice signal according to the microphone array may specifically include: Determine the preliminary beamforming direction of the microphone array according to the ambient speech; adjust the parameters of the beamforming algorithm in real time according to the user behavior; fine-tune the preliminary beamforming direction according to the parameters to obtain the beamforming direction of the microphone array; and obtain the speech signal according to the microphone array with the determined beamforming direction.

[0044] Beamforming is a signal processing technique used to enhance signals from a specific direction and suppress noise from other directions. In this embodiment, the preliminary beamforming direction is determined based on the distribution of ambient speech, with the goal of aligning the sensitive direction of the microphone array to the direction most likely to contain the desired speech signal while reducing noise interference from other directions.

[0045] In this embodiment, the head mounted device may also be provided with sensors for detecting user behavior, and the user behavior, such as the user's head rotation, gestures, etc., may be sensed based on the sensor data. The parameters of the beam forming algorithm are dynamically adjusted based on these behaviors, and these parameters may include the width, direction, gain, etc. of the beam. The beam direction is further fine-tuned based on the algorithm parameters adjusted in real time, and this fine-tuning is to ensure that the beam forming direction can be accurately aligned with the user's voice signal source while suppressing the ambient noise to the greatest extent.

[0046] Finally, when the beamforming direction is determined, the microphone array will capture the voice signal in this direction. Due to the beamforming technology, the head-mounted device can effectively enhance the voice signal from the beam direction and suppress the noise from other directions, thereby improving the clarity and quality of the voice signal.

[0047] Therefore, this embodiment dynamically adjusts the beamforming direction and parameters by detecting the ambient voice and user behavior to ensure that the user voice signal can be accurately captured and the ambient noise can be suppressed.

[0048] In some embodiments, the step of dividing the voice signal into roles according to the speaker to obtain the role voice may specifically include: The voice signal is divided into roles according to the speaker, and the user's own voice signal and the other's voice signal for other speakers other than the user are obtained.

[0049] In this embodiment, speech separation technology is used, which refers to the technology of separating speech signals of different speakers or different sound sources from mixed speech signals. The speech separation technology of this embodiment can easily separate different sound elements in the audio, such as different voices of different characters in the conversation, background music, etc.

[0050] This embodiment can further utilize deep learning technology, such as convolutional neural network (CNN) or recurrent neural network (RNN), to extract and enhance the features of the voice signal captured by the microphone array. The training model can be used to identify and separate the voice features of different speakers to improve the accuracy of voice separation. For example, the voiceprint features of the speaker can be distinguished to distinguish the speaker's identity. The language features of the speaker can be distinguished and corresponding translation operations can be performed to achieve a variety of display effects.

[0051] Deep neural network models can use various characteristics of speech data, such as temporal sequence, spatiotemporal correlation, long-short spectral dependence and autoregression, to model and analyze complex speech signals, thereby effectively separating the target speech. For example, a speech separation system based on a deep neural network can learn the speech feature patterns of different speakers and achieve accurate separation.

[0052] In some embodiments, speech separation can also be achieved through multimodal fusion. Specifically, other modal information other than audio, such as visual information (lip movement), can be combined for speech separation. Considering that humans can combine visual and auditory information to understand the target speech in a noisy environment, this embodiment can also combine audio and video information, and use visual cues to assist speech separation, which can significantly improve the accuracy and robustness of separation.

[0053] In some embodiments, the state space model can also be introduced into speech separation. The state space model has the advantage of establishing long-range dependencies with linear computational complexity, and is suitable for processing long-sequence speech tasks. Compared with the limitation of the local receptive field of convolutional neural networks (CNN) and the low computational efficiency of recurrent neural networks (RNN), it shows good performance and potential in speech separation.

[0054] With the continuous improvement of the algorithm and the increase of training data, the accuracy of speech separation in this embodiment continues to improve, and a very high separation accuracy rate can be achieved, close to or reaching the human hearing level, and the different speakers' voices in the mixed speech can be accurately separated, and good audio quality can be maintained. In addition, the performance of previous speech separation technology will drop sharply in complex environments such as noisy and reverberant. However, through the adaptive beamforming algorithm of this embodiment and the improvement of the neural network model, by training a large amount of speech data in complex environments, and using more advanced signal processing technology, the speech separation system can still maintain a good separation effect in the presence of interference factors such as noise and reverberation.

[0055] In some embodiments, converting the character's speech into text data, and displaying the text data with character-labeled subtitles on a near-eye display screen includes: Use speech recognition (ASR, Automatic Speech Recognition) technology and voiceprint recognition technology to convert the voices of different characters into text data.

[0056] Specifically, speech recognition technology can convert audio signals into text through algorithms. Voiceprint recognition technology can be used to distinguish the voices of different characters, ensuring that each character's voice is accurately mapped to the correct text data. Voiceprint recognition technology can be achieved by analyzing different voice features (such as pitch, timbre, etc.).

[0057] In this embodiment, the subtitle display also adds character annotations to distinguish the dialogues of different characters. This can be achieved by adding the character name (such as "Character A:"), different colors or icons before the subtitle text.

[0058] It should be noted that a voice signal may include multiple roles.

[0059] In some embodiments, dividing the voice signal into roles according to the speaker to obtain role voices may also include: Divide the speech signal into roles according to the speaker to obtain several preliminary role voices; Select the user's desired character voice from among several preliminary character voices.

[0060] In this embodiment, the character voice can also be selectively displayed according to user needs. Not all detected speakers' voices need to be translated and displayed as subtitles, thereby simplifying the content of the subtitle display interface.

[0061] In some embodiments, the step of converting the character's voice into text data may further include: Convert the character's speech in the first language into text data in the second language.

[0062] The first language and the second language can be different voices, that is, the head-mounted device can also realize the function of language translation, converting the character voice in the first language into text data in the second language, for example, translating English voice into Chinese text for subtitle display.

[0063] The first language and the second language may also be the same voice, for example, Chinese voice is translated into Chinese text for subtitle display.

[0064] In some embodiments, the subtitle display method may further include: In response to our voice signal, the control instruction is determined and executed.

[0065] For example, in applications such as smart homes and smart customer service, accurate speech separation technology can help intelligent voice systems better understand the commands of multiple users and improve the efficiency and accuracy of interactions.

[0066] In some embodiments, the step of acquiring a voice signal according to a microphone array may specifically include: The original signal of the first microphone, the original signal of the second microphone and the original signal of the third microphone are used as columns of a preliminary matrix, and the covariance matrix of the preliminary matrix is ​​calculated; the eigenvalues ​​and eigenvectors of the covariance matrix are determined, and the eigenvectors corresponding to the eigenvalues ​​exceeding the preset values ​​are selected as principal components to separate the speaker's voice; the original signal of the first microphone, the original signal of the second microphone and the original signal of the third microphone are respectively projected onto the principal components for fusion, and the new signals are obtained and then used as rows of the new matrix to obtain a target matrix; the independent components of the signal in the target matrix are determined, wherein each independent component represents a speaker in the original signal; and the preset independent components are selected as output signals to obtain enhanced speech signals.

[0067] As described above, an embodiment of the present invention provides a subtitle display method, which is applied to a head-mounted device. The head-mounted device includes a near-eye display screen, a main frame, a fixing component and a microphone array; the fixing component is used to fix the near-eye display screen arranged on the main frame in front of the user's eyes, and the microphone array includes at least a first microphone, a second microphone and a third microphone, the first microphone is arranged on the main frame, and the second microphone and the third microphone are arranged on the fixing component respectively. The subtitle display method specifically includes: determining the beamforming direction of the microphone array according to the environmental voice and user behavior to obtain the voice signal according to the microphone array; dividing the voice signal according to the role of the speaker to obtain the role voice; converting the role voice into text data, and displaying the text data with subtitles with role annotations on the near-eye display screen. In the above manner, the embodiment of the present invention uses the user behavior pattern to predict the direction of the voice source, and adjusts the beamforming direction of the microphone array in advance to improve the efficiency of voice capture; and it can also separate the roles of different speakers and display subtitles, thereby improving the accuracy and robustness of the separation.

[0068] In addition, the subtitle display method provided in this embodiment can be applied to the application scenarios of the communication field, intelligent voice assistants, and security and privacy protection. For example, in scenarios such as telephone communications and video conferencing, voice separation technology can effectively remove background noise and other interfering voices, and improve the quality and comprehensibility of calls. For example, some real-time voice communication software has begun to apply voice separation technology, allowing users to clearly hear the other party's voice even in a noisy environment. In some situations where personal privacy or sensitive information needs to be protected, voice separation technology can be used to isolate the voice of a specific person and prevent unauthorized audio leakage, such as in conference record encryption, telephone recording processing, etc. It has application value.

[0069] In order to better illustrate the subtitle display method provided by the embodiment of the present invention, it is described below in conjunction with AR glasses.

[0070] In AR glasses, the first microphone is set as a main mic just above the nose clip of the glasses, and the second and third microphones are set as two auxiliary mics on the two sides of the glasses. The voice is input through the microphone array, and after being pre-processed by the main control chip, it is sent to the algorithm processing unit for voice separation calculation. The algorithm should be processed as follows: the algorithm should first perform noise reduction processing; then perform you-me separation processing to distinguish my voice from others' voices. The separation algorithm can also be turned on and off, and only the noise-reduced sound is output when it is turned off.

[0071] Specifically, the step of performing speech separation by the microphone array may include: The multi-microphone array composed of MIC1, MIC2 and MIC3 is used to improve the accuracy of voice capture through signal fusion technology. MIC1 collects the main sound component for both the wearer and the speaker on the opposite side, while MIC2 and MIC3 collect auxiliary sound components. They are 85cm away from MIC1 and are placed symmetrically to facilitate left and right beamforming. The principal component analysis (PCA) and independent component analysis (ICA) dynamic adaptation methods are used to fuse the signals captured by the three microphones to enhance the separation effect of the target speech and suppress noise.

[0072] First, the signals of the three microphones are used as the columns of the matrix to calculate the covariance matrix of the matrix. Next, the eigenvalues ​​and eigenvectors of the covariance matrix are calculated, and the eigenvectors corresponding to the largest eigenvalues ​​are selected as the principal components to separate the main speaker's voice. Then, the original signal is projected onto these principal components, and the fused signals are used as the rows of the matrix to construct a new signal matrix. After that, the ICA algorithm is applied to estimate the independent components of the signal matrix, which represent the independent speakers in the original signal. It does not require prior knowledge about the noise characteristics and is still effective in unknown noise environments. Finally, specific independent components are selected as the output signal as needed to enhance the target speech.

[0073] The PCA algorithm is used to transform the original data into a set of new variables through linear transformation. These variables are called principal components. The principal components are linear combinations of the original data and are orthogonal (uncorrelated) to each other. The main goal of the PCA algorithm is dimensionality reduction, that is, explaining most of the variance in the original data with fewer variables.

[0074] The ICA algorithm is used to decompose a multivariate signal into statistically independent non-Gaussian signals. Unlike the PCA algorithm, the goal of the ICA algorithm is to find mutually independent components rather than simply uncorrelated components. The ICA algorithm assumes that the original signal is a linear mixture of multiple independent source signals and attempts to separate these source signals through an inverse process.

[0075] When AR glasses are transcribing speech, they separate you and me based on the glasses algorithm, and present my voice and his voice in different forms; further, through the background role recognition capability, his voice can be presented in different roles.

[0076] In addition, AR glasses also support intelligent voice assistants. For example, you can activate the intelligent voice assistant by long pressing the touchpad or voice wake-up words, and only retain your voice, so that the intelligent voice assistant can eliminate the noise interference of others during the conversation.

[0077] In addition, AR glasses can also support meeting recording mode. In meeting recording mode, the voice separation algorithm is turned off, my voice and his voice are mixed, and the large model automatically distinguishes the interlocutors and provides intelligent prompts.

[0078] For example, when the user wears the glasses and speaks, the glasses can filter the sound and not display any interference information, regardless of whether the wearer speaks Chinese or English. When the wearer speaks a voice command, the glasses can respond accurately without being affected by other people's speech and external noise, improving the accuracy of voice control. When other people speak in English, the glasses will display the translated English text. When others speak in Chinese, the glasses should not display anything and will be completely filtered.

[0079] The present invention further provides a subtitle display device. The subtitle display device provided by the present invention is described below. The subtitle display device described below and the subtitle display method described above can be referred to each other.

[0080] See also Figure 4 , Figure 4 2 is a schematic diagram of the structure of a subtitle display device provided by an embodiment of the present invention. In this embodiment, the subtitle display device is deployed on a head-mounted device, and the head-mounted device includes a near-eye display screen, a main frame, a fixing component and a microphone array.

[0081] The fixing component is used to fix the near-eye display screen arranged on the main frame in front of the user's eyes. The microphone array includes at least a first microphone, a second microphone and a third microphone. The first microphone is arranged on the main frame, and the second microphone and the third microphone are respectively arranged on the fixing component.

[0082] Specifically, the subtitle display device may include a voice signal acquisition module 410 , a character voice division module 420 and a subtitle display module 430 .

[0083] The speech signal acquisition module 410 is used to determine the beamforming direction of the microphone array according to the environmental speech and the user behavior, so as to acquire the speech signal according to the microphone array.

[0084] The role speech division module 420 is used to divide the speech signal into roles according to the speaker to obtain the role speech.

[0085] The subtitle display module 430 is used to convert the character's speech into text data, and display the text data with subtitles with character annotations on the near-eye display screen.

[0086] In some embodiments, the speech signal acquisition module 410 can be specifically used to: determine the preliminary beamforming direction of the microphone array according to the ambient speech; adjust the parameters of the beamforming algorithm in real time according to the user behavior; fine-tune the preliminary beamforming direction according to the parameters to obtain the beamforming direction of the microphone array; and acquire the speech signal based on the microphone array with the determined beamforming direction.

[0087] In some embodiments, the role voice division module 420 can be specifically used to: divide the voice signal into roles according to the speaker, and obtain the user's own voice signal and the other's voice signal for other speakers except the user himself.

[0088] In some embodiments, the subtitle display device may further include an instruction response module, which may be specifically used to: respond to our voice signal, determine the control instruction and execute it.

[0089] In some embodiments, the fixing assembly includes a first supporting component and a second supporting component, the first supporting component is arranged on one side of the main frame, and the second supporting component is arranged on the other side of the main frame; the second microphone is arranged on the first supporting component, and the third microphone is arranged on the second supporting component.

[0090] Among them, the distance between the second microphone and the third microphone ranges from 140mm to 160mm; the distance between the second microphone and the user's mouth ranges from 120mm to 140mm; the distance between the third microphone and the user's mouth ranges from 120mm to 140mm; the distance between the second microphone and the first microphone ranges from 110mm to 120mm; the front-to-back projection distance between the second microphone and the first microphone ranges from 80mm to 90mm; the distance between the two points of the third microphone and the first microphone ranges from 110mm to 120mm; the front-to-back projection distance between the third microphone and the first microphone ranges from 80mm to 90mm.

[0091] In some embodiments, the speech signal acquisition module 410 can be specifically used to: use the original signal of the first microphone, the original signal of the second microphone, and the original signal of the third microphone as the columns of the preliminary matrix, and calculate the covariance matrix of the preliminary matrix; determine the eigenvalues ​​and eigenvectors of the covariance matrix, and select the eigenvectors corresponding to the eigenvalues ​​exceeding the preset values ​​as the principal components to separate the speaker's speech; project the original signal of the first microphone, the original signal of the second microphone, and the original signal of the third microphone onto the principal components respectively for fusion, and then use the new signals as the rows of the new matrix to obtain the target matrix; determine the independent components of the signal in the target matrix, where each independent component represents the speaker in the original signal; select the preset independent components as the output signal to obtain the enhanced speech signal.

[0092] On the other hand, an embodiment of the present invention further provides an electronic device, see Figure 5 , Figure 5 is a schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention, such as Figure 5 As shown, the electronic device may include a memory 520, a processor 510, and a computer program stored in the memory 520 and executable on the processor 510. When the processor 510 executes the program, the subtitle display method provided by the above methods is implemented.

[0093] Optionally, the electronic device may further include a communication bus 530 and a communication interface (CommunicationsInterface) 540, wherein the processor 510, the communication interface 540, and the memory 520 communicate with each other via the communication bus 530. The processor 510 may call a computer program in the memory 520 to execute a subtitle display method, which may include: According to the environmental speech and user behavior, the beamforming direction of the microphone array is determined to obtain the speech signal according to the microphone array; the speech signal is divided into roles according to the speaker to obtain the role speech; the role speech is converted into text data, and the text data is displayed on the near-eye display screen with subtitles labeled with the role.

[0094] In addition, the logic instructions in the above-mentioned memory 520 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0095] On the other hand, the present invention also provides a head-mounted device, which may include a near-eye display screen, a main frame, a fixing component, a microphone array, and a controller.

[0096] Among them, the fixing component is used to fix the near-eye display screen arranged on the main frame in front of the user's eyes, the microphone array includes at least a first microphone, a second microphone and a third microphone, the first microphone is arranged on the main frame, the second microphone and the third microphone are respectively arranged on the fixing component, and the controller is used to execute the above-mentioned subtitle display method.

[0097] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the subtitle display method provided by the above methods. The steps and principles have been introduced in detail in the above methods and will not be repeated here.

[0098] Non-transitory computer-readable storage media can be any available media or data storage devices that can be accessed by the processor, including but not limited to magnetic storage (such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO)), optical storage (such as CDs, DVDs, BDs, HVDs, etc.), and semiconductor storage (such as ROM, EPROM, EEPROM, non-volatile memory (NANDFLASH), solid-state drives (SSDs)), etc.

[0099] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0100] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0101] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A subtitle display method, characterized in that: The invention is applied to a head-mounted device, the head-mounted device comprises a near-eye display screen, a main frame, a fixing component and a microphone array; the fixing component is used to fix the near-eye display screen arranged on the main frame in front of the user's eyes, the microphone array comprises at least a first microphone, a second microphone and a third microphone, the first microphone is arranged on the main frame, the second microphone and the third microphone are respectively arranged on the fixing component, and the subtitle display method comprises: Determining a beamforming direction of the microphone array according to ambient speech and user behavior to obtain a speech signal according to the microphone array; Dividing the speech signal into roles according to the speaker to obtain role speech; The character's voice is converted into text data, and the text data is displayed on the near-eye display screen with subtitles labeled with the character.

2. The subtitle display method according to claim 1, characterized in that: The step of determining the beamforming direction of the microphone array according to the ambient voice and the user behavior so as to obtain the voice signal according to the microphone array includes: Determining a preliminary beamforming direction of the microphone array according to the ambient speech; adjusting the parameters of the beamforming algorithm in real time according to the user behavior; Fine-tuning the preliminary beamforming direction according to the parameter to obtain the beamforming direction of the microphone array; The speech signal is acquired according to a microphone array that determines a beamforming direction.

3. The subtitle display method according to claim 1, characterized in that: The step of dividing the speech signal into roles according to the speaker to obtain role speech includes: The voice signals are divided into roles according to the speakers to obtain a user's own voice signal and an other's voice signal for speakers other than the user himself.

4. The subtitle display method according to claim 3, characterized in that: Also includes: In response to the own voice signal, a control instruction is determined and executed.

5. The subtitle display method according to any one of claims 1 to 4, characterized in that: The fixing assembly comprises a first supporting component and a second supporting component, wherein the first supporting component is arranged on one side of the main frame, and the second supporting component is arranged on the other side of the main frame; The second microphone and the third microphone are disposed on the first supporting member or the second supporting member.

6. The subtitle display method according to claim 5, characterized in that: The acquiring of a voice signal according to the microphone array comprises: Taking the original signal of the first microphone, the original signal of the second microphone, and the original signal of the third microphone as columns of a preliminary matrix, and calculating a covariance matrix of the preliminary matrix; Determining the eigenvalues ​​and eigenvectors of the covariance matrix, and selecting the eigenvectors corresponding to the eigenvalues ​​exceeding a preset value as principal components to separate the speaker's speech; The original signal of the first microphone, the original signal of the second microphone, and the original signal of the third microphone are respectively projected onto the principal component for fusion, and the obtained new signals are used as rows of a new matrix to obtain a target matrix; Determining independent components of the signal in the target matrix, wherein each independent component represents a speaker in the original signal; The preset independent components are selected as output signals to obtain enhanced speech signals.

7. A subtitle display device, characterized in that: Deployed on a head-mounted device, the head-mounted device includes a near-eye display screen, a main frame, a fixing component and a microphone array; the fixing component is used to fix the near-eye display screen arranged on the main frame in front of the user's eyes, the microphone array includes at least a first microphone, a second microphone and a third microphone, the first microphone is arranged on the main frame, the second microphone and the third microphone are respectively arranged on the fixing component, and the subtitle display device includes: A speech signal acquisition module, used to determine the beamforming direction of the microphone array according to the environmental speech and user behavior, so as to acquire the speech signal according to the microphone array; A role voice division module, used for dividing the voice signal into roles according to the speaker to obtain role voices; The subtitle display module is used to convert the character's voice into text data, and display the text data with subtitles marked with the character on the near-eye display screen.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the subtitle display method according to any one of claims 1 to 6 is implemented.

9. A head mounted device, characterized in that: It includes a near-eye display screen, a main frame, a fixing component, a microphone array and a controller; The fixing component is used to fix the near-eye display screen arranged on the main frame in front of the user's eyes. The microphone array includes at least a first microphone, a second microphone and a third microphone. The first microphone is arranged on the main frame, and the second microphone and the third microphone are respectively arranged on the fixing component. The controller is used to execute the subtitle display method as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the subtitle display method according to any one of claims 1 to 6 is implemented.

Images