Voice processing method, device, electronic device and computer readable storage medium

Abstract

The embodiment of the present application discloses a speech processing method, device, electronic device and computer-readable storage medium, and the method is applied to an extended reality device. The method includes: when a first speech acquisition device acquires a first speech, acquiring the current facial electromyogram of the wearer; acquiring a pre-acquired target facial electromyogram when the wearer is speaking; determining whether the wearer is speaking based on the current facial electromyogram and the target facial electromyogram; when the wearer is speaking, acquiring a second speech acquisition device to directionally acquire a second speech; distinguishing the speech of the non-wearer based on the first speech and the second speech; converting the speech of the non-wearer into text, and displaying the text on a virtual screen. In this way, the present solution can determine whether the wearer is speaking through the facial electromyogram of the wearer, thereby accurately distinguishing the speech of the wearer and the non-wearer, and then accurately displaying the text converted from the speech of the non-wearer on the virtual screen.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of extended reality technology, and specifically to a speech processing method, device, electronic device, and computer-readable storage medium. Background Art

[0002] An important application of Extended Reality (XR) devices is to convert the other person's words into text and display them on the screen. However, when the microphone picks up the sound, it cannot accurately determine whether the sound is from the other person or yourself, resulting in your own words being displayed on the lens of the XR device.

[0003] The related technology uses a microphone to monitor the location of the sound to make a directional distinction and display the distinguished words of the other party. However, because the other party's speaking volume varies and the head shapes of different users are different, the directional distinction is inaccurate, so the accuracy of this method in displaying the text converted from speech is not high. Summary of the invention

[0004] The embodiments of the present application provide a speech processing method, device, electronic device and computer-readable storage medium, which can determine whether the wearer is speaking through the wearer's facial electromyogram, thereby accurately distinguishing the voice of the wearer and that of non-wearers, and then accurately displaying the text converted from the non-wearer's voice on a virtual screen.

[0005] In a first aspect, an embodiment of the present application provides a voice processing method, which is applied to an extended reality device, wherein the extended reality device includes: a first voice collection device, a second voice collection device, and a virtual screen; the method includes:

[0006] When the first voice collection device collects the first voice, obtaining the current facial electromyogram of the wearer;

[0007] Acquire a pre-collected target facial electromyogram of the wearer when speaking;

[0008] Determining whether the wearer is speaking according to the current facial electromyogram and the target facial electromyogram;

[0009] When the wearer is speaking, obtaining a second voice directionally collected by the second voice collecting device;

[0010] Distinguishing a voice of a non-wearer according to the first voice and the second voice; and

[0011] The non-wearer's voice is converted into text, and the text is displayed on the virtual screen.

[0012] In one embodiment, converting the non-wearer's voice into text and displaying the text on the virtual screen includes:

[0013] converting the speech of the non-wearer into text;

[0014] Determining whether the text is in the target language;

[0015] When the text is not in the target language, translating the text into a target text in the target language;

[0016] The target text is displayed on the virtual screen.

[0017] In one embodiment, the method further comprises:

[0018] Distinguishing the wearer's voice according to the first voice and the second voice;

[0019] When the wearer's voice is a voice command, respond to the voice command.

[0020] In one embodiment, the extended reality device includes: a loudspeaker; the method further includes:

[0021] Distinguishing the wearer's voice according to the first voice and the second voice;

[0022] The wearer's voice is amplified and outputted through the loudspeaker.

[0023] In one embodiment, the extended reality device includes: headphones; the method further includes:

[0024] The voice of the non-wearer is played toward the ear of the wearer through the earphone.

[0025] In one embodiment, distinguishing the voice of the non-wearer according to the first voice and the second voice includes:

[0026] Acquire voice features of non-wearers; the voice features of non-wearers are: common features extracted from multiple voice data of non-wearer samples located at various positions;

[0027] The voice of the non-wearer is distinguished according to the first voice, the second voice and the voice features of the non-wearer.

[0028] In one embodiment, the method further comprises:

[0029] When the wearer is silent, converting the first speech into text;

[0030] The text is displayed on the virtual screen.

[0031] In a second aspect, an embodiment of the present application provides a speech processing device, which is applied to an extended reality device, wherein the extended reality device comprises: a first speech collection device, a second speech collection device, and a virtual screen; the device comprises:

[0032] an electromyogram acquisition module, configured to acquire the current facial electromyogram of the wearer when the first voice acquisition device acquires the first voice;

[0033] A target acquisition module, used to acquire a pre-collected target facial electromyogram of the wearer when speaking;

[0034] A determination module, used to determine whether the wearer is speaking according to the current facial electromyogram and the target facial electromyogram;

[0035] A collection module, used for acquiring the second voice directionally collected by the second voice collection device when the wearer is speaking;

[0036] a distinguishing module, configured to distinguish a voice of a non-wearer according to the first voice and the second voice; and

[0037] The display module is used to convert the voice of the non-wearer into text and display the text on the virtual screen.

[0038] In one embodiment, the display module comprises:

[0039] A conversion unit, used for converting the speech of the non-wearer into text;

[0040] A language determination unit, used to determine whether the text is in a target language;

[0041] A translation unit, configured to translate the text into a target text in the target language when the text is not in the target language;

[0042] A text display unit is used to display the target text on the virtual screen.

[0043] In one embodiment, the device further comprises:

[0044] A first voice distinguishing module, used to distinguish the wearer's voice according to the first voice and the second voice;

[0045] In one embodiment, when the wearer's voice is a voice command, respond to the voice command.

[0046] In one embodiment, the extended reality device includes: a loudspeaker; the device also includes:

[0047] A second voice distinguishing module, used to distinguish the wearer's voice according to the first voice and the second voice;

[0048] The amplification module is used to amplify and output the wearer's voice through the loudspeaker.

[0049] In one embodiment, the extended reality device includes: headphones; the apparatus also includes:

[0050] The playing module is used to play the voice of the non-wearer toward the ear of the wearer through the earphone.

[0051] In one embodiment, the distinguishing module includes:

[0052] A feature acquisition unit, used to acquire voice features of non-wearers; the voice features of non-wearers are: common features extracted from multiple voice data of each non-wearer sample located in each position;

[0053] A voice distinguishing unit is used to distinguish the voice of the non-wearer according to the first voice, the second voice and the voice features of the non-wearer.

[0054] In one embodiment, the device further comprises:

[0055] A text conversion module, used for converting the first speech into text when the wearer does not speak;

[0056] The text display module is used to display the text on the virtual screen.

[0057] In a third aspect, an embodiment of the present application further provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps in the above-mentioned speech processing method when executed by the processor.

[0058] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps in the above-mentioned speech processing method are implemented.

[0059] In a fifth aspect, the embodiments of the present application further provide a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method provided in various optional implementations described in the embodiments of the present application.

[0060] In summary, in the embodiment of the present application, when the first voice collection device collects the first voice, the current facial electromyogram of the wearer can be obtained, as well as the target facial electromyogram of the wearer when speaking that is collected in advance; then, according to the current facial electromyogram and the target facial electromyogram, it is determined whether the wearer is speaking; when the wearer is speaking, the second voice collection device directionally collects the second voice; according to the first voice and the second voice, the voice of the non-wearer is distinguished; the voice of the non-wearer is converted into text, and the text is displayed on the virtual screen. In this way, it is determined whether the wearer is speaking according to the facial electromyogram of the wearer, so when the direction distinction is made according to the first voice and the second voice collected directionally, it can be determined whether the first voice and the second voice contain the voice of the wearer, so as to accurately distinguish the voice of the wearer and the non-wearer, and then accurately display the text converted from the voice of the non-wearer on the virtual screen, thereby improving the accuracy rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the technical solutions in the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0062] Figure 1 It is a schematic diagram of the steps of a speech processing method provided by an embodiment of the present application;

[0063] Figure 2 is a schematic diagram of an electromyographic signal sensor provided in an embodiment of the present application;

[0064] Figure 3 is a structural diagram of a speech processing device provided by an embodiment of the present application;

[0065] Figure 4 It is a schematic diagram of the structure of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0066] The following will be combined with the drawings in this application to clearly and completely describe the technical solutions in this application. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments in the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0067] Extended reality technology combines reality and virtuality through computers to create a virtual environment that allows human-computer interaction. Extended reality includes but is not limited to: Augmented Reality (AR), Virtual Reality (VR) and Mixed Reality (MR). Extended reality devices refer to electronic devices with extended reality functions.

[0068] In one embodiment, Figure 1 As shown, a speech processing method is provided. Although a logical order is shown in the step diagram, in some cases, the steps shown or described may be performed in an order different from that shown in the figure. In one embodiment, the speech processing method can be applied to an extended reality device. In particular, the extended reality device refers to a wearable extended reality device, and in particular, refers to wearable optical see-through smart glasses.

[0069] The extended reality device may include but is not limited to a first voice collection device, a second voice collection device and a virtual screen. When the specific extended reality device is a wearable optical see-through smart glasses, in addition to the above structure, it may also include at least a glasses body frame, an optical display component, an electronic circuit component and / or a sensor, etc. The sensor built into the glasses may include at least an electromyographic signal sensor, a heart rate monitor, a blood glucose detector, a microphone, a camera and / or an eye tracker.

[0070] It should be noted that the order of description of the following embodiments is not intended to limit the priority order of the embodiments.

[0071] according to Figure 1 The speech processing method shown in the figure includes at least steps S110 to S160, which are described in detail as follows:

[0072] In step S110, when the first voice collection device collects the first voice, the current facial electromyogram of the wearer is obtained.

[0073] The first voice collection device and the second voice collection device may have different sound collection directions, so that the wearer's voice and the non-wearer's voice can be distinguished based on the first voice collected by the first voice collection device and the second voice collected by the second voice collection device. The wearer refers to the user wearing the augmented reality device.

[0074] The first voice is a voice containing the user's speech, and the user may include a wearer and a non-wearer. The first voice acquisition device acquires various audios in real time, and transmits the audios to the audio analysis module of the extended reality device. The audio analysis module roughly analyzes the various audios to analyze whether the audio contains the user's speech. When it is determined that the audio contains the user's speech, the audio is determined to be the first voice.

[0075] The extended reality device may include an electromyographic signal sensor, which collects the current facial electromyographic signal of the wearer and generates a current facial electromyogram according to the current facial electromyographic signal of the wearer.

[0076] Electromyogram (EMG) is a bioelectric signal generated when muscles contract or relax. When the brain issues a command, motor neurons transmit nerve impulses to the corresponding muscle fibers through nerve conduction, which causes changes in the muscle fiber membrane potential, thereby generating action potentials. The human face is composed of many different muscles, and various facial movements are actually a process of joint regulation and coordinated operation of muscles in various parts of the face. The generation of facial movements such as opening the mouth, shaking the head, nodding, and frowning will prompt the muscles of the entire head to participate in the collaborative processing, and the movement of the muscles will generate corresponding electromyographic signals.

[0077] Electromyographic signals can be collected using electromyographic signal sensors. Optionally, the electromyographic signal sensor can be a surface electromyographic (sEMG) electrode. Surface electromyographic electrodes are attached to the surface of the skin to detect changes in electrical activity generated by muscle contraction. Based on the principle that muscle action potentials form a potential difference on the surface of the skin, surface electromyographic electrodes can capture these weak potential changes and convert them into measurable electrical signals. Surface electromyographic electrodes can be easily attached to corresponding muscle positions on the face, such as the frontalis, orbicularis oculi, zygomaticus major, orbicularis oris, etc., and can detect electrical signals of muscles related to different facial expressions, can better adapt to the contours of different parts of the face, reduce interference from motion artifacts, and improve signal acquisition quality.

[0078] Optionally, the electromyographic signal sensor can be a flexible electronic sensor. The flexible electronic sensor is made of flexible conductive materials (such as conductive polymers, carbon nanotubes, etc.), which can deform accordingly with the movement of facial muscles, and during the deformation process, its electrical properties such as conductivity will change accordingly, thereby converting muscle movement and electrical activity into measurable electrical signals. Flexible electronic sensors are made of soft materials and have little foreign body sensation when attached to the skin, which is conducive to continuous monitoring of facial muscle electrical signals, or recording of users' expression-related electromyographic signals for a long time in psychological and emotional research.

[0079] like Figure 2 As shown, an electromyographic signal sensor can be set in the area where the extended reality device contacts the user's face. For example, an electromyographic sensor is set in the area where the temple of the smart glasses contacts the wearer. Because people need to use a lot of facial muscles when speaking, therefore, setting the electromyographic sensor in the area where the extended reality device contacts the wearer's face can enable the electromyographic signal sensor to collect facial electromyographic signals that are more closely related to the wearer's speech.

[0080] When a user wears an extended reality device, the electromyographic sensor can collect the wearer's facial electromyographic signals. In order to accurately analyze and apply the facial electromyographic signals later, the collected facial electromyographic signals can be preprocessed. The preprocessing may include but is not limited to: removing power frequency interference, filtering, signal amplification, noise reduction processing and / or signal segmentation.

[0081] The presence of 50Hz power frequency signals in the environment can easily interfere with the EMG signals, which are manifested as peaks at 50Hz and its harmonics in the signal spectrum. A notch filter can be used to remove power frequency interference. For example, a notch filter with a center frequency of 50Hz can be designed to filter out interference signals at this frequency and its vicinity, making the facial EMG signals purer.

[0082] Facial EMG signals may contain interference information such as baseline drift and motion artifacts. Filters can be used to filter out interference, reduce interference caused by electrode movement, slow body shaking, etc., filter out high-frequency noise, and retain only the main effective frequency components in facial EMG signals. Signals within this passband will be considered to be relatively pure EMG signals.

[0083] Facial electromyographic signals are usually weak, and a signal amplification circuit can be integrated into the extended reality device to amplify the facial electromyographic signals so that the amplitude of the facial electromyographic signals reaches an appropriate range.

[0084] Noise reduction can be achieved through wavelet transform, which can decompose facial electromyographic signals into wavelet coefficients of different scales and frequencies. By thresholding these coefficients, those wavelet coefficients that mainly reflect noise are set to zero or adjusted appropriately, and then the signal is reconstructed to achieve the purpose of removing noise. For example, using appropriate wavelet basis functions and threshold selection methods, wavelet denoising is performed according to the characteristics of facial electromyographic signals, which can effectively reduce random noise interference such as white noise in the signal.

[0085] According to the actual action situation or research needs, the continuously collected facial electromyographic signals are segmented into meaningful segments. For example, when the facial electromyographic signals correspond to the blinking action, the start point is the beginning of the blinking action and the end point is the end of the blinking action. The signals during this period are intercepted as a separate segment to facilitate the subsequent analysis of the electromyographic signal characteristics corresponding to each action. Segmentation can be performed by manually observing the signal waveform combined with the time mark of the action, or some automatic algorithms can be used, such as threshold detection, energy change and other methods to determine the start and end of the action, to achieve automatic signal segmentation.

[0086] In this way, the current facial electromyographic signal is preprocessed to obtain a purer current facial electromyographic signal, so as to generate an accurate current facial electromyogram based on the current facial electromyographic signal.

[0087] The coordinate axis is set, and based on the preprocessed current facial electromyography signal, the facial electromyography of the wearer is drawn on the coordinate axis. The facial electromyography can reflect the change of the facial electromyography signal over time.

[0088] In step S120, a pre-collected target facial electromyogram of the wearer when speaking is obtained.

[0089] The target facial electromyogram is generated based on the target facial electromyogram when the wearer is speaking. The target facial electromyogram can be stored in a storage device of the augmented reality device or in a server, so that when it is necessary to determine whether the wearer is speaking, the target facial electromyogram can be directly obtained from the storage device or the server.

[0090] In one embodiment, multiple reference words can be preset to guide the wearer to speak the multiple reference words in advance, and when the wearer speaks the multiple reference words, each target facial electromyogram of the wearer is collected. Each facial electromyogram signal is preprocessed, and the electromyogram common features of each preprocessed target facial electromyogram signal are extracted, and the target facial electromyogram of the wearer when speaking is generated based on the electromyogram common features.

[0091] In step S130, it is determined whether the wearer is speaking based on the current facial electromyogram and the target facial electromyogram.

[0092] The similarity is calculated based on the current facial electromyogram of the wearer and the target facial electromyogram. When the similarity between the current facial electromyogram and the target facial electromyogram is greater than a similarity threshold, it is determined that the wearer is speaking.

[0093] In step S140, when the wearer speaks, the second voice directionally collected by the second voice collecting device is obtained.

[0094] When the wearer is speaking, the second voice collected by the second voice collection device is acquired. The second voice collection device can collect the second voice in a direction toward the wearer's mouth.

[0095] It is understandable that the contents of the first voice and the second voice are almost the same, but because the first voice collection device and the second voice collection device have different sound collection directions, the time and volume of the voice collected by the first voice collection device and the second voice collection device from the same sound source are different.

[0096] In step S150, the voice of a person other than the wearer is distinguished based on the first voice and the second voice.

[0097] When the wearer is speaking, the first voice and the second voice contain the wearer's voice. According to the time and volume of the first voice acquisition device to the first voice, and the time and volume of the second voice acquisition device to the second voice, the wearer's voice and the non-wearer's voice can be directionally distinguished, and the non-wearer's voice can be distinguished from the first voice and the second voice.

[0098] Among them, the specific technical means for directional differentiation can refer to relevant technologies.

[0099] In step S160, the voice of the non-wearer is converted into text, and the text is displayed on the virtual screen.

[0100] The distinguished speech of the non-wearer is converted into text, and the text is displayed on the virtual screen of the extended reality. Displaying the text on the virtual screen of the extended reality can help the wearer to accurately know the specific content of the non-wearer's speech.

[0101] The virtual screen of the extended reality device is not a real physical screen in the traditional sense, but a display area that is simulated and generated through technical means, using elements such as light and images to allow users to perceive various virtual information, images, videos, interactive interfaces and other content.

[0102] By adopting the technical solution of the embodiment of the present application, when the first voice collection device collects the first voice, the current facial electromyogram of the wearer can be obtained, as well as the target facial electromyogram of the wearer when speaking that is collected in advance; then, according to the current facial electromyogram and the target facial electromyogram, it is determined whether the wearer is speaking; when the wearer is speaking, the second voice is directionally collected by the second voice collection device; according to the first voice and the second voice, the voice of the non-wearer is distinguished; the voice of the non-wearer is converted into text, and the text is displayed on the virtual screen. In this way, it is determined whether the wearer is speaking according to the facial electromyogram of the wearer, so when the direction distinction is made according to the first voice and the second voice collected directionally, it can be determined whether the first voice and the second voice contain the voice of the wearer, so as to accurately distinguish the voice of the wearer and the non-wearer, and then accurately display the text converted from the voice of the non-wearer on the virtual screen, thereby improving the accuracy rate.

[0103] Based on the above technical solution, as an embodiment, the method may further include: when the wearer does not speak, converting the first voice into text; and displaying the text on the virtual screen.

[0104] When it is determined that the wearer is not speaking based on the current facial electromyogram and the target facial electromyogram, it can be determined that the user voices included in the first voice are all voices of non-wearers. Therefore, the first voice can be directly converted into text and the text can be displayed on the virtual screen.

[0105] By adopting the technical solution of the embodiment of the present application, when it is determined that the wearer is not speaking based on the current facial electromyogram and the target facial electromyogram, the first voice is directly converted into text and displayed on the virtual screen. This can avoid recognizing the user voice in the first voice as the wearer's voice, thereby causing omissions in the text displayed on the virtual screen.

[0106] Based on the above technical solution, as an embodiment, distinguishing the voice of the non-wearer according to the first voice and the second voice may include: acquiring voice features of the non-wearer; the voice features of the non-wearer are: common features extracted from multiple segments of voice data of each non-wearer sample located at each position; distinguishing the voice of the non-wearer according to the first voice, the second voice and the voice features of the non-wearer.

[0107] In advance, various extended reality devices can be used to collect a large amount of voice data of non-wearer samples speaking different words in various positions relative to the extended reality devices. Big data analysis is performed on the large amount of collected voice data to extract the voice features of the non-wearer from the large amount of voice data. The voice features of the non-wearer reflect the common features of the voices of the non-wearer.

[0108] The voice features of the non-wearer can be stored in the server, and the voice features of the non-wearer in the server can be updated. When the voice of the non-wearer is distinguished according to the first voice and the second voice, the voice features of the non-wearer can be obtained from the server.

[0109] In one embodiment, after the non-wearer's voice is initially distinguished based on the first voice and the second voice, the features of the non-wearer's voice can be extracted, and the features of the non-wearer's voice can be compared with the non-wearer's voice features obtained from the server, so as to further determine whether the determined non-wearer's voice is accurate, so as to make adjustments if it is inaccurate and determine the accurate non-wearer's voice.

[0110] In another embodiment, when distinguishing the voice of the non-wearer based on the first voice and the second voice, the voice features of the non-wearer are simultaneously considered so as to accurately determine the voice of the non-wearer directly from the first voice and the second voice.

[0111] By adopting the technical solution of the embodiment of the present application, the voice features of the non-wearer are obtained in advance based on big data, so that when determining the voice of the non-wearer, the accuracy of the determined voice of the non-wearer can be improved based on the voice features of the non-wearer, thereby improving the accuracy of the text converted from the voice of the non-wearer displayed on the virtual screen.

[0112] Based on the above technical solution, as an embodiment, converting the voice of the non-wearer into text and displaying the text on the virtual screen may include: converting the voice of the non-wearer into text; determining whether the text is in a target language; when the text is not in the target language, translating the text into a target text in the target language; and displaying the target text on the virtual screen.

[0113] When the text displays the voice of the non-wearer, the text can also be translated. After the voice of the non-wearer is converted into text, it can be determined whether the text is in the target language. The target language can be a language that can be selected by the user.

[0114] When the text is not in the target language, the text can be translated into the target text in the target language in real time, and the target text can be displayed on the virtual screen. Optionally, when the target text is displayed, the text can be displayed accordingly.

[0115] The target language may be the language that the wearer wants to learn. By converting the speech of the non-wearer into the target text of the target language and displaying it, the wearer can be helped to learn the target language.

[0116] The target language may be a language familiar to the wearer, and the language used by the non-wearer voice may be a language unfamiliar to the wearer. By converting the non-wearer voice into target text in the target language and displaying it, the wearer can understand what the non-wearer is saying specifically.

[0117] By adopting the technical solution of the embodiment of the present application, the voice of the non-wearer is converted into the target text of the target language, which can meet the various needs of the wearer and enhance the wearer's user experience.

[0118] Based on the above technical solution, as an embodiment, the method may further include: distinguishing the wearer's voice according to the first voice and the second voice; and responding to the voice command when the wearer's voice is a voice command.

[0119] The first voice and the second voice may include the voice of the wearer and the voice of the non-wearer. Therefore, when the voice of the non-wearer is distinguished based on the first voice and the second voice, the voice of the wearer may also be distinguished. The method of distinguishing the voice of the wearer may refer to the method of distinguishing the voice of the non-wearer.

[0120] When the wearer's voice is distinguished, it can be identified whether the wearer's voice contains a voice command. The voice command is a command used to operate the extended reality device.

[0121] In conventional technology, when using voice commands to operate an extended reality device, it is necessary to first wake up the voice command function of the extended reality device. In one embodiment of the present application, if it is recognized that the wearer's voice contains a voice command, the voice command can be directly responded to without waking up the voice command function of the extended reality device.

[0122] For example, without waking up the voice command function of the extended reality device, if it is recognized that the wearer's voice contains the voice command "turn on the camera function", the camera function of the extended reality device can be directly turned on.

[0123] By adopting the technical solution of the embodiment of the present application, the wearer can directly operate the extended reality device without waking up the voice command function of the extended reality device through the voice command of the wearer. In this way, the wearer does not need to wake up the voice command function of the extended reality device, thereby improving the user experience.

[0124] Based on the above technical solution, as an embodiment, the extended reality device may further include: a loudspeaker. The method may further include: distinguishing the voice of the wearer according to the first voice and the second voice; and amplifying and outputting the voice of the wearer through the loudspeaker.

[0125] The method for distinguishing the voice of the wearer can refer to the method for distinguishing the voice of the non-wearer. After distinguishing the voice of the wearer, the voice of the wearer can be amplified and output through the loudspeaker.

[0126] By adopting the technical solution of the embodiment of the present application, the wearer's voice and the non-wearer's voice are distinguished, so that only the wearer's voice is amplified and output. While amplifying the wearer's voice, the non-wearer's voice can also be avoided from being amplified at the same time, thereby avoiding the problem of high noise content in the amplified voice.

[0127] Based on the above technical solution, as an embodiment, the extended reality device may further include: headphones; the method may further include: playing the voice of the non-wearer toward the ears of the wearer through the headphones.

[0128] When the wearer wears the extended reality device, the earphone of the extended reality device can be directed toward the ear of the wearer. After the voice of the non-wearer is acquired, the voice of the non-wearer can be played toward the ear of the wearer through the earphone.

[0129] Optionally, the volume of the non-wearer's voice played by the headset can be adjusted.

[0130] By adopting the technical solution of the embodiment of the present application, the voice of the non-wearer is played toward the ear of the wearer through the earphone, which can reduce the impact of the noise in the environment on the voice of the non-wearer, thereby ensuring that the wearer can hear the voice of the non-wearer clearly.

[0131] In order to better implement the speech processing method of the present application, the present application also provides a speech processing device based on the above speech processing method. The meanings of the nouns are the same as those in the above speech processing method, and the specific implementation details can refer to the description in the method embodiment.

[0132] See also Figure 3 , Figure 3: is a schematic diagram of the structure of the voice processing device provided in an embodiment of the present application, wherein the voice processing device is applied to an extended reality device, and the extended reality device may include but is not limited to a first voice collection device, a second voice collection device, and a virtual screen. When the specific extended reality device is a wearable optical perspective smart glasses, in addition to the above structure, it may also include at least a glasses body frame, an optical display component, an electronic circuit component and / or a sensor, etc. The sensor built into the glasses may include at least an electromyographic signal sensor, a heart rate monitor, a blood glucose detector, a microphone, a camera and / or an eye tracker.

[0133] The speech processing device provided in the embodiment of the present application may include:

[0134] The electromyogram acquisition module 301 is used to acquire the current facial electromyogram of the wearer when the first voice acquisition device acquires the first voice;

[0135] A target acquisition module 302 is used to acquire a pre-collected target facial electromyogram of the wearer when speaking;

[0136] A determination module 303, for determining whether the wearer is speaking according to the current facial electromyogram and the target facial electromyogram;

[0137] A collection module 304, configured to obtain the second voice directionally collected by the second voice collection device when the wearer is speaking;

[0138] A distinguishing module 305, configured to distinguish a voice of a non-wearer according to the first voice and the second voice;

[0139] The display module 306 is used to convert the voice of the non-wearer into text and display the text on the virtual screen.

[0140] In one embodiment, the display module 306 includes:

[0141] A conversion unit, used for converting the speech of the non-wearer into text;

[0142] A language determination unit, used to determine whether the text is in a target language;

[0143] A translation unit, configured to translate the text into a target text in the target language when the text is not in the target language;

[0144] A text display unit is used to display the target text on the virtual screen.

[0145] In one embodiment, the device further comprises:

[0146] A first voice distinguishing module, used to distinguish the wearer's voice according to the first voice and the second voice;

[0147] In one embodiment, when the wearer's voice is a voice command, respond to the voice command.

[0148] In one embodiment, the extended reality device includes: a loudspeaker; the device also includes:

[0149] A second voice distinguishing module, used to distinguish the wearer's voice according to the first voice and the second voice;

[0150] The amplification module is used to amplify and output the wearer's voice through the loudspeaker.

[0151] In one embodiment, the extended reality device includes: headphones; the apparatus also includes:

[0152] The playing module is used to play the voice of the non-wearer toward the ear of the wearer through the earphone.

[0153] In one embodiment, the distinguishing module 305 includes:

[0154] A feature acquisition unit, used to acquire voice features of non-wearers; the voice features of non-wearers are: common features extracted from multiple voice data of each non-wearer sample located in each position;

[0155] A voice distinguishing unit is used to distinguish the voice of the non-wearer according to the first voice, the second voice and the voice features of the non-wearer.

[0156] In one embodiment, the device further comprises:

[0157] A text conversion module, used for converting the first speech into text when the wearer does not speak;

[0158] The text display module is used to display the text on the virtual screen.

[0159] By adopting the technical solution of the embodiment of the present application, when the first voice collection device collects the first voice, the current facial electromyogram of the wearer can be obtained, as well as the target facial electromyogram of the wearer when speaking that is collected in advance; then, according to the current facial electromyogram and the target facial electromyogram, it is determined whether the wearer is speaking; when the wearer is speaking, the second voice collection device directionally collects the second voice; according to the first voice and the second voice, the voice of the non-wearer is distinguished; the voice of the non-wearer is converted into text, and the text is displayed on the virtual screen. In this way, it is determined whether the wearer is speaking according to the facial electromyogram of the wearer, so when the direction distinction is made according to the first voice and the second voice collected directionally, it can be determined whether the first voice and the second voice contain the voice of the wearer, so as to accurately distinguish the voice of the wearer and the non-wearer, and then accurately display the text converted from the voice of the non-wearer on the virtual screen, thereby improving the accuracy rate.

[0160] For the specific definition of the speech processing device, please refer to the definition of the speech processing method above, which will not be repeated here. Each module in the above-mentioned speech processing device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0161] In addition, the present application also provides an electronic device, which may be an extended reality device. Figure 4 As shown, it shows a schematic diagram of the structure of the electronic device involved in this application, specifically:

[0162] The electronic device may include one or more processors 401 of processing cores and one or more computer-readable storage media memories 402 and other components. Those skilled in the art will appreciate that Figure 4 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0163] The processor 401 is the control center of the electronic device, and uses various interfaces and lines to connect various parts of the entire electronic device. By running or executing software programs and / or modules stored in the memory 402, and calling data stored in the memory 402, the processor 401 performs various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole. Optionally, the processor 401 may include one or more processing cores; preferably, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 401.

[0164] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 402 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0165] In one embodiment, the electronic device further includes a power supply 403 for supplying power to various components. Preferably, the power supply 403 can be logically connected to the processor 401 through a power management system, so as to manage charging, discharging, power consumption management and other functions through the power management system. The power supply 403 can also include any components such as one or more DC or AC power supplies, recharging systems, power supply device debugging circuits, power converters or inverters, and power status indicators.

[0166] In one embodiment, the electronic device may further include an input unit 404, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0167] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 401 in the electronic device will load the executable files corresponding to the processes of one or more application programs into the memory 402 according to the following instructions, and the processor 401 will run the application programs stored in the memory 402, thereby implementing the steps in any of the speech processing methods provided in the embodiments of the present application.

[0168] When the specific electronic device is wearable optical see-through smart glasses, in addition to the above structure, it also includes at least a glasses body frame, an optical display component, an electronic circuit component, a sensor, etc. The sensors built into the glasses include a heart rate monitor, a blood glucose tester, a microphone, a camera and / or an eye tracker.

[0169] Those skilled in the art will understand that Figure 4 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0170] In one embodiment, an electronic device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the method described in any embodiment of the present application is implemented.

[0171] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method described in any embodiment of the present application is implemented.

[0172] In some embodiments, a computer program product is also proposed, including a computer program or instructions, which implement the method described in any embodiment of the present application when executed by a processor.

[0173] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0174] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0175] To this end, the present application provides a computer-readable storage medium, on which a computer program is stored. The computer program can be loaded by a processor to execute the steps in any speech processing method provided in the present application.

[0176] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0177] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0178] Since the instructions stored in the computer-readable storage medium can execute the steps in any speech processing method provided in the present application, the beneficial effects that can be achieved by any speech processing method provided in the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0179] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0180] The above is a detailed introduction to a speech processing method, device, electronic device and computer-readable storage medium provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A speech processing method, characterized in that: Applied to an extended reality device, the extended reality device comprises: a first voice collection device, a second voice collection device and a virtual screen; the method comprises: When the first voice collection device collects the first voice, obtaining the current facial electromyogram of the wearer; Acquire a pre-collected target facial electromyogram of the wearer when speaking; Determining whether the wearer is speaking according to the current facial electromyogram and the target facial electromyogram; When the wearer is speaking, obtaining a second voice directionally collected by the second voice collecting device; Distinguishing a voice of a non-wearer according to the first voice and the second voice; and The non-wearer's voice is converted into text, and the text is displayed on the virtual screen.

2. The method according to claim 1, characterized in that The converting the non-wearer's voice into text and displaying the text on the virtual screen includes: converting the speech of the non-wearer into text; Determining whether the text is in the target language; When the text is not in the target language, translating the text into a target text in the target language; The target text is displayed on the virtual screen.

3. The method according to claim 1, characterized in that The method further comprises: Distinguishing the wearer's voice according to the first voice and the second voice; When the wearer's voice is a voice command, respond to the voice command.

4. The method according to claim 1, characterized in that: The extended reality device includes: a loudspeaker; the method further includes: Distinguishing the wearer's voice according to the first voice and the second voice; The wearer's voice is amplified and outputted through the loudspeaker.

5. The method according to claim 1, characterized in that The extended reality device includes: headphones; the method also includes: The voice of the non-wearer is played toward the ear of the wearer through the earphone.

6. The method according to claim 1, characterized in that The distinguishing the voice of the non-wearer according to the first voice and the second voice includes: Acquire voice features of non-wearers; the voice features of non-wearers are: common features extracted from multiple voice data of non-wearer samples located at various positions; The voice of the non-wearer is distinguished according to the first voice, the second voice and the voice features of the non-wearer.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: When the wearer is silent, converting the first speech into text; The text is displayed on the virtual screen.

8. A speech processing device, characterized in that: Applied to an extended reality device, the extended reality device comprises: a first voice collection device, a second voice collection device and a virtual screen; the device comprises: an electromyogram acquisition module, configured to acquire the current facial electromyogram of the wearer when the first voice acquisition device acquires the first voice; A target acquisition module, used to acquire a pre-collected target facial electromyogram of the wearer when speaking; A determination module, used to determine whether the wearer is speaking according to the current facial electromyogram and the target facial electromyogram; A collection module, used for acquiring the second voice directionally collected by the second voice collection device when the wearer is speaking; a distinguishing module, configured to distinguish a voice of a non-wearer according to the first voice and the second voice; and The display module is used to convert the voice of the non-wearer into text and display the text on the virtual screen.

9. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps in the speech processing method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the speech processing method according to any one of claims 1 to 7 are implemented.

Images