Speech enhancement methods, devices, electronic devices, storage media, and software products

Abstract

This application discloses a speech enhancement method, apparatus, electronic device, storage medium, and program product, belonging to the field of artificial intelligence technology. The method includes: during a call conducted through a head-mounted device, determining a first real-world object in a first scene based on the gaze information of a first user, wherein the first user is a user wearing the head-mounted device, and the first real-world object is a real-world object exhibiting speaking behavior and for which the first user intends to perform sound suppression processing; acquiring first voice feature information of the first real-world object; and suppressing the voice signal corresponding to the first real-world object in the detected voice signal of the first scene based on the first voice feature information and the first user's second voice feature information.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence technology, specifically relating to a speech enhancement method, device, electronic device, storage medium, and program product. Background Technology

[0002] Currently, in order to improve the quality of calls with other users, electronic devices can perform noise reduction processing on the voice signal collected by the microphone, thereby removing environmental noise signals from the voice signal and improving call quality.

[0003] In related technologies, during the process of acquiring speech signals, electronic devices can estimate environmental noise signals from the acquired speech signals because human voice signals have time-varying and non-stationary characteristics. They can also suppress environmental noise signals to remove them from the speech signals.

[0004] However, when the aforementioned voice signal includes human voice interference, it is impossible to identify the human voice interference signal contained in the voice signal by the above method because it also has time-varying and non-stationary characteristics. As a result, the interference of human voice interference signal cannot be removed, making it difficult to highlight the human voice signal of the user of the electronic device during the call; thus, the call quality of the electronic device is poor. Summary of the Invention

[0005] The purpose of this application is to provide a voice enhancement method, apparatus, electronic device, storage medium, and program product that can reduce interference from other voice signals during a call and improve call quality.

[0006] In a first aspect, embodiments of this application provide a voice enhancement method, which includes: during a call conducted through a head-mounted device, determining a first real-world object in a first scene based on the gaze information of a first user, wherein the first user is a user wearing a head-mounted electronic device, and the first real-world object is a real-world object that exhibits speaking behavior and the first user intends to perform sound suppression processing; acquiring first voice feature information of the first real-world object; and suppressing the voice signal corresponding to the first real-world object in the voice signal detected in the first scene based on the first voice feature information and the second voice feature information of the first user.

[0007] Secondly, embodiments of this application provide a voice enhancement device, which includes a determining module, an acquiring module, and a processing module. The determining module is used to determine a first real-world object in a first scene based on the gaze information of a first user during a call conducted through a head-mounted device. The first user is a user wearing a head-mounted electronic device, and the first real-world object is a real-world object exhibiting speaking behavior and for which the first user intends to perform sound suppression processing. The acquiring module is used to acquire first voice feature information of the first real-world object determined by the determining module. The processing module is used to perform suppression processing on the voice signal corresponding to the first real-world object in the detected voice signal in the first scene based on the first voice feature information acquired by the acquiring module and the second voice feature information of the first user.

[0008] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, the memory storing programs or instructions executable on the processor, the programs or instructions, when executed by the processor, implementing the steps of the method described in the first aspect.

[0009] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0010] Fifthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the method as described in the first aspect.

[0011] In a sixth aspect, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the method described in the first aspect.

[0012] In this embodiment, during a call using a head-mounted device, a first real-world object in a first scene can be determined based on the gaze information of a first user. The first user is the user wearing the head-mounted electronic device, and the first real-world object is the object that is speaking and the first user intends to suppress the sound. Then, first voice feature information of the first real-world object is acquired. Finally, based on the first voice feature information and the first user's second voice feature information, the voice signal corresponding to the first real-world object in the detected voice signal in the first scene is suppressed. In this solution, during a call using a head-mounted device, since the user's gaze information (i.e., eye tracking) can be used to identify the first real-world object for which the user wants to suppress the sound, and the voice feature information of that first real-world object can be obtained in a targeted manner, the voice signal corresponding to the first real-world object can be identified from the first scene using this voice feature information. This allows for voice suppression of the voice signal corresponding to the first real-world object, suppressing human voice interference signals other than the voice signal of the user wearing the head-mounted device during the call. This avoids the difficulty in highlighting the voice signal of the user wearing the head-mounted device due to human voice interference signals during the call, thus improving call quality. Attached Figure Description

[0013] Figure 1 This is one of the flowcharts of a speech enhancement method provided in the embodiments of this application;

[0014] Figure 2 This is a second flowchart of a speech enhancement method provided in the embodiments of this application;

[0015] Figure 3 This is the third flowchart of a speech enhancement method provided in the embodiments of this application;

[0016] Figure 4 This is one of the example schematic diagrams of a call interface for a head-mounted device provided in an embodiment of this application;

[0017] Figure 5 This is the fourth flowchart of a speech enhancement method provided in the embodiments of this application;

[0018] Figure 6 This is a second example of a call interface of a head-mounted device provided in an embodiment of this application;

[0019] Figure 7 This is a third example of a call interface for a head-mounted device provided in an embodiment of this application;

[0020] Figure 8 This is the fourth flowchart of a speech enhancement method provided in the embodiments of this application;

[0021] Figure 9 This is the fifth flowchart of a speech enhancement method provided in the embodiments of this application;

[0022] Figure 10 This is a schematic diagram of the structure of a speech enhancement device provided in an embodiment of this application;

[0023] Figure 11 This is one of the hardware structure diagrams of an electronic device provided in the embodiments of this application;

[0024] Figure 12 This is a second schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0025] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0026] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0027] The terms "at least one," "at least one of," etc., used in the specification and claims of this application refer to any one, any two, or a combination of two or more of the included items. For example, at least one of a, b, and c can mean: "a," "b," "c," "a and b," "a and c," "b and c," and "a, b, and c," where a, b, and c can be single or multiple. Similarly, "at least two" refers to two or more items, and its meaning is similar to that of "at least one."

[0028] The speech enhancement method, apparatus, electronic device, storage medium, and program product provided in this application will be described in detail below with reference to the accompanying drawings and through specific embodiments and application scenarios.

[0029] In today's era of increasing digitalization and remote work, video conferencing or voice conferencing has become an indispensable tool in fields such as business communication, distance education, and telemedicine. In these scenarios, clear voice transmission is crucial for effective communication. During a call, electronic devices such as Virtual Reality (VR), Augmented Reality (AR), or Mixed Reality (MR) comprehensively capture the voice and ambient noise to obtain audio. To improve call quality, these electronic devices can perform noise reduction processing on the audio, effectively suppressing or eliminating ambient noise while maintaining the naturalness and intelligibility of the voice.

[0030] Typically, electronic devices can use traditional methods such as spectral subtraction, beamforming, and adaptive filtering to reduce noise in the audio they capture. However, while these methods are relatively effective at handling stationary, non-speech noise, they are ineffective when there is interference from human voices during a call. This is because the spectra of multiple human voice signals are similar to those of the user's voice and are time-varying and non-stationary. Consequently, these methods cannot distinguish between the user's voice signal and the interference signals, resulting in poor call quality as the electronic device fails to highlight the user's voice.

[0031] For example, speaker A is video chatting with friend B in the living room. During the call, family members C and D, who are also in the living room, are also chatting. Therefore, friend B hears the voices of A, C, and D, making it difficult to hear A's voice clearly.

[0032] The voice enhancement methods, devices, electronic devices, storage media, and program products provided in this application can be applied to scenarios where users are making calls. Specifically, in scenarios where users are making calls in this application, when a user is in a scene or environment with multiple people (such as a shopping mall or amusement park), the user can make calls with other users by wearing a head-mounted device, so that the head-mounted device can suppress human voice interference signals present in the scene.

[0033] In this embodiment of the application, the aforementioned call may include any of the following: video call, voice call, telephone call, etc.

[0034] For example, when user A is in an amusement park and is having a video call with user B via an MR device, there are interfering users C and D around user A. If user A hears a loud sound from interfering user C, user A can look at user C. The MR device can determine the direction of user C through user A's eyeballs and perform directional sound pickup to obtain user C's voice signal. Through this voice signal, the MR device can obtain the sound feature information of the signal, and then process the sound signal in the current scene collected by the MR device using the sound feature information to suppress user C's voice during the video call.

[0035] In the voice enhancement methods, apparatus, electronic devices, storage media, and program products provided in the embodiments of this application, during a call using a head-mounted device, the user's gaze information, i.e., eye tracking, can be used to identify the first real-world object for which the user wants to perform voice suppression processing. Then, the voice feature information of the first real-world object can be directionally acquired. Therefore, through this voice feature information, the voice signal corresponding to the first real-world object can be identified from the aforementioned first scene. This allows for voice suppression processing of the voice signal corresponding to the first real-world object, suppressing voice interference signals other than the user's voice signal during the call. This avoids the difficulty in highlighting the user's voice signal due to voice interference signals during the call, thus improving call quality.

[0036] The speech enhancement method provided in this application can be implemented by a speech enhancement device, which can be an electronic device or a functional module within an electronic device. The following description uses an electronic device as an example to illustrate the technical solution provided in this application.

[0037] This application provides a speech enhancement method. Figure 1 A flowchart of a speech enhancement method provided in an embodiment of this application is shown, which can be applied to head-mounted devices. Figure 1 As shown, the speech enhancement method provided in this application embodiment may include the following steps 201 to 203.

[0038] Step 201: During a call using a head-mounted device, the head-mounted device determines the first real-world object in the first scene based on the first user's gaze information.

[0039] In this embodiment of the application, the first user is a user wearing a head-mounted device.

[0040] In this embodiment of the application, the first real object can be a real object that has the behavior of speaking and the first user intends to perform sound suppression processing.

[0041] Optionally, in the embodiments of this application, the head-mounted device described above can be any of the following: VR device, AR device, or MR device.

[0042] Optionally, in this embodiment, the call can be a voice call or a video call; the specific type can be determined according to the actual usage, and this embodiment does not impose any restrictions.

[0043] In this embodiment of the application, the first scenario can be the environment in which the first user is currently located.

[0044] Optionally, in this embodiment, the first scenario described above can be any of the following: shopping mall, parking lot, supermarket, scenic spot, or amusement park, etc. The specific scenario can be determined according to the actual situation, and this embodiment does not impose any limitations.

[0045] In this embodiment of the application, the first real-world object can be a real person existing in the first scene.

[0046] Optionally, in the embodiments of this application, the first real object can be any of the following: a person or an animal.

[0047] Optionally, in this embodiment of the application, the aforementioned first real-world object can be one or more. Specifically, it can be determined by the first user's selection.

[0048] Optionally, in the embodiments of this application, the above-mentioned sound suppression processing can be any of the following: sound noise reduction processing, sound attenuation processing, or sound shielding processing.

[0049] In this embodiment of the application, the gaze information may include the direction in which the first user's eyes are looking.

[0050] Optionally, in this embodiment, the direction can be any of the following: east, west, south, north, southeast, southwest, northeast, northwest, etc. The specific direction can be determined according to the actual situation, and this embodiment does not impose any restrictions.

[0051] In this embodiment of the application, the head-mounted device can obtain the direction in which the first user's eyes are looking through the front-facing camera, and determine the real object whose lips are moving among the real objects in the first scene in that direction as the first real object.

[0052] Optionally, in this embodiment, the front-facing camera can be any of the following: a wide-angle camera, a macro camera, an ultra-wide-angle camera, or an auto-zoom camera, etc. The specific type can be determined based on actual circumstances, and this embodiment does not impose any limitations.

[0053] Optionally, in this embodiment of the application, if the head-mounted device does not detect the first real object in the direction that the first user is looking at, the head-mounted device may display a first prompt message to indicate to the first user that there is no first real object in the current direction.

[0054] For example, the first prompt message could be "No people or animals found in the current direction, please adjust your direction".

[0055] Optionally, in this embodiment of the application, if the head-mounted device does not detect the first real object in the direction that the first user is looking at, the head-mounted device may display a second prompt message to prompt the first user that there is a real object in another direction; then, if the direction that the first user is looking at is the same as other directions, the head-mounted device may determine the first real object in the current direction.

[0056] In this embodiment of the application, the head-mounted device can determine which direction the real object is in by collecting the sound signal of the real object through the microphone, and generate a second prompt message.

[0057] For example, the second prompt message could be: "Someone is speaking in the southeast direction," and the head-mounted device could display a southeast direction indicator.

[0058] Step 202: The head-mounted device acquires the first voice feature information of the first real object.

[0059] Optionally, in this embodiment, the first sound feature information may include at least one of the following: linear prediction cepstral information, line spectrum pair information, Mel frequency cepstral information, pitch information, voiceprint information, and timbre information, etc. The specific details can be determined according to actual usage requirements, and this embodiment does not impose any limitations.

[0060] In this embodiment of the application, after identifying the first real object, the head-mounted device can directionally pick up the sound signal of the first real object, and then obtain the first sound feature information through the sound signal.

[0061] In this embodiment of the application, the electronic device can obtain the first sound feature information through a sound processing model.

[0062] Optionally, in this embodiment, the sound processing model can be any of the following: an artificial intelligence (AI) model, a neural network model, or a large-scale model, etc. The specific model can be determined according to actual usage requirements, and this embodiment does not impose any limitations.

[0063] For example, taking the above-mentioned sound processing model as a speech neural network model, the speech neural network model may include a speech coding network. The head-mounted device can input the sound signal of the first real object into the speech neural network model, and then perform convolution processing on the sound signal of the first real object through the speech coding network to obtain the first sound feature information.

[0064] Optionally, in the embodiments of this application, combined with Figure 1 ,like Figure 2 As shown, step 202 can be implemented through steps 202a and 202b below.

[0065] Step 202a: The head-mounted device acquires the first sound signal of the first real object in the first scene based on gaze information and the sound acquisition module array in the head-mounted device.

[0066] In this embodiment of the application, the sound acquisition module array can be a microphone array, which contains multiple microphones and the multiple microphones are located in different positions in the head-mounted device.

[0067] Optionally, in this embodiment, the microphone can be any of the following: an electrodynamic microphone, a condenser microphone, a piezoelectric microphone, or a pressure microphone, etc. The specific type can be determined according to actual usage requirements, and this embodiment does not impose any limitations.

[0068] In this embodiment of the application, the head-mounted device can directionally pick up the first sound signal of the first real object through the sound acquisition module array.

[0069] It should be noted that the aforementioned directional sound pickup is one of many methods for speech enhancement. Directional sound pickup is to pick up the target signal in a mixed signal according to the direction of the sound source. That is, it only picks up the sound signal that is propagating from a specific direction, while noise and interference signals from other directions can be ignored, attenuated, or shielded, thereby achieving the effect of directional sound signal acquisition.

[0070] In this embodiment of the application, the head-mounted device can determine the direction of directional sound pickup based on the above-mentioned gaze information, and then collect the sound signal of the first real object through multiple microphones at different positions in the microphone array in that direction. Then, the sound signals collected by the multiple microphones at different positions are fused together to obtain the above-mentioned first sound signal.

[0071] Optionally, in this embodiment of the application, when there are multiple first real objects, the head-mounted device can pick up the first sound signals of the multiple first real objects in a directional manner.

[0072] For example, when there are three first real objects, the head-mounted device can pick up the first sound signals of the three first real objects in a directional manner at preset intervals, such as 3 seconds.

[0073] For example, after the head-mounted device identifies three first real-world objects, it can directionally pick up the first sound signal of the first real-world object within 0 to 3 seconds; within 4 to 6 seconds, it can directionally pick up the first sound signal of the second real-world object; and within 7 to 9 seconds, it can directionally pick up the first sound signal of the third real-world object.

[0074] It should be noted that the specific process of the head-mounted device picking up the first sound signals of the three first real objects can be found in the above embodiments, and will not be repeated here to avoid repetition.

[0075] Step 202b: The head-mounted device performs feature extraction processing on the first sound signal to obtain the first sound feature information.

[0076] In this embodiment of the application, the head-mounted device can input the first sound signal into the sound processing model, and then identify and obtain the first sound feature information through the speech coding network.

[0077] Optionally, in this embodiment of the application, when the first sound signal includes multiple first sound signals, the head-mounted device can sequentially input the multiple first sound signals into the speech coding network to obtain multiple first sound feature information corresponding to the multiple first sound signals, with one first sound signal corresponding to one first sound feature information.

[0078] It should be noted that the process by which the head-mounted device obtains the first sound feature information through the sound processing model can be found in the above embodiments, and will not be repeated here to avoid repetition.

[0079] In this embodiment, the head-mounted device can use gaze information to directionally pick up the first sound signal of the first real object, so that the first sound signal can include only the sound of the first real object. Based on the first sound signal, accurate first sound feature information can be extracted. Then, the head-mounted device can use the first sound feature information to accurately identify the voice signal that interferes with the user from the voice signal detected in the first scene.

[0080] Step 203: The head-mounted device performs suppression processing on the voice signal corresponding to the first real object in the voice signal detected in the first scene, based on the first voice feature information and the second voice feature information of the first user.

[0081] Optionally, in this embodiment, the second sound feature information may include at least one of the following: linear prediction cepstral information, line spectrum pair information, Mel frequency cepstral information, pitch information, voiceprint information, and timbre information, etc. The specific details can be determined according to actual usage requirements, and this embodiment does not impose any limitations.

[0082] In this embodiment of the application, the voice signal detected in the first scenario may include: the voice signal of the first user, the voice signal of the first real object, and the environmental noise signal.

[0083] It should be noted that the specific implementation process can be found in the following embodiments, and will not be repeated here to avoid repetition.

[0084] In the speech enhancement method provided in this application embodiment, during a call via a head-mounted device, the head-mounted device can determine a first real-world object in a first scene based on the gaze information of a first user. The first user is a user wearing a head-mounted electronic device, and the first real-world object is a real-world object that exhibits speaking behavior and that the first user intends to perform sound suppression processing. Then, the head-mounted device can acquire first voice feature information of the first real-world object. Finally, based on the first voice feature information and the second voice feature information of the first user, the speech signal corresponding to the first real-world object in the detected speech signal in the first scene is suppressed. In this solution, during a call using a head-mounted device, the electronic device can identify the first real-world object for which the user wants to perform sound suppression processing based on the user's gaze information, i.e., through eye tracking. Then, it can directionally acquire the sound feature information of the first real-world object. Therefore, through this sound feature information, the voice signal corresponding to the first real-world object can be identified from the aforementioned first scene, thereby performing sound suppression processing on the voice signal corresponding to the first real-world object. This suppresses human voice interference signals other than the voice signal of the user wearing the head-mounted device during the call, avoiding the difficulty in highlighting the voice signal of the user wearing the head-mounted device due to human voice interference signals during the call, and improving call quality.

[0085] Optionally, in the embodiments of this application, combined with Figure 1 ,like Figure 3 As shown, before "determining the first real-world object in the first scene based on the first user's gaze information" in step 201 above, the speech enhancement method provided in this application embodiment further includes the following step 301, and the "determining the first real-world object in the first scene based on the first user's gaze information" in step 201 above can be specifically implemented through the following steps 201a to 201c.

[0086] Step 301: During a call using the headset, the headset displays the first screen.

[0087] In this embodiment of the application, the first screen includes at least one object identifier, and each of the at least one object identifier is used to indicate a real object in the first scene.

[0088] Optionally, in this embodiment of the application, the first screen can be a real screen corresponding to the first scene.

[0089] Optionally, in this embodiment of the application, the first screen may further include a virtual call interface.

[0090] Optionally, in this embodiment, the object identifier may include at least one of the following: image identifier, text identifier, and emoticon identifier, etc. The specific identifier can be determined based on actual usage, and this embodiment does not impose any limitations.

[0091] In this embodiment of the application, the head-mounted device can display a first image through its screen.

[0092] For example, such as Figure 4 As shown, taking the first scenario as a living room with two interfering users as an example, and using a head-mounted device as an MR device, during a call between the first user and user B through the MR device, the MR device can display a first screen 10 on its screen. This first screen 10 can display the actual layout of the living room and the user icons of the two interfering users. Figure 4 User C and User D are represented in the middle; and the video call interface 11 is displayed in the first screen 10.

[0093] Step 201a: The head-mounted device marks at least N object identifiers from at least one object identifier in the first screen based on gaze information.

[0094] In this embodiment of the application, the above N object identifiers are used to indicate real objects that are speaking.

[0095] In this embodiment of the application, the head-mounted device can determine the gaze direction in the first screen based on the gaze information of the first user, and then detect whether the mouth of at least one real object in the gaze direction has changed; if the mouths of N real objects in at least one real object in the gaze direction have changed, the head-mounted device can mark N object identifiers corresponding to the N real objects in the first screen.

[0096] Optionally, in this embodiment, the aforementioned markings can be any of the following: color markings, control addition markings, text markings, or pattern markings, etc. The specific markings can be determined based on actual usage, and this embodiment does not impose any limitations.

[0097] Step 201b: The head-mounted device receives a first input for the first object identifier among N object identifiers.

[0098] In this embodiment of the application, the first input is used to select a first object identifier from N object identifiers.

[0099] Optionally, in this embodiment, the first input includes, but is not limited to: a user clicking on the first object identifier with their finger, a voice command input by the user, a specific gesture input by the user, or other feasible inputs. The specific input can be determined according to the actual usage situation, and this embodiment does not limit it.

[0100] In some embodiments of this application, the aforementioned specific gesture can be any one of a single-click gesture, a swipe gesture, a drag gesture, a pressure-recognition gesture, a long-press gesture, an area-changing gesture, a double-press gesture, or a double-tap gesture.

[0101] In some embodiments of this application, the aforementioned click input can be a single click, a double click, or any number of clicks, and can also be a long press or a short press. For example, the aforementioned first input can be a single click by the user on the aforementioned first object identifier.

[0102] Optionally, in this embodiment of the application, the first object identifier mentioned above can be one or more.

[0103] Optionally, in this embodiment of the application, when the first object identifier is multiple object identifiers, the head-mounted device can receive multiple first inputs from the first user to determine multiple first object identifiers from N object identifiers.

[0104] Step 201c: The head-mounted device responds to the first input and determines the real object indicated by the first object identifier as the first real object.

[0105] Optionally, in embodiments of this application, when there are multiple first object identifiers, the head-mounted device can respond to multiple first inputs respectively and determine the real object indicated by each of the multiple first object identifiers as the first real object.

[0106] Optionally, in this embodiment of the application, after the head-mounted device determines the real object indicated by the first object identifier as the first real object, the head-mounted device may display a third prompt message around the first object identifier to prompt the first user that the real object indicated by the first object identifier has been determined as the first real object.

[0107] For example, the aforementioned third prompt message could be: "It has been determined that this user is a disruptive user."

[0108] Optionally, in this embodiment of the application, after the head-mounted device determines the real object indicated by the first object identifier as the first real object, it can receive a second input from the first user to deselect the real object indicated by the first object identifier as the first real object.

[0109] Optionally, in this embodiment, the second input includes, but is not limited to: a user clicking on the first object identifier with their finger, a voice command input by the user, a specific gesture input by the user, or other feasible inputs. The specific input can be determined according to the actual usage situation, and this embodiment does not limit it.

[0110] Optionally, in this embodiment of the application, when there are multiple first object identifiers, the head-mounted device can, based on the input of the first user to the second object identifier among the multiple first object identifiers, deselect the real object indicated by the second object identifier, and determine the real object indicated by the multiple first object identifiers other than the second object identifier as the first real object.

[0111] In this embodiment, the head-mounted device can select a real user as the interference user based on the input of the first user, thereby improving the flexibility and accuracy of the electronic device in determining the interference user.

[0112] Optionally, in the embodiments of this application, combined with Figure 3 ,like Figure 5 As shown, step 201a can be implemented by step 201a1, and step 201b can be implemented by step 201b1.

[0113] Step 201a1: Based on gaze information, the head-mounted device displays N controls at the positions corresponding to N object identifiers in the first screen.

[0114] In this embodiment of the application, the above N controls correspond one-to-one with N object identifiers, and each of the N controls is used to indicate at least one sound suppression processing method.

[0115] Optionally, in this embodiment of the application, each of the above N controls may include at least one sub-control, and each of the at least one sub-control is used to indicate a sound suppression processing method.

[0116] Optionally, in this embodiment of the application, the above-mentioned position can be preset by the head-mounted device.

[0117] For example, the head-mounted device may display N controls above N object identifiers in the first screen.

[0118] For example, combined Figure 4 ,like Figure 6 As shown, after the MR device identifies two first object identifiers, the MR device can display a control 16 above the first object identifier 15, which includes three sub-control identifiers. Figure 6 The numbers 161, 162, and 163 are used to represent these; and control 18, which includes three sub-control identifiers, is displayed above the second first object identifier 17. Figure 6 The numbers 181, 182, and 183 are used to represent them.

[0119] Optionally, in this embodiment of the application, after the head-mounted device displays N controls in the first screen, the head-mounted device can update the display position of the N controls in the first screen based on the input of the first user to the N controls.

[0120] Step 201b1: The head-mounted device receives the first input to the control corresponding to the first object identifier.

[0121] In this embodiment of the application, the first input is used to select the control corresponding to the first object identifier from N controls.

[0122] In this embodiment of the application, the control corresponding to the first object identifier can be used to indicate a sound suppression processing method.

[0123] Optionally, in this embodiment of the application, the first input can be an input to a first sub-control among at least one sub-control.

[0124] In this embodiment of the application, the head-mounted device can determine the real object corresponding to the first object identifier as the first real object through the first input, and process the sound signal of the first real object according to the sound suppression processing method of the control corresponding to the first object identifier.

[0125] For example, combined Figure 6 ,like Figure 7 As shown, the first user can interact with the third sub-control in control 16 displayed on user C of the MR device, namely... Figure 7 Clicking the sub-control corresponding to 163 allows the head-mounted device to identify user C as the first real-world object, and the sound signal of the first real-world object is processed according to the sound suppression processing method corresponding to the third sub-control 163.

[0126] Optionally, in this embodiment of the application, when there are multiple first object identifiers, the head-mounted device can receive multiple first inputs from the first user to the multiple first object identifiers in order to determine the sound suppression processing method for each object identifier among the multiple first object identifiers that interferes with the user.

[0127] For example, taking two first object identifiers as an example, the first user can click on the third sub-control in the control corresponding to the first first object identifier to input, so that the head-mounted device can determine that the sound suppression processing method for the user interfering with the first first object identifier is speech noise reduction processing; then, the first user can click on the second sub-control in the control corresponding to the second first object identifier to input, so that the head-mounted device can determine that the sound suppression processing method for the user interfering with the second first object identifier is speech attenuation processing.

[0128] It should be noted that all inputs in the above embodiments are virtual inputs, that is, they are not actual touch inputs on the screen of the head-mounted device, but rather inputs to a virtual interface displayed on the head-mounted device.

[0129] In this embodiment, the head-mounted device can determine the sound suppression processing method for the voice signal that interferes with the user based on the user's input to the control, and then process the voice signal that interferes with the user according to the sound suppression processing method selected by the user, thereby improving the flexibility of the head-mounted device in determining the sound suppression processing method.

[0130] Optionally, in this embodiment of the application, the first input is the input of selecting the sound suppression processing method through the control corresponding to the first object identifier.

[0131] For example, combined Figure 1 ,like Figure 8 As shown, step 203 above can be specifically implemented through step 203a below.

[0132] Step 203a: The head-mounted device adopts the sound suppression processing method corresponding to the first input, and suppresses the speech signal corresponding to the first real object in the speech signal detected in the first scene based on the first sound feature information and the second sound feature information.

[0133] Optionally, in this embodiment, when the sound suppression processing method corresponding to the first object identifier is a sound attenuation method, the head-mounted device can input the sound attenuation processing identifier corresponding to the sound attenuation processing method, the first sound feature information, the second sound feature information, and the aforementioned speech signal into the sound processing model, so that the sound processing model can calculate at least one sound gain parameter; and multiply the at least one sound gain parameter by a preset attenuation parameter to obtain at least one second sound gain parameter; then, perform frequency domain transformation processing on the speech signal to obtain a frequency domain transformed speech signal; finally, multiply the frequency domain transformed speech signal by attenuation and at least one second sound gain parameter to obtain the speech signal with sound attenuation processing corresponding to the first object identifier.

[0134] Optionally, in this embodiment of the application, when the above-mentioned sound suppression processing method is a sound noise reduction processing method, the head-mounted device can input the sound noise reduction processing identifier, the first sound feature information, the second sound feature information, and the above-mentioned speech signal corresponding to the sound noise reduction processing method into the above-mentioned sound processing model, so that the sound processing model can calculate at least one sound gain parameter, and perform frequency domain transformation processing on the speech signal to obtain the frequency domain transformed speech signal. Finally, the frequency domain transformed speech signal and at least one sound gain parameter are multiplied to obtain the sound noise-reduced speech signal.

[0135] It should be noted that the process of calculating at least one sound gain parameter for the head-mounted device can be found in the following embodiments, and will not be repeated here to avoid repetition.

[0136] In this embodiment, the head-mounted device can process the voice signal detected in the first scene based on the first voice feature information and the second voice feature information by using the voice suppression processing method corresponding to the first input. It can be understood that different voice suppression processing methods can produce voice signals with different effects. The head-mounted device can flexibly determine the voice suppression processing method according to the user's selection, thereby improving the flexibility of voice signal processing.

[0137] Optionally, in the embodiments of this application, combined with Figure 1 ,like Figure 9 As shown, step 203 can be implemented through steps 401 to 403 as described below.

[0138] Step 401: The head-mounted device calculates at least one sound gain parameter based on the first sound feature information, the second sound feature information, and the speech signal.

[0139] In this embodiment of the application, each of the at least one sound gain parameter corresponds to a time-frequency point of the speech signal.

[0140] In this embodiment of the application, the aforementioned time-frequency point refers to the coordinate point formed by the combination of the amplitude of the aforementioned speech signal in the time domain and its frequency in the frequency domain.

[0141] In this embodiment, the head-mounted device can input first sound feature information, second sound feature information, and a speech signal into the aforementioned sound processing model. Then, the sound processing model can identify human voice interference speech signals from the speech signal based on the first sound feature information, and identify the first user's human voice speech signal from the speech signal through the second sound feature information. Next, the sound processing model can calculate at least one first ratio between the amplitude of the human voice interference speech signal and the amplitude of the human voice speech signal in the time domain, and calculate at least one second ratio between the frequency of the human voice interference speech signal and the frequency of the human voice speech signal in the frequency domain. Finally, the sound processing model can multiply at least one first ratio and at least one second ratio to obtain at least one sound gain parameter.

[0142] In the embodiments of this application, each of the at least one first ratio value corresponds one-to-one with each of the at least one second ratio value.

[0143] Step 402: The head-mounted device performs frequency domain conversion on the speech signal to obtain the frequency domain converted speech signal.

[0144] In this embodiment of the application, the head-mounted device can perform a Fast Fourier Transform (FFT) on the speech signal using the above-mentioned sound processing model to obtain the speech signal after frequency domain transformation.

[0145] Step 403: The head-mounted device performs suppression processing on the speech signal corresponding to the first real object in the frequency domain converted speech signal based on at least one sound gain parameter.

[0146] In this embodiment of the application, the head-mounted device can perform a dot product of at least one sound gain parameter with the frequency domain speech signal corresponding to at least one frequency point of the frequency domain converted speech signal through the above-described sound processing model to obtain the processed frequency domain speech signal. Then, the processed frequency domain speech signal is subjected to an inverse fast fourier transform (IFFT) to obtain the processed speech signal.

[0147] In this embodiment, the head-mounted device can identify the interfering user's voice signal from the aforementioned speech signal by using the first voice feature information corresponding to the interfering user's voice signal, and then suppress the interfering user's voice signal, thereby highlighting the user's voice signal in the suppressed speech signal, thus improving the call quality.

[0148] Optionally, in this embodiment of the application, before step 201 above, the speech enhancement method provided in this embodiment of the application further includes the following steps 501 and 502.

[0149] Step 301: The head-mounted device acquires the second sound signal and obtains the sound parameters corresponding to the second sound signal.

[0150] In this embodiment of the application, the head-mounted device can acquire a second sound signal through the microphone module in the head-mounted device and analyze the second sound signal to obtain the above-mentioned sound parameters.

[0151] Optionally, in embodiments of this application, the aforementioned sound parameters may include at least one of signal-to-noise ratio and volume.

[0152] In this embodiment of the application, after the head-mounted device acquires the second sound signal, the head-mounted device can identify the intensity of the human voice signal and the intensity of the environmental noise signal in the second sound signal, and then determine the ratio between the intensity of the human voice signal and the intensity of the environmental noise signal as the signal-to-noise ratio.

[0153] Step 302: When the sound parameters meet the first condition, the head-mounted device acquires the sound feature information corresponding to the second sound signal and determines the sound feature information corresponding to the second sound signal as the second sound feature information.

[0154] Optionally, in this embodiment of the application, when the sound parameters meet the first condition, the head-mounted device can register the second sound signal, that is, store the second sound signal and obtain the sound feature information corresponding to the second sound signal.

[0155] Optionally, in this embodiment of the application, when the above-mentioned sound parameters include signal-to-noise ratio, and when the signal-to-noise ratio of the second sound signal is greater than or equal to a preset signal-to-noise ratio threshold, the head-mounted device can acquire the sound feature information corresponding to the second sound signal.

[0156] Optionally, in this embodiment, the preset signal-to-noise ratio threshold can be preset by the head-mounted device. For example, the preset signal-to-noise ratio threshold can be 3KdB.

[0157] For example, if the signal-to-noise ratio (SNR) of the second audio signal acquired by the head-mounted device is 5 kdB, after acquiring the SNR of the second audio signal, the head-mounted device can compare the SNR of 5 kdB with a preset SNR threshold of 3 kdB. At this time, the head-mounted device determines that the SNR of the second audio signal is greater than the preset SNR threshold; then it registers the second audio signal and acquires the audio feature information of the second audio signal.

[0158] Optionally, in this embodiment of the application, when the above-mentioned sound parameters include volume, and when the volume of the second sound signal is greater than or equal to a preset volume threshold, the head-mounted device can obtain the sound feature information corresponding to the second sound signal.

[0159] Optionally, in this embodiment, the preset volume threshold can be preset by the head-mounted device. For example, the preset volume threshold can be 80 dB.

[0160] For example, if the volume of the second sound signal acquired by the head-mounted device is 90dB, after acquiring the volume of the second sound signal, the head-mounted device can compare the volume of 90dB with a preset volume threshold of 80dB. At this time, the head-mounted device determines that the volume of the second sound signal is greater than the preset volume threshold; then it registers the second sound signal and acquires the sound feature information of the second sound signal.

[0161] Optionally, in this embodiment of the application, when the above-mentioned sound parameters include signal-to-noise ratio and volume, when the signal-to-noise ratio of the second sound signal is greater than or equal to a preset signal-to-noise ratio threshold and the volume of the second sound signal is greater than or equal to a preset volume threshold, the head-mounted device can obtain the sound feature information corresponding to the second sound signal.

[0162] For example, if the signal-to-noise ratio (SNR) of the second audio signal acquired by the head-mounted device is 5 kdB, after acquiring the SNR of the second audio signal, the head-mounted device can compare the SNR of 5 kdB with a preset SNR threshold of 3 kdB. At this time, the head-mounted device determines that the SNR of the second audio signal is greater than the preset SNR threshold. Then, the head-mounted device can compare the volume of 90 dB with a preset volume threshold of 80 dB. At this time, the head-mounted device determines that the volume of the second audio signal is greater than the preset volume threshold. Finally, the head-mounted device can register the second audio signal and obtain the sound feature information of the second audio signal.

[0163] In this embodiment, the head-mounted device can obtain the voice feature information of the first user wearing the head-mounted device in advance, thereby determining the voice signal of the first user from the voice signal detected in the first scenario; and by determining the first user through threshold judgment, the voice of the wearer can be recognized and registered during use, avoiding the problem that the existing offline registration method is only applicable to the device owner, and improving the universality and flexibility of the device.

[0164] The above-described method embodiments, or various possible implementations of the method embodiments, can be executed individually, or, provided there are no contradictions, they can be combined with each other. The specific implementation can be determined according to actual usage requirements, and this application embodiment does not impose any restrictions on this.

[0165] It should be noted that the speech enhancement method provided in this application embodiment can be executed by a speech enhancement device. This application embodiment uses a speech enhancement device executing the speech enhancement method as an example to illustrate the speech enhancement device provided in this application embodiment.

[0166] Figure 10 A schematic diagram of a possible structure of the speech enhancement device involved in an embodiment of this application is shown. For example... Figure 10 As shown, the voice enhancement device 70 may include: a determination module 71, an acquisition module 72, and a processing module 73.

[0167] The determination module 71 is used to determine a first real-world object in a first scene based on the gaze information of a first user during a call via a head-mounted device. The first user is the user wearing the head-mounted device, and the first real-world object is a real-world object exhibiting speaking behavior and for which the first user intends to perform sound suppression processing. The acquisition module 72 is used to acquire first voice feature information of the first real-world object determined by the determination module. The processing module 73 is used to perform suppression processing on the voice signal corresponding to the first real-world object in the detected voice signal in the first scene based on the first voice feature information acquired by the acquisition module and the second voice feature information of the first user.

[0168] In one possible implementation, the voice enhancement device 70 provided in this application embodiment further includes: a display module, a marking module, and a receiving module. The display module is used to display a first screen before the determining module 71 determines a first real-world object in a first scene based on the gaze information of a first user. The first screen includes at least one object identifier, each of the at least one object identifier indicating a real-world object in the first scene. The marking module is used to mark N object identifiers from the at least one object identifier in the first screen based on the gaze information. The N object identifiers indicate a real-world object exhibiting speaking behavior. The receiving module is used to receive a first input to the first object identifier among the N object identifiers marked by the marking module. The determining module is specifically used to determine the real-world object indicated by the first object identifier as the first real-world object in response to the first input received by the receiving module.

[0169] In one possible implementation, the display module is further configured to display N controls at positions corresponding to N object identifiers in the first screen based on gaze information. Each of the N controls corresponds one-to-one with one of the N object identifiers, and each of the N controls indicates at least one sound suppression processing method. The receiving module is specifically configured to receive a first input to the control corresponding to the first object identifier.

[0170] In one possible implementation, the first input is an input that selects a sound suppression processing method through a control corresponding to the first object identifier. Specifically, the processing module 73 is used to suppress the speech signal corresponding to the first real-world object in the speech signal detected in the first scene, based on the first sound feature information and the second sound feature information, using the sound suppression processing method corresponding to the first input.

[0171] In one possible implementation, the processing module 73 is specifically used to calculate at least one sound gain parameter based on the first sound feature information, the second sound feature information, and the speech signal, wherein each sound gain parameter corresponds to a time-frequency point of the speech signal; and to perform frequency domain transformation processing on the speech signal to obtain the frequency domain transformed speech signal; and to perform suppression processing on the speech signal corresponding to the first real object in the frequency domain transformed speech signal based on at least one sound gain parameter.

[0172] In one possible implementation, the acquisition module 73 is further configured to acquire a second sound signal and obtain sound parameters corresponding to the second sound signal before the determining module 71 determines the first real-world object in the first scene based on the first user's gaze information during a call via the head-mounted device. The processing module 73 is further configured to acquire sound feature information corresponding to the second sound signal if the sound parameters meet a first condition; and to determine the sound feature information corresponding to the second sound signal as the second sound feature information.

[0173] This application provides a voice enhancement device. During a call using the voice enhancement device, the user can identify the first real-world object for which they wish to perform voice suppression processing based on their gaze information, i.e., through eye tracking. The device can then directionally acquire the voice feature information of this first real-world object. Based on this voice feature information, the voice signal corresponding to the first real-world object can be identified from the aforementioned first scene. This allows for voice suppression processing of the voice signal corresponding to the first real-world object, suppressing voice interference signals other than the voice signal of the user wearing the head-mounted device during the call. This avoids the difficulty in highlighting the voice signal of the user wearing the head-mounted device due to voice interference signals, thus improving call quality.

[0174] The voice enhancement device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, a mobile electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the device.

[0175] The voice enhancement device in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system used.

[0176] The speech enhancement device provided in this application embodiment can realize the various processes implemented in the above embodiments, and will not be described again here to avoid repetition.

[0177] Optionally, such as Figure 11 As shown, this application embodiment also provides an electronic device 90, including a processor 91 and a memory 92. The memory 92 stores a program or instructions that can run on the processor 91. When the program or instructions are executed by the processor 91, they implement the various steps of the above-described speech enhancement method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0178] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0179] Figure 12 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application.

[0180] The electronic device 100 includes, but is not limited to, components such as: radio frequency unit 101, network module 102, audio output unit 103, input unit 104, sensor 105, display unit 106, user input unit 107, interface unit 108, memory 109, and processor 110.

[0181] Those skilled in the art will understand that the electronic device 100 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 110 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 12 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0182] The processor 110 is configured to, during a call conducted through the head-mounted device, determine a first real-world object in a first scene based on the gaze information of a first user, wherein the first user is a user wearing the head-mounted device, and the first real-world object is a real-world object that is speaking and the first user intends to perform sound suppression processing; acquire first voice feature information of the first real-world object; and perform suppression processing on the voice signal corresponding to the first real-world object in the voice signal detected in the first scene based on the first voice feature information and the second voice feature information of the first user.

[0183] Optionally, in this embodiment, the display unit 106 is further configured to display a first screen before determining the first real-world object in the first scene based on the first user's gaze information. The first screen includes at least one object identifier, each object identifier indicating a real-world object in the first scene. The processor 110 is specifically configured to mark N object identifiers from the at least one object identifier in the first screen based on the gaze information. These N object identifiers indicate a real-world object exhibiting speaking behavior. The user input unit 107 is configured to receive a first input to the first object identifier from the N object identifiers. The processor 110 is further configured to, in response to the first input, determine the real-world object indicated by the first object identifier as the first real-world object.

[0184] Optionally, in this embodiment, the display unit 106 is specifically used to display N controls at positions corresponding to N object identifiers in the first screen based on gaze information. Each of the N controls corresponds one-to-one with one of the N object identifiers, and each control is used to indicate at least one sound suppression processing method. The user input unit 107 is specifically used to receive a first input to the control corresponding to the first object identifier.

[0185] Optionally, in this embodiment of the application, the first input is an input that selects a sound suppression processing method through the control corresponding to the first object identifier; the processor 110 is specifically used to use the sound suppression processing method corresponding to the first input to suppress the speech signal corresponding to the first real object in the speech signal based on the first sound feature information and the second sound feature information.

[0186] Optionally, in this embodiment of the application, the processor 110 is specifically configured to calculate at least one sound gain parameter based on the first sound feature information, the second sound feature information, and the speech signal, wherein each sound gain parameter corresponds to a time-frequency point of the speech signal; and to perform frequency domain conversion processing on the speech signal to obtain a frequency domain converted speech signal; and to perform suppression processing on the speech signal corresponding to the first real object in the frequency domain converted speech signal based on at least one sound gain parameter.

[0187] Optionally, in this embodiment of the application, the processor 110 is specifically used to collect a second sound signal and obtain sound parameters corresponding to the second sound signal before determining the first real object in the first scene based on the gaze information of the first user during a call through the head-mounted device; and to obtain sound feature information corresponding to the second sound signal when the sound parameters meet the first condition, and to determine the sound feature information corresponding to the second sound signal as the second sound feature information.

[0188] This application provides a head-mounted device. During a call using the head-mounted device, the user can identify the first real-world object for which they wish to suppress sound based on their gaze information (i.e., eye tracking). The device can then directionally acquire the sound feature information of this first real-world object. This sound feature information allows for the identification of the voice signal corresponding to the first real-world object within the aforementioned first scene. Consequently, voice suppression processing can be applied to the voice signal corresponding to this first real-world object, suppressing voice interference signals other than the user's voice during the call. This avoids the difficulty in highlighting the user's voice due to voice interference signals during the call, thus improving call quality.

[0189] The electronic device provided in this application embodiment can implement the various processes implemented in the above method embodiments and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0190] For details on the beneficial effects of the various implementation methods in this embodiment, please refer to the beneficial effects of the corresponding implementation methods in the above method embodiments. To avoid repetition, these will not be repeated here.

[0191] It should be understood that, in this embodiment, the input unit 104 may include a graphics processing unit (GPU) 1041 and a microphone 1042. The GPU 1041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 106 may include a display panel 1061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 107 includes at least one of a touch panel 1071 and other input devices 1072. The touch panel 1071 is also called a touch screen. The touch panel 1071 may include a touch detection device and a touch controller. Other input devices 1072 may include, but are not limited to, a physical keyboard, function keys (such as volume control buttons, power buttons, etc.), a trackball, a mouse, and a joystick, which will not be described in detail here.

[0192] The memory 109 can be used to store software programs and various data. The memory 109 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 109 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 109 in the embodiments of this application includes, but is not limited to, these and any other suitable types of memory.

[0193] Processor 110 may include one or more processing units; optionally, processor 110 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 110.

[0194] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0195] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0196] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above method embodiments and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0197] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0198] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the above-described speech enhancement method embodiments, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0199] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0200] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0201] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A speech enhancement method, characterized by, The method is executed by a head-mounted device, and the method comprises: In the process of calling through the head-mounted device, a first real object in a first scene is determined based on gaze information of a first user, the first user is a user wearing the head-mounted device, and the first real object is a real object that has a speaking behavior and for which the first user intends to perform sound suppression processing; First sound feature information of the first real object is obtained; Based on the first sound feature information and second sound feature information of the first user, a voice signal corresponding to the first real object in a voice signal detected in the first scene is suppressed; The gaze information comprises a direction in which an eye of the first user gazes, and the method further comprises: In a case where the first real object is not detected in the direction in which the eye of the first user gazes by the head-mounted device, second prompt information is displayed to prompt the first user that there is a real object in another direction.

2. The method of claim 1, wherein, Before the first real object in the first scene is determined based on the gaze information of the first user, the method further comprises: A first image is displayed, and the first image comprises at least one object identifier, and each object identifier is used to indicate a real object in the first scene; The first real object in the first scene is determined based on the gaze information of the first user, and the method comprises: Based on the gaze information, N object identifiers in the at least one object identifier are marked in the first image, and the N object identifiers are used to indicate real objects that have a speaking behavior; A first input to a first object identifier in the N object identifiers is received; In response to the first input, a real object indicated by the first object identifier is determined as the first real object.

3. The method of claim 2, wherein, The N object identifiers in the at least one object identifier are marked in the first image based on the gaze information, and the method comprises: Based on the gaze information, N controls corresponding to the N object identifiers are displayed in positions of the N object identifiers in the first image, the N controls correspond to the N object identifiers one by one, and each control is used to indicate at least one sound suppression processing mode; The first input to the first object identifier in the N object identifiers is received, and the method comprises: A first input to a control corresponding to the first object identifier is received.

4. The method of claim 3, wherein, The first input is an input of selecting a sound suppression processing mode through the control corresponding to the first object identifier; Based on the first sound feature information and the second sound feature information, the voice signal corresponding to the first real object in the voice signal detected in the first scene is suppressed, and the method comprises: Based on the first sound feature information and the second sound feature information, a voice signal corresponding to the first real object in the voice signal is suppressed by using a sound suppression processing mode corresponding to the first input.

5. The method according to any one of claims 1 to 4, characterized in that, Based on the first sound feature information and the second sound feature information, the voice signal corresponding to the first real object in the voice signal detected in the first scene is suppressed, and the method comprises: Based on the first sound feature information, the second sound feature information and the speech signal, at least one sound gain parameter is calculated, each sound gain parameter corresponding to a time-frequency point of the speech signal respectively; The speech signal is processed by frequency domain conversion to obtain the speech signal after frequency domain conversion; Based on the at least one sound gain parameter, the speech signal corresponding to the first real object in the speech signal after frequency domain conversion is suppressed.

6. The method of claim 1, wherein, Before determining the first real object in the first scene based on the gaze information of the first user in the process of calling through the head-mounted device, the method further comprises: Collecting a second sound signal and obtaining sound parameters corresponding to the second sound signal; In the case that the sound parameters meet the first condition, the sound feature information corresponding to the second sound signal is obtained, and the sound feature information corresponding to the second sound signal is determined as the second sound feature information.

7. A speech enhancement device, characterized by The speech enhancement device comprises a determination module, an acquisition module and a processing module. The determination module is used to determine the first real object in the first scene based on the gaze information of the first user in the process of calling through the head-mounted device, the first user being the user wearing the head-mounted device, and the first real object being a real object with speaking behavior and intended to be subjected to sound suppression processing by the first user. The acquisition module is used to acquire the first sound feature information of the first real object determined by the determination module. The processing module is used to suppress the speech signal corresponding to the first real object in the speech signal detected in the first scene based on the first sound feature information acquired by the acquisition module and the second sound feature information of the first user. The gaze information comprises the direction in which the eyes of the first user gaze, and the device further comprises a display module. The display module is used to display second prompt information to prompt the first user that there is a real object in other directions in the case that the head-mounted device does not detect the first real object in the direction in which the eyes of the first user gaze.

8. The apparatus of claim 7, wherein, The speech enhancement device further comprises a marking module and a receiving module. The display module is further used to display a first picture before the determination module determines the first real object in the first scene based on the gaze information of the first user, the first picture comprising at least one object identifier, each object identifier being used to indicate a real object in the first scene. The marking module is used to mark N object identifiers in the at least one object identifier in the first picture based on the gaze information, the N object identifiers being used to indicate real objects with speaking behavior. The receiving module is used to receive a first input to a first object identifier in the N object identifiers marked by the marking module. The determination module is specifically used to determine the real object indicated by the first object identifier as the first real object in response to the first input received by the receiving module.

9. The apparatus of claim 8, wherein, The display module is further configured to display N controls in positions corresponding to the N object identifiers in the first picture based on the gaze information, the N controls corresponding one-to-one to the N object identifiers, and each control being configured to indicate at least one sound suppression processing mode. The receiving module is specifically configured to receive a first input on the control corresponding to the first object identifier.

10. The apparatus of claim 9, wherein, The first input is an input of selecting a sound suppression processing mode through the control corresponding to the first object identifier. The processing module is specifically configured to perform suppression processing on the voice signal corresponding to the first real object in the voice signal detected in the first scene based on the first sound feature information and the second sound feature information by using the sound suppression processing mode corresponding to the first input.

11. The apparatus of any one of claims 7 to 10, wherein, The processing module is specifically configured to: calculate at least one sound gain parameter corresponding to one time-frequency point of the voice signal based on the first sound feature information, the second sound feature information, and the voice signal; perform frequency domain conversion processing on the voice signal to obtain the voice signal after frequency domain conversion; perform suppression processing on the voice signal corresponding to the first real object in the voice signal after frequency domain conversion based on the at least one sound gain parameter.

12. The apparatus of claim 7, wherein, The obtaining module is further configured to, before the determining module determines the first real object in the first scene based on the gaze information of the first user during a call through the head-mounted device, collect a second voice signal and obtain a sound parameter corresponding to the second voice signal. The processing module is further configured to, when the sound parameter meets a first condition, obtain sound feature information corresponding to the second voice signal, and determine the sound feature information corresponding to the second voice signal as the second sound feature information.

13. An electronic device, comprising: The processor, the memory, and the program or instructions stored in the memory and executable on the processor are included, and the program or instructions are executed by the processor to implement the steps of the voice enhancement method in any one of claims 1 to 6.

14. A readable storage medium, characterized by, The program or instructions are stored on the readable storage medium, and the program or instructions are executed by the processor to implement the steps of the voice enhancement method in any one of claims 1 to 6.

15. A computer program product, characterised in that, The computer program product is stored in the storage medium, and the computer program product is executed by at least one processor to implement the voice enhancement method in any one of claims 1 to 6.

Images