Speech enhancement method and device, electronic equipment, storage medium and program product

Abstract

The invention discloses a speech enhancement method and device, electronic equipment, a storage medium and a program product, and belongs to the technical field of artificial intelligence. The method comprises: in a process of performing a call through a head-mounted device, based on gaze information of a first user, determining a first real object in a first scene, the first user being a user wearing the head-mounted device, and the first real object being a real object which has a speaking behavior and which the first user intends to perform sound suppression processing; acquiring first sound feature information of the first real object; and based on the first sound feature information and second sound feature information of the first user, performing suppression processing on the voice signal corresponding to the first real object in the voice signals detected in the first scene.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence technology, and specifically relates to a speech enhancement method, device, electronic device, storage medium and program product. Background Art

[0002] Currently, during a call, in order to improve the call quality with other users, the electronic device can perform noise reduction processing on the voice signal collected by the microphone, thereby removing the environmental noise signal in the voice signal to improve the call quality.

[0003] In the related art, when electronic devices collect voice signals, since human voice signals have time-varying and non-stationary characteristics, the electronic devices can estimate the ambient noise signal from the collected voice signals through these characteristics, and suppress the ambient noise signal to remove the ambient noise signal from the voice signals.

[0004] However, when the above-mentioned voice signal includes a human voice interference signal, due to the same time-varying and non-stationary characteristics, the above-mentioned method cannot identify the human voice interference signal contained in the voice signal, and thus cannot remove the interference of the human voice interference signal, resulting in difficulty in highlighting the human voice signal of the user of the electronic device during the call; thus, the call quality of the electronic device is poor. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a speech enhancement method, device, electronic device, storage medium and program product, which can reduce the interference of other voice signals during a call and improve the call quality.

[0006] In a first aspect, an embodiment of the present application provides a speech enhancement method, which includes: during a call through a head-mounted device, determining a first real object in a first scene based on gaze information of a first user, the first user being a user wearing a head-mounted electronic device, and the first real object being a real object that has a speaking behavior and that the first user intends to perform sound suppression processing on; obtaining first sound feature information of the first real object; and suppressing a speech signal corresponding to the first real object in a speech signal detected in the first scene based on the first sound feature information and second sound feature information of the first user.

[0007] In a second aspect, an embodiment of the present application provides a speech enhancement device, which includes: a determination module, an acquisition module and a processing module. The determination module is used to determine a first real object in a first scene based on gaze information of a first user during a call through a head-mounted device, wherein the first user is a user wearing a head-mounted electronic device, and the first real object is a real object that has a speaking behavior and the first user intends to perform sound suppression processing. The acquisition module is used to obtain first sound feature information of the first real object determined by the determination module. The processing module is used to suppress the speech signal corresponding to the first real object in the speech signal detected in the first scene based on the first sound feature information acquired by the acquisition module and the second sound feature information of the first user.

[0008] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0009] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0010] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0011] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.

[0012] In an embodiment of the present application, during a call through a head-mounted device, a first real object in a first scene can be determined based on the gaze information of a first user, the first user is a user wearing a head-mounted electronic device, and the first real object is a real object that has a speaking behavior and the first user intends to perform sound suppression processing; then, the first sound feature information of the first real object is obtained; finally, based on the first sound feature information and the second sound feature information of the first user, the voice signal corresponding to the first real object in the voice signal detected in the first scene is suppressed. In this solution, during a call through a head-mounted device, since the first real object that the user wants to perform sound suppression processing can be obtained based on the user's line of sight information, that is, through eye tracking, and then the sound feature information of the first real object can be directionally obtained, the voice signal corresponding to the first real object can be identified from the above-mentioned first scene through the sound feature information, so that the voice signal corresponding to the first real object can be voice suppressed, so as to suppress the human voice interference signal other than the human voice signal of the user to whom the head-mounted device belongs during the call, thereby avoiding the difficulty in highlighting the human voice signal of the user wearing the head-mounted device due to the human voice interference signal during the call, and improving the call quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 This is one of the flow charts of a speech enhancement method provided in an embodiment of the present application;

[0014] Figure 2 This is the second flowchart of a speech enhancement method provided in an embodiment of the present application;

[0015] Figure 3 This is the third flowchart of a speech enhancement method provided in an embodiment of the present application;

[0016] Figure 4 This is one of the example schematic diagrams of a call interface of a head mounted device provided in an embodiment of the present application;

[0017] Figure 5 This is a fourth flow chart of a speech enhancement method provided in an embodiment of the present application;

[0018] Figure 6 This is a second example schematic diagram of a call interface of a head mounted device provided in an embodiment of the present application;

[0019] Figure 7 This is a third example schematic diagram of a call interface of a head mounted device provided in an embodiment of the present application;

[0020] Figure 8 This is a fourth flowchart of a speech enhancement method provided in an embodiment of the present application;

[0021] Fig. 9 This is the fifth flow chart of a speech enhancement method provided in an embodiment of the present application;

[0022] Fig.10 is a structural schematic diagram of a speech enhancement device provided in an embodiment of the present application;

[0023] Fig.11 This is one of the hardware structure diagrams of an electronic device provided in an embodiment of the present application;

[0024] Fig.12 This is the second schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0026] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0027] The terms "at least one (item)", "at least one of" and the like in the specification and claims of the present application refer to any one, any two or a combination of more than two of the objects included therein. For example, at least one (item) of a, b, and c can be represented by: "a", "b", "c", "a and b", "a and c", "b and c" and "a, b and c", where a, b, and c can be single or multiple. Similarly, "at least two (items)" refers to two or more, and its meaning is similar to that of "at least one (item)".

[0028] The speech enhancement method, device, electronic device, storage medium and program product provided in the embodiments of the present application are described in detail below in conjunction with the accompanying drawings through specific embodiments and their application scenarios.

[0029] At present, in today's era of increasing digitalization and remote work, video conferencing or voice conferencing has become an indispensable tool in the fields of business communication, distance education, and telemedicine. In the above scenarios, clear voice transmission is essential for effective communication. During a call, electronic devices such as virtual reality (VR), augmented reality (AR) or mixed reality (MR) will comprehensively collect the caller's voice and environmental noise to collect audio. In order to improve the quality of the call, the electronic device can perform noise reduction processing on the above audio, which can effectively suppress or eliminate environmental noise while keeping the naturalness and intelligibility of the human voice intact.

[0030] Usually, electronic devices can perform noise reduction processing on the audio collected by the electronic devices through traditional methods such as spectrum subtraction, beamforming, and adaptive filtering. However, the above methods are relatively effective in processing stable, non-voice noise. However, when the human voice in the call contains human voice interference signals, since the spectrum of multiple human voice signals is similar to the spectrum of the user to whom the electronic device belongs, and has the characteristics of time-varying and non-stationary; the above methods cannot distinguish between the user voice signal and the human voice interference voice signal, resulting in the electronic device being unable to highlight the user voice signal during the call, and thus the call quality of the electronic device is poor.

[0031] For example, speaker A is video chatting with friend B in the living room. During the call, family members C and D who are also in the living room are chatting. Therefore, the sound heard by friend B includes the voices of A, C, and D, and it may be difficult to hear A's voice clearly.

[0032] The speech enhancement method, device, electronic device, storage medium, and program product provided in the embodiments of the present application can be applied to scenarios where users make calls. Specifically, in the scenario where users make calls in the embodiments of the present application, when the user is in a scene or environment with multiple people (such as a shopping mall or an amusement park), the user can talk to other users by wearing a head-mounted device, so that the head-mounted device can suppress the human voice interference signal in the scene.

[0033] In the embodiment of the present application, the above-mentioned call may include any of the following: a video call, a voice call, a telephone call, etc.

[0034] For example, when user A is in an amusement park and is making a video call with user B through the MR device, there are interfering users C and D around the users; if user A hears a loud sound from interfering user C, user A can look at user C, and the MR device can determine the direction of user C through user A's eyes, and perform directionally receiving sound from user C to obtain the sound signal of user C; through the sound signal, the MR device can obtain the sound characteristic information of the signal, and then process the sound signal in the current scene collected by the MR device through the sound characteristic information to suppress the sound of user C during the video call.

[0035] In the voice enhancement method, device, electronic device, storage medium and program product provided in the embodiments of the present application, during a call through a head-mounted device, since the first real object that the user wants to perform sound suppression processing on can be obtained based on the user's line of sight information, that is, through eye tracking, and then the sound feature information of the first real object can be directionally acquired, the voice signal corresponding to the first real object can be identified from the above-mentioned first scene through the sound feature information, so that the voice signal corresponding to the first real object can be voice suppressed to suppress the human voice interference signal other than the human voice signal of the user of the head-mounted device during the call, thereby avoiding the difficulty in highlighting the human voice signal of the user wearing the head-mounted device due to the human voice interference signal during the call, and improving the call quality.

[0036] The execution subject of the speech enhancement method provided in the embodiment of the present application may be a speech enhancement device, and the speech enhancement device may be an electronic device, or a functional module in an electronic device. The technical solution provided in the embodiment of the present application is described below using an electronic device as an example.

[0037] The present application provides a method for speech enhancement. Figure 1 FIG. 1 is a flow chart of a method for speech enhancement provided by an embodiment of the present application, which method can be applied to a head mounted device. Figure 1 As shown, the speech enhancement method provided in the embodiment of the present application may include the following steps 201 to 203.

[0038] Step 201: During a call through a head mounted device, the head mounted device determines a first real object in a first scene based on gaze information of a first user.

[0039] In the embodiment of the present application, the first user is a user wearing a head mounted device.

[0040] In the embodiment of the present application, the first real object may be a real object that has a speaking behavior and for which the first user intends to perform sound suppression processing.

[0041] Optionally, in the embodiment of the present application, the head-mounted device may be any one of the following: a VR device, an AR device or an MR device.

[0042] Optionally, in an embodiment of the present application, the above-mentioned call can be a voice call or a video call; the specific amount can be determined based on actual usage, and is not limited in the embodiment of the present application.

[0043] In the embodiment of the present application, the above-mentioned first scene may be the environment scene in which the first user is currently located.

[0044] Optionally, in the embodiment of the present application, the first scene may be any one of the following: a shopping mall, a parking lot, a supermarket, a scenic spot or an amusement park, etc. The specific scene may be determined according to actual conditions, and the embodiment of the present application does not limit the scene.

[0045] In the embodiment of the present application, the first real object may be a real person in the first scene.

[0046] Optionally, in the embodiment of the present application, the first real object may be any one of the following: a person or an animal.

[0047] Optionally, in the embodiment of the present application, the first real object may be one or more, which may be specifically determined by the first user's selection.

[0048] Optionally, in an embodiment of the present application, the above-mentioned sound suppression processing can be any one of the following: sound noise reduction processing, sound attenuation processing or sound shielding processing.

[0049] In the embodiment of the present application, the above-mentioned gaze information may include the direction in which the eyes of the first user are gazed.

[0050] Optionally, in the embodiment of the present application, the above direction may be any one of the following: east, west, south, north, southeast, southwest, northeast, northwest, etc. The specific direction may be determined according to actual conditions, and the embodiment of the present application does not limit it.

[0051] In the embodiment of the present application, the head-mounted device can obtain the direction in which the first user's eyes are looking through the front camera, and determine the real object with lips moving among the real objects existing in the first scene in that direction as the first real object.

[0052] Optionally, in the embodiment of the present application, the front camera may be any one of the following: a wide-angle camera, a macro camera, an ultra-wide-angle camera, or an automatic zoom camera, etc. The specific configuration may be determined according to actual conditions, and the embodiment of the present application does not limit this.

[0053] Optionally, in an embodiment of the present application, when the head-mounted device does not detect the first real object in the direction that the first user's eyes are looking at, the head-mounted device may display first prompt information to prompt the first user that there is no first real object in the current direction.

[0054] Exemplarily, the first prompt information may be “No person or animal is found in the current direction, please adjust the direction”.

[0055] Optionally, in an embodiment of the present application, when the head-mounted device does not detect the first real object in the direction in which the eyes of the first user are looking, the head-mounted device can display second prompt information to prompt the first user that there are real objects in other directions; then, when the direction in which the eyes of the first user are looking is the same as the other directions, the head-mounted device can determine the first real object in the current direction.

[0056] In the embodiment of the present application, the head mounted device can determine in which direction the real object exists through the sound signal of the real object collected by the microphone, and generate the second prompt information.

[0057] Exemplarily, the second prompt information may be: "Someone is talking in the southeast direction", and the head mounted device may display a southeast direction sign.

[0058] Step 202: The head mounted device obtains first sound feature information of a first real object.

[0059] Optionally, in the embodiment of the present application, the first sound feature information may include at least one of the following: linear prediction cepstrum information, line spectrum pair information, Mel frequency cepstrum information, pitch information, voiceprint information, and timbre information, etc. The specific information may be determined according to actual use requirements, and the embodiment of the present application does not limit it.

[0060] In the embodiment of the present application, after determining the first real object, the head-mounted device can directionally pick up the sound signal of the first real object, and then obtain the first sound feature information through the sound signal.

[0061] In the embodiment of the present application, the electronic device can obtain the first sound feature information through a sound processing model.

[0062] Optionally, in the embodiment of the present application, the above-mentioned sound processing model can be any one of the following: artificial intelligence (AI) model, neural network model or large model, etc. The specific model can be determined according to actual use requirements, and the embodiment of the present application does not limit it.

[0063] Exemplarily, taking the above-mentioned sound processing model as a speech neural network model, the speech neural network model may include a speech coding network, and the head-mounted device may input the sound signal of the first real object into the speech neural network model, and then perform convolution processing on the sound signal of the first real object through the speech coding network to obtain first sound feature information.

[0064] Optionally, in the embodiment of the present application, combined with Figure 1 ,like Figure 2 As shown, the above step 202 can be specifically implemented through the following steps 202a and 202b.

[0065] Step 202a: The head mounted device collects a first sound signal of a first real object in a first scene based on the gaze information and an array of sound collection modules in the head mounted device.

[0066] In an embodiment of the present application, the above-mentioned sound collection module array can be a microphone array, which includes multiple microphones, and the positions of the multiple microphones in the head-mounted device are different.

[0067] Optionally, in the embodiment of the present application, the microphone may be any one of the following: an electrodynamic microphone, a capacitive microphone, a piezoelectric microphone, or a pressure microphone, etc. The specific type may be determined according to actual use requirements, and the embodiment of the present application does not limit this.

[0068] In the embodiment of the present application, the head mounted device can directionally pick up the first sound signal of the first real object through the sound collection module array.

[0069] It should be noted that the above-mentioned directional sound pickup can be one of the many methods of speech enhancement. Directional sound pickup is to pick up the target signal from the mixed signal according to the direction of the sound source, that is, only the sound signal propagated from a specific direction is picked up, while the noise and interference signals from other directions may not be picked up or may be attenuated or shielded, thereby achieving the effect of directional collection of sound signals.

[0070] In an embodiment of the present application, the head-mounted device can determine the direction of directional sound pickup based on the above-mentioned gaze information, and then collect the sound signal of the first real object in this direction through multiple microphones at different positions in the microphone array, and then fuse the sound signals collected by the multiple microphones at different positions to obtain the above-mentioned first sound signal.

[0071] Optionally, in an embodiment of the present application, when there are multiple first real objects, the head mounted device may respectively and directionally pick up the first sound signals of the multiple first real objects.

[0072] Exemplarily, when there are three first real objects, the head mounted device may directionally pick up the first sound signals of the three first real objects every preset time duration, such as 3 seconds.

[0073] For example, after the head-mounted device determines three first real objects, the head-mounted device can directionally pick up the first sound signal of the first real object within 0 to 3 seconds; within 4 to 6 seconds, the head-mounted device can directionally pick up the first sound signal of the second real object; within 7 to 9 seconds, the head-mounted device can directionally pick up the first sound signal of the third real object.

[0074] It should be noted that the specific process of the head-mounted device directionally picking up the first sound signals of the three first real objects can be found in the above embodiment, and will not be repeated here to avoid repetition.

[0075] Step 202b: The head mounted device performs feature extraction processing on the first sound signal to obtain first sound feature information.

[0076] In an embodiment of the present application, the head-mounted device can input the above-mentioned first sound signal into the sound processing model, and then recognize and obtain the above-mentioned first sound feature information through the above-mentioned speech coding network.

[0077] Optionally, in an embodiment of the present application, when the above-mentioned first sound signal includes multiple first sound signals, the head-mounted device can input the multiple first sound signals into the speech coding network in sequence to obtain multiple first sound feature information corresponding to the multiple first sound signals, and one first sound signal corresponds to one first sound feature information.

[0078] It should be noted that the process of the head mounted device obtaining the first sound feature information through the sound processing model can be found in the above embodiment in detail, and will not be described again here to avoid repetition.

[0079] In an embodiment of the present application, the head-mounted device can directionally pick up the first sound signal of the first real object through gaze information, so that the first sound signal can only include the sound of the first real object, thereby extracting accurate first sound feature information based on the first sound signal. Further, the head-mounted device can accurately identify the voice signal that interferes with the user from the voice signal detected in the first scene through the first sound feature information.

[0080] Step 203: The head mounted device suppresses the voice signal corresponding to the first real object in the voice signal detected in the first scene based on the first sound feature information and the second sound feature information of the first user.

[0081] Optionally, in the embodiment of the present application, the second sound feature information may include at least one of the following: linear prediction cepstrum information, line spectrum pair information, Mel frequency cepstrum information, pitch information, voiceprint information, and timbre information, etc. The specific information may be determined according to actual use requirements, and the embodiment of the present application does not limit it.

[0082] In the embodiment of the present application, the voice signal detected in the above-mentioned first scene may include: a voice signal of the first user, a voice signal of the above-mentioned first real object, and an environmental noise signal.

[0083] It should be noted that the specific implementation process can be found in the following embodiments, and will not be described again here to avoid repetition.

[0084] In the speech enhancement method provided in an embodiment of the present application, during a call through a head-mounted device, the head-mounted device can determine a first real object in a first scene based on gaze information of a first user, where the first user is a user wearing a head-mounted electronic device, and the first real object is a real object that has a speaking behavior and that the first user intends to perform sound suppression processing on; then, the head-mounted device can obtain first sound feature information of the first real object; finally, based on the first sound feature information and the second sound feature information of the first user, the speech signal corresponding to the first real object in the speech signal detected in the first scene is suppressed. In this solution, during a call through a head-mounted device, since the electronic device can obtain the first real object that the user wants to suppress sound based on the user's line of sight information, that is, through eye tracking, and then can directionally obtain the sound feature information of the first real object, the voice signal corresponding to the first real object can be identified from the above-mentioned first scene through the sound feature information, so that the voice signal corresponding to the first real object can be voice suppressed to suppress voice interference signals other than the voice signal of the user of the head-mounted device during the call, thereby avoiding the difficulty in highlighting the voice signal of the user wearing the head-mounted device due to the voice interference signal during the call, and improving the call quality.

[0085] Optionally, in the embodiment of the present application, combined with Figure 1 ,like Figure 3 As shown, before the "determining the first real object in the first scene based on the gaze information of the first user" in the above step 201, the speech enhancement method provided in the embodiment of the present application also includes the following step 301, and the "determining the first real object in the first scene based on the gaze information of the first user" in the above step 201 can be specifically implemented through the following steps 201a to 201c.

[0086] Step 301: During a call through a head mounted device, the head mounted device displays a first screen.

[0087] In the embodiment of the present application, the first picture includes at least one object identifier, and each of the at least one object identifier is used to indicate a real object in the first scene.

[0088] Optionally, in an embodiment of the present application, the first picture may be a real picture corresponding to the first scene.

[0089] Optionally, in the embodiment of the present application, the first screen may also include a virtual call interface.

[0090] Optionally, in the embodiment of the present application, the object identifier may include at least one of the following: an image identifier, a text identifier, an expression identifier, etc. The specific identifier may be determined according to actual usage, and the embodiment of the present application does not limit the identifier.

[0091] In the embodiment of the present application, the head-mounted device can display the first picture through the screen of the head-mounted device.

[0092] For example, Figure 4 As shown, taking the first scene as a living room including two interfering users as an example, taking the head mounted device as an MR device as an example, during the process in which the first user is talking to user B through the MR device, the MR device can display a first picture 10 on the screen, and the first picture 10 can display the real layout of the living room and the user identifications of the two interfering users, Figure 4 User C and user D are represented in the figure; and a video call interface 11 is displayed in the first screen 10.

[0093] Step 201a: The head mounted device marks N object identifiers of at least one object identifier in the first picture based on the gaze information.

[0094] In the embodiment of the present application, the above-mentioned N object identifiers are used to indicate real objects that have speaking behavior.

[0095] In an embodiment of the present application, the head-mounted device can determine a gaze direction in the first screen based on the gaze information of the first user, and then detect in the gaze direction whether the mouth of at least one real object in the gaze direction changes; when the mouths of N real objects among at least one real object in the gaze direction change, the head-mounted device can mark N object identifiers corresponding to the N real objects in the first screen.

[0096] Optionally, in the embodiment of the present application, the above mark can be any of the following: color mark, added control mark, text mark or pattern mark, etc. The specific mark can be determined according to actual usage, and the embodiment of the present application does not limit it.

[0097] Step 201b: The head mounted device receives a first input of a first object identifier among the N object identifiers.

[0098] In the embodiment of the present application, the first input is used to select a first object identifier from N object identifiers.

[0099] Optionally, in an embodiment of the present application, the above-mentioned first input includes but is not limited to: the user's click input on the above-mentioned first object identifier through a finger, or a voice command input by the user, or a specific gesture input by the user, or other feasible input, which can be determined based on actual usage and is not limited in the embodiment of the present application.

[0100] In some embodiments of the present application, the above-mentioned specific gesture can be any one of a single-click gesture, a sliding gesture, a drag gesture, a pressure recognition gesture, a long press gesture, an area change gesture, a double-press gesture, and a double-click gesture.

[0101] In some embodiments of the present application, the click input may be a single click input, a double click input, or any number of click inputs, etc., and may also be a long press input or a short press input. For example, the first input may be a single click input of the user to the first object identifier.

[0102] Optionally, in an embodiment of the present application, the first object identifier may be one or more.

[0103] Optionally, in an embodiment of the present application, when the above-mentioned first object identifier is multiple object identifiers, the head-mounted device can receive multiple first inputs from the first user to determine multiple first object identifiers from the N object identifiers.

[0104] Step 201c: The head mounted device determines, in response to the first input, the real object indicated by the first object identifier as the first real object.

[0105] Optionally, in an embodiment of the present application, when there are multiple first object identifiers, the head-mounted device can respond to multiple first inputs respectively and determine the real object indicated by each first object identifier in the multiple first object identifiers as the first real object.

[0106] Optionally, in an embodiment of the present application, after the head-mounted device determines the real object indicated by the first object identifier as the first real object, the head-mounted device may display third prompt information around the first object identifier to prompt the first user that the real object indicated by the first object identifier has been determined as the first real object.

[0107] Exemplarily, the third prompt information may be: "Determine that the user is an interfering user."

[0108] Optionally, in the embodiment of the present application, after the head mounted device determines the real object indicated by the first object identifier as the first real object, it can receive a second input from the first user to cancel the selection of the real object indicated by the first object identifier as the first real object.

[0109] Optionally, in an embodiment of the present application, the second input includes but is not limited to: a click input of the first object identifier by the user through a finger, or a voice command input by the user, or a specific gesture input by the user, or other feasible input, which can be determined based on actual usage and is not limited in the embodiment of the present application.

[0110] Optionally, in an embodiment of the present application, when there are multiple first object identifiers, the head-mounted device can, based on the first user's input of a second object identifier among the multiple first object identifiers, deselect the real object indicated by the second object identifier, and determine the real objects indicated by the multiple first object identifiers other than the second object identifier as the first real object.

[0111] In the embodiment of the present application, the head mounted device can select a real user required by the first user as an interfering user based on the input of the first user, thereby improving the flexibility and accuracy of the electronic device in determining the interfering user.

[0112] Optionally, in the embodiment of the present application, combined with Figure 3 ,like Figure 5 As shown, the above step 201a can be specifically implemented by the following step 201a1, and the above step 201b can be specifically implemented by the following step 201b1.

[0113] Step 201a1: Based on the gaze information, the head mounted device displays N controls at positions corresponding to N object identifiers in the first screen.

[0114] In an embodiment of the present application, the above-mentioned N controls correspond one-to-one to N object identifiers, and each of the N controls is used to indicate at least one sound suppression processing method.

[0115] Optionally, in an embodiment of the present application, each of the above-mentioned N controls may include at least one sub-control, and each sub-control of the at least one sub-control is used to indicate a sound suppression processing method.

[0116] Optionally, in the embodiment of the present application, the above-mentioned position can be preset for the head-mounted device.

[0117] Exemplarily, the head mounted device may display N controls above the N object identifiers in the first screen.

[0118] For example, in combination Figure 4 ,like Figure 6 As shown, after the MR device determines the two first object identifiers, the MR device may display a control 16 above the first first object identifier 15, wherein the control 16 includes three sub-control identifiers. Figure 6 161, 162 and 163 are indicated; and a control 18 is displayed above the second first object identifier 17, the control 18 includes 3 sub-control identifiers, Figure 6 Indicated by 181, 182 and 183.

[0119] Optionally, in the embodiment of the present application, after the head-mounted device displays N controls in the first screen, the head-mounted device may update display positions of the N controls in the first screen based on input of the first user to the N controls.

[0120] Step 201b1: The head mounted device receives a first input to a control corresponding to a first object identifier.

[0121] In the embodiment of the present application, the above-mentioned first input is used to select a control corresponding to the first object identifier from N controls.

[0122] In an embodiment of the present application, the control corresponding to the above-mentioned first object identifier can be used to indicate a sound suppression processing method.

[0123] Optionally, in the embodiment of the present application, the first input may be an input to a first sub-control among at least one sub-control.

[0124] In the embodiment of the present application, the head-mounted device can determine the real object corresponding to the first object identifier as the first real object through the first input, and process the sound signal of the first real object according to the sound suppression processing method of the control corresponding to the first object identifier.

[0125] For example, in combination Figure 6 ,like Figure 7 As shown, the first user can control the third sub-control in the control 16 displayed on the MR device user C, namely Figure 7 The sub-control corresponding to 163 is clicked and inputted so that the head mounted device can identify the user C as the first real object, and the sound signal of the first real object is processed according to the sound suppression processing method corresponding to the third sub-control 163.

[0126] Optionally, in an embodiment of the present application, when there are multiple first object identifiers, the head-mounted device can receive multiple first inputs from the first user to the multiple first object identifiers to determine the sound suppression processing method that interferes with the user corresponding to each object identifier in the multiple first object identifiers.

[0127] Exemplarily, taking the case where there are two first object identifiers, the first user can click and input the third sub-control in the control corresponding to the first first object identifier, so that the head-mounted device can determine that the sound suppression processing method for the interfering user corresponding to the first first object identifier is voice noise reduction processing; then, the first user can click and input the second sub-control in the control corresponding to the second first object identifier, so that the head-mounted device can determine that the sound suppression processing method for the interfering user corresponding to the second first object identifier is voice attenuation processing.

[0128] It should be noted that all inputs in the above embodiments are virtual inputs, that is, they are not real inputs by touching the screen of the head-mounted device, but inputs to the virtual interface in the virtual interface displayed by the head-mounted device.

[0129] In an embodiment of the present application, the head-mounted device can determine a sound suppression processing method for voice signals that interfere with the user based on the user's input to the control, and then perform corresponding processing on the voice signals that interfere with the user based on the sound suppression processing method selected by the user, thereby improving the flexibility of the head-mounted device in determining the sound suppression processing method.

[0130] Optionally, in an embodiment of the present application, the above-mentioned first input is an input for selecting a sound suppression processing method through a control corresponding to the first object identifier.

[0131] For example, in combination Figure 1 ,like Figure 8 As shown, the above step 203 can be specifically implemented through the following step 203a.

[0132] Step 203a: The head mounted device adopts the sound suppression processing method corresponding to the first input, and suppresses the voice signal corresponding to the first real object in the voice signal detected in the first scene based on the first sound feature information and the second sound feature information.

[0133] Optionally, in an embodiment of the present application, when the sound suppression processing method corresponding to the above-mentioned first object identifier is a sound attenuation method, the head-mounted device can input the sound attenuation processing identifier corresponding to the sound attenuation processing method, the first sound feature information, the second sound feature information and the above-mentioned voice signal into the above-mentioned sound processing model, so that the sound processing model can calculate at least one sound gain parameter; and perform dot multiplication of at least one sound gain parameter with a preset attenuation parameter to obtain at least one second sound gain parameter; then, perform frequency domain conversion processing on the voice signal to obtain a voice signal after frequency domain conversion, and finally, perform dot multiplication of the frequency domain converted voice signal and at least one second sound gain parameter to obtain a voice signal after sound attenuation processing corresponding to the first object identifier.

[0134] Optionally, in an embodiment of the present application, when the above-mentioned sound suppression processing method is a sound noise reduction processing method, the head-mounted device can input the sound noise reduction processing identifier, the first sound feature information, the second sound feature information and the above-mentioned voice signal corresponding to the sound noise reduction processing method into the above-mentioned sound processing model, so that the sound processing model can calculate at least one sound gain parameter, and perform frequency domain conversion processing on the voice signal to obtain a voice signal after frequency domain conversion, and finally, perform dot multiplication on the voice signal after frequency domain conversion and at least one sound gain parameter to obtain a voice signal after sound noise reduction processing.

[0135] It should be noted that the specific process of the head mounted device calculating at least one sound gain parameter can be found in the following embodiments, which will not be described again here to avoid repetition.

[0136] In an embodiment of the present application, the head-mounted device can process the voice signal detected in the first scene based on the first sound feature information and the second sound feature information through the sound suppression processing method corresponding to the first input; it can be understood that different sound suppression processing methods can obtain voice signals with different effects, and the head-mounted device can flexibly determine the sound suppression processing method according to the user's choice, thereby improving the flexibility of processing voice signals.

[0137] Optionally, in the embodiment of the present application, combined with Figure 1 ,like Fig. 9 As shown, the above step 203 can be specifically implemented through the following steps 401 to 403.

[0138] Step 401: The head mounted device calculates at least one sound gain parameter based on the first sound feature information, the second sound feature information and the speech signal.

[0139] In the embodiment of the present application, each sound gain parameter of the at least one sound gain parameter corresponds to a time-frequency point of the speech signal.

[0140] In the embodiment of the present application, the time-frequency point refers to a coordinate point formed by combining the amplitude of the speech signal in the time domain and the frequency in the frequency domain.

[0141] In an embodiment of the present application, the head-mounted device can input the first sound feature information, the second sound feature information and the speech signal into the above-mentioned sound processing model, and then, the sound processing model can identify the human voice interference speech signal from the speech signal based on the first sound feature information, and identify the human voice speech signal of the first user from the speech signal through the second sound feature information; then, the sound processing model can calculate at least one first ratio between the amplitude of the human voice interference speech signal and the amplitude of the human voice speech signal in the time domain, and calculate at least one second ratio between the frequency of the human voice interference speech signal and the frequency of the human voice speech signal in the frequency domain; finally, the sound processing model can perform a dot product of at least one first ratio and at least one second ratio to obtain at least one sound gain parameter.

[0142] In the embodiment of the present application, each first ratio in the at least one first ratio corresponds one-to-one to each second ratio in the at least one second ratio.

[0143] Step 402: The head mounted device performs frequency domain conversion processing on the speech signal to obtain a speech signal after frequency domain conversion.

[0144] In the embodiment of the present application, the head mounted device can perform a fast Fourier transform (FFT) on the speech signal through the above-mentioned sound processing model to obtain a speech signal after frequency domain conversion.

[0145] Step 403: The head mounted device suppresses the speech signal corresponding to the first real object in the speech signal after frequency domain conversion based on at least one sound gain parameter.

[0146] In an embodiment of the present application, the head-mounted device can use the above-mentioned sound processing model to perform a point multiplication of at least one sound gain parameter with the frequency domain speech signal corresponding to at least one frequency point of the speech signal after frequency domain conversion to obtain a processed frequency domain speech signal, and then perform an inverse fast Fourier transform (IFFT) on the processed frequency domain speech signal to obtain a processed speech signal.

[0147] In an embodiment of the present application, the head-mounted device can identify the sound signal of the interfering user from the above voice signal through the first sound feature information corresponding to the sound signal of the interfering user, and then suppress the sound signal of the interfering user, so that the human voice signal of the user wearing the head-mounted device can be highlighted in the suppressed voice signal, thereby improving the call quality.

[0148] Optionally, in an embodiment of the present application, before the above step 201, the speech enhancement method provided in the embodiment of the present application further includes the following steps 501 and 502.

[0149] Step 301: The head mounted device collects a second sound signal and obtains sound parameters corresponding to the second sound signal.

[0150] In the embodiment of the present application, the head-mounted device can collect the second sound signal through the microphone module in the head-mounted device, and analyze the second sound signal to obtain the above-mentioned sound parameters.

[0151] Optionally, in the embodiment of the present application, the above-mentioned sound parameter may include at least one of a signal-to-noise ratio and a volume.

[0152] In an embodiment of the present application, after the head-mounted device collects the second sound signal, the head-mounted device can identify the intensity of the human voice signal and the intensity of the ambient noise signal in the second sound signal, and then determine the ratio between the intensity of the human voice signal and the intensity of the ambient noise signal as the signal-to-noise ratio.

[0153] Step 302: When the sound parameter satisfies the first condition, the head mounted device obtains sound characteristic information corresponding to the second sound signal, and determines the sound characteristic information corresponding to the second sound signal as the second sound characteristic information.

[0154] Optionally, in an embodiment of the present application, when the sound parameters meet the first condition, the head-mounted device can register the second sound signal, that is, store the second sound signal, and obtain sound feature information corresponding to the second sound signal.

[0155] Optionally, in an embodiment of the present application, when the above-mentioned sound parameters include a signal-to-noise ratio, and when the signal-to-noise ratio of the above-mentioned second sound signal is greater than or equal to a preset signal-to-noise ratio threshold, the head-mounted device can obtain sound feature information corresponding to the second sound signal.

[0156] Optionally, in the embodiment of the present application, the preset signal-to-noise ratio threshold may be preset by the head mounted device. For example, the preset signal-to-noise ratio threshold may be 3Kdb.

[0157] Exemplarily, the signal-to-noise ratio of the second sound signal obtained by the head-mounted device is 5Kdb. After obtaining the signal-to-noise ratio of the second sound signal, the head-mounted device can compare the signal-to-noise ratio of 5Kdb with a preset signal-to-noise ratio threshold of 3Kdb. At this time, the head-mounted device determines that the signal-to-noise ratio of the second sound signal is greater than the preset signal-to-noise ratio threshold; then register the second sound signal, and obtain the sound feature information of the second sound signal.

[0158] Optionally, in an embodiment of the present application, when the above-mentioned sound parameters include volume, when the volume of the above-mentioned second sound signal is greater than or equal to a preset volume threshold, the head-mounted device can obtain sound feature information corresponding to the second sound signal.

[0159] Optionally, in the embodiment of the present application, the preset volume threshold may be preset by the head mounted device. For example, the preset volume threshold may be 80db.

[0160] Exemplarily, the volume of the second sound signal obtained by the head-mounted device is 90db. After obtaining the volume of the second sound signal, the head-mounted device can compare the volume 90db with the preset volume threshold 80db. At this time, the head-mounted device determines that the volume of the second sound signal is greater than the preset volume threshold; then register the second sound signal, and obtain the sound characteristic information of the second sound signal.

[0161] Optionally, in an embodiment of the present application, when the above-mentioned sound parameters include signal-to-noise ratio and volume, when the signal-to-noise ratio of the above-mentioned second sound signal is greater than or equal to a preset signal-to-noise ratio threshold, and the volume of the second sound signal is greater than or equal to a preset volume threshold, the head-mounted device can obtain sound feature information corresponding to the second sound signal.

[0162] Exemplarily, the signal-to-noise ratio of the second sound signal obtained by the head-mounted device is 5Kdb. After obtaining the signal-to-noise ratio of the second sound signal, the head-mounted device can compare the signal-to-noise ratio of 5Kdb with the preset signal-to-noise ratio threshold value 3Kdb. At this time, the head-mounted device determines that the signal-to-noise ratio of the second sound signal is greater than the preset signal-to-noise ratio threshold value; then, the head-mounted device can compare the volume of 90db with the preset volume threshold value 80db. At this time, the head-mounted device determines that the volume of the second sound signal is greater than the preset volume threshold value; finally, the head-mounted device can register the second sound signal and obtain the sound characteristic information of the second sound signal.

[0163] In an embodiment of the present application, the head-mounted device can obtain sound feature information of the first user wearing the head-mounted device in advance, so that the voice signal of the first user can be determined from the voice signal detected in the first scene; and the first user is determined by threshold judgment, and the wearer's voice can be recognized and registered during use, avoiding the problem that the existing offline registration method is only applicable to the device owner, thereby improving the universality and flexibility of the device.

[0164] The above-mentioned method embodiments, or various possible implementation methods in each method embodiment, can be executed separately, or, under the premise that there is no contradiction, can also be executed in combination with each other. The specific implementation can be determined according to actual usage requirements, and the embodiments of the present application do not limit this.

[0165] It should be noted that the speech enhancement method provided in the embodiment of the present application can be executed by a speech enhancement device. In the embodiment of the present application, the speech enhancement device provided in the embodiment of the present application is described by taking the speech enhancement method executed by the speech enhancement device as an example.

[0166] Fig.10 FIG. 1 is a schematic diagram showing a possible structure of a speech enhancement device involved in an embodiment of the present application. Fig.10 As shown, the speech enhancement device 70 may include: a determination module 71, an acquisition module 72 and a processing module 73.

[0167] Among them, the determination module 71 is used to determine the first real object in the first scene based on the gaze information of the first user during the call through the head-mounted device, the first user is a user wearing the head-mounted device, and the first real object is a real object that has a speaking behavior and the first user intends to suppress the sound. The acquisition module 72 is used to obtain the first sound feature information of the first real object determined by the determination module. The processing module 73 is used to suppress the voice signal corresponding to the first real object in the voice signal detected in the first scene based on the first sound feature information obtained by the acquisition module and the second sound feature information of the first user.

[0168] In a possible implementation, the speech enhancement device 70 provided in the embodiment of the present application also includes: a display module, a marking module and a receiving module. The display module is used to display a first picture before the determination module 71 determines the first real object in the first scene based on the gaze information of the first user, and the first picture includes at least one object identifier, and each object identifier in the at least one object identifier is used to indicate a real object in the first scene. The marking module is used to mark N object identifiers in the at least one object identifier in the first picture based on the gaze information, and the N object identifiers are used to indicate a real object with a speaking behavior. The receiving module is used to receive a first input of a first object identifier among the N object identifiers marked by the marking module. The determination module is specifically used to determine the real object indicated by the first object identifier as the first real object in response to the first input received by the receiving module.

[0169] In a possible implementation, the display module is further used to display N controls at positions corresponding to the N object identifiers in the first screen based on the gaze information, the N controls corresponding to the N object identifiers one by one, and each of the N controls is used to indicate at least one sound suppression processing method. The receiving module is specifically used to receive a first input to the control corresponding to the first object identifier.

[0170] In a possible implementation, the first input is an input for selecting a sound suppression processing mode through a control corresponding to the first object identifier. The processing module 73 is specifically configured to suppress a speech signal corresponding to the first real object in the speech signal detected in the first scene based on the first sound feature information and the second sound feature information by adopting the sound suppression processing mode corresponding to the first input.

[0171] In one possible implementation, the processing module 73 is specifically used to calculate at least one sound gain parameter based on the first sound feature information, the second sound feature information and the speech signal, each sound gain parameter corresponding to a time-frequency point of the speech signal; and perform frequency domain conversion processing on the speech signal to obtain a speech signal after frequency domain conversion; and based on at least one sound gain parameter, suppress the speech signal corresponding to the first real object in the speech signal after frequency domain conversion.

[0172] In a possible implementation, the acquisition module 73 is further configured to collect the second sound signal and obtain the sound parameters corresponding to the second sound signal before the determination module 71 determines the first real object in the first scene based on the gaze information of the first user during a call through the head mounted device. The processing module 73 is further configured to obtain the sound feature information corresponding to the second sound signal when the sound parameters meet the first condition; and determine the sound feature information corresponding to the second sound signal as the second sound feature information.

[0173] An embodiment of the present application provides a voice enhancement device. During a call through the voice enhancement device, since the first real object that the user wants to suppress sound can be obtained based on the user's line of sight information, that is, through eye tracking, and then the sound feature information of the first real object can be directionally acquired, the voice signal corresponding to the first real object can be identified from the above-mentioned first scene through the sound feature information, so that the voice signal corresponding to the first real object can be voice suppressed to suppress voice interference signals other than the voice signal of the user of the head-mounted device during the call, thereby avoiding the difficulty in highlighting the voice signal of the user wearing the head-mounted device due to the voice interference signal during the call, and improving the call quality.

[0174] The speech enhancement device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the mobile electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0175] The speech enhancement device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0176] The speech enhancement device provided in the embodiment of the present application can implement each process implemented in the above embodiment, and to avoid repetition, it will not be described again here.

[0177] Alternatively, if Fig.11 As shown, an embodiment of the present application further provides an electronic device 90, including a processor 91 and a memory 92, wherein the memory 92 stores programs or instructions that can be executed on the processor 91, and when the program or instructions are executed by the processor 91, the various steps of the above-mentioned speech enhancement method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0178] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0179] Fig.12 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0180] The electronic device 100 includes but is not limited to components such as a radio frequency unit 101, a network module 102, an audio output unit 103, an input unit 104, a sensor 105, a display unit 106, a user input unit 107, an interface unit 108, a memory 109, and a processor 110.

[0181] Those skilled in the art will appreciate that the electronic device 100 may also include a power source (such as a battery) for supplying power to various components, and the power source may be logically connected to the processor 110 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig.12 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0182] Among them, the above-mentioned processor 110 is used to determine the first real object in the first scene based on the gaze information of the first user during a call through the head-mounted device, the first user is a user wearing the head-mounted device, and the first real object is a real object that has a speaking behavior and the first user intends to perform sound suppression processing; and obtain first sound feature information of the first real object; and based on the first sound feature information and the second sound feature information of the first user, suppress the voice signal corresponding to the first real object in the voice signal detected in the first scene.

[0183] Optionally, in an embodiment of the present application, the display unit 106 is further used to display a first picture before determining the first real object in the first scene based on the gaze information of the first user, wherein the first picture includes at least one object identifier, and each object identifier is used to indicate a real object in the first scene. The processor 110 is specifically used to mark N object identifiers out of at least one object identifier in the first picture based on the gaze information, and the N object identifiers are used to indicate a real object with a speaking behavior. The user input unit 107 is used to receive a first input of a first object identifier among the N object identifiers. The processor 110 is also used to determine the real object indicated by the first object identifier as the first real object in response to the first input.

[0184] Optionally, in an embodiment of the present application, the display unit 106 is specifically used to display N controls at positions corresponding to N object identifiers in the first screen based on the gaze information, the N controls corresponding to the N object identifiers one by one, and each control is used to indicate at least one sound suppression processing method. The user input unit 107 is specifically used to receive a first input to the control corresponding to the first object identifier.

[0185] Optionally, in an embodiment of the present application, the first input is an input for selecting a sound suppression processing method through a control corresponding to a first object identifier; the processor 110 is specifically used to adopt the sound suppression processing method corresponding to the first input, and based on the first sound feature information and the second sound feature information, suppress the voice signal corresponding to the first real object in the voice signal.

[0186] Optionally, in an embodiment of the present application, the above-mentioned processor 110 is specifically used to calculate at least one sound gain parameter based on the first sound feature information, the second sound feature information and the speech signal, each sound gain parameter corresponding to a time-frequency point of the speech signal; and perform frequency domain conversion processing on the speech signal to obtain a speech signal after frequency domain conversion; and based on at least one sound gain parameter, suppress the speech signal corresponding to the first real object in the speech signal after frequency domain conversion.

[0187] Optionally, in an embodiment of the present application, the above-mentioned processor 110 is specifically used to collect a second sound signal and obtain sound parameters corresponding to the second sound signal before determining the first real object in the first scene based on the gaze information of the first user during a call through the head-mounted device; and when the sound parameters meet the first condition, obtain sound feature information corresponding to the second sound signal, and determine the sound feature information corresponding to the second sound signal as the second sound feature information.

[0188] An embodiment of the present application provides a head-mounted device. During a call via the head-mounted device, since the first real object that the user wants to perform sound suppression processing on can be obtained based on the user's line of sight information, that is, through eye tracking, and then the sound feature information of the first real object can be directionally acquired, the voice signal corresponding to the first real object can be identified from the above-mentioned first scene through the sound feature information, so that the voice signal corresponding to the first real object can be voice suppressed to suppress voice interference signals other than the voice signal of the user of the head-mounted device during the call, thereby avoiding the difficulty in highlighting the voice signal of the user wearing the head-mounted device due to the voice interference signal during the call, and improving the call quality.

[0189] The electronic device provided in the embodiment of the present application can implement each process implemented in the above method embodiment and can achieve the same technical effect. To avoid repetition, it will not be described here.

[0190] The beneficial effects of various implementations in this embodiment can be specifically referred to the beneficial effects of the corresponding implementations in the above method embodiment. To avoid repetition, they will not be described again here.

[0191] It should be understood that in the embodiment of the present application, the input unit 104 may include a graphics processor (Graphics Processing Unit, GPU) 1041 and a microphone 1042, and the graphics processor 1041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 106 may include a display panel 1061, and the display panel 1061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 107 includes a touch panel 1071 and at least one of other input devices 1072. The touch panel 1071 is also called a touch screen. The touch panel 1071 may include two parts: a touch detection device and a touch controller. Other input devices 1072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0192] The memory 109 can be used to store software programs and various data. The memory 109 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 109 may include a volatile memory or a non-volatile memory, or the memory 109 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 109 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0193] The processor 110 may include one or more processing units; optionally, the processor 110 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 110.

[0194] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0195] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0196] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0197] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0198] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned speech enhancement method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0199] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0200] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0201] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A speech enhancement method, characterized in that: The method is performed by a head mounted device and includes: During a call through the head mounted device, determining a first real object in a first scene based on gaze information of a first user, the first user being a user wearing the head mounted device, and the first real object being a real object that is in a speaking behavior and for which the first user intends to perform sound suppression processing; Acquire first sound feature information of the first real object; Based on the first sound feature information and the second sound feature information of the first user, a suppression process is performed on the voice signal corresponding to the first real object in the voice signal detected in the first scene.

2. The method according to claim 1, characterized in that Before determining the first real object in the first scene based on the gaze information of the first user, the method further includes: Displaying a first picture, wherein the first picture includes at least one object identifier, each object identifier being used to indicate a real object in the first scene; The determining, based on the gaze information of the first user, a first real object in the first scene includes: Based on the gaze information, marking N object identifiers of the at least one object identifier in the first picture, wherein the N object identifiers are used to indicate real objects with speech behavior; Receiving a first input of a first object identifier among the N object identifiers; In response to the first input, the real object indicated by the first object identifier is determined as the first real object.

3. The method according to claim 2, characterized in that The marking, based on the gaze information, N object identifiers of the at least one object identifier in the first picture includes: Based on the gaze information, displaying N controls at positions corresponding to the N object identifiers in the first picture, the N controls corresponding to the N object identifiers one by one, and each control being used to indicate at least one sound suppression processing mode; The receiving a first input of a first object identifier among the N object identifiers includes: A first input to a control corresponding to the first object identifier is received.

4. The method according to claim 3, characterized in that The first input is an input for selecting a sound suppression processing mode through a control corresponding to the first object identifier; The processing of the voice signal detected in the first scene based on the first voice feature information and the second voice feature information of the first user includes: The sound suppression processing method corresponding to the first input is adopted to suppress the speech signal corresponding to the first real object in the speech signal based on the first sound feature information and the second sound feature information.

5. The method according to any one of claims 1 to 4, characterized in that The suppressing process of the voice signal corresponding to the first real object in the voice signal detected in the first scene based on the first voice feature information and the second voice feature information of the first user includes: Based on the first sound feature information, the second sound feature information and the speech signal, at least one sound gain parameter is calculated, each sound gain parameter corresponding to a time-frequency point of the speech signal; Performing frequency domain conversion processing on the speech signal to obtain the speech signal after frequency domain conversion; Based on the at least one sound gain parameter, a suppression process is performed on the speech signal corresponding to the first real object in the speech signal after frequency domain conversion.

6. The method according to claim 1, characterized in that During the call through the head mounted device, before determining the first real object in the first scene based on the gaze information of the first user, the method further includes: Collecting a second sound signal, and obtaining sound parameters corresponding to the second sound signal; In a case where the sound parameter satisfies the first condition, sound feature information corresponding to the second sound signal is acquired, and the sound feature information corresponding to the second sound signal is determined as the second sound feature information.

7. A speech enhancement device, characterized in that: The speech enhancement device comprises: a determination module, an acquisition module and a processing module; The determining module is configured to determine, during a call through the head mounted device, a first real object in a first scene based on gaze information of a first user, the first user being a user wearing the head mounted device, the first real object being a real object that has a speaking behavior and for which the first user intends to perform sound suppression processing; The acquisition module is used to acquire the first sound feature information of the first real object determined by the determination module; The processing module is used to suppress the voice signal corresponding to the first real object in the voice signal detected in the first scene based on the first sound feature information acquired by the acquisition module and the second sound feature information of the first user.

8. The device according to claim 7, characterized in that The speech enhancement device also includes: a display module, a marking module and a receiving module; The display module is used to display a first picture before the determination module determines the first real object in the first scene based on the gaze information of the first user, wherein the first picture includes at least one object identifier, each object identifier is used to indicate a real object in the first scene; The marking module is used to mark N object identifiers of the at least one object identifier in the first picture based on the gaze information, wherein the N object identifiers are used to indicate real objects with speech behavior; The receiving module is used to receive a first input of a first object identifier among the N object identifiers marked by the marking module; The determining module is specifically configured to determine the real object indicated by the first object identifier as the first real object in response to the first input received by the receiving module.

9. The device according to claim 8, characterized in that The display module is further used to display N controls at positions corresponding to the N object identifiers in the first picture based on the gaze information, the N controls corresponding to the N object identifiers one by one, and each control is used to indicate at least one sound suppression processing method; The receiving module is specifically configured to receive a first input to a control corresponding to the first object identifier.

10. The device according to claim 9, characterized in that The first input is an input for selecting a sound suppression processing mode through a control corresponding to the first object identifier; The processing module is specifically used to adopt the sound suppression processing method corresponding to the first input, and based on the first sound feature information and the second sound feature information, suppress the voice signal corresponding to the first real object in the voice signal detected in the first scene.

11. The device according to any one of claims 7 to 10, characterized in that The processing module is specifically used for: Based on the first sound feature information, the second sound feature information and the speech signal, at least one sound gain parameter is calculated, each sound gain parameter corresponding to a time-frequency point of the speech signal; Performing frequency domain conversion processing on the speech signal to obtain the speech signal after frequency domain conversion; Based on the at least one sound gain parameter, a suppression process is performed on the speech signal corresponding to the first real object in the speech signal after frequency domain conversion.

12. The device according to claim 7, characterized in that The acquisition module is further configured to collect a second sound signal and obtain a sound parameter corresponding to the second sound signal before the determination module determines the first real object in the first scene based on the gaze information of the first user during a call through the head mounted device; The processing module is further configured to, when the sound parameter satisfies the first condition, obtain sound feature information corresponding to the second sound signal; and determine the sound feature information corresponding to the second sound signal as the second sound feature information.

13. An electronic device, characterized in that: The method comprises a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the speech enhancement method as claimed in any one of claims 1 to 6.

14. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the speech enhancement method according to any one of claims 1 to 6 are implemented.

15. A computer program product, characterized in that The computer program product is stored in a storage medium, and the computer program product is executed by at least one processor to implement the speech enhancement method according to any one of claims 1 to 6.

Images