Voice processing method, device and system, storage medium and program product

Abstract

The embodiment of the invention provides a voice processing method, device and system, a storage medium and a program product, and relates to the technical field of terminals. According to the voice processing method, in a face-to-face communication scene and a conversation scene, the speech barrier voice of the speech barrier user can be restored, the restored voice is obtained, the intelligibility of the restored voice is higher than that of the speech barrier voice, so that the speech barrier user can communicate by inputting the voice, and the user experience is improved. According to the invention, the technical problem of insufficient communication convenience of the speech-impaired users can be solved, so that the technical effect of improving the communication convenience of the speech-impaired users is realized.

Description

[0001] This application claims the priority of the Chinese patent application filed with the China Patent Office on June 20, 2024, with application number 202410808992.4 and application name “A method and device for assisting communication”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of terminal technology, and in particular to a voice processing method, device, system, storage medium and program product. Background Art

[0003] With the development of the field of terminal technology, more and more functions are applied to terminals, such as a user communication function applied to terminals.

[0004] At present, there are some users who cannot communicate as smoothly as normal people due to congenital or acquired factors. For example, people with special speech impairments such as hearing impairment, ALS, and vocal cord damage have defects in their pronunciation, which makes their speech intelligible and difficult for listeners to understand, which greatly affects their daily social life. Therefore, for people with special speech impairments, they need to input text on electronic devices in communication scenarios, and then the electronic devices can convert the text input by the user into voice and output it.

[0005] However, for people with special speech impairments, they need to communicate by inputting text on electronic devices, which makes communication inconvenient. Summary of the invention

[0006] The present application provides a speech processing method, device, system, storage medium and program product, which are conducive to improving the convenience of communication between users.

[0007] In a first aspect, an embodiment of the present application provides a speech processing method, which may include:

[0008] Receive a setting input, where the setting input is used to indicate a scenario for enabling voice repair, a contact for enabling voice repair, or an application for enabling voice repair, where the scenario includes a face-to-face communication scenario or a remote communication scenario. Then, obtain the user's first voice feature through sound registration. The order of sound registration and setting input can be swapped. Then, when the user speaks, the first voice input by the user can be received. Since the setting input has been received before, the first voice can be repaired according to the setting input and the first voice feature.

[0009] In an embodiment of the present application, the first voice is repaired according to the setting input and the first voice feature, and because the setting input is used to indicate the scene in which voice repair is turned on, the contact for which voice repair is turned on, or the application for which voice repair is turned on, when it is detected that the electronic device is in a scene in which voice repair is turned on, or when the contact communicated by the electronic device is the contact for which voice repair is turned on, or when the application being run by the electronic device is the application for which voice repair is turned on, the first voice is repaired according to the first voice feature, so that the repaired first voice can be broadcast with the first voice feature, and the intelligibility of the repaired first voice is higher than the intelligibility of the first voice before repair. Among them, intelligibility can indicate the accuracy of expressing what the user wants to express, and can also be understood as the listener's understanding of the voice signal transmitted by the speaker.

[0010] In an embodiment of the present application, by receiving a setting input, the setting input is used to indicate a scenario for turning on voice repair, a contact for turning on voice repair, or an application for turning on voice repair, and then the user's first voice feature is obtained through sound registration. After receiving the first voice input by the user, the first voice can be repaired according to the setting input and the first voice feature, so that speech-impaired users or hearing-impaired users can communicate by inputting voice, thereby improving the convenience of communication for speech-impaired users. In addition, the repaired first voice is generated through the first voice feature registered by the user, so that the repaired first voice can be broadcast according to the first voice feature, so that the repaired first voice can be broadcast according to the user's own voice feature, which is closer to the user's own timbre, etc., thereby further improving the user experience.

[0011] In a possible implementation, the remote communication scenario includes a call scenario. In the call scenario, the method further includes:

[0012] A first prompt message is sent to the first contact of the call, thereby prompting that the voice repair function has been turned on. And / or, a second prompt message is sent to the first contact when the first voice is received, thereby prompting that the first voice is being repaired.

[0013] In the embodiment of the present application, by sending a first prompt message to the first contact of the call to prompt the user that the voice repair function has been turned on, the first contact can be informed that the user has turned on the voice repair function, thereby improving the call experience. In addition, by sending a second prompt message to the first contact when the first voice is received to prompt that the first voice is being repaired, the first contact can be informed that the delay is caused by the voice repair, thereby reducing the conflict of speaking time of two or more parties in the call, thereby improving the call experience.

[0014] In a possible implementation, the second prompt information includes a prompt tone, and sending the second prompt information to the first contact when the first voice is received may include:

[0015] When the first voice starts to be received, the prompt tone is continuously sent to the first contact. And, when the repaired first voice starts to be sent to the first contact, the prompt tone is stopped.

[0016] In an embodiment of the present application, when the first voice is initially received, a prompt tone is continuously sent to the first contact. Thus, when the first contact hears the prompt tone, he or she can know that the user on the other end is speaking. When the repaired first voice is initially sent to the first contact, the prompt tone is stopped and the first contact can subsequently hear the repaired first voice. This can reduce the situation where two or more parties speak in a call and thereby improve the experience of both parties on the call.

[0017] In a possible implementation, the remote communication scenario includes a call scenario. In the call scenario, the method further includes:

[0018] The first voice after repair is sent to the first contact of the call. Then, a first interface can be displayed, the first interface is an interface for making a call with the first contact, the first interface includes a first control, and the first control is used to control the closing of the voice repair function. Then, in response to the first operation on the first control, the voice repair function is closed to send a second voice to the first contact, the second voice includes the user's voice received after the voice repair function is closed.

[0019] In the embodiment of the present application, during a call, the voice repair function can also be controlled to be turned off, so that the user can control the voice repair function to be turned off when the voice repair function is not needed, thereby improving the flexibility of switching between using or not using the voice repair function. For example, the user can control the voice repair function to be turned off through the first control when the electronic device is low on power or the electronic device is stuck, thereby improving the turning off of the voice repair function during a call and improving the user experience.

[0020] In a possible implementation, in response to the first operation, the first control is also switched from a first state to a second state, the first state is used to indicate that the voice repair function is turned on, and the second state is used to indicate that the voice repair function is turned off, and the method further includes:

[0021] In response to the second operation on the first control, the first control is then switched from the second state to the first state, and the voice repair function is turned on, so as to send a repaired third voice to the first contact, where the third voice includes the voice received from the user after the voice repair function is turned on.

[0022] In the embodiment of the present application, the state of the first control can be used to know whether the voice repair function is currently turned on, thereby improving the accuracy of the user's choice to use or not use the voice repair function. In addition, the embodiment of the present application can not only turn off the voice repair function during a call, but also turn on the voice repair function again during a call, so that the flexibility of turning on or off the voice repair can be improved. In addition, the voice repair function is turned on or off through the same control, thereby improving the simplicity of the interface during a call.

[0023] It should be understood that the WeChat application and the face-to-face communication application can also support turning off or on the voice repair function in the application interface. Please refer to the relevant description of how to turn off or on the voice repair function in the call interface, which will not be repeated here.

[0024] In a possible implementation, the method further includes:

[0025] A second interface is displayed, and the second interface includes playback controls for the repaired first voice and a first text corresponding to the repaired first voice. Then, if the user selects a first target character in the first text, a third operation is received, and the third operation is used to select the first target character in the first text. In response to the third operation, at least one candidate character related to the first target character is displayed. Then, if the user selects one of the candidate characters, a fourth operation may be received, and the fourth operation is used to select a second target character from at least one candidate character. In response to the fourth operation, a second text and playback controls for the voice corresponding to the second text are displayed, and the second text is obtained by replacing the first target character in the first text with the second target character.

[0026] In an embodiment of the present application, by displaying a second interface, the second interface includes a playback control of the repaired first voice and a first text corresponding to the repaired first voice. A third operation is received, and the third operation is used to select a first target character in the first text. In response to the third operation, at least one candidate character related to the first target character is displayed. A fourth operation is received, and the fourth operation is used to select a second target character from at least one candidate character. In response to the fourth operation, the second text and the playback control of the voice corresponding to the second text are displayed, so that when the result of the voice repair is inaccurate, the user can manually correct it quickly, thereby improving the accuracy of communication through voice.

[0027] In a possible implementation, the method further includes:

[0028] Disabled display of playback controls for the first voice after repair.

[0029] In the embodiment of the present application, by canceling the display of the playback control of the repaired first voice, there can be only one playback control for the latest voice at a time, which can improve the convenience of selecting the appropriate voice for broadcasting.

[0030] In a possible implementation, in a remote communication scenario, the method further includes:

[0031] The third interface is displayed, and the third interface is an interface for communicating with the second contact. The third interface includes a second control and a third control. The second control is used to indicate that the first voice is sent to the second contact, and the third control is used to indicate that the repaired first voice is sent to the second contact. Then, in response to an operation on the third control, the repaired first voice can be sent to the second contact. Alternatively, in response to an operation on the second control, the first voice can be sent to the second contact.

[0032] In an embodiment of the present application, the user can use the second control and the third control to selectively send the repaired voice or the voice before repair to the second contact. That is, even if the voice repair function is turned on, the user can still choose to send the original voice to the second contact, thereby improving the user's flexibility in remote communication.

[0033] In a possible implementation, in a face-to-face communication scenario, the method further includes:

[0034] A fourth interface is displayed, the fourth interface including a virtual keyboard. Then, the user can operate the virtual keyboard, and the third text can be displayed in response to the operation on the virtual keyboard. Then, the user can operate the fourth control, and a fifth operation on the fourth control in the fourth interface can be received, the fourth control is used to instruct the generation of speech. In response to the fifth operation, speech corresponding to the third text is generated according to the first speech feature.

[0035] In an embodiment of the present application, by displaying a fourth interface, the fourth interface includes a virtual keyboard. In response to an operation on the virtual keyboard, a third text is displayed. A fifth operation on a fourth control in the fourth interface is received, and the fourth control is used to instruct the generation of speech. In response to the fifth operation, a speech corresponding to the third text is generated according to the first speech feature, that is, the user can also input text and then generate speech according to the registered first speech feature, that is, text-to-speech conversion is realized, and the user can communicate according to the choice of inputting voice or inputting text, thereby improving the selectivity and flexibility of the user in communicating.

[0036] In a possible implementation, the third text includes punctuation marks and / or emoticons. When generating speech corresponding to the third text according to the first speech feature, the tone of the speech corresponding to the third text can also be controlled according to the punctuation marks and / or emoticons.

[0037] In an embodiment of the present application, the tone of the voice corresponding to the third text can be controlled by punctuation marks and / or emoticons in the third text input by the user, thereby enabling the tone of the output voice to be adaptively adjusted according to the user input text, thereby increasing the flexibility of voice broadcasting and improving user experience.

[0038] In a possible implementation, obtaining the first voice feature of the user through voice registration includes:

[0039] A fifth interface is displayed, the fifth interface is an interface for voice registration, and the fifth interface includes a third prompt information and a fifth control, the third prompt information is used to prompt the recording content of the voice registration, and the fifth control is used to indicate the recording voice. If the user operates the fifth control, a fourth voice can be recorded in response to the operation on the fifth control. The first voice feature is extracted from the fourth voice.

[0040] In an embodiment of the present application, the recording of the voice is triggered by the fifth control, thereby improving the accuracy and effectiveness of the voice recording, and the subsequent comparison between the recorded content and the voice recognition result is also more accurate, thereby improving the accuracy of the repair model selection.

[0041] In a possible implementation, the method further includes:

[0042] Perform speech recognition on the fourth speech to obtain a fourth text. Then, compare the fourth text with the recorded content to obtain the similarity between the fourth text and the recorded content. Then, a target repair model can be selected from multiple repair models based on the similarity, and the target repair model is used to repair the first speech. The multiple repair models have different speech repair capabilities, and the speech repair capability of the target repair model is negatively correlated with the similarity.

[0043] In an embodiment of the present application, the text obtained by performing speech recognition on the voice registered by the user is compared with the recorded content, and then the similarity is obtained, and then the target repair model is selected from multiple repair models based on the similarity. In this way, a suitable repair model can be selected for repair according to the degree of the user's speech disorder, thereby balancing the accuracy of speech repair and the computing power resources required for speech repair.

[0044] In a second aspect, the present application also provides another speech processing method. The method may include:

[0045] Receive a setting input, the setting input is used to indicate a scenario for enabling voice repair, a contact for enabling voice repair, or an application for enabling voice repair, the scenario including a face-to-face communication scenario or a remote communication scenario. Then, a fifth voice from a target contact can be received. When voice repair is required, a second voice feature can be obtained, the second voice feature being a preset voice feature or a voice feature extracted from the fifth voice. Then, the fifth voice is repaired according to the setting input and the second voice feature.

[0046] In an embodiment of the present application, the fifth voice is repaired according to the setting input and the second voice feature, and because the setting input is used to indicate the scene in which voice repair is turned on, the contact for which voice repair is turned on, or the application for which voice repair is turned on, when it is detected that the electronic device is in the scene in which voice repair is turned on, when the contact communicated by the electronic device is the contact for which voice repair is turned on, or when the application being run by the electronic device is the application for which voice repair is turned on, the fifth voice is repaired according to the second voice feature, so that the intelligibility of the repaired fifth voice is higher than the intelligibility of the fifth voice before repair. Among them, intelligibility can indicate the degree of understanding of the voice when it is listened to, and can also be understood as the accuracy of the voice in expressing what the contact wants to express.

[0047] In a possible implementation, in a remote communication scenario, the method further includes:

[0048] The sixth interface is displayed, and the sixth interface is an interface for communicating with the target contact, and the sixth interface includes the fifth voice. Then, if the user operates the fifth voice, the sixth control and the seventh control can be displayed in response to the operation on the fifth voice, and the sixth control is used to indicate that the fifth voice is converted into text, and the seventh control is used to indicate that the repaired fifth voice is converted into text. Then, the user can operate the sixth control or the seventh control, and the text corresponding to the repaired fifth voice can be displayed in response to the operation on the seventh control. Alternatively, the text corresponding to the fifth voice can be displayed in response to the operation on the sixth control.

[0049] In an embodiment of the present application, after receiving a voice from a target contact, the fifth voice can be selectively converted into text, which can also be understood as converting the original voice of the target contact into text; the repaired fifth voice can also be converted into text, and the user can select the text to be displayed as needed, thereby improving the flexibility and experience of user communication.

[0050] In a possible implementation, in a face-to-face communication scenario, the method further includes:

[0051] The seventh interface is displayed, and the seventh interface is an interface for communicating with the target contact, and the seventh interface includes an eighth control. Then, the user can operate the eighth control, and the fifth text can be displayed in response to the sixth operation on the eighth control. Then, the user can operate the eighth control again, and the display of the fifth text can be stopped in response to the seventh operation on the eighth control. The fifth text includes the text corresponding to the fifth voice within the target time period or the text corresponding to the repaired fifth voice, and the target time period includes the time period between responding to the sixth operation and responding to the seventh operation.

[0052] In the embodiment of the present application, the voice of the target contact may also be converted into text, thereby increasing the flexibility of the communication method between the user and the contact.

[0053] In a third aspect, the embodiment of the present application further provides another speech processing method, which may include:

[0054] The target speech and speech features are obtained. Then, the target speech and speech features are input into the target restoration model, which is used to extract the content of the target speech, obtain speech continuity features according to the content and speech features of the target speech, and synthesize the restored target speech according to the speech continuity features. Then, the restored target speech output by the target restoration model can be obtained.

[0055] In an embodiment of the present application, after inputting the target speech, the electronic device can obtain the target speech and speech features, and then input the target speech and speech features into a target repair model. The target repair model is used to extract the content of the target speech, obtain speech continuity features based on the content and speech features of the target speech, and synthesize the repaired target speech based on the speech continuity features. In this way, people with speech impairments can communicate by inputting speech, thereby improving the convenience of communication for people with speech impairments.

[0056] In one possible implementation, the target repair model includes a first module, a second module and a third module. The first module is used to extract the content of the target speech, the second module is used to obtain speech continuity features based on the content and speech features of the target speech, and the third module is used to synthesize the repaired target speech based on the speech continuity features.

[0057] In a possible implementation, inputting the target speech and speech features into the target restoration model includes:

[0058] The speech features and the first part of the target speech are input into the target restoration model, the first module is used to extract the content of the first part of the speech, the second module is used to obtain the target speech continuous features according to the speech features and the content of the first part of the speech, and the third module is used to synthesize the restored first part of the speech according to the target speech continuous features. Then, the speech features and the second part of the target speech are input into the target restoration model, the first module is used to extract the content of the second part of the speech, the second module is used to obtain the second speech continuous features according to the speech features and the content of the second part of the speech, and the third module is used to synthesize the restored second part of the speech according to the second speech continuous features.

[0059] In an embodiment of the present application, a part of the target speech is first repaired, and then another part of the target speech is repaired. In this way, speech repair can be started when a part of the speech is obtained. In other words, speech repair can be started without obtaining the complete target speech, thereby improving the efficiency of speech repair.

[0060] In one possible implementation, the target repair model also includes a fourth module, which is used to discretize the first part of the speech to obtain discrete features of the target speech. The second module is also used to predict based on the discrete features of the target speech to obtain discrete features of the repaired target speech, and to obtain continuous features of the second speech based on the repaired discrete features of the target speech, speech features and the content of the second part of the speech.

[0061] In an embodiment of the present application, the first part of the speech is discretized by the fourth module to obtain discrete features of the target speech, and then the second module performs prediction based on the discrete features of the target speech to obtain the repaired discrete features of the target speech, and then the second speech continuous features are obtained based on the repaired discrete features of the target speech, the speech features and the content of the second part of the speech. In this way, the second speech continuous features can be obtained by combining the discrete features of the target speech, the speech features and the content of the second part of the speech, thereby improving the accuracy of the obtained second speech continuous features, thereby improving the accuracy of speech repair.

[0062] In a possible implementation, the fourth module is used to discretize the first part of speech to obtain discrete features of the target speech, including:

[0063] The fourth module is used to discretize the first part of speech according to the content of the first part of speech to obtain discrete features of the target speech.

[0064] In an embodiment of the present application, discrete features of the target speech are obtained by discretizing the first part of speech according to its content. That is, the content of the first part of speech is used as a reference for the discretization processing, thereby improving the accuracy of the obtained discrete features of the target speech and further improving the accuracy of speech restoration.

[0065] It should be noted that the solution of the embodiment of the present application can not only be used in sound restoration tasks, but can also be extended to tasks such as converting dialects to Mandarin and cross-language translation.

[0066] In a fourth aspect, another speech processing device is provided, including a processor, the processor is coupled to a memory, and can be used to execute instructions in the memory to implement the method in any possible implementation of the first aspect. Optionally, the device also includes a memory. Optionally, the device also includes a communication interface, and the processor is coupled to the communication interface.

[0067] In a fifth aspect, a processor is provided, comprising: an input circuit, an output circuit and a processing circuit. The processing circuit is used to receive a signal through the input circuit and transmit a signal through the output circuit, so that the processor executes the method in any possible implementation of the first aspect.

[0068] In the specific implementation process, the processor can be a chip, the input circuit can be an input pin, the output circuit can be an output pin, and the processing circuit can be a transistor, a gate circuit, a trigger, and various logic circuits. The input signal received by the input circuit can be, for example, but not limited to, received and input by a receiver, and the signal output by the output circuit can be, for example, but not limited to, output to a transmitter and transmitted by the transmitter, and the input circuit and the output circuit can be the same circuit, which is used as an input circuit and an output circuit at different times. The embodiments of the present application do not limit the specific implementation methods of the processor and various circuits.

[0069] In a sixth aspect, a processing device is provided, comprising a processor and a memory. The processor is used to read instructions stored in the memory, and can receive signals through a receiver and transmit signals through a transmitter to execute the method in any possible implementation of the first aspect.

[0070] Optionally, there are one or more processors and one or more memories.

[0071] Optionally, the memory may be integrated with the processor, or the memory may be provided separately from the processor.

[0072] In the specific implementation process, the memory can be a non-transitory memory, such as a read-only memory (ROM), which can be integrated with the processor on the same chip or can be set on different chips respectively. The embodiments of the present application do not limit the type of memory and the setting method of the memory and the processor.

[0073] It should be understood that the relevant data interaction process, such as sending indication information, can be a process of outputting indication information from a processor, and receiving capability information can be a process of receiving input capability information from a processor. Specifically, the processed output data can be output to a transmitter, and the input data received by the processor can come from a receiver. Among them, the transmitter and the receiver can be collectively referred to as a transceiver.

[0074] The processing device in the fourth aspect mentioned above can be a chip. The processor can be implemented by hardware or by software. When implemented by hardware, the processor can be a logic circuit, an integrated circuit, etc.; when implemented by software, the processor can be a general-purpose processor, which is implemented by reading the software code stored in the memory. The memory can be integrated in the processor or can be located outside the processor and exist independently.

[0075] In a seventh aspect, a computer program product is provided, the computer program product comprising: a computer program (also referred to as code, or instruction), which, when executed, enables a computer to execute a method in any possible implementation of the first aspect.

[0076] In an eighth aspect, a computer-readable storage medium is provided, which stores a computer program (also referred to as code, or instruction) which, when executed on a computer, enables the computer to execute a method in any possible implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 A schematic diagram of a system architecture provided for an embodiment of the present application;

[0078] Figure 2 A schematic diagram of a sound registration interface provided in an embodiment of the present application;

[0079] Figure 3 A schematic diagram of an interface of an application configured to use a sound repair function provided in an embodiment of the present application;

[0080] Figure 4 A schematic diagram of an interface for configuring a whitelist for sound repair provided in an embodiment of the present application;

[0081] Figure 5A schematic diagram of an interface for a remote real-time communication scenario provided in an embodiment of the present application;

[0082] Figure 6 A schematic diagram of a call scenario provided in an embodiment of the present application;

[0083] Figure 7 A schematic diagram of an interface for a remote non-real-time communication scenario provided in an embodiment of the present application;

[0084] Figure 8 A schematic diagram of an interface for a face-to-face communication scenario provided in an embodiment of the present application;

[0085] Fig. 9 A schematic diagram of another interface for a face-to-face communication scenario provided in an embodiment of the present application;

[0086] Fig.10 A flowchart of a speech processing method provided in an embodiment of the present application;

[0087] Fig.11 A flow chart of a sound registration method provided in an embodiment of the present application;

[0088] Fig.12 A schematic diagram of the architecture of a repair model provided in an embodiment of the present application;

[0089] Fig.13 A flowchart of another speech processing method provided in an embodiment of the present application;

[0090] Fig.14 A flowchart of another speech processing method provided in an embodiment of the present application;

[0091] Fig.15 A flowchart of another speech processing method provided in an embodiment of the present application;

[0092] Fig.16 A schematic diagram of the structure of a speech processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0093] In order to clearly describe the technical solutions of the embodiments of the present application, some terms and technologies involved in the embodiments of the present application are briefly introduced below:

[0094] 1. Electronic equipment:

[0095] The electronic devices in the embodiments of the present application may also be referred to as: terminal equipment, user equipment (UE), mobile station (MS), mobile terminal (MT), access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication equipment, user agent or user device, etc.

[0096] The electronic device may be a device that provides voice / data connectivity to a user, such as a handheld device, a vehicle-mounted device, etc., with a wireless connection function. At present, some examples of terminals are: mobile phones, tablet computers, laptop computers, PDAs, mobile internet devices (MIDs), wearable devices, virtual reality (VR) devices, augmented reality (AR) devices, wireless terminals in industrial control, wireless terminals in self driving, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, cellular phones, cordless phones, session initiation protocol (SIP) phones, wireless local loop (WLL) stations, personal digital assistants (PDAs), handheld devices with wireless communication capabilities, computing devices or other processing devices connected to wireless modems, vehicle-mounted devices, wearable devices, electronic devices in 5G networks or future evolved public land mobile communication networks (public land mobile The electronic devices in the network (PLMN) and the like are not limited to this in the embodiments of the present application.

[0097] As an example but not limitation, in the embodiments of the present application, the electronic device may also be a wearable device. Wearable devices may also be referred to as wearable smart devices, which are a general term for wearable devices that are intelligently designed and developed using wearable technology for daily wear, such as glasses, gloves, watches, clothing, and shoes. A wearable device is a portable device that is worn directly on the body or integrated into the user's clothes or accessories. Wearable devices are not only hardware devices, but also powerful functions achieved through software support, data interaction, and cloud interaction. Broadly speaking, wearable smart devices include full-featured, large-sized, and fully or partially independent of smartphones, such as smart watches or smart glasses, as well as devices that only focus on a certain type of application function and need to be used in conjunction with other devices such as smartphones, such as various types of smart bracelets and smart jewelry for vital sign monitoring.

[0098] In addition, in the embodiments of the present application, the electronic device can also be an electronic device in the Internet of Things (IoT) system. IoT is an important part of the future development of information technology. Its main technical feature is to connect objects to the network through communication technology, so as to realize the intelligent network of human-machine interconnection and object-to-object interconnection. The electronic device of the present application can also be a vehicle-mounted unit, a vehicle-mounted module, a vehicle-mounted component, a vehicle-mounted chip or a vehicle-mounted unit built into the vehicle as one or more components or units. The vehicle can implement the method of the present application through the built-in vehicle-mounted unit, vehicle-mounted module, vehicle-mounted component, vehicle-mounted chip or vehicle-mounted unit. Therefore, the embodiments of the present application can be applied to the Internet of Vehicles, such as vehicle to everything (V2X), long-term evolution-vehicle (LTE-V), vehicle-to-vehicle (V2V), etc. Optionally, the electronic device of the embodiment of the present application can also be referred to as a device.

[0099] 2. Access network equipment:

[0100] The access network device in the embodiment of the present application may also be referred to as a wireless access network device, which may be a transmission reception point (TRP), an evolved NodeB (eNB or eNodeB) in an LTE system, a home base station (for example, home evolved NodeB, or home Node B, HNB), a base band unit (BBU), or a wireless controller in a cloud radio access network (CRAN) scenario, or the access network device may be a relay station, an access point, a vehicle-mounted device, a wearable device, an access network device in a 5G network, or an access network device in a future evolved PLMN network, etc., may be an access point (AP) in a WLAN, may be a gNB in ​​a new radio (NR) system, may be a satellite base station in a satellite communication system, etc., and the embodiment of the present application is not limited.

[0101] The access network device in this embodiment may include a centralized unit (CU) node, or a distributed unit (DU) node, or an access network device including a CU node and a DU node, or a control plane CU node (CU-CP node) and a user plane CU node (CU-UP node) and an access network device of a DU node. The access network device including the CU node and the DU node can split the protocol layer of the access network device, put the functions of some protocol layers in the CU centralized control, and distribute the functions of the remaining part or all of the protocol layers in the DU, and the DU is centrally controlled by the CU. As an implementation method, the CU deployment protocol stack includes a radio resource control (RRC) layer, a packet data convergence protocol (PDCP) layer, and a service data adaptation protocol (SDAP) layer. The radio link control (RLC) layer, the media access control (MAC) layer, and the physical (PHY) layer in the DU deployment protocol stack. Thus, the CU has the processing capabilities of RRC, PDCP and SDAP. DU has the processing capabilities of RLC, MAC and PHY. The above-mentioned functional division is only an example and does not constitute a limitation on CU and DU. That is to say, there may be other functional divisions between CU and DU, which will not be described in detail in the embodiments of the present application. The functions of CU can be implemented by one entity or by different entities. For example, the functions of CU can be further divided, for example, the control plane (CP) and the user plane (UP) are separated, that is, the control plane (CU-CP) of CU and the user plane (CU-UP) of CU. For example, CU-CP and CU-UP can be implemented by different functional entities, and CU-CP and CU-UP can be coupled with DU to jointly complete the functions of access network equipment. In one possible way, CU-CP is responsible for control plane functions, mainly including RRC and PDCP-C, wherein PDCP-C is mainly responsible for encryption and decryption of control plane data, integrity protection, data transmission, etc. CU-UP is responsible for user plane functions, mainly including SDAP and PDCP-U, wherein SDAP is mainly responsible for processing the data of core network equipment and mapping data flows to bearers. PDCP-U is mainly responsible for data plane encryption and decryption, integrity protection, header compression, sequence number maintenance, data transmission, etc. CU-CP and CU-UP are connected through the E1 interface. CU-CP represents the access network equipment and is connected to the core network equipment through the interface between the core network equipment and the access network equipment.The DU is connected via F1-C (control plane). The CU-UP is connected via F1-U (user plane) to the DU. In addition, another possible implementation is that the PDCP-C is also in the CU-UP, which is not limited in this application. The access network device may be, for example, a base station.

[0102] 3. Artificial intelligence (AI):

[0103] AI is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. The fields involved in AI include but are not limited to at least one of automatic speech recognition (ASR), text to speech (TTS), image recognition or natural language processing. ASR is a technology that converts speech into text. By processing and analyzing speech signals, the language content is identified and converted into a readable text form. TTS is a technology that converts text information into speech output. By converting text information into speech feature vectors, and then converting speech features into audio signals, it can also provide personalized pronunciation services for enterprises and individuals through timbre selection, custom volume, and speech speed. Speech can also be called sound, audio, etc.

[0104] 4. Other terms

[0105] In the embodiments of the present application, words such as "first" and "second" are used to distinguish the same or similar items with substantially the same functions and effects. For example, the first voice and the second voice are only used to distinguish different voices, and their order is not limited. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit them to be different.

[0106] It should be noted that in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0107] In the embodiments of the present application, "at least one" refers to one or more, and "plurality" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0108] At present, there are some users who cannot communicate as smoothly as normal people due to congenital or acquired factors. For example, people with special speech impairments such as hearing impairment, ALS, and vocal cord damage have defects in their pronunciation, which leads to low intelligibility of their speech and difficulty in being understood by the listener, which greatly affects their daily social life. For the above-mentioned people with special speech impairments, in order to solve the problem of daily communication, it is necessary to provide a system to assist communication. People with special speech impairments can also be called speech-impaired users.

[0109] For the above-mentioned special speech-impaired people, in order to solve the problem of daily communication, a system that assists language understanding is needed. In the related art, for the above-mentioned special speech-impaired people, text can be input, and then the electronic device converts the input text into voice for output, so that the special speech-impaired people can also communicate through voice.

[0110] However, people with special speech impairments need to input text to communicate through voice, but inputting text is troublesome, which makes it inconvenient for people with special speech impairments to communicate through voice. For people with special speech impairments, their speech is difficult to understand and difficult for listeners to understand, not that they cannot make any sound.

[0111] In view of this, the embodiments of the present application propose a speech processing method, device, system, storage medium and program product, which can assist the communication of speech-impaired users by repairing the speech input by speech-impaired users, that is, dysarthric speech reconstruction (DSR). DSR can convert the speech of special speech-impaired groups with low intelligibility into normal speech with higher intelligibility. Speech can also be called sound, audio or sound source, etc. The speech processing method of the embodiments of the present application can convert the difficult speech-impaired speech into a more intelligible speech, and keep the timbre consistent with the original. In both short-distance communication scenarios and long-distance communication scenarios, the sound repair function has a high use value.

[0112] In one possible implementation, the voice repair method may be, for example, that an electronic device performs voice repair processing (referred to as voice repair or repair) on the voice input by a speech-impaired user to obtain a repaired voice. The repaired voice is more intelligible than the input voice. The electronic device may then output the repaired voice, which enables the party communicating with the speech-impaired user to better understand the expression of the speech-impaired user.

[0113] In another possible implementation, after obtaining the repaired speech, the repaired speech can be subjected to speech recognition processing to obtain recognized text. The recognized text converted from the repaired speech can be understood as the repaired recognized text, and then the electronic device can output the repaired recognized text. This can also enable the party communicating with the speech-impaired user to better understand the expression of the speech-impaired user.

[0114] In another possible implementation, the voice repair method may be, for example, an electronic device performing voice recognition processing on the input voice to obtain recognized text, and then repairing the recognized text to obtain repaired recognized text. The repaired recognized text is more intelligible than the recognized text, which can also enable the party communicating with the speech-impaired user to better understand the expression of the speech-impaired user.

[0115] It can be seen that through the voice repair method, speech-impaired users can communicate by inputting voice, which improves the convenience of communication for speech-impaired users and improves the communication experience of speech-impaired users.

[0116] See also Figure 1 , Figure 1 A schematic diagram of a system architecture provided for an embodiment of the present application.

[0117] like Figure 1 As shown in (a), the system architecture of the embodiment of the present application may include a first device 101, a second device 102, an access network device 103 and a cloud 104. The first device 101 and the second device 102 can communicate with each other through the access network device 103, and at least one of the first device 101 and the second device 102 is a device used by a speech-impaired user. The cloud 104 has a sound repair function, which can repair the input voice to obtain the output voice. The cloud 104 can be, for example, a server.

[0118] In one possible implementation, the first device 101 may be a device used by a speech-impaired user. After receiving the voice input by the speech-impaired user, the first device 101 sends the input voice to the cloud 104. The cloud 104 can repair the input voice, and then the cloud 104 sends the repaired voice to the first device 101. The first device 101 then sends the repaired voice to the second device 102 through the access network device 103. After receiving the repaired voice, the second device 102 can broadcast the repaired voice.

[0119] In another possible implementation, the second device 102 may be a device used by a speech-impaired user. After receiving the voice input by the speech-impaired user, the second device 102 sends the input voice to the first device 101. After receiving the input voice, the first device 101 sends the input voice to the cloud 104. The cloud 104 can repair the input voice, and then the cloud 104 sends the repaired voice to the first device 101. After receiving the repaired voice, the first device 101 can broadcast the repaired voice.

[0120] It should be understood that the above-mentioned voice repair processing can also be performed on at least one device in the first device 101 or the second device 102, so that the cloud 104 may not be required.

[0121] It should be noted that the cloud 104 may be configured with multiple speech repair models, which may also be referred to as repair models, for performing repair processing on the input speech to obtain repaired speech. Optionally, the speech repair capabilities of the multiple repair models are different.

[0122] like Figure 1 As shown in (b) , the system architecture of the embodiment of the present application may include a first device 101 and a cloud 104 .

[0123] In one possible implementation, after receiving the voice input by the speech-impaired user, the first device 101 sends the input voice to the cloud 104, and the cloud 104 can repair the input voice. Then the cloud 104 sends the repaired voice to the first device 101, and the first device 101 can output the repaired voice, or convert the repaired voice into text and output it.

[0124] It should be understood that the above-mentioned voice repair processing can also be performed on the first device 101, so that the cloud 104 may not be required.

[0125] It should be noted that the scenarios in which the solution of the embodiment of the present application can be used include but are not limited to the following scenarios:

[0126] Use scenario 1: Remote communication scenario. The remote communication scenario may include but is not limited to real-time or non-real-time remote communication scenarios. Real-time remote communication scenarios may include but are not limited to real-time call scenarios. Non-real-time remote communication scenarios may include but are not limited to non-real-time remote voice communication scenarios.

[0127] Usage scenario 2: short-distance communication scenario. The short-distance communication scenario may include but is not limited to face-to-face communication scenarios.

[0128] That is to say, in the embodiments of the present application, both the call scene and the face-to-face communication scene can integrate the sound repair capability into the interaction. In general, the embodiments of the present application can be applied to special speech-impaired people whose users cannot speak clearly, to achieve real-time repair of the input disabled voice, and play the repaired audio to the listener or transmit it to the call end device; the usage scenarios are long-distance communication scenes such as making calls and video calls, as well as short-distance communication scenes such as face-to-face communication.

[0129] The following are examples of remote communication scenarios and face-to-face communication scenarios.

[0130] First, an exemplary description is given for a remote communication scenario. This embodiment is described by taking a remote real-time communication scenario as an example.

[0131] It is understood that the terms "interface" and "user interface" in this application refer to the media interface for interaction and information exchange between an application or operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. The commonly used form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operation displayed in a graphical manner. It can be an interface element such as an icon, window, control, etc. displayed on the display screen of an electronic device, where the control can include icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, etc. Visual interface elements.

[0132] See also Figure 2 , Figure 2 A schematic diagram of a sound registration interface provided in an embodiment of the present application.

[0133] like Figure 2The interface shown in (a) in FIG. 1 shows a page with application icons, which may include multiple application icons (e.g., at least one of the settings application icon 201, the weather application icon, the calendar application icon, the photo album application icon, the email application icon, or the application store application icon, etc.). A page indicator may also be displayed below the multiple application icons to indicate the positional relationship between the currently displayed page and other pages. There are multiple application icons (e.g., the camera application icon, the address book application icon, the information application icon, and the dialing application icon) below the page indicator, and these application icons remain displayed when the page is switched.

[0134] It can be understood that the application icon of the embodiment of the present application is an icon for launching an application or an icon for jumping to a certain interface. For example, the camera application icon is an icon of a camera application (i.e., a camera application), that is, the camera application icon can be used to trigger the launch of the camera application. For another example, the setting application icon 201 can be used to trigger jumping to the setting interface.

[0135] In the embodiment of the present application, the electronic device may detect a user operation on the setting application icon 201, and in response to the user operation, the electronic device may display the following Figure 2 The user interface shown in (b) in FIG. Figure 2 The interface shown in (b) may be, for example, an interface for setting an application, on which a user may query information of an electronic device or perform at least one of user configurations.

[0136] It is understandable that the user operations mentioned in this application may include but are not limited to touch (for example, click, etc.), voice control, gestures and other operations, and this application does not limit this.

[0137] like Figure 2 The interface shown in (b) in the figure displays a page including setting options. The page may include one or more setting options. Among them, the setting options in the page may include at least one of accessibility setting options 202, battery and performance setting options, security and privacy setting options, display and brightness setting options, or network connection management options. The accessibility setting option 202 may be designed for users with special needs, and through a series of auxiliary functions, it helps users use electronic devices more conveniently. The functions that can be set in the accessibility setting option 202 of the embodiment of the present application may include but are not limited to at least one of sound optimization, screen reading, or color correction. Sound optimization may also be referred to as sound repair.

[0138] It should be noted that in Figure 2In the interface shown in (b), for the setting options for configuration, such as setting option 2, an indication of whether the option is turned on or off may also be included. Optionally, when the indication corresponding to the setting option is "off", the configuration corresponding to the setting option is not turned on, and when the indication corresponding to the setting option is "on", the configuration corresponding to the setting option is turned on. It should be understood that the indication of turning a setting option on or off may be different indications, and is not limited to specific text-style indications such as "off" or "on". By displaying the indication of whether the option is turned on or off on the interface, the user can quickly know whether the setting option is turned on, thereby improving the user experience. For setting options used to query information, the setting option may not include an indication of whether the option is turned on or off. In addition, if Figure 2 The interface shown in (b) may also include an indication that the option has a next-level interface, such as a ">" mark. By displaying an indication that the option has a next-level interface on the interface, the user can be informed of the next-level interface through the indication, thereby improving the user experience.

[0139] In the embodiment of the present application, the electronic device may detect a user operation on the accessibility setting option 202, and in response to the operation, the electronic device may display a Figure 2 The user interface shown in (c) in FIG. Figure 2 The interface shown in (c) may be, for example, an interface of the barrier-free setting option 202 , and the user may implement the setting function of the barrier-free setting option 202 on this interface.

[0140] like Figure 2 The interface shown in (c) in the figure shows a page including barrier-free functions. The page may include one or more barrier-free functions controls, such as the sound repair function control 203. The sound repair function control 203 may be used to turn the sound repair function on or off, that is, the sound repair function control 203 may be understood as the master switch of the sound repair function. Optionally, the interface may also include an indication of whether the sound repair function is turned on or off, and the indication of whether the sound repair function is turned on or off may be used to determine whether the sound repair function is turned on. Figure 2 As shown in (c) in FIG. 1 , the sound repair function is in a closed state, that is, the sound repair function is not turned on. Figure 2 The interface shown in (c) may also include an indication that the sound repair function has a next level interface. Figure 2 The interface shown in (c) may also include a control for returning to the previous interface, for example, including a control for returning to Figure 2 Controls of the interface shown in (b).

[0141] In the embodiment of the present application, the electronic device can detect the user operation of the control 203 acting on the sound repair function, and in response to the operation, the electronic device can display the following Figure 2 The user interface shown in (d) in FIG. Figure 2 The interface shown in (d) in the figure may be, for example, an interface of a sound repair function, which may be understood as a detailed setting page of the sound repair function. The user may implement relevant configurations of the sound repair function on the interface, such as implementing at least one of the sound recording of the sound repair function or the application or scene configuration supported by the sound repair function.

[0142] like Figure 2 The interface shown in (d) includes a sound recording control 204 and a sound repair application configuration control 205. The sound repair application configuration control 205 may include but is not limited to a call application configuration control 2051, a face-to-face communication application configuration control 2052 and a WeChat application configuration control 2053.

[0143] The voice recording control 204 may be used to record the user's voice. Figure 2 The interface shown in (d) may also include a sound recording status indicator. The sound recording status indicator is used to indicate whether sound has been recorded. Optionally, if the sound recording status indicator is "to be recorded", it means that the user has not recorded sound; if the sound recording status indicator is "recorded", it means that the user has already recorded sound. Optionally, when the user has already recorded sound, he or she can continue to add sound, which can increase the number of recorded sounds. It should be understood that the indicator indicating that sound has been recorded can be different from the indicator indicating that sound has not been recorded, and is not limited to the above example. Figure 2 The sound recording status indicator in (d) can indicate that no sound has been recorded. In the embodiment of the present application, the sound recording status indicator indicates whether the user has recorded sound, so the user can use the sound recording status indicator to quickly know whether the sound has been recorded, thereby improving the user experience. In another possible implementation, it is also not necessary to display the sound recording status indicator, which can reduce the content displayed on the interface and improve the efficiency of the interface display.

[0144] The sound repair application configuration control 205 is used to configure the sound repair function to be turned on or off. The sound repair application configuration control 205 can also be understood as a switch for applications supported by the sound repair function. When the sound repair application configuration control 205 is in the sound repair on state, the sound repair function in the corresponding application or scene is activated. Figure 2In the interface shown in (d), the application or scenario in which the sound repair function can be configured may include but is not limited to at least one of a call application, a face-to-face communication application, or a WeChat application. The WeChat application can be understood as a non-real-time remote communication application. Figure 2 (d) in may include the sound repair application configuration control 205 of each application, such as the sound repair application configuration control 205 of the call application, the sound repair application configuration control 205 of the face-to-face communication application, and the sound repair application configuration control 205 of the WeChat application. Optionally, when the sound repair application configuration control 205 of an application or scene is in the closed sound repair state, the application or scene cannot use the sound repair function; when the sound repair application configuration control 205 of an application or scene is in the open sound repair state, the application or scene can use the sound repair function. The embodiment of the present application configures each application or scene with a sound repair application configuration control 205, that is, the sound repair function of each application or scene can be turned on or off separately, so that the user can choose to turn on or off the application or scene of the sound repair function as needed, thereby improving the user experience. In another possible implementation, the sound repair function of multiple applications or scenes can also be turned on or off by a sound repair application configuration control 205, so that the sound repair function of multiple applications or scenes can be turned on or off with one key, thereby improving the efficiency of turning on or off the sound repair function of multiple applications or scenes. In another possible implementation, the applications or scenarios that support the sound repair function configuration and the corresponding sound repair application configuration control 205 may not be displayed. In other words, the sound repair function may be enabled for all applications or scenarios by default. In this way, the content displayed on the interface can be reduced and the efficiency of the interface display can be improved.

[0145] For example, Figure 2 The sound repair application configuration control 205 shown in (d) is in a sound repair closed state, that is, each application cannot use the sound repair function.

[0146] It should be noted that the sound recording and the supported application or scene configuration may be mutually constrained, and the constraint method may be that after one of the sound recording and the supported application or scene configuration is completed, the other of the sound recording and the supported application or scene configuration may be performed. Exemplary, it may be that the application or scene supported by the sound repair function can be configured after the sound recording is completed, that is, if the sound recording is not completed, any application or scene cannot enable the sound repair function through the sound repair application configuration control 205. In the embodiment of the present application, the configuration of the application or scene supported by the sound recording and the supported application or scene may be mutually constrained, and the user may be prompted to perform sound recording and enable the configuration of the application or scene of the sound repair function, thereby improving the user experience.

[0147] In the embodiment of the present application, the electronic device may detect a user operation on the sound recording control 204, and in response to the operation, the electronic device may display a Figure 2 In the embodiment of the present application, since the sound optimization function requires the user to register the sound, after the sound repair is turned on, the user can automatically jump to the sound registration interface, for example Figure 2 The interface shown in (e) in FIG. 1 may be an interface for voice registration. Figure 2 The interface shown in (e) in FIG. 1 may include a new voiceprint control 206. The new voiceprint control 206 may be used to trigger the collection of voiceprint features (referred to as voiceprints). Voiceprints may be used to indicate the timbre of the voice of the registered user. Optionally, Figure 2 The interface shown in (e) may also include a control for returning to the previous interface, for example, including returning to Figure 2 Controls of the interface shown in (d).

[0148] Then, the electronic device may detect a user operation on the new voiceprint control 206, and in response to the operation, the electronic device may display a Figure 2 The interface shown in (f) in the figure. Figure 2 The interface shown in (f) may include sound recording content 207, a recording control 208, and a save control. Among them, the sound recording content 207 can guide the user to the content that needs to be recorded, for example, the sound recording content 207 can guide the text content such as "Giant pandas do not have a fixed sleeping time, they sleep wherever they go". The recording control 208 can be used to trigger sound collection. Optionally, the sound collection can be triggered by long pressing the recording control 208, and the sound collected by the electronic device is the sound recorded by the user during the long pressing of the recording control 208; or the user clicks the recording control 208 to trigger the sound collection, and then the electronic device starts to collect sound, and the user clicks the recording control 208 again to stop the sound collection. The save control is used to include the collected sound. Optionally, the user can choose to record the sound in segments. When the electronic device detects the user operation acting on the save control, the segmented recorded sound is spliced ​​in the order of recording time to obtain a recording.

[0149] In the embodiment of the present application, the electronic device detects a user operation on the recording control 208, and in response to the operation, starts collecting the user's voice.

[0150] Then, if Figure 2 As shown in (g), the electronic device detects a user operation on the save control, saves the collected sound in response to the operation, and extracts the voiceprint of the collected sound. It should be understood that the process of extracting the voiceprint can be performed at any time after the sound is collected, and is not limited to extracting immediately after the sound is saved.

[0151] Optionally, the electronic device may compare the content of the recording with the sound recording content 207. If the similarity between the content of the recording and the sound recording content 207 is higher than the similarity threshold, the voiceprint of the recording is used as the voiceprint of the subsequent broadcast voice. If the similarity between the content of the recording and the sound recording content 207 is not higher than the similarity threshold, the user is prompted to re-record the voice according to the guidance of the sound recording content 207. Optionally, the electronic device in the embodiment of the present application supports recording 1 to n sentences of text, that is, when recording such as Figure 2 After recording the text content shown in (g), another text content may be displayed for the user to record, where n is an integer not less than 2. After recording, the user may save the text content through the save control. It should be understood that the user may also choose to save the text content after recording m sentences, where 1≤m<n, and m is an integer.

[0152] Then, if Figure 2 As shown in (h) in FIG. 1 , the electronic device detects a user operation on a control for returning to the previous level interface, and in response to the operation, a display such as Figure 2 The interface shown in (i) in . Figure 2 In the interface shown in (i), the sound recording status indicator is marked as "recorded". Figure 2 (i) and Figure 2 The similar part of (d) in can be referred to Figure 2 The description of (d) in the above description will not be repeated here.

[0153] In another possible implementation, Figure 2 The interface shown in (f) in the figure may not include the sound recording content 207, so that the content displayed on the interface can be reduced and the efficiency of the interface display can be improved. The embodiment of the present application can guide the user to record according to the sound recording content 207 when recording sound by displaying the sound recording content 207. This is conducive to the electronic device to judge the validity of the sound recording and improve the accuracy of voiceprint extraction. In addition, the collected sound content and the voiceprint can be extracted according to the content of the sound. Figure 2 The sound recording content 207 of the interface shown in (f) is compared, and then a suitable repair model is selected according to the similarity of the comparison to repair the subsequent user input voice. The similarity of the comparison can also be called intelligibility. The greater the similarity of the comparison, the more intelligible the content of the collected sound is. Figure 2 The more similar the sound recording content 207 of the interface shown in (f) is, the less the impairment of the user's speaking ability is. In other words, the smaller the similarity of the comparison, the more similar the content of the collected sound is to the user. Figure 2The less similar the sound recording content 207 of the interface shown in (f) is, the greater the damage to the user's speaking ability is. Generally speaking. The more complex the architecture of the repair model, the stronger the voice repair ability of the repair model, and the more accurate the voice repair, but the greater the computing power resources required by the electronic device. Therefore, when selecting a suitable repair model based on the similarity of the comparison, the smaller the similarity of the comparison, the more complex the architecture of the selected repair model, and the greater the similarity of the comparison, the simpler the architecture of the selected repair model. For example, when the similarity of the comparison is the first similarity, the first repair model is selected; when the similarity of the comparison is the second similarity, the second repair model is selected; wherein the first similarity is higher than the second similarity, and the model architecture complexity of the first repair model is lower than the complexity of the second repair model, then the corresponding ability of the first repair model is lower than the ability of the second repair model.

[0154] In another possible implementation, Figure 2 The interface shown in (f) may also not include the recording control 208, and then enter the Figure 2 In this way, the content displayed on the interface can be reduced and the efficiency of the interface display can be improved. In the embodiment of the present application, by displaying the recording control 208 and starting to collect sound after detecting that the recording control 208 is acted upon, the resource utilization rate of the electronic device can be reduced and the service life of the electronic device can be increased.

[0155] In another possible implementation, Figure 2 The interface shown in (f) may not include a save control, and the recorded sound is directly saved after the recording is completed. For example, if no recording is detected within a certain period of time, the recording is considered to be completed, or the length of the recording content is Figure 2 If the length of the content shown in (f) is consistent, the recording is considered complete. The embodiment of the present application allows users to flexibly record sound by setting a save control.

[0156] It should be understood that the interfaces of the embodiments of the present application can be increased or decreased as needed, and are not limited to Figure 2 The number of interfaces and the styles of interfaces shown are not limited here.

[0157] In an embodiment of the present application, by allowing the user to register a voice, the user's voiceprint can be extracted from the collected sound. In this way, when broadcasting, the repaired voice can be broadcast with the user's voiceprint. That is, the user can select a specific voiceprint to broadcast the repaired voice as needed, thereby improving the user experience.

[0158] In another possible implementation, Figure 2The sound recording process may not be required, so that the electronic device can use the built-in voiceprint to broadcast the repaired voice. Since the sound recording process is reduced, the ease of communication is also improved.

[0159] The following is an example of configuring an application that uses the sound repair function.

[0160] See also Figure 3 , Figure 3 A schematic diagram of an interface for an application configured to use a sound repair function provided in an embodiment of the present application.

[0161] like Figure 3 The interface shown in (a) includes a sound recording control 204 and a sound repair application configuration control 205. Figure 3 The interface shown in (a) can be referred to Figure 2 The description of the interface shown in (i) is not repeated here.

[0162] In the embodiment of the present application, the electronic device can detect a user operation acting on the sound repair application configuration control 205, for example, a user operation acting on the call application configuration control 2051 is detected, and in response to the operation, a display Figure 3 The interface shown in (b). Figure 3 In the interface shown in (b) of FIG. 1 , the call application configuration control 2051 in the sound repair application configuration control 205 is switched from the sound repair off state to the sound repair on state. For example, Figure 3 In the interface shown in (b), that is, the call application configuration control 2051 is in the sound repair on state, then the call application can use the sound repair function; and the face-to-face communication application configuration control 2052 and the WeChat application configuration control 2053 are both in the sound repair off state, then the face-to-face communication application and the WeChat application cannot use the sound repair function, but when using the face-to-face communication application or the WeChat application, in the interface of the face-to-face communication application or the WeChat application, the voice repair function can be turned on or off through the control. In the embodiment of the present application, optionally, the face-to-face communication application, the WeChat application, and the call application can control the sound repair function to be turned on or off through the control in the interface corresponding to each application. This embodiment does not explain how to control the sound repair function to be turned on or off through the control in the application interface, and further explanation will be given in the following embodiments.

[0163] It should be noted that the electronic device can detect the user operation acting on the sound repair application configuration control 205, and the sound repair application configuration control 205 can also switch from the sound repair on state to the sound repair off state.

[0164] In the embodiment of the present application, by configuring a sound repair control for each application, the turning on or off of the sound repair function of each application can be controlled individually, and the user can choose to turn on or off the application of the sound repair function as needed, thereby improving the user experience.

[0165] exist Figure 3 In the illustrated interface, the voice repair function of the call application has been enabled. Therefore, the following embodiment is described by example with respect to the call scenario.

[0166] Figure 4 A schematic diagram of an interface for configuring a whitelist for sound repair provided in an embodiment of the present application. In an embodiment of the present application, for people with special speech impairments, there may be some friends who are accustomed to understanding their pronunciation; for these friends, when the sound repair switch is turned on, Figure 4 As shown, users with special speech impairments can set up a whitelist in their mobile phone address book so that sound repair is not enabled by default when talking to these friends.

[0167] like Figure 4 The interface shown in (a) in FIG. 4 shows a page with application icons placed thereon, and the page may include multiple application icons (eg, a contact book application 401 ). Figure 4 The interface shown in (a) can be referred to Figure 2 The description of the interface shown in (a) is not repeated here.

[0168] In the embodiment of the present application, the electronic device can detect the user operation acting on the address book application 401, and in response to the operation, the electronic device can display the following Figure 4 The interface shown in (b) in the figure. Figure 4 The interface shown in (b) may be, for example, the main interface of the address book. Figure 4 The interface shown in (b) may include a contact configuration control 403, a contact search bar, or at least one item in a contact list. The contact configuration control 403 is used to configure the same operation for the selected contact, such as adding the selected contact to the sound repair whitelist, or enabling the call record function by default for the selected contact during a call. The contact search bar can be used to quickly search for contacts. The contact list includes one or more contacts. Optionally, the contacts in the contact list can be classified according to certain rules, such as by the first letter of the contact name, and Figure 4 The interface shown in (b) also includes controls for quickly locating categories. Figure 4 The interface shown in (b) may also include a dial control and a favorites control. Figure 4 The interface shown in (b) may also include a contact adding control.

[0169] Then, if Figure 4 As shown in (c) of FIG. 1 , the electronic device may detect a user operation on the interface, and in response to the operation, the electronic device may display a contact selection control 402. The contact selection control 402 may be used to select a contact to add the selected contact to the whitelist. In the embodiment of the present application, each contact has a corresponding contact selection control 402. Figure 4 The contact selection control 402 shown in (c) is in an unselected state.

[0170] In the embodiment of the present application, the electronic device may detect a user operation on the contact selection control 402, and in response to the operation, the electronic device may display the following information: Figure 4 The interface shown in (d). Figure 4 In the interface shown in (d) in FIG. 1 , the contact selection control 402 being acted upon is in a selected state, for example, the contact selection control 402 whose contact is “B2” is in a selected state.

[0171] Then, the electronic device may detect a user operation on the contact configuration control 403, and in response to the operation, the electronic device may display a Figure 4 The contact configuration control 403 of the embodiment of the present application can be used to configure the selected contact. Figure 4 The interface shown in (e) in FIG. 4 may also include a sound repair whitelist option 404 and a call recording option. The call recording option is used to set the contact whose contact selection control 402 is selected as the contact for saving the call recording. The sound repair whitelist option 404 is used to add the contact whose contact selection control 402 is selected to the sound repair whitelist, that is, to add the selected contact to the sound repair whitelist.

[0172] Then, if Figure 4 As shown in (f) in the figure, the electronic device can detect the user operation acting on the sound repair whitelist option 404, and the electronic device can add the contact whose contact selection control 402 is in the selected state to the sound repair whitelist. Optionally, the sound repair whitelist of the embodiment of the present application is a list where sound repair is not enabled, that is, the contacts in the sound repair whitelist do not enable the sound repair function when making a call; or the sound repair whitelist is a list where sound repair is enabled, that is, the contacts in the sound repair whitelist enable the sound repair function when making a call.

[0173] Through the configuration of the sound repair whitelist in the embodiment of the present application, the user can choose the contacts who use the sound repair function or not use the sound repair function in the call scenario according to the actual situation. In this way, the user can flexibly adjust the contacts who need to perform sound repair or not perform sound repair in the call scenario, thereby improving the user experience.

[0174] It should be noted that the configuration of the sound repair whitelist may not be performed, so that sound repair may be performed or not performed for all contacts.

[0175] After the configuration of the sound repair whitelist is completed, a call can be initiated with one of the contacts. For ease of understanding, the following embodiment is described by taking the case where the contacts in the sound repair whitelist are not subjected to sound repair as an example.

[0176] The embodiment of the present application sets a sound repair whitelist, so that the sound repair function can be used or not used for different contacts, thereby improving the flexibility of using the sound repair function.

[0177] See also Figure 5 , Figure 5 A schematic diagram of an interface of a remote real-time communication scenario provided in an embodiment of the present application. In an embodiment of the present application, the contacts in the voice repair whitelist are the contacts for which voice repair is not enabled by default.

[0178] like Figure 5 The interface shown in (a) shows a page with application icons placed on it, and the page may include multiple application icons (for example, phone application 501). Figure 5 The interface shown in (a) can be referred to Figure 2 The description of the interface shown in (a) is not repeated here.

[0179] In the embodiment of the present application, the electronic device can detect the user operation acting on the phone application 501, and in response to the operation, can display the following Figure 5 The interface shown in (b) in the figure. Figure 5 The interface shown in (b) shows the user's recent call records, which may include call contacts and corresponding call times. Figure 5 The interface shown in (b) in FIG. 1 may include multiple contacts. Then, if the electronic device can detect a user operation on one of the contacts, for example, a user operation on contact "B2", a call with the contact "B2" may be initiated in response to the operation. Figure 4 As shown in (d) in FIG. 1 , since the contact “B2” is added to the sound repair whitelist, that is, the contact “B2” is configured not to use the sound repair function, then Figure 5As shown in (c) of FIG. 1 , when initiating a call with the contact "B2", the call is made in the original voice by default, that is, the call is made in the voice before the repair. Figure 5 The interface shown in (c) also includes a sound switching control 502, an avatar of the contact in the call, a sound blocking control, a keyboard call control, a sound speaker control, and a call end control. Figure 5 The sound switching control 502 shown in (c) in the figure is in the original sound playing state, that is, the other end of the call plays the original sound collected by the local end. The sound switching control 502 can be used to control the switching between the original sound and the restored voice. Figure 5 As shown in (d), the electronic device can detect the user operation acting on the sound switching control 502. In response to the operation, the sound switching control 502 of the electronic device is in the repaired voice playback state, that is, the other end of the call plays the repaired sound, and the sound heard by the contact "B2" of the call is the repaired sound. In the embodiment of the present application, the repaired sound can be obtained by repairing the original sound. The original sound can also be called the original pronunciation, and the original pronunciation can be the sound emitted by the collected user.

[0180] Optionally, if the electronic device detects a user operation on the sound switching control 502 again, the state of the sound switching control 502 switches to the original sound playing state, and the corresponding user makes a call using the original sound. In the embodiment of the present application, by setting the sound switching control 502 in the call interface, the user can choose to use the original sound or the repaired sound through the sound switching control 502, which can improve the user experience.

[0181] It should be noted that if the electronic device initiates a call with a contact outside the sound repair whitelist, the sound repair function will be enabled by default on the call interface.

[0182] It should be noted that if the call is initiated by the other electronic device, the local electronic device can also display Figure 5 (c) or Figure 5 In the interface shown in (d), the local end can control whether to play the original sound collected by the other end or the sound after the original sound collected by the other end is repaired through the sound switching control 502.

[0183] In another possible implementation, if the contact of the call is a contact outside the voice repair whitelist, that is, the contact of the call is configured to turn on the voice repair function, the voice switching control 502 is in the repaired voice playback state when the call is connected.

[0184] In another possible implementation, the sound switching control 502 may not be needed, that is, the original sound or the repaired voice is used for the call according to the configuration of the sound repair whitelist.

[0185] In some example situations, when an electronic device needs a certain amount of time to repair a sound, there will be a certain delay from the time the local user speaks to the time the other user hears what the local user is saying. This delay is related to the time required for the electronic device to repair the sound. This will cause the other user to take a longer time to receive the repaired voice of the local user. The other user may think that the local user did not speak or that the signal was bad and the other user could not hear.

[0186] It should be noted that when making a call with a contact in the sound repair whitelist, the sound transmitted from the contact in the sound repair whitelist can be repaired and then broadcast. That is, the local device repairs the sound transmitted from the other end and then broadcasts it, for example, the sound of contact "B2" is repaired and then broadcast.

[0187] See also Figure 6 , Figure 6 A schematic diagram of a call scenario provided for an embodiment of the present application. In the embodiment of the present application, there are a local user and an opposite user, and the electronic device used by the local user is called a local device, and the electronic device used by the opposite user is called an opposite device. The local device may also be called a first electronic device, and the opposite device may also be called a second electronic device. The local user may also be called a first user, and the opposite user may also be called a second user. In the embodiment of the present application, the call delay will increase after the call audio passes through the sound repair model, and the call delay may affect the call effect of multiple parties. Therefore, the embodiment of the present application is described by improving the call effect of multiple parties in a call scenario.

[0188] like Figure 6As shown in (a), the voice input by the local user is "Hello, hello", then the local device needs to perform repair processing, and then send the repaired voice "Hello, hello" to the peer device, then the peer device receives the repaired voice "Hello, hello" and broadcasts it, then the peer user can hear the repaired voice, but it takes a long time from the voice input by the local user to the peer user hearing the repaired voice, the reason is that it takes a certain amount of time for the local device to perform voice repair, that is, processing delay. Then, the peer device can send the voice input by the peer user as "Hello, your takeaway, I'm downstairs" to the local device, and the local device will broadcast the voice input by the peer user. At this time, when the local user wants to reply, he can input the voice "Thank you, just leave it downstairs". At this time, the local device needs to process the voice, but the processing takes a certain amount of time, that is, the processing delay. At this time, the other user finds that the local user has not replied for a long time, and may think that the local user did not hear his voice, and will repeat the voice "Hello, your takeaway, I'm downstairs". In fact, the local user has heard it, but because the repair takes a certain amount of time, the other user may not hear the local user's feedback for a long time.

[0189] Therefore, in one possible implementation, during a call, the local device can send a prompt to the other device, and the other device broadcasts the prompt, thereby prompting the other user that the local device needs to process the voice input by the local user, so that the other user can know that the local device needs to repair the voice.

[0190] like Figure 6 As shown in (b), after the local device establishes a call channel with the opposite device, the local device can send a prompt to the opposite device. After receiving the prompt, the opposite device can broadcast the prompt, and the opposite user can hear the prompt. The prompt can be used to prompt that the local device needs to repair the voice input by the local user. Exemplarily, the prompt can include, for example, "The other party has enabled the sound optimization function, there is a delay in the call, please wait patiently for the prompt tone to end and then reply" to notify the other party that the call may be delayed.

[0191] It should be understood that the timing for the local device to send the prompt message to the other device may be immediately after establishing a call channel with the other device or after a certain interval, and there is no limitation here.

[0192] In the embodiment of the present application, by sending a prompt message from the local device to the opposite device, the experience of both parties in the call can be improved.

[0193] like Figure 6As shown in (b) in FIG. 1 , although the local device can send a prompt to the other device, during the communication process, there is still a situation where the user at the other end believes that the local user has not heard his voice.

[0194] In an embodiment of the present application, in a call scenario, such as a mobile phone call or a video call scenario, the voice changing, audio repair and other functions are turned on through the switch on the call interface. After turning on, the answering party will receive a corresponding prompt to remind the other party that the sound heard by the other party has been optimized and will cause a delay to improve the call experience between the two parties.

[0195] Therefore, in a possible implementation, after receiving each voice segment input by the local user, the local device sends a prompt tone to the opposite device, and then the opposite device can broadcast the prompt tone, which is used to prompt the local user that the voice has been input.

[0196] like Figure 6 As shown in (c), exemplarily, after the local device establishes a call channel with the opposite device, the local device can send a prompt to the opposite device. Then, after the local device detects the voice of "Hello, hello" input by the local user, the local device sends a prompt tone to the opposite device, and the opposite device broadcasts the prompt tone. Then, after the local device detects the voice of "Thank you, just put it downstairs" input by the local user, the local device sends a prompt tone to the opposite device, and the opposite device broadcasts the prompt tone. The opposite user can know that the local user has input the voice through the prompt tone, and then wait until the voice of "Thank you, just put it downstairs" is received, and then the voice of "OK, goodbye" can be replied. Exemplarily, the prompt tone can be, for example, a prompt tone such as "ding", "di" or "tick". In the call scenario, the AI ​​model converts the sound, which will cause a call delay. After the local end speaks, the "di" prompt tone is broadcast to the opposite end to indicate waiting, guiding to reduce the overlap of the two sides' speech. Optionally, during a call, when the sound repair result has not yet been given, a call prompt tone is used to guide the other party to wait patiently and respond.

[0197] In the embodiment of the present application, the timing for sending the prompt tone by the local end may be to send the prompt tone after starting to receive the voice input by the user, or to send the prompt tone after receiving a segment of voice, and there is no limitation here.

[0198] It should be noted that the prompt in this embodiment can also be used to remind the role of the prompt tone during the call. For example, the prompt can include "You will hear a prompt tone during the call, indicating that the user is inputting a voice at this time, please wait patiently" and the like.

[0199] In an embodiment of the present application, after receiving each voice segment input by the local user, the local device sends a prompt tone to the other device, which can improve the experience of both parties in the call when using the sound repair function and reduce the problem of inefficient communication caused by delays.

[0200] In a possible implementation, only one of the prompt words or the prompt tone may be sent, which is not limited here.

[0201] The above embodiment illustrates a remote real-time communication scenario. The following embodiment illustrates a remote non-real-time communication scenario.

[0202] See also Figure 7 , Figure 7 A schematic diagram of an interface for a remote non-real-time communication scenario provided in an embodiment of the present application.

[0203] like Figure 7 The interface shown in (a) in FIG. 1 shows a page with application icons, which may include multiple application icons (eg, a chat application 701). The chat application 701 of this embodiment may be a non-real-time remote chat application 701. Figure 7 The interface shown in (a) can be referred to Figure 2 The description of the interface shown in (a) is not repeated here.

[0204] In the embodiment of the present application, the electronic device can detect the user operation acting on the chat application 701, and in response to the operation, the electronic device can display the following Figure 7 The interface shown in (b) in FIG. Optional: Figure 7 The interface shown in (b) may include a search bar, a chat function control, a contact book function control, a send function control, and a personal center function control. The search bar may be used to quickly search for information, such as quickly searching for contacts. Figure 7 In the interface shown in (b) of FIG. 1 , the chat function control is selected. Figure 7 The interface shown in (b) in the figure can be understood as an interface corresponding to the chat function, which includes one or more contacts and contact information corresponding to each contact. Optionally, the contact information may include the contact name and chat content, and the chat content may be, for example, the latest chat content. Exemplarily, the name of one of the contacts may be, for example, "Xiao Ming", and the latest chat record with him may be, for example, "What did you do today?"

[0205] In the embodiment of the present application, the electronic device may detect a user operation on one of the contacts, and in response to the operation, the electronic device may display a Figure 7The interface shown in (c) in the figure may be a chat interface with the selected contact. The chat interface may include each chat content and the user corresponding to the chat content. Optionally, the chat content may include but is not limited to at least one of text, voice, video or file. Figure 7 The interface shown in (c) may include a chat content input control, and the chat content input control may include a voice input control 702. Optionally, the chat content input control may also include a text input control, a file / video transmission control, and a video shooting control. Among them, the voice input control 702 is used to trigger the recording of the voice. How the voice input control 702 triggers the recording of the voice can be referred to Figure 2 The description of the recording control triggering the recording of the sound in the embodiment is not repeated here.

[0206] In the embodiment of the present application, the electronic device can detect the user operation acting on the voice input control 702, and in response to the operation, the voice input by the user can be recorded. Figure 7 As shown in (d) in the figure, the electronic device can also display a prompt message that recording is in progress during the process of recording voice. Optionally, the prompt message in recording can include text prompt information such as "recording" and can also include text content converted from the recorded voice. It should be understood that by displaying the text content converted from the recorded voice during the process of recording voice, the user can know whether the electronic device has collected his voice and whether the text content converted by the electronic device matches the recorded voice, thereby improving the user experience.

[0207] Then, if Figure 7 As shown in (e) of FIG. 1 , after the voice recording is finished, the electronic device may display a prompt message indicating that the recording is completed. The prompt message indicating that the recording is completed may include text prompt messages such as “Recording completed”, and may also include text content converted from the recorded voice. Figure 7 The interface shown in (e) includes a direct send option 703 and a repair and send option 704. The direct send option 703 is used to trigger the sending of the collected user's original voice, that is, if the electronic device detects a user operation acting on the direct send option 703, the unrepaired voice is sent in the chat interface. The repair and send option 704 is used to trigger the sending of the repaired voice 705, that is, if the electronic device detects a user operation acting on the repair and send option 704, the repaired voice 705 is sent in the chat interface.

[0208] In the embodiment of the present application, the electronic device can detect the user operation acting on the repair and send option 704, and in response to the operation, the repaired voice 705 can be sent in the chat interface. Figure 7In the interface shown in (f) of FIG. 1 , the restored voice 705 is sent. The electronic device can detect the user operation acting on the restored voice 705 and play the restored voice 705.

[0209] Optionally, if the user chooses to send the original sound, the electronic device may also play the original sound after detecting the user operation acting on the original sound.

[0210] The embodiment of the present application allows the user to send the repaired voice 705 to the contact, so that the experience of non-real-time remote communication between the user and the contact can be improved.

[0211] Then, if Figure 7 In the interface shown in (g), a voice 706 is received from a contact.

[0212] In the embodiments of the present application, Figure 7 As shown in (h), the electronic device can detect a user operation on a contact's voice 706, and in response to the operation, can display a voice repair and text conversion option 707 and a voice to text conversion option 708. The voice repair and text conversion option 707 is used to trigger: displaying the text converted after the selected voice is repaired. The voice to text conversion option 708 is used to trigger: displaying the text converted from the selected voice.

[0213] It should be noted that, optionally, clicking on the voice can play the voice, and long pressing the voice can display the voice repaired text conversion option 707 and the voice conversion text option 708.

[0214] Then, if Figure 7 As shown in (i), the electronic device can detect the user operation on the voice repair and text conversion option 707, and in response to the operation, the selected voice can be sent in the chat interface to be converted into text after repair.

[0215] It should be noted that, upon detecting the user operation of the voice repair and text conversion option 707, the electronic device may repair the voice and then convert it into text for display; or, after receiving the voice, the electronic device may repair the voice to obtain the repaired voice, and then upon detecting the user operation of the voice repair and text conversion option 707, the electronic device may display the text converted from the selected voice after the repair.

[0216] In an embodiment of the present application, if the voice message received from the other party is the audio of a user with special speech impairment, then the local user can act on the user operation of the contact's voice 706 to display the voice repair and convert to text option 707 and the voice repair and convert to text option 708, and then act on the user operation of the voice repair and convert to text option 707 to obtain more accurate text.

[0217] In another possible implementation, after the electronic device detects a user operation acting on the voice 706 of a contact, it may also display a voice repair option, and the voice repair option is used to trigger the broadcast of the repaired voice 705. Then, after the electronic device detects a user operation acting on the voice repair option, it may broadcast the repaired voice 705, that is, broadcast the repaired voice 706 from the contact.

[0218] The embodiment of the present application can repair the contact's voice 706 after receiving the voice 706 from the contact, so that even if the other party is a speech-impaired user, normal communication can be achieved, thereby improving the experience of non-real-time remote communication between the user and the contact.

[0219] In general, in remote communication scenarios, the other party can hear the restored sound with the same timbre in real time with higher intelligibility, making the call smoother. In addition, in remote communication scenarios, the other party will receive a "prompt broadcast that the sound restoration function is turned on on this end", and the local end can also switch between the original sound source and the restored sound source at any time by pressing a button, so that the user on the other end can obtain the original sound information, which may include at least one of the original timbre, ambient sound or original pronunciation.

[0220] In another possible implementation, Figure 7 In (h), a play voice option and a play repaired voice option may also be displayed. If a user operation acting on the play voice option is detected, the voice from the contact is played; if an operation acting on the play repaired voice option is detected, the voice from the contact is repaired before being played. The embodiment of the present application can repair the voice from the contact by displaying the play voice option and the play repaired voice option, thereby improving the communication experience of both parties.

[0221] The above describes the remote communication scenario. The following is an example of the short-distance communication scenario. Different from real-time calls, face-to-face repair is used by people with special speech impairments to communicate with others face to face offline.

[0222] See also Figure 8 , Figure 8 A schematic diagram of an interface for a face-to-face communication scenario provided in an embodiment of the present application.

[0223] like Figure 8 The interface shown in (a) in FIG. 1 shows a page with application icons, which may include multiple application icons (eg, face-to-face repair application 801). The face-to-face repair application 801 of this embodiment may be used for voice repair in a face-to-face communication scenario. Figure 8 The interface shown in (a) can be referred to Figure 2 The description of the interface shown in (a) is not repeated here.

[0224] In the embodiment of the present application, the electronic device can detect the user operation acting on the face-to-face repair application 801, and in response to the operation, can display the following Figure 8 The interface shown in (b) in the figure. Figure 8 The interface shown in (b) may include a recording control 802, and the recording control 802 is used to trigger the collection of voice. Exemplarily, the recording sound control may be, for example, in the style of a recording ball. In an embodiment of the present application, the electronic device may detect a user operation acting on the recording control 802, and start recording voice in response to the operation. Optionally, the user operation of this embodiment may be, for example, a long press operation, and during the long press of the recording control 802, a swipe up to cancel prompt and a release to repair prompt may be displayed. The swipe up to cancel prompt is used to prompt that the voice recording can be canceled by swiping up. The release to repair prompt is used to prompt that the recording can be completed by releasing the recording control 802.

[0225] like Figure 8 As shown in (c) of FIG. 1 , after the electronic device detects the release operation on the recording control 802, a playback control 803 of a repaired voice is displayed, and the playback control 803 of the repaired voice is the voice obtained by repairing the recorded voice. Figure 8 The interface shown in (c) may also include a text display area 804, which is used to display the text content corresponding to the playback control 803 of the repaired voice. Figure 8 The text content displayed in the interface shown in (c) of FIG. 1 may be, for example, "Do you want to go this afternoon?" It should be noted that if the electronic device detects that the Figure 8 The user operation of the playback control 803 of the repaired voice in the interface shown in (c) can broadcast the repaired voice, for example, broadcast the voice corresponding to "Do you want to go this afternoon?". Optionally, after detecting the release operation on the recording control 802, the animation of the recording control 802 scrolls for a short period of time, and then displays the repaired text and automatically plays the repaired audio. If the user is not satisfied with the recording during the recording process, the user can swipe up in the long press state to cancel the repair.

[0226] However, since the sound repair may not be very accurate, in order to improve the effect of face-to-face communication, the embodiment of the present application also supports the rapid correction of the repair result. Exemplarily, if the repaired text contains a misidentified word, the user can click on the word to be modified, and a candidate word list 806 will pop up, from which the user can select the desired word for rapid replacement. The user can also edit the text directly through the input method. After completing the error correction (also called correction), the user can choose to play the corrected voice.

[0227] like Figure 8 As shown in (d) in FIG. 1 , the electronic device may detect a user operation on a portion of the words 805 in the text content, and in response to the operation, may display a Figure 8 The interface shown in (e) in the figure. Figure 8 The interface shown in (e) includes a candidate word list 806 corresponding to the part of words 805 selected by the user, and the candidate word list 806 includes one or more candidate words. Exemplarily, the part of words 805 may be, for example, "afternoon", and the corresponding candidate words may include "morning", "good at dancing" and "fog" and other candidate words.

[0228] Then, if Figure 8 As shown in (f) of FIG. 1 , the electronic device may detect a user operation on one of the candidate words, and in response to the operation, may display a Figure 8 The interface shown in (g) in Figure 8 In the interface shown in (g), the part of words 805 in the displayed text content is replaced with the candidate word affected by the user operation. For example, if the electronic device detects the user operation acting on the candidate word "morning", the "afternoon" in the displayed text content "Do you want to go this afternoon?" is replaced with "morning", and the corresponding displayed text content is replaced with "Do you want to go this morning?".

[0229] Then, if Figure 8 As shown in (h), the electronic device can detect the effect on Figure 8 In the interface shown in (h), the user operates the playback control 803 of the restored voice, and in response to the operation, the voice corresponding to "Do you want to go this morning?" can be broadcast. In this way, the other party communicating with the user can know the user's voice expression.

[0230] That is to say, in the embodiment of the present application, the voice can be broadcast according to the text content displayed in the text display area 804.

[0231] In this embodiment, during face-to-face communication, the user's voice can be recorded and repaired, so that the voice broadcast is also the playback control 803 of the repaired voice, which can improve the user's experience of communicating with others. In addition, after recording the voice input by the user, the text content corresponding to the recorded voice can also be displayed, so that the user can know whether the repair result of the voice is accurate through the text content. If the repair result is considered inaccurate, the text content can be adjusted, and then the content of the voice broadcast can be adjusted. In other words, the face-to-face communication scene can be quickly corrected for semantic errors, making the user operation more convenient and fast and improving the accuracy of voice communication. In addition, in the embodiment of the present application, voice can be input during face-to-face communication, and the repaired audio can be broadcast, without the need for pure text input, which is faster. And for the words with errors after repair, the operation of quickly replacing the words in the recognized text is supported, and the broadcast is re-broadcast, which improves the efficiency of face-to-face communication.

[0232] It should be noted that in Figure 8 In the interface shown in (c)-(e) in the figure, one possible implementation method may be: when performing ASR recognition processing on the voice, multiple ASR recognition results may be obtained, and each of the multiple ASR recognition results corresponds to a probability value, and the electronic device can display the ASR recognition result with the largest probability value among the multiple ASR recognition results. For example, the ASR recognition results obtained by the ASR recognition processing may include ASR recognition results such as "Shan Dancing Today", "Will You Go Today It's Foggy", "Will You Go This Morning", and "Will You Go This Afternoon", and the probability value of the recognition result "Will You Go This Afternoon" is the largest, then the ASR recognition result "Will You Go This Afternoon" is displayed, and the words of the remaining ASR recognition results except the displayed ASR recognition result in the multiple ASR recognition results are also saved. Then, the displayed ASR recognition results can be segmented to obtain one or more words. When a user operation acting on one of the words (that is, part of the words 805) is detected, words different from the part of the words 805 are searched from the remaining ASR recognition results as candidate words for display, for example, candidate words such as "morning", "Shan Dancing" and "Fog" are displayed. This embodiment saves the remaining ASR recognition results, and then searches for words different from the word selected by the user (ie, part of the word 805) from the remaining ASR recognition results to display as candidate words, which is helpful to improve the efficiency of obtaining candidate words.

[0233] In another possible implementation, the remaining ASR recognition results may not be saved, and the candidate words may be directly predicted based on the words selected by the user. In this way, since the remaining ASR recognition results do not need to be stored, the storage resources required for storing the ASR recognition results can be reduced.

[0234] It should be noted that in the embodiments of the present application, Figure 8 In the interface shown, there may be at most one playback control 803 of the repaired voice, for example, there is a playback control 803 of the latest repaired voice, so that the simplicity of the interface display can be improved. In another possible implementation, a new voice can be displayed for each recorded voice, and a new voice can be displayed for each modified text content, so that each voice of the user and the modification process of the user can be recorded, thereby improving the user experience.

[0235] In another possible implementation, Figure 8 In the interface shown in (c) of FIG. 1 , a voice or text display area 804 may also be displayed, so that the simplicity of the interface display can be improved. In the embodiment of the present application, by displaying the voice and text display area 804, the user can quickly know whether the voice repair result of the electronic device is accurate through the text content displayed in the text display area 804, thereby improving the user experience.

[0236] In another possible implementation, the user may select punctuation marks in the text content, so that candidate punctuation marks may be displayed, and the candidate punctuation marks selected by the user may be used to replace the punctuation marks selected by the user in the text content, and the voice announcement tone may be changed according to the updated punctuation marks. For example, if "?" is replaced with "!", the voice announcement tone changes from a questioning tone to an exclamatory tone.

[0237] It should be noted that speech-impaired users generally also have hearing impairments to a certain extent, that is, speech-impaired users may also be hearing-impaired users. Therefore, the following embodiments illustrate how to improve the communication experience of hearing-impaired or speech-impaired users.

[0238] See also Fig. 9 , Fig. 9 A schematic diagram of an interface for another face-to-face communication scenario provided in an embodiment of the present application.

[0239] like Fig. 9 The interface shown in (a) in FIG. 1 may be, for example, an interface for a face-to-face repair application. Fig. 9 The interface shown in (a) includes a recording control 802, which can be used to record voice. Optionally, the recording control 802 can be used to record the voice of a hearing-impaired user / speech-impaired user, or can be used to record the voice of a normal user. A normal user can be a user with normal hearing ability or a user with normal speaking ability. It should be noted that although Fig. 9 and Figure 8 The styles of the recording controls 802 are different, but their functions are the same, and they are both used to trigger voice recording.

[0240] In the embodiment of the present application, the electronic device can detect a click operation on the recording control 802, and in response to the operation, enter a continuous sound recording mode, at which time the other party's voice will be converted into text in real time. Fig. 9 In the interface shown in (a) of FIG. 1 , during the recording process, a prompt message for prompting that recording is in progress may also be displayed, such as a prompt "Listening now, can help you convert to text" or other text prompt content. Then, as Fig. 9 In the interface shown in (a), the electronic device can display the text obtained by converting the voice of a normal user, for example, the text "I am a hearing person and I am talking." Optionally, the content of the conversation can be displayed in a dialog box (chat box). The electronic device can cancel the continuous reception mode by detecting the click operation on the recording control 802 again. Optionally, the electronic device can enter the continuous reception mode when it detects a long press operation on the recording control 802, and then cancel the continuous reception mode after detecting the operation of leaving the recording control 802, that is, it is in the continuous reception mode during the process of long pressing the recording control 802. Optionally, in an embodiment of the present application, the voice of a normal user can be repaired and then converted into text for display, thereby improving the experience of face-to-face communication.

[0241] Then, if Fig. 9 As shown in (b) of FIG. 1 , the electronic device can detect a user operation acting on the interface, for example, a user operation acting on a blank area of ​​the interface, and in response to the operation, display a Fig. 9 The interface shown in (c) in the figure. Fig. 9 The interface shown in (a) may be, for example, an interface of a voice recording function. The interface of the voice recording function may refer to Figure 8 The description of the embodiments will not be repeated here. Fig. 9 The interface shown in (c) in FIG. 1 may be, for example, an interface for a text input function. Fig. 9 The interface shown in (a) in the figure switches to Fig. 9 The interface shown in (c) in FIG. 1 can be understood as switching from the voice recording function to the text input function. Optionally, the user operation can be, for example, a swipe up operation. Fig. 9 The interface shown in (c) includes a virtual keyboard 808 for inputting text, a speech generation control 809 for generating speech, and a text display box 810 for displaying text. Optionally, the recording control 802 can also be used as a control for generating speech, that is, after inputting text through the virtual keyboard 808, the speech is displayed in the dialog box through the user operation acting on the recording control 802.

[0242] In another possible implementation, Fig. 9The interface shown in (b) in FIG. 1 may also include a keyboard trigger control, which is used to trigger the display of the virtual keyboard 808. After the electronic device detects the user operation acting on the keyboard trigger control, it may display the following Fig. 9 The interface shown in (c) in .

[0243] In the embodiments of the present application, Fig. 9 As shown in (c) , the electronic device can detect a user operation on the virtual keyboard 808 , and in response to the operation, display the text input by the user in the text display box 810 .

[0244] Then, if Fig. 9 In the interface shown in (d) of FIG. 8 , the electronic device can detect a user operation on the speech generation control 809, and in response to the operation, generate speech according to the text displayed in the text display box 810. Fig. 9 The interface shown in (d) in the figure may include the voice of the text displayed in the text display box 810. For example, the text displayed in the text display box 810 is "Hello, I am Zhang San", and the electronic device generates the voice of "Hello, I am Zhang San" after detecting the user operation acting on the voice generation control 809. In addition, when generating the voice of "Hello, I am Zhang San", the tone of the voice may also be generated according to the punctuation marks or emoticons of the text.

[0245] For example, if the text includes ".", the voice segment "Hello, I am Zhang San" may be broadcasted in a plain tone; if the text includes "!", the voice segment "Hello, I am Zhang San" may be broadcasted in an exclamatory tone; if the text includes "?", the voice segment "Hello, I am Zhang San" may be broadcasted in a questioning tone. In addition, if the text includes a smiling face, the voice segment "Hello, I am Zhang San" may be broadcasted in a joyful tone; if the text includes a crying face, the voice segment "Hello, I am Zhang San" may be broadcasted in a sad tone.

[0246] Then, if Fig. 9 In the interface shown in (e), the electronic device can detect the user operation acting on the voice, and in response to the operation, it can broadcast the voice of "Hello, I am Zhang San".

[0247] In the embodiment of the present application, the user can continue to input text. Then, Fig. 9 In the interface shown in (f) in FIG. 8 , the electronic device can detect a user operation on the virtual keyboard 808 , and in response to the operation, display the text input by the user in the text display box 810 .

[0248] Then, if Fig. 9In the interface shown in (g), the electronic device can detect the user operation acting on the speech generation control 809, and in response to the operation, generate speech according to the text displayed in the text display box 810, for example, displaying Fig. 9 The interface shown in (h) is shown in Fig. 9 The interface shown in (h) may include a voice of the text displayed in the text display box 810. For example, the text displayed in the text display box 810 is "I want to ask, what time does the store close today", and the electronic device generates a voice of "I want to ask, what time does the store close today" after detecting a user operation acting on the voice generation control 809.

[0249] Then, if Fig. 9 In the interface shown in (i), the electronic device can detect a user operation on the interface, such as detecting a user operation on a blank area of ​​the interface, and in response to the operation, can switch to the interface of the voice recording function. Optionally, the user operation can be, for example, a sliding operation.

[0250] In an embodiment of the present application, the voice input by a normal user can be converted into text, and the hearing-impaired / speech-impaired user can input text, and then the electronic device displays the voice corresponding to the text. This is conducive to communication between the hearing-impaired / speech-impaired user and normal users.

[0251] In general, in face-to-face communication scenarios, it supports functions such as converting the other party's voice to text, converting the local user's text to speech, and repairing the local user's voice.

[0252] In another possible implementation, the face-to-face repair application may only include an interface with a text input function, but not a voice input function, which can simplify the functions of the face-to-face repair application, reduce the package size of the face-to-face repair application, and thus reduce the storage resources required by the electronic device. The embodiment of the present application configures the voice recording function and the text input function in the face-to-face repair application, so that the user can choose the communication method according to the needs, thereby improving the flexibility of communication and improving the user experience.

[0253] It should be noted that when the face-to-face repair application has a voice recording function and a text input function, the default interface for entering the face-to-face repair application can be the interface for the voice recording function or the interface for the text input function, and this is not limited here. Optionally, the default interface for the face-to-face repair application can be the interface for exiting the face-to-face repair application. For example, if the interface for the voice recording function is displayed when exiting the face-to-face repair application, the default interface for re-entering the face-to-face repair application is the interface for the voice recording function; if the interface for the text input function is displayed when exiting the face-to-face repair application, the default interface for re-entering the face-to-face repair application is the interface for the text input function.

[0254] It should be noted that the sound repair capability can be integrated into the system of electronic devices, so that the sound repair capability can be provided for some or all applications or scenarios involving voice input functions. That is, the voice can be repaired and then sent out for some or all applications or scenarios involving voice input functions, thereby comprehensively improving the user experience.

[0255] The above embodiment describes the interface of the communication scene. The following embodiment describes how to specifically implement voice processing.

[0256] See also Fig.10 , Fig.10 A flowchart of a speech processing method provided in an embodiment of the present application. Fig.10 The method shown can be performed by at least one of an electronic device or a cloud. Fig.10 The methods shown may include:

[0257] S1001. Obtain sound input.

[0258] In the embodiment of the present application, the received input may be a user's voice input. For example, the voice input method may refer to Figure 5 , Figure 7 , Figure 8 or Fig. 9 The method shown will not be described in detail here.

[0259] S1002: Input the acquired sound into the ASR model.

[0260] The ASR model is used to convert sound into text. In an embodiment of the present application, the ASR model may be a trained model. In this embodiment, the acquired sound is input into the ASR model, and the ASR model can convert the sound input into the ASR model into text output.

[0261] S1003: Obtain text output by the ASR model.

[0262] S1004: Correct the text.

[0263] In an embodiment of the present application, if the acquired sound is input by a speech-impaired user, the converted text may be biased, and therefore the text needs to be corrected. Optionally, the text correction method may be automatic correction or manual correction. Optionally, the automatic text correction method may include but is not limited to statistical-based text correction or context-aware text correction. Statistical-based text correction may be to use a large amount of text data to learn the frequency and context information of words, and predict the most likely correct text through a statistical model. Context-aware text correction may be to correct the text in combination with context information, and the context information may be, for example, the context before and after of a sentence. The manual correction method may be, for example, Figure 8 The method shown.

[0264] S1005. Obtain the result of the sound registration.

[0265] In the embodiment of the present application, the result of the sound registration may include the registered sound or the voiceprint feature extracted from the registered sound. Optionally, when the user uses the sound repair function, if it is detected that the user has not registered the sound, the user is redirected to the sound registration interface for registration; or the user has registered the sound before using the sound repair function. Exemplary methods for registering sound can refer to Figure 2 Related description.

[0266] S1006: Perform TTS broadcast based on the corrected text and sound registration results.

[0267] In the embodiment of the present application, after the corrected text is converted into speech, the speech is broadcast according to the voiceprint of the registered voice, so that not only the output speech is more intelligible than the acquired sound, but also the speech can be broadcast according to the voiceprint of the registered user, which can improve the user experience. Optionally, TTS can convert the text into audio of the user's tone through a neural network model and play it out.

[0268] It should be understood that the embodiments of the present application can obtain voiceprint features in advance by registering voiceprints, and the pre-registered voiceprints can be obtained when voice broadcasting. In this way, the time for extracting voiceprints after obtaining sound can be reduced, and then the time from obtaining sound to broadcasting voice can be reduced, thereby improving the efficiency of voice processing.

[0269] In a possible implementation, S1005 may not be necessary, and accordingly, S1006 may broadcast a voice based on the corrected text. In this case, the voiceprint of the broadcast voice may be a voiceprint built into the electronic device or a voiceprint of the acquired sound.

[0270] It should be noted that since the embodiment of the present application converts the acquired sound into text, the solution of the embodiment of the present application can be applied to scenarios where text needs to be displayed, such as Figure 7 , Figure 8 or Fig. 9 In addition, since it is also necessary to convert into text, the solution of the embodiment of the present application can be applied to the scene where the real-time requirement is not particularly high.

[0271] In another possible implementation, the acquired sound may be corrected and then converted into text through an ASR model.

[0272] The following is an example of the specific implementation of sound restoration.

[0273] See also Fig.11 , Fig.11 A flowchart of a sound registration method provided in an embodiment of the present application. Fig.11 The method shown can be performed by at least one of an electronic device or a cloud. Fig.11 The methods shown may include:

[0274] S1101, voiceprint registration.

[0275] In an embodiment of the present application, the electronic device may trigger the voiceprint registration process in response to the user's voiceprint registration operation. Exemplarily, the voiceprint registration operation may be, for example, Figure 2 The operation shown in (e) is not limited here.

[0276] S1102: Sound input.

[0277] In an embodiment of the present application, after the electronic device responds to the user's voiceprint registration operation, it can collect the user's voice input. Exemplary methods of receiving the user's voice input can be, for example, Figure 2 The method shown in (e) is not limited here.

[0278] S1103, ASR recognition rate classification and determination.

[0279] In an embodiment of the present application, after collecting the user's voice input, the electronic device can perform ASR recognition on the voice input to obtain an ASR recognition result. It should be understood that the ASR recognition result can understand the recognition text corresponding to the voice input by the user. Then, the ASR recognition result is compared with the pre-made voice recording content to obtain the ASR recognition rate. The ASR recognition rate can also be called the similarity of the comparison, which is used to indicate the degree of similarity between the ASR recognition result and the pre-made voice recording content. Exemplarily, the voice recording content can be, for example, Figure 2The text content shown in (f) is not limited here. After obtaining the ASR recognition rate, it can be determined which recognition rate the ASR recognition rate is in. Optionally, multiple recognition rate levels may be pre-divided, and there is no common recognition rate between the multiple recognition rate levels, such as (0, 70%] and (70%, 100%], etc. If the ASR recognition rate is 50%, it belongs to the (0, 70%] level, and if the ASR recognition rate is 80%, it belongs to the (70%, 100%] level.

[0280] The embodiment of the present application can improve the efficiency of repair model selection by performing ASR recognition rate classification and then selecting a repair model.

[0281] S1104, repair model selection.

[0282] Among them, the repair model is used to repair the input speech to obtain the repaired speech. In the embodiment of the present application, the repair model can be selected according to the grading judgment result of the ASR recognition rate. Optionally, a mapping relationship between each recognition rate gear and the repair model is pre-configured, and after the grading judgment result of the ASR recognition rate is obtained, the repair model corresponding to the grading judgment result of the ASR recognition rate can be obtained from the mapping relationship. Optionally, the speech repair capabilities of the repair models corresponding to different recognition rate gears are different. Optionally, the higher the recognition rate in the recognition rate gear, the weaker the speech repair capability of the corresponding repair model, that is, the lower the recognition rate in the recognition rate gear, the stronger the speech repair capability of the corresponding repair model.

[0283] For example, by calculating the recognition rate of the user's recorded audio, the intelligibility of the user's voice can be directly determined, so that at least one suitable repair model can be selected from multiple alternative repair models for use. For example, users with an ASR recognition rate lower than 70% use a repair model with stronger repairability, while users with an ASR recognition rate higher than 70% use a repair model with weaker repairability that tends to beautify the voice and correct the accent.

[0284] The embodiment of the present application selects a repair model according to the grading judgment result of the ASR recognition rate, that is, the repair model can be selected according to the degree of impairment of the user's speaking ability. In this way, a repair model with weaker repair ability can be selected for users with less impaired speaking ability. Accordingly, the computing power required for the repair model to perform speech repair is smaller; and a repair model with weaker repair ability can be selected for users with more impaired speaking ability. Accordingly, the effect of speech repair and the computing power resources required for speech repair are also better. In this way, both the effect of speech repair and the computing power resources required for speech repair can be taken into account.

[0285] S1105, voiceprint feature extraction module.

[0286] Among them, the voiceprint feature extraction module is used to extract voiceprint features (referred to as voiceprint). In an embodiment of the present application, the voiceprint feature extraction module is used to extract the voiceprint of the sound input by the user. Optionally, the voiceprint feature extraction module can convert the input audio of any length into a fixed-length voiceprint feature vector. The voiceprint feature extraction module can be an algorithm or a neural network model. Optionally, the voiceprint extraction module can select a voiceprint extraction model. The voiceprint extraction model of this embodiment can include but is not limited to an emphasized channel attention, propagation and aggregation in time delay neural network (ECAPA-TDNN) structure.

[0287] S1106. Fixed-length feature vector.

[0288] In the embodiment of the present application, the fixed-length feature vector is a vector obtained by extracting voiceprint features from the voice recorded by the user, and is used to indicate the voiceprint features of the user. The embodiment of the present application converts the extracted voiceprint features into a fixed-length voiceprint feature vector, which is conducive to reducing the storage resources required to store the feature vector.

[0289] S1107. Save.

[0290] In the embodiment of the present application, the feature vector can be saved so that when the voice is subsequently broadcast, it can be broadcast according to the voiceprint.

[0291] In another possible implementation, the length of the feature vector saved in S1106 may be related to the input sound length, and is not limited to a fixed-length feature vector. This allows adaptive saving according to the input sound length, thereby improving the accuracy of the saved voiceprint feature vector.

[0292] In another possible implementation, in S1103, after obtaining the ASR recognition rate, the gear position judgment may not be performed, and the repair model may be selected directly based on the ASR recognition rate.

[0293] In another possible implementation, S1103 and S1104 may not be necessary, and a pre-configured repair model may be used to repair the sound, which can reduce the time required for ASR recognition rate classification and repair model selection.

[0294] The architecture of one of the repair models is described below.

[0295] See also Fig.12 , Fig.12The schematic diagram of the architecture of a repair model provided in an embodiment of the present application. The repair model of this embodiment can be deployed in at least one of an electronic device or a cloud, such as Fig.12 The repair model shown may include a prosody encoder 1201 , a streaming ASR encoder 1202 , an audio discretization unit 1203 , a streaming audio repair large model 1204 , and a vocoder 1205 .

[0296] Wherein, the prosody encoder 1201 is used to extract prosody features (referred to as prosody). Exemplarily, the prosody encoder 1201 can extract prosody features from the input audio signal. Prosody features may include but are not limited to at least one of pitch, rhythm or intonation. Optionally, an audio signal of a normal user may be obtained, and then the prosody features may be extracted from the audio signal of the normal user as the prosody features when broadcasting audio.

[0297] The streaming ASR encoder 1202 is an encoder that can convert an audio signal into a text sequence in real time or near real time. In an embodiment of the present application, the streaming ASR encoder 1202 is used to convert the audio input by the user into text in real time or near real time. The streaming ASR encoder 1202 may include a neural network model.

[0298] The audio discretization unit 1203 is used to convert a continuous audio signal into a discrete symbol or unit sequence, for example, a continuous audio signal (such as a waveform signal) is decomposed into a series of discrete, identifiable units or symbols by a certain method. These units can be based on phonemes, features, or obtained by other methods (such as cluster analysis).

[0299] Among them, the streaming audio repair model 1204 is used to repair or optimize audio features. Audio features may include but are not limited to at least one of content features, rhythm features or voiceprint features. Optionally, the streaming audio repair model 1204 uses the audio conversion of the streaming language model (streaming LM-based vc) to repair or optimize, which combines the advantages of streaming processing (streaming) and language model (Language Model, LM) to achieve real-time and efficient audio conversion. Optionally, the streaming audio repair model 1204 may include but is not limited to the streaming rhythm repair model 1204.

[0300] Among them, the vocoder 1205 is a synthesizer that converts audio features into playable audio signals.

[0301] In an embodiment of the present application, the repair model may be a pre-trained model.

[0302] In a possible implementation, the repair model may perform the following processing when performing audio repair:

[0303] The prosody encoder 1201 can extract prosody features from the built-in normal user voice. The streaming ASR encoder 1202 performs ASR recognition processing on the input audio to be repaired to obtain an ASR recognition result (also called content feature or content). Then, the result of splicing the ASR recognition result, the prosody feature, and the voiceprint feature registered by the user is input into the streaming audio repair model 1204. In addition, the streaming audio repair model 1204 fuses the voiceprint features, prosody features, and ASR recognition results in the spliced ​​result with the initial audio discretization features (also called discretization features), and generates repaired discretization features and repaired continuous audio features through autoregressive reasoning. The discretization features generated by autoregressive reasoning will be used as the input of subsequent autoregressive reasoning, and will be fused and reasoned by the streaming audio repair model 1204, and the generated continuous audio features will be used as the input of the vocoder 1205. Then, the audio continuous features are converted into repaired audio by the voice coder 1205, and the repaired audio can be broadcast through an audio broadcasting device (such as a speaker). The voiceprint of the broadcasted audio can be the voiceprint registered by the user, and the rhythm of the broadcasted audio can be the rhythm extracted by the rhythm encoder 1201.

[0304] The audio to be repaired may be audio input by a user, which is collected by a sound collection device (such as a microphone) of an electronic device. Figure 5 , Figure 7 , Figure 8 or Fig. 9 The input speech in the scene shown. Optionally, the initial audio discretization feature can be obtained by the audio discretization unit 1203 based on at least one of the ASR recognition result or the audio to be repaired.

[0305] In an embodiment of the present application, since the ASR recognition result is obtained by performing ASR recognition processing on the audio to be repaired, and the audio discretization feature is also obtained by performing discretization processing on the audio to be repaired, the ASR recognition result is fused with the audio discretization feature, that is, the processing results of the audio to be repaired from different dimensions can be fused, thereby improving the accuracy of the obtained audio continuous features, and then improving the accuracy of the content of the obtained audio to be repaired.

[0306] Optionally, when the user inputs audio, the input audio can be repaired, which can improve the real-time performance of the audio repair process. Optionally, after a certain period of audio is acquired, the acquired audio can be repaired. For example, when the user inputs audio, if 1 second of audio has been acquired, the 1 second of audio can be repaired, and then the next 1 second of audio can be repaired, until a complete audio segment input by the user is repaired.

[0307] In another possible implementation, it is not necessary to generate the repaired discretized features, which can simplify the architecture of the repair model and improve the efficiency of sound repair. In the embodiment of the present application, the repaired discretized features are generated, and the repaired discretized features can be used as input for subsequent autoregressive reasoning, and the streaming audio repair model 1204 performs fusion and reasoning, thereby improving the accuracy of sound repair.

[0308] Optionally, the length of each audio segment to be repaired may be consistent, which can improve the accuracy of audio repair; or the lengths of different audio segments to be repaired may be inconsistent, which can improve the flexibility of audio repair.

[0309] Exemplarily, assuming that the audio input by the user is "Should I go this afternoon?", after obtaining "this afternoon", "this afternoon" is used as the audio to be repaired. Then, the streaming ASR encoder 1202 performs ASR recognition processing on the input audio segment "this afternoon" to obtain the ASR recognition result, and then splices the ASR recognition result, the prosodic feature, and the voiceprint feature registered by the user, and inputs the result into the streaming audio repair model 1204. In addition, the audio discretization unit 1203 discretizes the audio segment "this afternoon" based on the ASR recognition result to obtain the audio discretization feature, and then inputs the audio discretization feature into the streaming audio repair model 1204. The streaming audio repair model 1204 can output the repaired audio discretization feature and the repaired audio continuous feature corresponding to the audio segment "this afternoon", and the vocoder 1205 can output the repair result of the audio segment "this afternoon" based on the repaired audio continuous feature. In addition, the streaming audio repair model 1204 can use the repaired audio discretization features corresponding to the audio segment "this afternoon" as a reference for repairing the audio segment "whether to remove" or not, for example, as a reference for discretizing the audio segment "whether to remove".

[0310] The following is an exemplary description of the training of the repair model.

[0311] A training sample set is obtained, where the training sample set includes one or more training samples, and each training sample includes an audio sample, a text sample corresponding to the audio sample, and a restored audio sample corresponding to the audio sample.

[0312] During training, the audio sample is used as the input of the streaming ASR encoder 1202, and the streaming ASR encoder 1202 outputs the predicted text, and the predicted text is compared with the text sample corresponding to the audio sample to calculate the first training loss. If the first training loss meets the first training end condition, the streaming ASR encoder 1202 ends the training; if the first training loss does not meet the first training end condition, the parameters of the streaming ASR encoder 1202 are updated and the training continues until the first training loss meets the first training end condition. Optionally, the first training end condition may include that the first training loss is less than a first threshold.

[0313] Then, continue to train the repair model, use the audio sample as the input of the streaming ASR encoder 1202, or use the text sample corresponding to the audio sample as the input of the streaming audio repair model 1204, obtain the repaired audio output by the vocoder 1205, and then compare the repaired audio output by the vocoder 1205 with the repaired audio sample corresponding to the audio sample to calculate the second training loss. If the second training loss meets the second training end condition, the repair model ends training; if the second training loss does not meet the second training end condition, the parameters of the repair model are updated and the training continues until the second training loss meets the second training end condition. Optionally, the second training end condition may include that the second training loss is less than the second threshold. Optionally, at least one of the vocoder 1205 or the prosody encoder 1201 of the embodiment of the present application may be pre-trained, then updating the parameters of the repair model may be updating at least one of the streaming audio repair model 1204 or the audio discretization unit 1203.

[0314] Optionally, the audio sample may include a first audio sample and a second audio sample. Exemplarily, the audio sample may include "Today's weather is very sunny", then the first audio sample may include "Today's weather", and the second audio sample may include "very sunny". Correspondingly, the text sample corresponding to the audio sample includes the text of the first audio sample and the text of the second audio sample, and the restored audio sample corresponding to the audio sample includes the restored audio sample corresponding to the first audio sample and the restored audio sample corresponding to the second audio sample.

[0315] Then, during training, the repaired audio corresponding to the first audio sample output by the repair model is compared with the repaired audio sample to calculate the second training loss, and the repaired audio discretization features corresponding to the first audio sample output by the repair model are compared with the audio discretization features of the second audio sample to calculate the third training loss. If the third training loss meets the third training end condition, the repair model ends training; if the third training loss does not meet the third training end condition, the training continues after updating the parameters of the repair model until the third training loss meets the third training end condition. Optionally, the third training end condition may include that the third training loss is less than a third threshold.

[0316] It should be understood that the processing flow of the repair model in the training process of the embodiment of the present application can refer to the description of the processing flow of sound repair in the above embodiment, and will not be repeated here.

[0317] It should be noted that users can enter the personalized portal and record a large amount of corpus as training samples to train a personalized repair model, which can improve the user's personal experience.

[0318] In another possible implementation, the audio discretization unit 1203 may not need the ASR recognition result when discretizing the audio to be repaired. This can improve the efficiency of the discretization processing and thus improve the efficiency of audio repair. The embodiment of the present application uses the ASR recognition result to discretize the audio to be repaired by the audio discretization unit 1203, which can improve the accuracy of audio repair.

[0319] In another possible implementation, the repair model may include one of the audio discretization unit 1203 or the streaming ASR encoder 1202, which can improve the efficiency of the discretization process, and the audio repair through the audio discretization unit 1203 and the streaming ASR encoder 1202 can improve the accuracy of the audio repair. It should be noted that if the repair model includes the audio discretization unit 1203 but does not include the streaming ASR encoder 1202, the result of splicing the audio discretization features, the prosody features, and the voiceprint features registered by the user can be input into the streaming audio repair large model 1204.

[0320] In another possible implementation, when performing audio repair, the prosody encoder 1201 may not be needed. By pre-storing prosody features, the pre-stored prosody features can be used to perform audio repair, which can improve the efficiency of audio repair.

[0321] In another possible implementation, the streaming audio repair model 1204 may not need to predict the audio discretization features, which can improve the efficiency of audio repair, while the accuracy of audio repair can be improved by predicting the audio discretization features.

[0322] It should be noted that the prosody encoder 1201 and the streaming prosody repair model 1204 can both adopt a generative pre-trained transformers (GPT) structure, the speech discretization unit can use a vector quantized variational autoencoder (VQVAE) model, and the streaming ASR encoder 1202 can use a recurrent neural network transducer (RNNT). The vocoder 1205 may include an efficient and high-fidelity generative adversarial network (HiFi GAN). Among them, VQVAE can convert audio into frame-level discretization units (discretization features). HiFi GAN can convert audio features into audio sampling points (audio signals).

[0323] The following embodiments illustrate the process of the embodiments of the present application based on the above embodiments.

[0324] See also Fig.13 , Fig.13 A flowchart of another speech processing method provided by an embodiment of the present application. The method of the embodiment of the present application can be executed by an electronic device, a server, or a system including an electronic device and a server. The embodiment of the present application can repair the speech input by the user. Fig.13 The methods shown may include:

[0325] S1301: Receive a setting input, where the setting input is used to indicate a scenario for enabling voice repair, a contact for enabling voice repair, or an application for enabling voice repair, where the scenario includes a face-to-face communication scenario or a remote communication scenario.

[0326] Among them, the face-to-face communication scenario can be a face-to-face communication scenario, such as a scenario in which at least two users communicate face-to-face. The remote communication scenario can be a scenario in which communication is conducted by means of communication means. In an embodiment of the present application, at least one of a scene in which voice repair is enabled, a contact in which voice repair is enabled, or an application in which voice repair is enabled can be selectively enabled. Optionally, if a scenario in which an input is set to enable voice repair, then if it is detected that the electronic device is in a scenario in which voice repair is enabled, such as a face-to-face communication scenario or a remote communication scenario, the voice input by the user is repaired. Exemplarily, a face-to-face communication scenario can be, for example, Figure 8 or Fig. 9 The scenarios described in the embodiments are not described in detail here. The remote communication scenario can be, for example, Figure 5 or Figure 7 The scenario described in the embodiment is not limited here. If a contact for enabling voice repair is set to be input, then when the contact communicated with by the electronic device is the contact for enabling voice repair, the electronic device will repair the voice input by the user. Exemplary, the method of enabling voice repair for a contact may be, for example, Figure 4 The description of the embodiment is not repeated here. If the input is set to start the voice repair application, then when the application running on the electronic device is the application that starts the voice repair, the voice input by the user is repaired. Exemplary, the method of starting the voice repair application can be, for example, Figure 3 The embodiments are not described in detail here.

[0327] S1302: Acquire a first voice feature of the user through sound registration.

[0328] Among them, the first voice feature may be a voice feature of the user. In an embodiment of the present application, the first voice feature is used to represent the characteristics of the user's pronunciation. Optionally, the first voice feature may include at least one of a voiceprint feature or a prosodic feature. A voiceprint feature may be a sound wave spectrum carrying speech information displayed by an electroacoustic instrument. Prosodic features may refer to those features of speech that are not directly manifested as changes in sound quality, but are reflected through changes in factors such as pitch, length, and intensity. These features appear as suprasegmental components in the speech signal, and together with the sound quality components (such as vowels and consonants), they constitute a complete speech signal. The first voice feature may be obtained by extracting features from the voice registered by the user. Exemplarily, the way in which the user registers the voice can refer to Figure 2 The description of the embodiments is not repeated here. It should be noted that the users of the embodiments of the present application may include speech-impaired users or hearing-impaired users. The speech-impaired users may be, for example, users who have certain difficulties or defects in pronunciation. The hearing-impaired users may be, for example, users who have certain difficulties or defects in hearing.

[0329] S1303: Receive a first voice input by a user.

[0330] S1304: Repair the first voice according to the setting input and the first voice feature.

[0331] In an embodiment of the present application, the first voice is repaired according to the setting input and the first voice feature, and because the setting input is used to indicate the scene in which voice repair is turned on, the contact for which voice repair is turned on, or the application for which voice repair is turned on, when it is detected that the electronic device is in a scene in which voice repair is turned on, or when the contact communicated by the electronic device is the contact for which voice repair is turned on, or when the application being run by the electronic device is the application for which voice repair is turned on, the first voice is repaired according to the first voice feature, so that the repaired first voice can be broadcast with the first voice feature, and the intelligibility of the repaired first voice is higher than the intelligibility of the first voice before repair. Among them, intelligibility can indicate the accuracy of expressing what the user wants to express, and can also be understood as the listener's understanding of the voice signal transmitted by the speaker.

[0332] In an embodiment of the present application, by receiving a setting input, the setting input is used to indicate a scenario for turning on voice repair, a contact for turning on voice repair, or an application for turning on voice repair, and then the user's first voice feature is obtained through sound registration. After receiving the first voice input by the user, the first voice can be repaired according to the setting input and the first voice feature, so that speech-impaired users or hearing-impaired users can communicate by inputting voice, thereby improving the convenience of communication for speech-impaired users. In addition, the repaired first voice is generated through the first voice feature registered by the user, so that the repaired first voice can be broadcast according to the first voice feature, so that the repaired first voice can be broadcast according to the user's own voice feature, which is closer to the user's own timbre, etc., thereby further improving the user experience.

[0333] In another possible implementation, voice repair can also be performed using preset voice features. For example, before the user registers a voice, voice repair is first performed according to the preset voice features, so that the repaired voice is broadcast according to the preset voice features; and after the user registers a voice, the voice repair is performed according to the user's first voice features. That is, even if the user has not registered a voice, the voice repair function can be used, thereby improving the applicable scenarios of the voice repair function. In addition, the process of extracting voice features from the registered voice is reduced, thereby improving the efficiency of voice repair.

[0334] In a possible implementation, the remote communication scenario includes a call scenario. In the call scenario, the method further includes:

[0335] A first prompt message is sent to the first contact of the call, where the first prompt message is used to prompt that the voice repair function has been turned on.

[0336] and / or,

[0337] When the first voice is received, a second prompt message is sent to the first contact, where the second prompt message is used to prompt that the first voice is being repaired.

[0338] The first prompt information may include prompt text or prompt sound. Optionally, the prompt text in the first prompt information may be displayed on the display screen of the terminal corresponding to the first contact, and the prompt sound in the first prompt information may be broadcast through the speaker of the terminal corresponding to the first contact. Figure 6 The relevant description of the prompt in the embodiment is not repeated here. The second prompt information may include prompt text or prompt sound. Optionally, the prompt text in the second prompt information may be displayed on the display screen of the terminal corresponding to the first contact, and the prompt sound in the first prompt information may be broadcast through the speaker of the terminal corresponding to the first contact. Exemplarily, the prompt sound in the second prompt information may be, for example, Figure 6 The prompt sounds such as "ding" etc. illustrated in the embodiments are not described in detail here.

[0339] In the embodiment of the present application, by sending a first prompt message to the first contact of the call to prompt the user that the voice repair function has been turned on, the first contact can be informed that the user has turned on the voice repair function, thereby improving the call experience. In addition, by sending a second prompt message to the first contact when the first voice is received to prompt that the first voice is being repaired, the first contact can be informed that the delay is caused by the voice repair, thereby reducing the conflict of speaking time of two or more parties in the call, thereby improving the call experience.

[0340] In another possible implementation, at least one of the first prompt information or the second prompt information may not be sent, which can reduce the occupancy of transmission resources during a call.

[0341] In a possible implementation, the second prompt information includes a prompt tone, and sending the second prompt information to the first contact when the first voice is received includes:

[0342] When the first voice starts to be received, the prompt tone is continuously sent to the first contact.

[0343] The method further includes:

[0344] When the repaired first voice starts to be sent to the first contact, the prompt tone is stopped to be sent to the first contact.

[0345] For example, the embodiments of the present application can refer to Figure 6 The description of (c) in the above description will not be repeated here.

[0346] In an embodiment of the present application, when the first voice is initially received, a prompt tone is continuously sent to the first contact. Thus, when the first contact hears the prompt tone, he or she can know that the user on the other end is speaking. When the repaired first voice is initially sent to the first contact, the prompt tone is stopped and the first contact can subsequently hear the repaired first voice. This can reduce the situation where two or more parties speak in a call and thereby improve the experience of both parties on the call.

[0347] In another possible implementation, a short prompt tone may be sent to the first contact when the first voice is initially received, or after a certain period of time after the first voice is received, so as to reduce the transmission resources required for transmitting the prompt tone.

[0348] In a possible implementation, the remote communication scenario includes a call scenario. In the call scenario, the method further includes:

[0349] The first voice after repair is sent to the first contact of the call. Then, a first interface can be displayed, the first interface is an interface for making a call with the first contact, the first interface includes a first control, and the first control is used to control the closing of the voice repair function. Then, in response to the first operation on the first control, the voice repair function is closed to send a second voice to the first contact, the second voice includes the user's voice received after the voice repair function is closed.

[0350] Exemplarily, the first interface may be, for example, Figure 5 The first control may be, for example, a sound switching control 502. The first operation may be an operation of a user. In an embodiment of the present application, before the voice repair function is turned off, that is, when the voice repair function is turned on, the repaired voice is sent to the first contact, and after the voice repair function is turned off, the voice before the repair is sent to the first contact.

[0351] In the embodiment of the present application, during a call, the voice repair function can also be controlled to be turned off, so that the user can control the voice repair function to be turned off when the voice repair function is not needed, thereby improving the flexibility of switching between using or not using the voice repair function. For example, the user can control the voice repair function to be turned off through the first control when the electronic device is low on power or the electronic device is stuck, thereby improving the turning off of the voice repair function during a call and improving the user experience.

[0352] In another possible implementation, the closing of the voice repair function may not be displayed on the first interface, that is, the activation of the voice repair function is not supported during a call, thereby improving the display simplicity of the call interface.

[0353] In another possible implementation, the voice repair function may be turned off by controlling voice commands.

[0354] In a possible implementation, in response to the first operation, the first control is also switched from a first state to a second state, the first state is used to indicate that the voice repair function is turned on, and the second state is used to indicate that the voice repair function is turned off, and the method further includes:

[0355] In response to a second operation on the first control, the first control is switched from the second state to the first state, and a voice repair function is turned on to send a repaired third voice to the first contact, where the third voice includes the voice received from the user after the voice repair function is turned on.

[0356] The second operation may be an operation by a user. The first state may be, for example, Figure 5 The state of the sound switching control 502 shown in (d) in FIG. 1 , the second state may be, for example, Figure 5 The state of the sound switching control 502 shown in (c) in FIG.

[0357] In the embodiment of the present application, the state of the first control can be used to know whether the voice repair function is currently turned on, thereby improving the accuracy of the user's choice to use or not use the voice repair function. In addition, the embodiment of the present application can not only turn off the voice repair function during a call, but also turn on the voice repair function again during a call, so that the flexibility of turning on or off the voice repair can be improved. In addition, the voice repair function is turned on or off through the same control, thereby improving the simplicity of the interface during a call.

[0358] It should be understood that the WeChat application and the face-to-face communication application can also support turning off or on the voice repair function in the application interface. Please refer to the relevant description of how to turn off or on the voice repair function in the call interface, which will not be repeated here.

[0359] In another possible implementation, after the voice repair function is turned off during a call, re-enabling the voice repair function is no longer supported.

[0360] In another possible implementation, the voice repair function may be turned on or off by different controls. For example, one control may be used to turn on the voice repair function during a call, and another control may be used to turn off the voice repair function. This may improve the control independence of turning on or off the voice repair function.

[0361] In a possible implementation, the method further includes:

[0362] A second interface is displayed, the second interface including a playback control of the repaired first voice and a first text corresponding to the repaired first voice. A third operation is received, the third operation is used to select a first target character in the first text. In response to the third operation, at least one candidate character related to the first target character is displayed. A fourth operation is received, the fourth operation is used to select a second target character from at least one candidate character. In response to the fourth operation, a second text and a playback control of the voice corresponding to the second text are displayed, the second text being obtained by replacing the first target character in the first text with the second target character.

[0363] Exemplarily, the second interface may be, for example, Figure 8 In the interface shown, the playback control of the repaired first voice may be, for example, the playback control 803, and the first text corresponding to the first voice may be, for example, "Do you want to go this afternoon?". The third operation may be the user's operation. The first target character may be the character selected by the user. Optionally, the first target character may include, but is not limited to, text or punctuation marks or emoticons. Exemplarily, the first target character may be, for example, "afternoon." At least one candidate character may be displayed in a list. Exemplarily, at least one candidate character may be, for example, the characters displayed in the candidate word list 806, such as candidate characters such as "morning," "good at dancing," and "fog." The fourth operation may be the user's operation. The second target character may be the character selected by the user from at least one candidate character. Exemplarily, the second target character may be, for example, "morning." Exemplarily, the second text may be, for example, "Do you want to go this morning?". The playback control of the voice corresponding to the second text may also be, for example, the playback control 803.

[0364] In an embodiment of the present application, by displaying a second interface, the second interface includes a playback control of the repaired first voice and a first text corresponding to the repaired first voice. A third operation is received, and the third operation is used to select a first target character in the first text. In response to the third operation, at least one candidate character related to the first target character is displayed. A fourth operation is received, and the fourth operation is used to select a second target character from at least one candidate character. In response to the fourth operation, the second text and the playback control of the voice corresponding to the second text are displayed, so that when the result of the voice repair is inaccurate, the user can manually correct it quickly, thereby improving the accuracy of communication through voice.

[0365] In another possible implementation, when the user finds that the voice repair result is inaccurate, the user may re-input the voice.

[0366] In another possible implementation, the user may generate the second text and the playback controls corresponding to the second text with one click, and at least one character or word in the first text may be replaced to generate the second text and the playback controls corresponding to the second text.

[0367] In a possible implementation, the method further includes:

[0368] Disabled display of playback controls for the first voice after repair.

[0369] In the embodiment of the present application, by canceling the display of the playback control of the repaired first voice, there can be only one playback control for the latest voice at a time, which can improve the convenience of selecting the appropriate voice for broadcasting.

[0370] In another possible implementation, the display of the playback controls of the repaired first voice may not be cancelled, that is, not only the playback controls of the latest voice but also the playback controls of the historical voices are retained.

[0371] In a possible implementation, in a remote communication scenario, the method further includes:

[0372] The third interface is displayed, and the third interface is an interface for communicating with the second contact. The third interface includes a second control and a third control. The second control is used to indicate that the first voice is sent to the second contact, and the third control is used to indicate that the repaired first voice is sent to the second contact. Then, in response to an operation on the third control, the repaired first voice can be sent to the second contact. Alternatively, in response to an operation on the second control, the first voice can be sent to the second contact.

[0373] Exemplarily, the third interface may be, for example, Figure 7 (a)- Figure 7 The second control may be, for example, a direct send option 703, and the third control may be, for example, a repair and send option 704.

[0374] In an embodiment of the present application, the user can use the second control and the third control to selectively send the repaired voice or the voice before repair to the second contact. That is, even if the voice repair function is turned on, the user can still choose to send the original voice to the second contact, thereby improving the user's flexibility in remote communication.

[0375] In another possible implementation, there may be only one voice sending control, and the electronic device sends the repaired voice after detecting the action on the voice sending control. This can improve the efficiency of sending the repaired voice in a remote communication scenario.

[0376] In a possible implementation, in a face-to-face communication scenario, the method further includes:

[0377] A fourth interface is displayed, the fourth interface including a virtual keyboard. In response to an operation on the virtual keyboard, a third text is displayed. A fifth operation on a fourth control in the fourth interface is received, the fourth control is used to instruct to generate a voice. In response to the fifth operation, a voice corresponding to the third text is generated according to the first voice feature.

[0378] Exemplarily, the fourth interface may be, for example, Fig. 9 The third text may be text input by the user through the virtual keyboard, for example, the third text may be "Hello, I am Zhang San" or "I would like to ask what time the store closes today", etc. The fourth control may be, for example, control 809. The fifth operation may be an operation of the user. In an embodiment of the present application, the speech corresponding to the third text is generated according to the first speech feature, and the speech corresponding to the third text may be broadcast according to the first speech feature.

[0379] In an embodiment of the present application, by displaying a fourth interface, the fourth interface includes a virtual keyboard. In response to an operation on the virtual keyboard, a third text is displayed. A fifth operation on a fourth control in the fourth interface is received, and the fourth control is used to instruct the generation of speech. In response to the fifth operation, a speech corresponding to the third text is generated according to the first speech feature, that is, the user can also input text and then generate speech according to the registered first speech feature, that is, text-to-speech conversion is realized, and the user can communicate according to the choice of inputting voice or inputting text, thereby improving the selectivity and flexibility of the user in communicating.

[0380] In another possible implementation, after the user finishes entering the third text through the virtual keyboard, the voice corresponding to the input third text is automatically generated, which can reduce the user's operations. Optionally, if no operation acting on the virtual keyboard is detected within a certain period of time, it can be considered that the third text entry is completed.

[0381] In a possible implementation, the third text includes punctuation marks and / or emoticons. When generating speech corresponding to the third text according to the first speech feature, the tone of the speech corresponding to the third text is also controlled according to the punctuation marks and / or emoticons.

[0382] In an embodiment of the present application, the tone of the speech corresponding to the third text is generated according to the punctuation marks and / or emoticons, so that the tone of the text corresponding to the third speech matches the punctuation marks and / or emoticons. Exemplarily, if the punctuation mark of the last character of the third text is "!", the tone of the speech corresponding to the third text may be an exclamatory tone; if the punctuation mark of the last character of the third text is "?", the tone of the speech corresponding to the third text may be an interrogative tone. Exemplarily, if the emoticons in the third text include a smiling face, the tone of the speech corresponding to the third text is a joyful tone; if the emoticons in the third text include a crying face, the tone of the speech corresponding to the third text is a sad tone.

[0383] In an embodiment of the present application, the tone of the voice corresponding to the third text can be controlled by punctuation marks and / or emoticons in the third text input by the user, thereby enabling the tone of the output voice to be adaptively adjusted according to the user input text, thereby increasing the flexibility of voice broadcasting and improving user experience.

[0384] In another possible implementation, punctuation marks or emoticons in the third text may be ignored, which can reduce the computing resources required to determine the tone that matches the punctuation marks or the tone that matches the emoticons. Optionally, a mapping relationship between punctuation marks and tone, and a mapping relationship between emoticons and tone can be pre-set to achieve the determination of the tone that matches the punctuation marks or the tone that matches the emoticons.

[0385] In a possible implementation, obtaining the first voice feature of the user through voice registration includes:

[0386] A fifth interface is displayed, the fifth interface is an interface for voice registration, the fifth interface includes a third prompt information and a fifth control, the third prompt information is used to prompt the recording content of the voice registration, and the fifth control is used to indicate the recording voice. In response to the operation on the fifth control, a fourth voice is recorded. The first voice feature is extracted from the fourth voice.

[0387] Exemplarily, the fifth interface may be, for example, Figure 2 The interface shown. The third prompt information may be, for example, text prompt information such as "where the giant panda walks, there it sleeps", or voice prompt information, which is not limited here. The fifth control may be, for example, a control recording control 208. The fourth voice may be a voice recorded by the user, which is used to register the user's voice so as to extract the user's first voice feature, so that when the repaired voice is broadcast, it can be broadcast according to the user's first voice feature.

[0388] In a possible implementation, the method further includes:

[0389] Perform speech recognition on the fourth speech to obtain a fourth text. Then, compare the fourth text with the recorded content to obtain the similarity between the fourth text and the recorded content. Then, a target repair model can be selected from multiple repair models based on the similarity, and the target repair model is used to repair the first speech. The multiple repair models have different speech repair capabilities, and the speech repair capability of the target repair model is negatively correlated with the similarity.

[0390] In an embodiment of the present application, the text obtained by performing speech recognition on the voice registered by the user is compared with the recorded content, and then the similarity is obtained, and then the target repair model is selected from multiple repair models based on the similarity. In this way, a suitable repair model can be selected for repair according to the degree of the user's speech disorder, thereby balancing the accuracy of speech repair and the computing power resources required for speech repair.

[0391] It should be understood that multiple repair models can be deployed on one server, and the corresponding repair model can be obtained from the server through the model identifier for voice repair. In another possible implementation, they can also be deployed in multiple servers respectively, and then according to the corresponding relationship between the server and the repair model, the server corresponding to the target repair model can be called to perform voice repair. In addition, multiple repair models can also be deployed in electronic devices, which is not limited here.

[0392] In another possible implementation, only one restoration model may be configured. In this way, similarity matching with the recorded content may not be performed, thereby improving the efficiency of speech restoration.

[0393] The above embodiment describes an embodiment of repairing the user's voice, and the following embodiment describes an embodiment of repairing the voice of a contact who is in contact with the user.

[0394] See also Fig.14 , Fig.14 A flowchart of another voice processing method provided in an embodiment of the present application. The method of the embodiment of the present application can be executed by an electronic device, a server, or a system including an electronic device and a server. This embodiment can also be used to repair the voice of a contact with whom the user is communicating. Fig.14 The methods shown may include:

[0395] 1401. Receive a setting input, where the setting input is used to indicate a scenario for enabling voice repair, a contact for enabling voice repair, or an application for enabling voice repair, where the scenario includes a face-to-face communication scenario or a remote communication scenario.

[0396] Among them, S1401 can refer to the description of S1301 and will not be repeated here.

[0397] 1402. Receive a fifth voice from the target contact.

[0398] The target contact may be a contact with whom the user communicates, for example, the target contact may be the first contact or the second contact.

[0399] 1403. Obtain a second voice feature, where the second voice feature is a preset voice feature or a voice feature extracted from the fifth voice.

[0400] The second voice feature may include at least one of a voiceprint feature or a prosody feature. In another possible implementation, the second voice feature may be a voice feature extracted from a voice registered by a user. For example, if the user of the electronic device is a first user and the first user wants to hear the voice of a second user during voice repair, the second user may be asked to register a voice and then voice features may be extracted from the voice registered by the second user.

[0401] 1404. Repair the fifth voice according to the set input and the second voice feature.

[0402] In an embodiment of the present application, the fifth voice is repaired according to the setting input and the second voice feature, and because the setting input is used to indicate the scene in which voice repair is turned on, the contact for which voice repair is turned on, or the application for which voice repair is turned on, when it is detected that the electronic device is in the scene in which voice repair is turned on, when the contact communicated by the electronic device is the contact for which voice repair is turned on, or when the application being run by the electronic device is the application for which voice repair is turned on, the fifth voice is repaired according to the second voice feature, so that the intelligibility of the repaired fifth voice is higher than the intelligibility of the fifth voice before repair. Among them, intelligibility can indicate the degree of understanding of the voice when it is listened to, and can also be understood as the accuracy of the voice in expressing what the contact wants to express.

[0403] In a possible implementation, in a remote communication scenario, the method further includes:

[0404] The sixth interface is displayed, the sixth interface is an interface for communicating with the target contact, and the sixth interface includes the fifth voice. Then, in response to the operation on the fifth voice, the sixth control and the seventh control can be displayed, the sixth control is used to indicate that the fifth voice is converted into text, and the seventh control is used to indicate that the repaired fifth voice is converted into text. Then, in response to the operation on the seventh control, the text corresponding to the repaired fifth voice can be displayed. Alternatively, in response to the operation on the sixth control, the text corresponding to the fifth voice can be displayed.

[0405] Exemplarily, the sixth interface may be, for example, Figure 7In the interface shown in (g)-(i), the sixth control may be, for example, a speech-to-text option 708, and the seventh control may be, for example, a speech-to-text after repair option 707.

[0406] In an embodiment of the present application, after receiving a voice from a target contact, the fifth voice can be selectively converted into text, which can also be understood as converting the original voice of the target contact into text; the repaired fifth voice can also be converted into text, and the user can select the text to be displayed as needed, thereby improving the flexibility and experience of user communication.

[0407] In another possible implementation, the sixth control may be used to instruct to play the fifth voice, and the seventh control may be used to instruct to play the restored fifth voice. In the embodiment of the present application, the user can choose to play the original voice or the restored voice of the target contact, thereby improving the flexibility and experience of user communication.

[0408] In a possible implementation, in a face-to-face communication scenario, the method further includes:

[0409] The seventh interface is displayed, the seventh interface is an interface for communicating with the target contact, and the seventh interface includes an eighth control. Then, in response to the sixth operation on the eighth control, the fifth text is displayed. Then, in response to the seventh operation on the eighth control, the display of the fifth text is stopped, and the fifth text includes the text corresponding to the fifth voice within the target time period or the text corresponding to the repaired fifth voice, and the target time period includes the time period between responding to the sixth operation and responding to the seventh operation.

[0410] Exemplarily, the seventh interface may be, for example, Fig. 9 In the interface shown in (a)-(b) in FIG. 8 , the eighth control may be, for example, a recording control 802. The sixth operation may be an operation input by a user. The fifth text may be, for example, “I am a hearing person, and I am speaking”.

[0411] In the embodiment of the present application, the voice of the target contact may also be converted into text, thereby increasing the flexibility of the communication method between the user and the contact.

[0412] In another possible implementation, the eighth control may not be set to control the voice-to-text conversion of the contact, for example, all the voices of the contact may be converted into text, which can reduce the operations of converting the voice of the contact into text.

[0413] Based on the above embodiments, an exemplary description is given below on how to implement voice repair.

[0414] See also Fig.15 , Fig.15A flowchart of another speech processing method provided in an embodiment of the present application. The method in the embodiment of the present application can be executed by an electronic device, a server, or a system including an electronic device and a server. Fig.15 The methods shown may include:

[0415] S1501. Obtain target speech and speech features.

[0416] The target voice may be a voice that has not been restored, which may be understood as the original voice or the audio to be restored. For example, the target voice may include but is not limited to the first voice or the fifth voice. The voice feature may include but is not limited to the first voice feature or the second voice feature.

[0417] S1502. Input the target speech and speech features into a target restoration model, the target restoration model is used to extract the content of the target speech, obtain speech continuity features according to the content and speech features of the target speech, and synthesize the restored target speech according to the speech continuity features.

[0418] Optionally, the content of the target speech can be obtained through speech recognition. In the embodiment of the present application, the obtained target speech can be broadcasting the content of the extracted target speech according to the speech feature.

[0419] S1503: Obtain the repaired target speech output by the target repair model.

[0420] For example, this embodiment can refer to Fig.12 The relevant description is not repeated here.

[0421] In an embodiment of the present application, after inputting the target speech, the electronic device can obtain the target speech and speech features, and then input the target speech and speech features into a target repair model. The target repair model is used to extract the content of the target speech, obtain speech continuity features based on the content and speech features of the target speech, and synthesize the repaired target speech based on the speech continuity features. In this way, people with speech impairments can communicate by inputting speech, thereby improving the convenience of communication for people with speech impairments.

[0422] In one possible implementation, the target repair model includes a first module, a second module and a third module. The first module is used to extract the content of the target speech, the second module is used to obtain speech continuity features based on the content and speech features of the target speech, and the third module is used to synthesize the repaired target speech based on the speech continuity features.

[0423] Exemplarily, the first module may include, for example, a streaming ASR encoder, the second module may include, for example, a streaming audio repair model, and the third module may include, for example, a vocoder.

[0424] In a possible implementation, inputting the target speech and speech features into the target restoration model includes:

[0425] The speech features and the first part of the target speech are input into the target restoration model, the first module is used to extract the content of the first part of the speech, the second module is used to obtain the target speech continuous features according to the speech features and the content of the first part of the speech, and the third module is used to synthesize the restored first part of the speech according to the target speech continuous features. Then, the speech features and the second part of the target speech are input into the target restoration model, the first module is used to extract the content of the second part of the speech, the second module is used to obtain the second speech continuous features according to the speech features and the content of the second part of the speech, and the third module is used to synthesize the restored second part of the speech according to the second speech continuous features.

[0426] For example, the first part of speech and the second part of speech in the embodiment of the present application can refer to Figure 6 In the description of (c), for example, the first part of the speech includes "Thank you, I'll put it downstairs", and the second part of the speech may include "That's it".

[0427] The repaired target speech includes a repaired first part of speech and a repaired second part of speech.

[0428] In an embodiment of the present application, a part of the target speech is first repaired, and then another part of the target speech is repaired. In this way, speech repair can be started when a part of the speech is obtained. In other words, speech repair can be started without obtaining the complete target speech, thereby improving the efficiency of speech repair.

[0429] In another possible implementation, the voice repair may be started only after the complete target voice is acquired, which can reduce the situation where the target voice is acquired and the target voice is repaired at the same time, thereby reducing the resources required for voice repair.

[0430] In one possible implementation, the target repair model also includes a fourth module, which is used to discretize the first part of the speech to obtain discrete features of the target speech. The second module is also used to predict based on the discrete features of the target speech to obtain discrete features of the repaired target speech, and to obtain continuous features of the second speech based on the repaired discrete features of the target speech, speech features and the content of the second part of the speech.

[0431] Exemplarily, the fourth module may include, for example, an audio discretization unit.

[0432] In an embodiment of the present application, the first part of the speech is discretized by the fourth module to obtain discrete features of the target speech, and then the second module performs prediction based on the discrete features of the target speech to obtain the repaired discrete features of the target speech, and then the second speech continuous features are obtained based on the repaired discrete features of the target speech, the speech features and the content of the second part of the speech. In this way, the second speech continuous features can be obtained by combining the discrete features of the target speech, the speech features and the content of the second part of the speech, thereby improving the accuracy of the obtained second speech continuous features, thereby improving the accuracy of speech repair.

[0433] In another possible implementation, the fourth module may not be required, which can improve the efficiency of speech restoration.

[0434] In a possible implementation, the fourth module is used to discretize the first part of speech to obtain discrete features of the target speech, including:

[0435] The fourth module is used to discretize the first part of speech according to the content of the first part of speech to obtain discrete features of the target speech.

[0436] In an embodiment of the present application, discrete features of the target speech are obtained by discretizing the first part of speech according to its content. That is, the content of the first part of speech is used as a reference for the discretization processing, thereby improving the accuracy of the obtained discrete features of the target speech and further improving the accuracy of speech restoration.

[0437] In another possible implementation, only the fourth module may be used to discretize the first part of speech according to the content of the first part of speech to obtain discrete features of the target speech. This can improve the efficiency of obtaining discrete features of the target speech, thereby improving the efficiency of speech restoration.

[0438] It should be noted that the solution of the embodiment of the present application can not only be used in sound restoration tasks, but can also be extended to tasks such as converting dialects to Mandarin and cross-language translation.

[0439] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0440] It should also be understood that the steps of the above embodiments may be coupled to each other, and this application does not limit this. Moreover, the order of the sequence numbers of the above processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0441] The embodiment of the present application also proposes a speech processing device, which can perform the steps of the above method embodiment. For example, the speech processing device includes a receiving module and an output module, wherein the receiving module is used to receive a setting input, obtain a first speech feature of a user through sound registration, receive a first speech input by a user, etc.; the output module is used to repair the first speech according to the setting input and the first speech feature, etc.

[0442] In another possible implementation, the receiving module is used to receive a setting input, receive a fifth voice from a target contact, obtain a second voice feature, etc., and the output module is used to repair the fifth voice according to the setting input and the second voice feature, etc.

[0443] In another possible implementation, the receiving module is used to obtain the target speech and speech features, the output module is used to input the target speech and speech features into the target repair model, the target repair model is used to extract the content of the target speech, obtain speech continuity features based on the content and speech features of the target speech, and synthesize the repaired target speech based on the speech continuity features; the repaired target speech output by the target repair model is obtained.

[0444] The steps performed by the speech processing device can refer to the description of the method embodiment and will not be described in detail here.

[0445] It should be understood that the steps performed by the device of the embodiment of the present application can refer to the description of the above method embodiment and will not be repeated here.

[0446] It should be understood that the device here is embodied in the form of a functional module. The term "module" here may refer to an application specific integrated circuit (ASIC), an electronic circuit, a processor (such as a shared processor, a proprietary processor or a group processor, etc.) and a memory for executing one or more software or firmware programs, a merged logic circuit and / or other suitable components that support the described functions. In an optional example, those skilled in the art will understand that the device may be specifically the first electronic device, the second electronic device or the fourth electronic device in the above-mentioned embodiment, or the functions described in the above-mentioned embodiment may be integrated in the device, and the device may be used to execute the various processes and / or steps corresponding to the electronic device or the server in the above-mentioned method embodiment, and to avoid repetition, it will not be repeated here.

[0447] The above-mentioned device has the function of implementing the corresponding steps of the electronic device or server in the above-mentioned method; the above-mentioned function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-mentioned function.

[0448] In the embodiments of the present application, the device may also be a chip or a chip system, such as a system on chip (SoC).

[0449] The embodiment of the present application also provides a schematic block diagram of another speech processing device. The device includes a processor 1601, a transceiver 1602, and a memory 1603. The processor 1601, the transceiver 1602, and the memory 1603 communicate with each other through an internal connection path, the memory 1603 is used to store instructions, and the processor 1601 is used to execute the instructions stored in the memory 1603 to control the transceiver 1602 to send signals and / or receive signals.

[0450] It should be understood that the device can be specifically an electronic device or a server in the above embodiment, and can be used to execute the various steps and / or processes corresponding to the electronic device or the server in the above method embodiment. Optionally, the memory 1603 may include a read-only memory 1603 and a random access memory 1603, and provide instructions and data to the processor 1601. A part of the memory 1603 may also include a non-volatile random access memory 1603. For example, the memory 1603 may also store information about the device type. The processor 1601 may be used to execute the instructions stored in the memory 1603, and when the processor 1601 executes the instructions stored in the memory 1603, the processor 1601 is used to execute the various steps and / or processes of the above method embodiment. The transceiver 1602 may include a transmitter and a receiver, the transmitter may be used to implement the various steps and / or processes corresponding to the above transceiver 1602 for performing the sending action, and the receiver may be used to implement the various steps and / or processes corresponding to the above transceiver 1602 for performing the receiving action.

[0451] It should be understood that in the embodiment of the present application, the processor 1601 may be a central processing unit (CPU), and the processor 1601 may also be other general-purpose processors 1601, digital signal processors 1601 (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor 1601 may be a microprocessor 1601 or the processor 1601 may also be any conventional processor 1601, etc.

[0452] In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor 1601 or an instruction in the form of software. The steps of the method disclosed in conjunction with the embodiment of the present application can be directly embodied as being executed by the hardware processor 1601, or can be executed by a combination of hardware and software modules in the processor 1601. The software module can be located in a mature storage medium in the art such as a random access memory 1603, a flash memory, a read-only memory 1603, a programmable read-only memory 1603 or an electrically erasable programmable memory 1603, a register, etc. The storage medium is located in the memory 1603, and the processor 1601 executes the instructions in the memory 1603, and completes the steps of the above method in conjunction with its hardware. To avoid repetition, it is not described in detail here.

[0453] The embodiment of the present application further provides a speech processing system, including a terminal device and a server. The steps performed by the terminal device and the server can refer to the description of the above method embodiment, and will not be repeated here.

[0454] An embodiment of the present application also provides a computer-readable storage medium, which is used to store a computer program, and the computer program is used to implement the method shown in the above method embodiment.

[0455] The embodiment of the present application also provides a computer program product, which includes a computer program (also referred to as code or instruction). When the computer program runs on a computer, the computer can execute the method shown in the above method embodiment.

[0456] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0457] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0458] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0459] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0460] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0461] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., and other media that can store program codes.

Claims

1. A speech processing method, characterized in that: include: Receive a setting input, where the setting input is used to indicate a scenario for enabling voice repair, a contact for enabling voice repair, or an application for enabling voice repair, where the scenario includes a face-to-face communication scenario or a remote communication scenario; Acquiring a first voice feature of the user through voice registration; Receiving a first voice input by a user; The first speech is repaired according to the setting input and the first speech feature.

2. The method according to claim 1, characterized in that The remote communication scenario includes a call scenario. In the call scenario, the method further includes: Sending a first prompt message to the first contact of the call, where the first prompt message is used to prompt that the voice repair function has been turned on; and / or, When the first voice is received, a second prompt message is sent to the first contact, where the second prompt message is used to prompt that the first voice is being repaired.

3. The method according to claim 2, characterized in that The second prompt information includes a prompt tone, and sending the second prompt information to the first contact when the first voice is received includes: When the first voice starts to be received, continuously sending the prompt tone to the first contact; The method further comprises: When the repaired first voice starts to be sent to the first contact, the prompt tone is stopped to be sent to the first contact.

4. The method according to any one of claims 1 to 3, characterized in that The remote communication scenario includes a call scenario. In the call scenario, the method further includes: Sending the repaired first voice to the first contact of the call; Displaying a first interface, where the first interface is an interface for talking with the first contact, and the first interface includes a first control, where the first control is used to control the closing of the voice repair function; In response to a first operation on the first control, the voice repair function is turned off to send a second voice to the first contact, where the second voice includes the user's voice received after the voice repair function is turned off.

5. The method according to claim 4, characterized in that In response to the first operation, the first control is also switched from a first state to a second state, the first state is used to indicate that the voice repair function is turned on, and the second state is used to indicate that the voice repair function is turned off, and the method further includes: In response to a second operation on the first control, the first control is switched from the second state to the first state, and the voice repair function is turned on to send a repaired third voice to the first contact, wherein the third voice includes the voice received from the user after the voice repair function is turned on.

6. The method according to any one of claims 1 to 5, characterized in that The method further comprises: Displaying a second interface, wherein the second interface includes a playback control of the restored first voice and a first text corresponding to the restored first voice; receiving a third operation, wherein the third operation is used to select a first target character in the first text; In response to the third operation, displaying at least one candidate character related to the first target character; receiving a fourth operation, wherein the fourth operation is used to select a second target character from the at least one candidate character; In response to the fourth operation, a second text and a playback control of a voice corresponding to the second text are displayed, wherein the second text is obtained by replacing the first target characters in the first text with the second target characters.

7. The method according to claim 6, characterized in that The method further comprises: The display of the playback control of the repaired first voice is cancelled.

8. The method according to any one of claims 1 to 7, characterized in that In the remote communication scenario, the method further includes: Displaying a third interface, the third interface is an interface for communicating with a second contact, the third interface includes a second control and a third control, the second control is used to instruct sending the first voice to the second contact, and the third control is used to instruct sending the repaired first voice to the second contact; In response to the operation on the third control, sending the repaired first voice to the second contact; or, In response to an operation on the second control, the first voice is sent to the second contact.

9. The method according to any one of claims 1 to 8, characterized in that In the face-to-face communication scenario, the method further includes: Displaying a fourth interface, wherein the fourth interface includes a virtual keyboard; In response to an operation on the virtual keyboard, displaying a third text; Receiving a fifth operation on a fourth control in the fourth interface, the fourth control being used to instruct generation of speech; In response to the fifth operation, a speech corresponding to the third text is generated according to the first speech feature.

10. The method according to claim 9, characterized in that The third text includes punctuation marks and / or emoticons. When the speech corresponding to the third text is generated according to the first speech feature, the tone of the speech corresponding to the third text is also controlled according to the punctuation marks and / or emoticons.

11. The method according to any one of claims 1 to 10, characterized in that The obtaining of the first voice feature of the user through voice registration includes: Displaying a fifth interface, the fifth interface is an interface for voice registration, the fifth interface includes third prompt information and a fifth control, the third prompt information is used to prompt the recording content of the voice registration, and the fifth control is used to indicate the recording voice; In response to an operation on the fifth control, recording a fourth voice; The first speech feature is extracted from the fourth speech.

12. A speech processing method, characterized in that: include: Receive a setting input, where the setting input is used to indicate a scenario for enabling voice repair, a contact for enabling voice repair, or an application for enabling voice repair, where the scenario includes a face-to-face communication scenario or a remote communication scenario; Receiving a fifth voice from the target contact; Acquire a second voice feature, where the second voice feature is a preset voice feature or a voice feature extracted from the fifth voice; The fifth voice is repaired according to the setting input and the second voice feature.

13. The method according to claim 12, characterized in that In the remote communication scenario, the method further includes: Displaying a sixth interface, the sixth interface being an interface for communicating with the target contact, the sixth interface including the fifth voice; In response to the operation on the fifth voice, displaying a sixth control and a seventh control, the sixth control is used to instruct to convert the fifth voice into text, and the seventh control is used to instruct to convert the repaired fifth voice into text; In response to the operation on the seventh control, displaying the text corresponding to the restored fifth voice; or, In response to the operation on the sixth control, the text corresponding to the fifth voice is displayed.

14. The method according to claim 12 or 13, characterized in that In the face-to-face communication scenario, the method further includes: Displaying a seventh interface, wherein the seventh interface is an interface for communicating with the target contact, and the seventh interface includes an eighth control; In response to a sixth operation on the eighth control, displaying a fifth text; In response to the seventh operation on the eighth control, the display of the fifth text is stopped, and the fifth text includes the text corresponding to the fifth voice within the target time period or the text corresponding to the repaired fifth voice, and the target time period includes the time period between responding to the sixth operation and responding to the seventh operation.

15. A speech processing method, characterized in that: include: Acquire target speech and speech features; Inputting the target speech and the speech features into a target restoration model, the target restoration model is used to extract the content of the target speech, obtain speech continuity features according to the content of the target speech and the speech features, and synthesize the restored target speech according to the speech continuity features; The repaired target speech output by the target repair model is obtained.

16. A speech processing device, characterized in that: include: A processor, wherein the processor is coupled to a memory, the memory stores computer-executable instructions, and the processor executes the computer-executable instructions stored in the memory, so that the processor executes the method according to any one of claims 1 to 11, or executes the method according to any one of claims 12 to 14, or executes the method according to claim 15.

17. A speech processing system, characterized in that: The invention comprises an electronic device and a server, wherein the electronic device executes the method according to any one of claims 1 to 11, or executes the method according to any one of claims 12 to 14, and the server executes the method according to claim 15.

18. A computer-readable storage medium, characterized in that: Used to store a computer program, the computer program comprising instructions for implementing the method according to any one of claims 1 to 11, or comprising instructions for implementing the method according to any one of claims 12 to 14, or comprising instructions for implementing the method according to claim 15.

19. A computer program product, comprising computer program code, characterized in that: When the computer program code is executed on a computer, the computer is caused to execute the method according to any one of claims 1 to 11, or the method according to any one of claims 12 to 14, or the method according to claim 15.

Images