Display method and device and electronic equipment

Abstract

The invention discloses a display method and device and electronic equipment, and the method comprises the steps: displaying a voice interface, the voice interface displays a sub-interface, the sub-interface comprises a first region and a second region, and the first region displays a conversion control; receiving a first input to the conversion control; in response to the first input, a first text is displayed in the second area according to input parameters of the first input, the first text comprises at least one of an identification text and a translation text of an audio, the audio is acquired based on the voice interface, and the language types of the identification text and the translation text are different.

Description

Technical Field

[0001] The present application belongs to the field of speech processing technology, and specifically relates to a display method, device and electronic equipment. Background Art

[0002] With the rapid development of electronic technology, electronic devices have been widely used. Electronic devices can be installed with audio translation software, and accordingly, users can use the audio translation software installed in the electronic devices to perform voice translation.

[0003] At present, when users use certain applications such as conference software and multimedia playback software, they can usually use audio translation software for voice translation. In the related art, when using audio translation software for voice translation, users are usually required to manually input audio, and when the voice translation is completed, users are required to copy, paste, etc., so that users can view the recognized text and translated text of the audio in the interface of the conference software or multimedia playback software.

[0004] It can be seen that the current efficiency of obtaining audio recognition text and translation text is relatively low. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a display method, device and electronic device to solve the technical problem that the application scenarios of audio translation software for voice translation in the related art are limited.

[0006] In a first aspect, an embodiment of the present application provides a display method, the method comprising:

[0007] Displaying a voice interface, wherein the voice interface displays a sub-interface, wherein the sub-interface includes a first area and a second area, wherein the first area displays a conversion control;

[0008] receiving a first input to the conversion control;

[0009] In response to the first input, a first text is displayed in the second area according to input parameters of the first input, the first text including at least one of a recognized text and a translated text of the audio, the audio is acquired based on the voice interface, and the recognized text and the translated text are in different language types.

[0010] In a second aspect, an embodiment of the present application provides a display device, the device comprising:

[0011] A first display module, used for displaying a voice interface, wherein the voice interface displays a sub-interface, wherein the sub-interface includes a first area and a second area, wherein the first area displays a conversion control;

[0012] A receiving module, configured to receive a first input to the conversion control;

[0013] A second display module is used to display a first text in the second area in response to the first input and according to the input parameters of the first input, wherein the first text includes at least one of a recognized text and a translated text of the audio, the audio is obtained based on the voice interface, and the recognized text and the translated text are in different language types.

[0014] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the display method described in the first aspect are implemented.

[0015] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the display method described in the first aspect are implemented.

[0016] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the steps of the display method described in the first aspect.

[0017] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the steps of the display method as described in the first aspect.

[0018] In an embodiment of the present application, a voice interface is displayed; a first input of a conversion control displayed in a first area of ​​a sub-interface of the voice interface is received; in response to the first input, a first text is displayed in a second area of ​​the sub-interface according to an input parameter of the first input, and the first text includes at least one of a recognition text and a translation text of the audio acquired based on the voice interface. In this way, by displaying a voice interface, acquiring audio based on the voice interface, and inputting controls in the voice interface, the audio can be voice processed, for example, voice recognition or voice translation can be performed on the audio, and then the recognition text or translation text of the audio can be displayed in the sub-interface of the voice interface, so that when the voice interface and the sub-interface belong to different applications, the user can conveniently view the recognition text and translation text of the audio in the voice interface without the need for the user to perform operations such as copying and pasting, thereby improving the acquisition efficiency of the recognition text and translation text of the audio. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a flow chart of a display method provided by some embodiments of the present application;

[0020] Figure 2 It is one of the display diagrams of the voice interface;

[0021] Figure 3 This is the second display diagram of the voice interface;

[0022] Figure 4 This is the third display diagram of the voice interface;

[0023] Figure 5 This is a schematic diagram of processing silent segments in a speech data block;

[0024] Figure 6 It is a schematic diagram of separating the audios of different call locations;

[0025] Figure 7 is an architecture diagram for calling the first model to perform speech processing on audio;

[0026] Figure 8 This is a schematic diagram of the principle of audio clustering based on the speaker conversion label;

[0027] Fig. 9 is a schematic diagram of a process of feature clustering provided by some embodiments of the present application;

[0028] Fig.10 It is a schematic diagram of the structure of the input data of the third sub-model;

[0029] Fig.11 It is a schematic diagram of the structure of the output data of the third sub-model;

[0030] Fig.12 This is the fourth display diagram of the voice interface;

[0031] Fig.13 This is the fifth display diagram of the voice interface;

[0032] Fig.14 This is the sixth display diagram of the voice interface;

[0033] Fig.15 This is the seventh display diagram of the voice interface;

[0034] Fig.16 This is the eighth display diagram of the voice interface;

[0035] Fig.17 This is the ninth display diagram of the voice interface;

[0036] Fig.18 This is the tenth display diagram of the voice interface;

[0037] Fig.19 This is the eleventh display diagram of the voice interface;

[0038] Fig. 20 is a schematic diagram of the structure of a display device provided in some embodiments of the present application;

[0039] Fig.21 is a schematic diagram of the structure of an electronic device provided by some embodiments of the present application;

[0040] Fig. 22 A schematic diagram of the hardware structure of an electronic device provided for some embodiments of the present application. DETAILED DESCRIPTION

[0041] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0042] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0043] The display method provided in the embodiment of the present application is described in detail below through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0044] Figure 1 is a flow chart of the display method provided in the embodiment of the present application, such as Figure 1 As shown, the method comprises the following steps:

[0045] Step 101, displaying a voice interface, wherein the voice interface displays a sub-interface, wherein the sub-interface includes a first area and a second area, and the first area displays a conversion control.

[0046] The voice interface can be the interface of the target application, and the target application can be a first application or a second application, wherein the first application can be an application with a call mode, such as conference software, call software, etc., and the second application can be multimedia playback software, such as video playback software, audio playback software, etc.

[0047] The voice interface may be the interface of the target application in voice playback mode. For example, the voice interface may be the interface of the first application in call mode. For another example, the voice interface may be the interface of the second application in multimedia playback mode.

[0048] The sub-interface may be an interface of a voice translation software, and the user may use the voice translation software to perform voice translation.

[0049] The first area and the second area may be different display areas, and the first area may be used to display some setting controls. The first area may display a conversion control, and in some embodiments, the conversion control may be a control for setting translation language information, for example, the conversion control may be used to set the source language and the target language, and the translation direction of the speech translation software is to translate the audio from the source language into the text of the target language.

[0050] In some embodiments, the conversion control can also be a control for setting the display text of the second area. For example, the translated text of the audio can be displayed in the second area by default, and the conversion control can be used to set the recognition text and the translated text of the audio to be displayed simultaneously in the second area. Alternatively, when the recognition text and the translated text of the audio are displayed simultaneously in the second area, the conversion control can be used to set the translated text of the audio to be displayed in the second area.

[0051] In some embodiments, the sub-interface can be displayed in a window manner in the voice interface. Figure 2 As shown, the voice interface can be the interface of the conference software, and the sub-interface 201 is displayed in a floating window manner on the upper layer of the conference software interface 202. The sub-interface 201 displays a first area 2011 and a second area 2012, and a conversion control 2013 is displayed on the first area, and the conversion control 2013 is used to set the translation language information.

[0052] In some embodiments, the interface of the speech translation software and the interface of the target application program may also be displayed in a split-screen format, such as Figure 3 shown.

[0053] In some embodiments, the voice interface simultaneously displays the interface of the target application and the sub-interface, and step 101 specifically includes at least one of the following:

[0054] On the interface of the target application, the sub-interface is displayed in a floating window manner, and a display setting control is also displayed in the first area;

[0055] When a second input to the display setting control is received, the interface of the target application and the sub-interface are displayed in a split-screen manner.

[0056] The second input may be voice input, gesture input or touch input. Taking touch input as an example, the second input may be click input.

[0057] In some embodiments, the speech translation software can be started through the speech translation software start control in the sidebar of the target application interface while the target application interface is displayed. As a speech translation tool of the target application, the speech translation software can display a sub-interface in a floating window by default so as not to affect the interface display of the target application. Figure 2 shown.

[0058] In some embodiments, Figure 2 As shown, a display setting control 2014 is also displayed in the first area 2011 of the sub-interface 201. The user can click the display setting control 2014, and accordingly, the electronic device can display the interface and sub-interface of the target application in a split-screen manner. Figure 3 As shown, the sub-interface 201 and the interface 202 of the conference software are displayed in a split-screen manner.

[0059] In this way, the display of the sub-interface and the interface of the target application program can be made more flexible, thereby improving the user experience.

[0060] Step 102: Receive a first input to the conversion control.

[0061] The first input may be voice input, gesture input or touch input. Taking touch input as an example, the first input may be click input.

[0062] In some embodiments, the user can set the source language and the target language based on the conversion control, such as Figure 4 As shown, a conversion control 401 may be displayed in the first area. The conversion control 401 may display a language pair by clicking a drop-down menu. The language pair may include a source language and a target language for the user to click and select, thereby obtaining the translation language information selected by the user. Figure 2 and Figure 3 As shown, the language pair selected by the user in this scenario is "Chinese->English", that is, the set source language is Chinese and the target language is English.

[0063] In some embodiments, Figure 4 As shown, the language type pair may also include "Auto->Auto", which indicates automatic recognition of the translation language direction during voice translation.

[0064] Step 103, in response to the first input, display a first text in the second area according to the input parameters of the first input, the first text including at least one of a recognized text and a translated text of the audio, the audio is obtained based on the voice interface, and the recognized text and the translated text are in different language types.

[0065] In some embodiments, before step 103, the method further includes at least one of the following:

[0066] Audio is collected through a microphone, and the voice interface also displays a control for calling the microphone;

[0067] receiving audio through a first application, the voice interface also displaying an interface of the first application;

[0068] The audio is played through the second application, and the voice interface also displays the interface of the second application.

[0069] In some embodiments, the voice translation software may have a microphone function, and a control for calling the microphone may be displayed on the sub-interface, and the user may click the control. Accordingly, the electronic device may call the microphone function of the voice translation software and collect audio through the microphone of the electronic device.

[0070] In some embodiments, the audio may also be audio provided by the target application, for example, the audio may be call audio received by the first application, or for another example, the audio may be audio played by the second application.

[0071] In some embodiments, an audio link can be used to acquire and distinguish audio acquired based on a voice interface. The audio link can be divided into three, namely a first audio link, a second audio link and a third audio link. The first audio link is for audio collection through a microphone, the second audio link is for audio reception through a first application, and the third audio link is for audio playback through a second application.

[0072] In some embodiments, if the electronic device detects that the voice translation software has turned on the microphone function, the audio can be obtained through the first audio link.

[0073] In some embodiments, if the electronic device detects that the voice interface is the interface of the first application, it can receive audio through the second audio link. In this scenario, if the electronic device detects that the voice translation software also turns on the microphone function, it can also obtain audio through the first audio link. At this time, the audio obtained based on the voice interface also includes audio obtained through the first audio link and the second audio link respectively.

[0074] In some embodiments, if the electronic device detects that the voice interface is the interface of a second application, audio can be obtained through a third audio link.

[0075] In this way, the electronic device can set up multiple audio links to obtain audio, so that the voice translation software can perform voice translation on the audio obtained in multiple ways, thereby making the voice translation more flexible.

[0076] In some embodiments, the conversion control is used to set the source language and the target language, and the electronic device can respond to the first input and determine the translation language information according to the input parameters of the first input. After that, the electronic device can perform voice processing on the audio using the first model according to the translation language information to obtain data output by the first model, which may include the recognized text and the translated text of the audio. Accordingly, the electronic device can display the first text in the second area. Among them, the input parameters of the first input can indicate the language type pair selected by the user, and the electronic device can determine the translation language information based on the voice type pair selected by the user.

[0077] In some embodiments, the conversion control is used to set the display text of the second area, and the electronic device can display the first text in the second area in response to the first input according to the input parameters of the first input.

[0078] In some embodiments, the recognized text and translated text of the audio may be displayed in the form of bubbles or in other ways, which are not specifically limited here.

[0079] In this embodiment, a voice interface is displayed; a first input of a conversion control displayed in a first area of ​​a sub-interface of the voice interface is received; in response to the first input, a first text is displayed in a second area of ​​the sub-interface according to an input parameter of the first input, and the first text includes at least one of a recognition text and a translation text of the audio acquired based on the voice interface. In this way, by displaying a voice interface, acquiring audio based on the voice interface, and inputting controls in the voice interface, the audio can be voice processed, for example, voice recognition or voice translation can be performed on the audio, and then the recognition text or translation text of the audio can be displayed in the sub-interface of the voice interface. In this way, when the voice interface and the sub-interface belong to different applications, the user can conveniently view the recognition text and translation text of the audio in the voice interface without the need for the user to perform operations such as copying and pasting, thereby improving the acquisition efficiency of the recognition text and translation text of the audio.

[0080] In some embodiments, step 103 specifically includes:

[0081] Determining translation language information of the audio according to the input parameter of the first input, the translation language information indicating the language type of the recognized text and the language type of the translated text;

[0082] According to the translation language information, the audio is voice processed using a first model to obtain data output by the first model, the data comprising: the recognized text obtained by voice recognition of the audio based on the first model and the translated text obtained by voice translation of the audio based on the first model, the voice processing comprising the voice recognition and the voice translation;

[0083] The first text is displayed in the second area.

[0084] The input parameter of the first input may indicate the language pair selected by the user, and the electronic device may determine the translation language information based on the voice pair selected by the user. The translation language information includes a source language and a target language. The source language indicates the language type of the recognized text, and the target language indicates the language type of the translated text.

[0085] In some embodiments, a first model may be arranged on a terminal. When acquiring audio, the terminal may directly input the translation language information and the audio into the first model for speech processing, obtain data output by the first model, and display the first text in the second area.

[0086] In some embodiments, the first model can be arranged on the server, and the terminal can send the translated language information and audio to the server after obtaining the translated language information and audio. The server inputs the translated language information and audio into the first model for speech processing, obtains the data output by the first model, and sends it to the terminal. Accordingly, the terminal can display the first text in the second area. In this way, by arranging the first model in the server, with the help of the powerful processing power of the server, the operating efficiency of calling the first model is improved, thereby reducing the delay of speech translation and improving the accuracy of speech translation.

[0087] The speech processing may include speech recognition and speech translation, and accordingly, the first model may have a multi-task processing capability of simultaneously performing speech recognition and speech translation. Speech recognition refers to recognizing audio to obtain the recognized text of the audio, and speech translation refers to translating audio to obtain the translated text of the audio.

[0088] The first model may be a large model, and the first model has the capability of performing multiple tasks simultaneously.

[0089] In some embodiments, the audio may be acquired in the form of a speech data chunk, and the acquired audio may be the accumulation of at least one speech data chunk acquired sequentially in time sequence. The time length of a chunk may be 320ms. When a chunk is acquired, the silent segments in the chunk may be eliminated by voice activity detection (VAD) technology, and then the speech data chunks with the silent segments eliminated may be sequentially input into the first model for speech processing. Figure 5 As shown, Figure 5 The upper middle picture shows the chunk without silence elimination. Figure 5 The figure below shows a speech data block with silence segments removed from the chunk. Since the first model is an end-to-end translation, that is, the first model can directly perform speech translation based on audio, the theoretical delay of audio translation is usually only 320ms, which can greatly reduce the delay of speech translation.

[0090] In some embodiments, each voice data block in the audio can be sequentially input into the first model for voice processing, and the first model can directly output data. In some embodiments, each voice data block in the audio can also be sequentially input into the first model for voice processing, and the first model can output data only when it is determined that the audio meets the voice translation conditions, which can improve the accuracy of the voice translation of the audio and improve the user experience.

[0091] In some embodiments, the voice interface is an interface of a first application in a call mode, and the method of performing voice processing on the audio using a first model according to the translation language information to obtain data output by the first model includes:

[0092] Inputting the first information and the second information into the first model for voice processing respectively, obtaining the first data and the second data output by the first model, wherein the first information includes the first audio of the local call location and the translation language information in the call mode, and the second information includes the second audio of the other party call location in the call mode;

[0093] The data includes the first data and the second data, the first data includes the recognized text and the translated text of the first audio, and the second data includes the recognized text and the translated text of the second audio.

[0094] In some embodiments, the voice interface displays the interface of the first application in call mode, and when the audio includes the audio collected by the microphone and the audio received by the first application, the audio is mixed with audio from different call locations. In order to achieve audio separation, an audio link can be used to distinguish audio from different call locations. The method also includes:

[0095] Obtain the first audio of the local call location and the second audio of the other party call location;

[0096] The first audio is obtained by performing echo cancellation processing on audio collected by a microphone, and the second audio is audio received by the first application.

[0097] That is, if the first application is in call mode and the microphone function of the voice translation software is turned on, the microphone of the terminal will also record media sound and call audio. The audio collected by the microphone will be mixed with the audio of the other end's call location. Therefore, it is necessary to eliminate the media sound and call audio through echo cancellation technology to eliminate the audio of the other end's call location in the audio collected by the microphone, and the audio of the other end's call scene can be received through the second audio link, so that the first model can perform voice processing on the audio of different call locations respectively.

[0098] Figure 6 This is a schematic diagram of audio separation in different call locations. Taking the local call location as an example, the audio collected by the microphone is mixed with the audio of the other call location. Figure 6 As shown, the audio represented by black color represents the audio of the local call location, and the audio represented by red color represents the audio of the other end call location. The audio collected by the microphone includes both the audio of the local call location and the audio of the other end call location. The audio of the local call location can be obtained by echo cancellation, and the audio of the other end call location can be received through the second audio link, so that the audio of the local call location and the audio of the other end call location can be distinguished. Since the audio of different call locations are separated from each other, the voice translation services under different threads can be non-interfering with each other, so that when people in different call locations talk together, it will not affect the voice translation, which can improve the accuracy of voice translation.

[0099] In some embodiments, there may be multiple opposite-end call locations. In this scenario, the audio of different opposite-end call locations can be distinguished according to the audio source of the opposite-end call location. For example, multiple second audio links can be set according to the number of audio sources of the opposite-end call location, and each second audio link corresponds to an audio of the opposite-end call location. Accordingly, the audio of different opposite-end call locations can also be distinguished.

[0100] In some embodiments, multiple threads may be started to call the first model, each thread may correspond to a speech translation service of a call location, and the multiple threads may run in parallel so that the first model can perform speech translation on audios of different call locations. The first model may perform speech translation on audios of different call locations in the same manner.

[0101] In some embodiments, two threads can be started to call the first model respectively, and the first information and the second information can be respectively input into the first model for voice processing, wherein one thread can output the first data, which may include the recognition text and translation text of the first audio of the local call location in the call mode, and the other thread can output the second data, which may include the recognition text and translation text of the second audio of the other end call location in the call mode.

[0102] In some embodiments, the translation language information can be input into the first model in the form of input text, and the input text can be a prompt of the large model to give the first model some useful prompts when performing speech translation. The input text can include the translation language information, and the translation direction of the speech translation performed by the large model is from the source language to the target language.

[0103] In some embodiments, the server can receive a translation request from the terminal, and call the first model for speech processing in response to the translation request. The server can input each subsequently received speech data block into the first model for speech processing, and the first model can continuously output data. In this way, the cost of calling the first model can be reduced.

[0104] In some embodiments, when the electronic device obtains the first data and the second data, the data corresponding to the other party's call location, i.e., the second data, may be displayed aligned to the left, and the data corresponding to the local call location, i.e., the first data, may be displayed aligned to the right. Figure 2 and Figure 3 As shown, the local call location is venue 2, data 203 corresponding to venue 1 is data corresponding to the other end call location, which is aligned to the left, and data 204 corresponding to venue 2 is data corresponding to the local call location, which is aligned to the right. In addition, the recognized text 2031 of the audio can be displayed in the upper area of ​​the translated text 2032 of the audio.

[0105] In some embodiments, the first model can also be called to perform task processing of voice object separation, that is, the first model can also have the task processing function of voice object separation when it has the task processing functions of voice recognition and voice translation, and the task processing of voice recognition, voice translation and voice object separation can be performed in parallel, wherein voice object separation can identify and separate the speaking object of the audio to perform role separation for different speaking objects in the call mode. The displaying of the first text in the second area includes:

[0106] Displaying the first text in the second area according to the speaker information of the audio;

[0107] The data also includes the speaking object information, which is obtained by performing speech object separation on the audio based on the first model, and the speech processing also includes the speech object separation.

[0108] In some embodiments, the speaker information may include a speaker identification, a speaker gender, and the like.

[0109] In some embodiments, displaying the first text in the second area according to the speaker information of the audio may mean displaying the speaker information of the audio at a corresponding position to the display position of the recognized text and translated text of the audio, so as to mark the speaker information of the recognized text and translated text of the audio. In this way, the display of the speaker information of the audio can be used with the help of voice translation software to improve the call efficiency of the first application.

[0110] In some embodiments, audio in different call locations can be distinguished, and speech processing can be performed on the audio in different call locations through different threads. At the same time, for each thread, the first model can be called to separate the speech objects of the audio, so that the speaking objects in the same call location can be recognized and separated, which can improve the accuracy of speech translation.

[0111] In some embodiments, the speaker identifier may be displayed in the area above the bubbles of the recognized text and the translated text. At the same time, since different call locations are distinguished, the call location identifier may also be displayed in the area above the bubbles of the recognized text and the translated text. Figure 2 and Figure 3 As shown, venue 1 and venue 2 are call location identifiers, and speaker 1 and speaker 3 are speaker object identifiers.

[0112] In some embodiments, when displaying the translated text, if the speaker has not changed, text can be added directly after the last text for display. If the speaker has changed, a new speaker can be output, and the output translated text can be displayed in a new line in the second area, and the time when the speaker changed can be displayed.

[0113] In some embodiments, performing speech processing on the audio using a first model according to the translation language information to obtain data output by the first model includes:

[0114] Input each piece of third information into the first model in sequence for speech processing to obtain data output by the first model, wherein the third information includes the translation language information and an acquired speech data block;

[0115] The data is output by the first model when the output decision result is determined to be a first decision result, the first decision result indicates that the audio meets the speech translation condition, and the audio is composed of the speech data blocks acquired sequentially.

[0116] In some embodiments, the electronic device may call the first model to perform speech processing on the sequentially input speech data blocks. The first model may output data when it is determined that the audio composed of each speech data block meets the speech translation condition, which may improve the accuracy of the speech translation.

[0117] In some embodiments, the speech translation condition may indicate that the audio is a semantically complete sentence. In some embodiments, the speech translation condition may also indicate that a speech object of the audio has undergone a speech object conversion. For example, when it is determined based on the first model that the speech object of the speech data block input this time is different from the speech object of the speech data block input last time, it is determined based on the first model that the audio composed of the speech data blocks meets the speech translation condition.

[0118] In some embodiments, the step of sequentially inputting each piece of third information into the first model for speech processing to obtain data output by the first model includes:

[0119] Inputting each of the speech data blocks into the first sub-model in sequence to extract speech features, thereby obtaining speech features of the audio;

[0120] Inputting the speech feature into the second sub-model for feature mapping to obtain the first text feature of the audio;

[0121] concatenating the first text feature and the second text feature to obtain a third text feature, wherein the second text feature is a text feature obtained by text encoding the input text;

[0122] Inputting the third text feature into a third sub-model for natural language processing to obtain the data, wherein the natural language processing includes the speech recognition and the speech translation, and the data includes the recognized text and the translated text of the audio output by the third sub-model;

[0123] Among them, the first model includes the first sub-model, the second sub-model and the third sub-model, and the input text includes the translation language information and the recognition text of the audio obtained by performing the natural language processing by the third sub-model.

[0124] Figure 7 This is an architecture diagram for calling the first model to perform speech processing on audio. The following process of the first model performing speech processing will be explained around this diagram.

[0125] In some embodiments, the first sub-model may be Figure 7 The speech encoder in the speech encoder may be a neural network model. The electronic device may input each speech data block into the speech encoder in sequence to extract speech features, and may obtain speech features of the audio, which may be frequency bands (Fbank) features. The speech encoder may obtain the context of the input speech data block through an audio sliding window to perform splicing of different speech features. The speech encoder may also perform operations such as downsampling to reduce the dimension of the speech features. The speech encoder may adopt an open source pre-trained model such as the Hubert model, which may be obtained through quantization and comparative learning of a large amount of unlabeled training data.

[0126] The second sub-model can be Figure 7 The speech coder in the example may be a neural network model. The speech coder may map speech features to a text space to obtain a first text feature of the audio. The speech coder may be obtained by a linear transformation layer. The speech coder may convert speech features into text features (tokens) that can be processed by a text decoder through feature mapping. The parameters of the model need to be learned through fine-tuning.

[0127] The third sub-model can be Figure 7 The text decoder in the code can be a trained large language model, or an open source or self-developed large language model. The large model can perform natural language processing based on input information. The input information can include a first text feature of the audio and a second text feature obtained by text encoding the input text. The first text feature and the second text feature can be concatenated to obtain a third text feature, and the third text feature can be input into the third sub-model for natural language processing to obtain data. In this way, the first model can realize speech processing of the audio.

[0128] Among them, the input text may include audio recognition text, translation language information, speaking object conversion information and task processing information, etc. The speaking object conversion information indicates whether the speaking object of the currently input voice data block has been converted from the speaking object of the previously input voice data block, and the task processing information includes the data required to execute the task and the type of task executed.

[0129] In some embodiments, the input text may include a processing start tag, a speaker switching tag, a source language tag, a target language tag, a speech recognition task tag, a speech translation task tag, etc.

[0130] When the first chunk is fed into the first model, the input text needs to include the processing start tag, using <sot>Indicates start of text, which means starting speech recognition and speech translation.

[0131] The source language label and the target language label are the translation language information. The source language label indicates the source language, and the target language label indicates the target language. For example, the source language label can be <en> / <cn>, and the target language label can be <en> / <cn>,in, <en>Indicates English, <cn>If you don't know the translation language information, the source language label and the target language label can be used <unk>Indicates, that is, unknow, which means that the translation direction is currently unknown.

[0132] Speech recognition task tags can include start tags and end tags. The start tag can be <asr> Indicates that the end tag can be used< / asr> Asr, which stands for automatic speech recognition, indicates that the task type to be performed is speech recognition. The recognition text is between the start tag and the end tag.

[0133] The speech translation task tag can also include a start tag and an end tag. The start tag is <ast> , the closing tag is< / ast> Ast, which stands for automatic speech translation, indicates that the task type to be performed is speech translation. The text to be translated is between the start tag and the end tag.

[0134] The speaker change tag is the speaker object change information, which can be <0> or <1> , <0> Indicates that the speaker of the current input voice data block has not been converted with the speaker of the previous voice data block. <1> Indicates that the speaker of the current input speech data block has changed from that of the previous speech data block. When the speech data block is first input, there is no speaker change, so the speaker change tag can be set to <0> .

[0135] In some embodiments, the input text may also include processing end tags <eot>, that is, end of text, means ending the recognition of the current sentence and entering the recognition of the next sentence, thereby realizing streaming speech processing.

[0136] In some embodiments, Byte-Pair Encoding (BPE) may be used to perform feature mapping on the input text to map the features to a text space recognizable by the third sub-model to obtain a second text feature.

[0137] In some embodiments, the user may set the translation direction to be automatically determined according to the language types of the two parties in the call. In the case where the language type of the recognized text and the language type of the translated text indicated by the translation language information are unknown, the language type of the recognized text is the language type of the first audio obtained by the third sub-model performing the natural language processing, and the language type of the translated text is the language type of the second audio obtained by the third sub-model performing the natural language processing;

[0138] The first audio is the audio of the local call location in the call mode of the first application, and the second audio is the audio of the other party call location in the call mode of the first application.

[0139] In some embodiments, by using different threads to call the first model to perform speech translation of audio in different call locations, automatic recognition of the translation language direction can be achieved, that is, the user does not need to manually set the source language and the target language, which can simplify user operations. Among them, the language type of the first audio can be obtained from the first sentence of speech in the first audio, and also represents the language type of the local call location. The language type of the second audio can be obtained from the first sentence of speech in the second audio, and also represents the language type of the opposite call location. In this way, the first models of different threads can determine the translation language direction according to the language types of different call locations, thereby performing speech translation on the audio, and the first model can support the recognition and translation of multiple languages.

[0140] In some embodiments, the third sub-model also has a task processing function of identifying a speech object conversion, and the third sub-model also outputs speech object conversion information, and the speech object conversion information is used to indicate whether the speech object of the speech data block has been converted. In this way, the third sub-model can combine acoustic and linguistic information at the same time to realize the detection of speech object conversion, which can improve the accuracy of speech object separation. After the speech data blocks are sequentially input into the first sub-model for speech feature extraction, the method further includes:

[0141] Inputting the speech features into a fourth sub-model for feature extraction to obtain feature vectors of each of the speech data blocks, wherein the first model also includes the fourth sub-model;

[0142] According to the speech object conversion information and the feature vectors of each speech data block, speech object separation is performed on each speech data block to obtain the speech object information of the audio.

[0143] The fourth sub-model may be a neural network model, which may be a speaking object feature extraction module for extracting features of the speaking object.

[0144] like Figure 7 As shown, the speech features output from the speech encoder are respectively input to the speech coder converter and the speaking object feature extraction module, so that the first model can process multiple tasks of speech recognition, speech translation and speaking object separation in parallel, which can reduce the delay of speech translation.

[0145] Afterwards, the speech object of each speech data block can be separated according to the speech object conversion information and the feature vector of each speech data block to obtain the speech object information of the audio. Thanks to the speech object conversion information, the clustering of each speech data block can be accurately realized, which can not only improve the accuracy of speech object separation, but also improve the accuracy of speech translation.

[0146] In some embodiments, performing speech object separation on each of the speech data blocks according to the speech object conversion information and the feature vector of each of the speech data blocks to obtain the speech object information of the audio includes:

[0147] In a case where the speaker conversion information is a first conversion label, feature clustering is performed on a feature vector of a speech data block of a first speaker to obtain a central feature vector, and the first conversion label indicates that the speaker of the speech data block has not been converted;

[0148] In the case where the speaking object conversion information is a second conversion label, the central feature vector is compared with the feature vector of the speech data block of the second speaking object to obtain the speaking object information of the second speaking object, and the second conversion label indicates that the speaking object of the speech data block has been converted.

[0149] In some embodiments, the speech object conversion information can be represented by a speech object conversion label, and the third sub-model can output the speech object conversion label. The speech object conversion label can be used to segment the input audio according to the speech object. The speech object conversion label includes a first conversion label and a second conversion label. The first conversion label can be represented by <0> Indicates that the second transition tag can be used <1> express.

[0150] like Figure 8 As shown, the feature vector obtained by each chunk is represented by X, X = (X1, X2, X3, X4...), and the speaker conversion label of each chunk is marked respectively. <0> or <1> In this way, the time point when the speaker starts speaking can be easily obtained, which is convenient for audio positioning and speech translation.

[0151] In some embodiments, the output features of the penultimate layer of a speech encoder such as Hubert can be input into a speech object feature extraction module. Because the last layer of Hubert usually contains text information, the information of the speech object is lost. In order to utilize the speech information, the output features of the last few layers of the speech encoder, such as the penultimate layer, can be utilized. The speech object feature extraction module can be a linear layer, such as a stack of 8 layers of linear networks, which can output a feature vector, such as a 1024-dimensional feature vector.

[0152] In some embodiments, when the speaker conversion information is a first conversion label, feature clustering can be performed on the feature vectors of the speech data block of the first speaker using a related or new clustering algorithm to obtain a central feature vector.

[0153] When the speaking object conversion information is the second conversion label, the central feature vector and the feature vector of the speech data block of the second speaking object can be compared. On the one hand, the second speaking object can be identified, and on the other hand, the second speaking object can be separated, thereby improving the accuracy of distinguishing the speaking objects.

[0154] For example, the clustering category can be S=(S1, S2, S3, S4...), which represents the speaker identifier; during training, the speaker identifier can be a digital identifier, such as 1 to 20, and the maximum number of speakers that can be supported is 20. The clustering process can also be stream-processed, and the features of the speaker can be continuously accumulated through the first conversion label to achieve accurate clustering.

[0155] In some embodiments, the clustering process is as follows Fig. 9 As shown, the following steps are included:

[0156] Step 901: The first feature X1 forms the first cluster category, denoted by S1. When the speaker conversion information of the adjacent feature is the first conversion label, the feature is clustered with the first feature X1 to obtain the center vector of the cluster category.

[0157] Step 902: Assume that K cluster categories are obtained by clustering, each cluster category represents a speaking object, and the center vector of the k-th speaking object is represented by ck;

[0158] Step 903: when the nth feature Xn is received and the speaker conversion information of the feature is the second conversion label, the similarity between Xn and all ck is calculated;

[0159] Step 904: If the similarity between Xn and all ck is less than the threshold t, then execute step 905; otherwise, execute step 906;

[0160] Step 905: Create a new cluster category, representing the K+1th cluster category, in which there is currently only one feature vector Xn;

[0161] Step 906: classify Xn into the cluster category with the highest similarity, and recalculate the center vector of the cluster category;

[0162] Afterwards, the speaking party identification may be output.

[0163] When the speaking object conversion information is the first conversion label, the speech features of each of the input speech data blocks can be spliced ​​to obtain the speech features of the audio. Thanks to the speaking object conversion information, the accuracy of feature clustering of the same speaking object can be significantly improved, and the accuracy of speaking object separation can be improved, thereby improving the accuracy of speech translation.

[0164] In some embodiments, after inputting each of the speech data blocks into the first sub-model in sequence to extract speech features and obtaining the speech features of the audio, the method further includes at least one of the following:

[0165] The speech feature of the audio and the second text feature are input into the fifth sub-model for output decision processing of the first model to obtain an output decision result, and the first model also includes the fifth sub-model; wherein the speech feature of the audio is obtained by splicing the speech features of each of the input speech data blocks when the speaking object conversion information output by the third sub-model is the first conversion label; the output decision processing includes: feature association of the speech feature of the audio and the second text feature to obtain a first probability, the first probability is used to indicate the probability of semantic completeness of the audio, and when the first probability is greater than or equal to a preset threshold, determining that the output decision result is the first decision result;

[0166] When the speaking object conversion information output by the third sub-model is the second conversion label, the output decision result is determined to be the first decision result.

[0167] In this embodiment, outputting the decision result may include two cases, namely outputting empty data and outputting data including the recognized text and the translated text of the audio. Wherein, when the output decision result is the first decision result, the first model may output data, in which case the output features of the third sub-model may be converted into data through BPE.

[0168] In some embodiments, the input text may include a task type of speech recognition and a task type of speech translation. When the output decision result is the second decision result, the recognized text obtained by the third sub-model performing natural language processing may be used as the input of the third sub-model again, that is, the input text may include the recognized text of the audio obtained by the third sub-model performing the natural language processing. The second decision result indicates that the audio does not meet the speech translation condition.

[0169] In this way, the first model can process the audio in a processing mode of information flow, thereby reducing the delay of speech translation. Among them, the processing mode of information flow involves two directions. On the one hand, it refers to that the audio is sequentially input into the first model in the form of speech data blocks in a time sequence, so that the features of the speech data blocks are accumulated in the first model. On the other hand, it refers to that when the output decision result is the second decision result, the speech data blocks are continuously input, and the recognition text obtained by the speech recognition of the third sub-model is used as the input of the third sub-model again to realize the accumulation of the recognition text, and when the output decision result is the first decision result, the data is output by the third sub-model to realize the speech processing of information flow. In this way, the translated text can be output in real time, and the accuracy of speech translation can also be improved.

[0170] In some embodiments, the input text and the output text are respectively as follows: Fig.10 and Fig.11 As shown, when the first chunk is entered, the input text is <sot> <0> <unk> <unk> <ask>< / ask> <ast>< / ast> , then the fifth sub-model determines that the audio does not meet the voice translation conditions, and inputs the information output by the third sub-model last time to the third sub-model. At this time, the input text can be the local call location: <sot> <0> <cn> <en> <ask> Let’s get started< / ask> <ast>< / ast> , the other end call location: <sot> <0> <en> <cn> <ask> We will talk about< / ask> <ast>< / ast> .

[0171] When the output decision result is the first decision result, the first model does not output data. For example, it does not output the location of the local call: <sot> <0> <cn> <en> <ask> Let’s get started< / ask> <ast>< / ast> , the other end call location: <sot> <0> <en> <cn> <ask> We will talk about< / ask> <ast>< / ast> Output, and only when the fifth sub-model determines that the audio meets the voice translation conditions, the first model outputs data, for example, outputs the following text, the local call location: <sot> <0> <cn> <en> <ask> Let's start today's meeting< / ask> <ast> Let'sstart the conference of today< / ast> , the other end call location: <sot> <0> <en> <cn> <ask> We will talk about the problems< / ask> <ast> We want to discuss the problems we encounter< / ast> .

[0172] In some embodiments, the speech feature of the audio and the second text feature can be input into the fifth sub-model for output decision processing of the first model. The fifth sub-model can be as follows: Figure 7 The translation decision module in the method may be a neural network model. The fifth sub-model may perform feature association on the speech feature and the second text feature of the audio to obtain a first probability, and the first probability is used to indicate the probability of semantic completeness of the audio. When the first probability is greater than or equal to a preset threshold, the output decision result is determined to be the first decision result, and when the first probability is less than the preset threshold, the output decision result is determined to be the second decision result. The preset threshold may be set according to actual conditions, such as being set to 0.5, and is not specifically limited here.

[0173] In this embodiment, the speech features output by the speech encoder are input into the translation decision module, and the output features of the text decoder are also input into the translation decision module. The translation decision module calculates the correlation value between the speech features output by the speech encoder and the text features output by the text decoder through a preset correlation function to obtain a first probability, which is used to judge the semantic integrity of the audio, and then to determine the output decision result. When the output decision result is determined to be the second decision result, the voice data block can continue to be received to achieve audio accumulation. The translation decision module not only ensures streaming speech translation, but also outputs data in real time. Moreover, by combining speech features and text features, the accuracy of judging the semantic integrity of the audio can be improved.

[0174] In some embodiments, the translation decision module may be trained using an MMA loss function to determine the alignment between speech and translated text in speech translation.

[0175] In some embodiments, when the speech object conversion information output by the third sub-model is the second conversion label, the output decision result can be determined to be the first decision result. When the speech object conversion information output by the third sub-model is the first conversion label, the output decision result can be determined by the fifth sub-model.

[0176] In some embodiments, the output text of the text decoder can be output in a specified order. The first feature is the speaker conversion label. If it is <0> , we can continue to extract and accumulate speech features. The more speech features of the speech data block are accumulated, the more accurate the detection of the speaker conversion label is. <1> , it can be determined that the audio meets the speech translation conditions and data can be output, which may include the source language label, such as <cn>Indicates Chinese; target language label, such as <en>Represents English; recognizes text, such as <asr> How is the weather today?< / asr> ; Translate text, such as <ast> How is the weather today?< / ast> .

[0177] After that, the recognized text and translated text of the audio are output in real time. If the processing end tag appears <eot>, the incrementally accumulated speech features and text features can be cleared.

[0178] If the translation decision module determines that the output decision result is the first decision result, the currently accumulated translation text is output, and the sound replication software can use the same timbre as the audio to generate the audio of the translation text, so that the electronic device has the same timbre as the source audio when broadcasting the audio of the translation text, so that different speakers can be distinguished, thereby improving the efficiency of voice communication between the two parties of the call with the help of the voice translation software. When generating the audio of the translation text, the duration of the input audio can be combined to adjust the broadcast speed, thereby reducing the delay.

[0179] In some embodiments, the first model is trained based on a target loss function, and the target loss function includes: a loss function for training a third sub-model in the first model, a loss function for training a fourth sub-model in the first model, and a loss function for training a fifth sub-model in the first model.

[0180] The first model needs to be trained before it can be used. During the training process, training data can be prepared in a certain format. The training data can consist of audio of multiple speakers and pseudo-generated data. In order to enhance the data and improve the model effect, the audio of a single speaker can be used to generate audio of multiple speakers by splicing. The data structure of the pseudo-generated data can be similar to the data structure of the input text, for example, <speaker conversion label><original language label><target language label> <asr> Recognize text< / asr> <ast> Translate text< / ast> , you can add information such as speaking object conversion tags and timestamps.

[0181] During the training process, it is necessary to construct a loss function. The superposition function of the loss function of the third sub-model, the loss function of the fourth sub-model, and the loss function of the fifth sub-model can be used as the loss function of the first model, that is, Totalloss=CE loss+MMAloss+AAM loss.

[0182] Among them, CE loss function: used to train the text decoder, which is the cross entropy loss function;

[0183] MMA loss function: used to train the translation decision module;

[0184] AAM loss function: used to train the speaking object feature extraction module.

[0185] In this embodiment, by superimposing three loss functions to train the first model, the first model has multi-task processing capabilities, so that the task processing of speech recognition, speech translation and speech object separation can be realized in parallel.

[0186] The voice interface simultaneously displays the interface of the target application and the sub-interface, and the display of the voice interface includes at least one of the following:

[0187] On the interface of the target application, the sub-interface is displayed in a floating window manner, and a display setting control is also displayed in the first area;

[0188] When a second input to the display setting control is received, the interface of the target application and the sub-interface are displayed in a split-screen manner.

[0189] In some embodiments, the sub-interface further includes a third area, and the third area further displays a function display control. After step 103, the method further includes:

[0190] upon receiving a third input to the function display control, displaying the function control in the third area;

[0191] In case a fourth input to the function control is received, a control operation associated with the first text is performed.

[0192] The third input may be voice input, gesture input or touch input, taking touch input as an example, the third input may be click input. The fourth input may also be voice input, gesture input or touch input, taking touch input as an example, the fourth input may be click input.

[0193] like Figure 2 As shown, a function display control 2016 is displayed in the third area 2015 of the sub-interface, and the user can click the function display control 2016. Accordingly, the electronic device can display at least one function control 217 in the third area 2015, such as Figure 3 shown.

[0194] The user can click on any function control displayed in the third area, and accordingly, the electronic device can perform a control operation associated with the first text. Figure 3 As shown, if the user clicks on the "different roles" function control, the electronic device can display the recognized text and translated text of the audio in the second area according to the speaker information of the audio.

[0195] In this way, the user can trigger the function control to realize the function of the voice translation software.

[0196] In some embodiments, the functional control includes at least one of the following:

[0197] A first control, wherein the control operation is to start or stop calling a microphone to obtain audio corresponding to the first text;

[0198] A second control, wherein the control operation is to take a screenshot of an interface of a target application, and the target application is associated with a program for displaying a sub-interface of the first text;

[0199] A third control, wherein the control operation is to save the first text;

[0200] A fourth control, wherein the control operation is to obtain and display the summary content of the first text;

[0201] A fifth control, wherein the control operation is to obtain and display the speaker information of the audio corresponding to the first text;

[0202] A sixth control, wherein the control operation is to edit the speaker information;

[0203] A seventh control, wherein the control operation is to obtain and broadcast the audio of the translated text, and the audio timbre of the translated text is the same as the audio timbre of the recognized text;

[0204] An eighth control, wherein the control operation is to set fourth information, wherein the fourth information is used to trigger display of reminder information, and the reminder information is used to remind the content associated with the fourth information in the first text.

[0205] In some embodiments, the first control may be Figure 3 The "Microphone Sound" control shown is usually turned off by default. Accordingly, the speech translation software usually only receives multimedia audio or call audio provided by other applications. If this control is turned on, the live sound can be collected through the microphone of the electronic device.

[0206] In some embodiments, the second control may be Figure 3 The "Screenshot" control shown, when the user triggers the control, the electronic device can quickly capture the interface of the target application;

[0207] In some embodiments, the third control may be Figure 3 As shown in the “Save” control, when the user triggers the control, the electronic device can save historical or currently obtained data, that is, it can save the audio recognition text, translation text, speaker information and voice time information, etc.

[0208] In some embodiments, the fourth control may be as follows: Figure 3 The "AI Summary" control shown supports intelligent summarization of all text on the upper screen. When the user triggers the control, the text content of the call content, i.e., the data output by the first model, can be summarized by artificial intelligence (AI) based on the natural language processing capability of the first model. Accordingly, the electronic device can display the summary content of the data in the sub-interface.

[0209] In some embodiments, the first model can perform intelligent summarization of data in two cases: one is to summarize the entire content of the data, and the other is to summarize the content of the data according to the speaker. Fig.12 As shown, the sub-interface displays the summary of the full text of the data, such as Fig.13 As shown, the sub-interface displays the summary content obtained by summarizing the data according to the speaking object.

[0210] In some embodiments, the fifth control may be as follows: Figure 3 As shown in the "different roles" control, when the user triggers the control, each sentence can be assigned a speaking object information, and accordingly, the electronic device can display the recognized text and the translated text according to the speaking object information.

[0211] Since the speech object separation is performed based on the first model, different speech objects can only be distinguished according to the speech object identifiers, and the actual names of the speech objects cannot be determined. Therefore, the speech translation software can add a speech object editing function to facilitate users to read the translation content.

[0212] In some embodiments, the sixth control may be as follows: Figure 3 The "Edit Role" control shown in the figure can support users to edit the speaker information because the first model can only output the speaker identifier, such as speaker 1, speaker 2, etc., and the specific speaker's name is unknown. At the same time, the control also has the function of modifying errors. If the first model identifies speaker 1 as speaker 2, the user can modify it through editing.

[0213] When the user triggers the sixth control, the electronic device may display a data display interface, such as Fig.14 As shown, the data display interface may include data and an editing control 1401 associated with the data. The user may trigger the editing control 1401 to edit the speaker identifier. When the user triggers the editing control 1401, the electronic device may display an editing window, such as Fig.15 When the editing is completed, the electronic device may display Fig.16 The data display interface shown is relative to Fig.14 In the display interface shown, the speaking object identifiers of speaker 1, speaker 2 and speaker 3 have all been changed.

[0214] In some embodiments, the seventh control and the eighth control can be respectively as follows: Figure 3 The "simultaneous interpretation" control and the "set reminder" control shown are hidden by default. However, if the electronic device starts a first application such as conference software or call software, these two function controls can be displayed in the third area.

[0215] When the user triggers the "simultaneous interpretation" control, the audio translation text can call the sound replication software to generate the audio of the translation text. In some embodiments, the sound replication software can use the same timbre as the source audio to generate the audio of the translation text, so that when the electronic device broadcasts the audio of the translation text, the timbre is the same as the source audio, so that different speakers can be distinguished, thereby improving the efficiency of voice communication between the two parties of the call with the help of the voice translation software.

[0216] It should be noted that after the simultaneous interpretation function of the voice translation software is turned on, if the distance between the two parties is close, such as face-to-face communication, if you want the voice translation software to perform low-latency translation, you need to wear noise-cancelling headphones so that the translated sound and the original sound can be played at the same time. For the user, only the translated sound can be heard. If the distance between the two parties is far, such as cross-border conference communication, only the translated sound can be transmitted, and the original sound is not transmitted to prevent the two sounds from mixing together.

[0217] During a call of the first application, such as a meeting, if the meeting time is relatively long and not much is related to the user, the user can set fourth information to trigger the display of reminder information, which is used to remind the content in the first text associated with the fourth information.

[0218] When the user triggers the eighth control, the electronic device may display a setting interface, such as Fig.17 As shown, the setting interface can be used to set the fourth information, and the fourth information can include at least one of a specific nickname, a keyword, and a speaker identifier.

[0219] When the fourth information is set, when a specific nickname, keyword, or speaker identifier in the fourth information appears in the first text, a reminder message may be displayed, such as Fig.18 As shown, the user can choose to view or ignore the reminder information, and the reminder method can be vibration or playing a reminder sound. If the user chooses to view, the information display interface can be displayed, such as Fig.19 shown.

[0220] In this embodiment, by setting these function controls for users to trigger and enabling the electronic device to perform corresponding functions, the convenience of users using the voice translation software can be improved, and the functions of the voice translation software can also be enriched.

[0221] The display method provided in the embodiment of the present application can be executed by a display device. In the embodiment of the present application, the display method is executed by a display device as an example to illustrate the display device provided in the embodiment of the present application.

[0222] See also Fig. 20 , Fig. 20 is a schematic diagram of the structure of a display device provided in an embodiment of the present application, the device comprising:

[0223] A first display module 2001 is used to display a voice interface, wherein the voice interface is displayed with a sub-interface, wherein the sub-interface includes a first area and a second area, wherein the first area is displayed with a conversion control;

[0224] A receiving module 2002, configured to receive a first input to the conversion control;

[0225] The second display module 2003 is used to respond to the first input and display a first text in the second area according to the input parameters of the first input, wherein the first text includes at least one of a recognized text and a translated text of the audio, the audio is obtained based on the voice interface, and the recognized text and the translated text are in different language types.

[0226] In some embodiments, the second display module 2003 includes:

[0227] A first determination submodule, configured to determine translation language information of the audio according to an input parameter of the first input, wherein the translation language information indicates a language type of the recognized text and a language type of the translated text;

[0228] a speech processing submodule, configured to perform speech processing on the audio using a first model according to the translation language information, and obtain data output by the first model, wherein the data includes: the recognized text obtained by performing speech recognition on the audio based on the first model and the translated text obtained by performing speech translation on the audio based on the first model, wherein the speech processing includes the speech recognition and the speech translation;

[0229] The display submodule is used to display the first text in the second area.

[0230] In some embodiments, the voice interface is an interface of a first application in a call mode, and the voice processing submodule includes:

[0231] a first speech processing unit, configured to input first information and second information into the first model for speech processing, respectively, to obtain first data and second data output by the first model, wherein the first information includes first audio of the local end conversation location and the translation language information in the conversation mode, and the second information includes second audio of the opposite end conversation location in the conversation mode;

[0232] The data includes the first data and the second data, the first data includes the recognized text and the translated text of the first audio, and the second data includes the recognized text and the translated text of the second audio.

[0233] In some embodiments, the display submodule is specifically used for:

[0234] Displaying the first text in the second area according to the speaker information of the audio;

[0235] The data also includes the speaking object information, which is obtained by performing speech object separation on the audio based on the first model, and the speech processing also includes the speech object separation.

[0236] In some embodiments, the speech processing submodule includes:

[0237] A second speech processing unit, used for sequentially inputting each third information into the first model for speech processing to obtain data output by the first model, wherein the third information includes the translation language information and an acquired speech data block;

[0238] The data is output by the first model when the output decision result is determined to be a first decision result, the first decision result indicates that the audio meets the speech translation condition, and the audio is composed of the speech data blocks acquired sequentially.

[0239] In some embodiments, the second speech processing unit is specifically used to:

[0240] Inputting each of the speech data blocks into the first sub-model in sequence to extract speech features, thereby obtaining speech features of the audio;

[0241] Inputting the speech feature into the second sub-model for feature mapping to obtain the first text feature of the audio;

[0242] concatenating the first text feature and the second text feature to obtain a third text feature, wherein the second text feature is a text feature obtained by text encoding the input text;

[0243] Inputting the third text feature into a third sub-model for natural language processing to obtain the data, wherein the natural language processing includes the speech recognition and the speech translation, and the data includes the recognized text and the translated text of the audio output by the third sub-model;

[0244] Among them, the first model includes the first sub-model, the second sub-model and the third sub-model, and the input text includes the translation language information and the recognition text of the audio obtained by performing the natural language processing by the third sub-model.

[0245] In some embodiments, the third sub-model further outputs speech object conversion information, where the speech object conversion information is used to indicate whether the speech object of the speech data block is converted; the speech processing sub-module further includes:

[0246] A feature extraction unit, used for inputting the speech feature into the fifth sub-model for feature extraction to obtain a feature vector of each of the speech data blocks, wherein the first model also includes the fifth sub-model;

[0247] The separation unit is used to perform speech object separation on each of the speech data blocks according to the speech object conversion information and the feature vector of each of the speech data blocks to obtain the speech object information of the audio.

[0248] In some embodiments, the separation unit is specifically used for:

[0249] In a case where the speaker conversion information is a first conversion label, feature clustering is performed on a feature vector of a speech data block of a first speaker to obtain a central feature vector, and the first conversion label indicates that the speaker of the speech data block has not been converted;

[0250] In the case where the speaking object conversion information is a second conversion label, the central feature vector is compared with the feature vector of the speech data block of the second speaking object to obtain the speaking object information of the second speaking object, and the second conversion label indicates that the speaking object of the speech data block has been converted.

[0251] In some embodiments, the speech processing submodule further includes:

[0252] A first decision processing unit is used to input the speech feature of the audio and the second text feature into the fourth sub-model to perform output decision processing of the first model to obtain an output decision result, and the first model also includes the fourth sub-model; wherein the speech feature of the audio is obtained by splicing the speech features of each of the input speech data blocks when the speaking object conversion information output by the third sub-model is the first conversion label; the output decision processing includes: performing feature association on the speech feature of the audio and the second text feature to obtain a first probability, the first probability is used to indicate the probability of semantic completeness of the audio, and when the first probability is greater than or equal to a preset threshold, determining that the output decision result is the first decision result;

[0253] The second decision processing unit is used to determine that the output decision result is the first decision result when the speaking object conversion information output by the third sub-model is the second conversion label.

[0254] In some embodiments, the voice interface simultaneously displays the interface of the target application and the sub-interface, and the first display module 2001 is specifically used to:

[0255] On the interface of the target application, the sub-interface is displayed in a floating window manner, and a display setting control is also displayed in the first area;

[0256] When a second input to the display setting control is received, the interface of the target application and the sub-interface are displayed in a split-screen manner.

[0257] In some embodiments, the sub-interface further includes a third area, the third area further displays a function display control, and the device further includes:

[0258] A third display module, configured to display the function control in the third area when receiving a third input to the function display control;

[0259] The function processing module is used to execute the control operation associated with the first text when receiving a fourth input to the function control.

[0260] In some embodiments, the functional control includes at least one of the following:

[0261] A first control, wherein the control operation is to start or stop calling a microphone to obtain audio corresponding to the first text;

[0262] A second control, wherein the control operation is to take a screenshot of an interface of a target application, and the target application is associated with a program for displaying a sub-interface of the first text;

[0263] A third control, wherein the control operation is to save the first text;

[0264] A fourth control, wherein the control operation is to obtain and display the summary content of the first text;

[0265] A fifth control, wherein the control operation is to obtain and display the speaker information of the audio corresponding to the first text;

[0266] A sixth control, wherein the control operation is to edit the speaker information;

[0267] A seventh control, wherein the control operation is to obtain and broadcast the audio of the translated text, and the audio timbre of the translated text is the same as the audio timbre of the recognized text;

[0268] An eighth control, wherein the control operation is to set fourth information, wherein the fourth information is used to trigger display of reminder information, and the reminder information is used to remind the content associated with the fourth information in the first text.

[0269] In this embodiment, a voice interface is displayed; a first input of a conversion control displayed in a first area of ​​a sub-interface of the voice interface is received; in response to the first input, a first text is displayed in a second area of ​​the sub-interface according to an input parameter of the first input, and the first text includes at least one of a recognition text and a translation text of the audio acquired based on the voice interface. In this way, by displaying a voice interface, acquiring audio based on the voice interface, and inputting controls in the voice interface, the audio can be voice processed, for example, voice recognition or voice translation can be performed on the audio, and then the recognition text or translation text of the audio can be displayed in the sub-interface of the voice interface. In this way, when the voice interface and the sub-interface belong to different applications, the user can conveniently view the recognition text and translation text of the audio in the voice interface without the need for the user to perform operations such as copying and pasting, thereby improving the acquisition efficiency of the recognition text and translation text of the audio.

[0270] The display device in the embodiment of the present application can be an electronic device or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0271] The display device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0272] The display device provided in the embodiment of the present application can achieve Figure 1 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0273] Alternatively, if Fig.21 As shown, an embodiment of the present application further provides an electronic device 2100, including a processor 2101 and a memory 2102, wherein the memory 2102 stores programs or instructions that can be executed on the processor 2101, and when the program or instructions are executed by the processor 2101, the various steps of the above-mentioned display method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described again here.

[0274] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0275] Fig. 22 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0276] The electronic device 2200 includes but is not limited to: a radio frequency unit 2201, a network module 2202, an audio output unit 2203, an input unit 2204, a sensor 2205, a display unit 2206, a user input unit 2207, an interface unit 2208, a memory 2209, and a processor 2210 and other components.

[0277] Those skilled in the art will appreciate that the electronic device 2200 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 2210 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig. 22 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0278] The display unit 2206 is used to display a voice interface, wherein the voice interface is displayed with a sub-interface, wherein the sub-interface includes a first area and a second area, wherein the first area is displayed with a conversion control;

[0279] A user input unit 2207, used to receive a first input to the conversion control;

[0280] The display unit 2206 is used to display a first text in the second area in response to the first input according to the input parameters of the first input, wherein the first text includes at least one of a recognized text and a translated text of the audio, the audio is obtained based on the voice interface, and the recognized text and the translated text are in different language types.

[0281] In this embodiment, a voice interface is displayed; a first input of a conversion control displayed in a first area of ​​a sub-interface of the voice interface is received; in response to the first input, a first text is displayed in a second area of ​​the sub-interface according to an input parameter of the first input, and the first text includes at least one of a recognition text and a translation text of the audio acquired based on the voice interface. In this way, by displaying a voice interface, acquiring audio based on the voice interface, and inputting controls in the voice interface, the audio can be voice processed, for example, voice recognition or voice translation can be performed on the audio, and then the recognition text or translation text of the audio can be displayed in the sub-interface of the voice interface. In this way, when the voice interface and the sub-interface belong to different applications, the user can conveniently view the recognition text and translation text of the audio in the voice interface without the need for the user to perform operations such as copying and pasting, thereby improving the acquisition efficiency of the recognition text and translation text of the audio.

[0282] In some embodiments, the processor 2210 is configured to determine, according to the input parameters of the first input, the translation language information of the audio, the translation language information indicating the language type of the recognized text and the language type of the translated text; according to the translation language information, perform speech processing on the audio using a first model to obtain data output by the first model, the data including: the recognized text obtained by performing speech recognition on the audio based on the first model and the translated text obtained by performing speech translation on the audio based on the first model, the speech processing including the speech recognition and the speech translation;

[0283] The display unit 2206 is further configured to display the first text in the second area.

[0284] In some embodiments, the voice interface is an interface of a first application in a call mode, and the processor 2210 is further configured to:

[0285] Inputting the first information and the second information into the first model for voice processing respectively, obtaining the first data and the second data output by the first model, wherein the first information includes the first audio of the local call location and the translation language information in the call mode, and the second information includes the second audio of the other party call location in the call mode;

[0286] The data includes the first data and the second data, the first data includes the recognized text and the translated text of the first audio, and the second data includes the recognized text and the translated text of the second audio.

[0287] In some embodiments, the display unit 2206 is further configured to:

[0288] Displaying the first text in the second area according to the speaker information of the audio;

[0289] The data also includes the speaking object information, which is obtained by performing speech object separation on the audio based on the first model, and the speech processing also includes the speech object separation.

[0290] In some embodiments, the processor 2210 is further configured to:

[0291] Input each piece of third information into the first model in sequence for speech processing to obtain data output by the first model, wherein the third information includes the translation language information and an acquired speech data block;

[0292] The data is output by the first model when the output decision result is determined to be a first decision result, the first decision result indicates that the audio meets the speech translation condition, and the audio is composed of the speech data blocks acquired sequentially.

[0293] In some embodiments, the processor 2210 is further configured to:

[0294] Inputting each of the speech data blocks into the first sub-model in sequence to extract speech features, thereby obtaining speech features of the audio;

[0295] Inputting the speech feature into the second sub-model for feature mapping to obtain the first text feature of the audio;

[0296] concatenating the first text feature and the second text feature to obtain a third text feature, wherein the second text feature is a text feature obtained by text encoding the input text;

[0297] Inputting the third text feature into a third sub-model for natural language processing to obtain the data, wherein the natural language processing includes the speech recognition and the speech translation, and the data includes the recognized text and the translated text of the audio output by the third sub-model;

[0298] Among them, the first model includes the first sub-model, the second sub-model and the third sub-model, and the input text includes the translation language information and the recognition text of the audio obtained by performing the natural language processing by the third sub-model.

[0299] In some embodiments, the third sub-model further outputs speech object conversion information, where the speech object conversion information is used to indicate whether the speech object of the speech data block is converted; the processor 2210 is further used to:

[0300] Inputting the speech features into a fourth sub-model for feature extraction to obtain feature vectors of each of the speech data blocks, wherein the first model also includes the fourth sub-model;

[0301] According to the speech object conversion information and the feature vectors of each speech data block, speech object separation is performed on each speech data block to obtain the speech object information of the audio.

[0302] In some embodiments, the processor 2210 is further configured to:

[0303] In a case where the speaker conversion information is a first conversion label, feature clustering is performed on a feature vector of a speech data block of a first speaker to obtain a central feature vector, and the first conversion label indicates that the speaker of the speech data block has not been converted;

[0304] In the case where the speaking object conversion information is a second conversion label, the central feature vector is compared with the feature vector of the speech data block of the second speaking object to obtain the speaking object information of the second speaking object, and the second conversion label indicates that the speaking object of the speech data block has been converted.

[0305] In some embodiments, the processor 2210 is further configured to:

[0306] The speech feature of the audio and the second text feature are input into the fifth sub-model for output decision processing of the first model to obtain an output decision result, and the first model also includes the fifth sub-model; wherein the speech feature of the audio is obtained by splicing the speech features of each of the input speech data blocks when the speaking object conversion information output by the third sub-model is the first conversion label; the output decision processing includes: feature association of the speech feature of the audio and the second text feature to obtain a first probability, the first probability is used to indicate the probability of semantic completeness of the audio, and when the first probability is greater than or equal to a preset threshold, determining that the output decision result is the first decision result;

[0307] When the speaking object conversion information output by the third sub-model is the second conversion label, the output decision result is determined to be the first decision result.

[0308] In some embodiments, the voice interface simultaneously displays the interface of the target application and the sub-interface, and the display unit 2206 is further configured to display the sub-interface in a floating window manner on the interface of the target application, and a display setting control is also displayed in the first area;

[0309] The user input unit 2207 is further used to receive a second input to the display setting control;

[0310] The display unit 2206 is further configured to display the interface of the target application and the sub-interface in a split-screen manner when a second input to the display setting control is received.

[0311] In some embodiments, the sub-interface further includes a third area, and the third area further displays a function display control, and the display unit 2206 is further configured to display the function control in the third area when receiving a third input to the function display control;

[0312] The processor 2210 is further configured to, upon receiving a fourth input to the function control, execute a control operation associated with the first text.

[0313] In some embodiments, the functional control includes at least one of the following:

[0314] A first control, wherein the control operation is to start or stop calling a microphone to obtain audio corresponding to the first text;

[0315] A second control, wherein the control operation is to take a screenshot of an interface of a target application, and the target application is associated with a program for displaying a sub-interface of the first text;

[0316] A third control, wherein the control operation is to save the first text;

[0317] A fourth control, wherein the control operation is to obtain and display the summary content of the first text;

[0318] A fifth control, wherein the control operation is to obtain and display the speaker information of the audio corresponding to the first text;

[0319] A sixth control, wherein the control operation is to edit the speaker information;

[0320] A seventh control, wherein the control operation is to obtain and broadcast the audio of the translated text, and the audio timbre of the translated text is the same as the audio timbre of the recognized text;

[0321] An eighth control, wherein the control operation is to set fourth information, wherein the fourth information is used to trigger display of reminder information, and the reminder information is used to remind the content associated with the fourth information in the first text.

[0322] It should be understood that in the embodiment of the present application, the input unit 2204 may include a graphics processor (Graphics Processing Unit, GPU) 22041 and a microphone 22042, and the graphics processor 22041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 2206 may include a display panel 22061, and the display panel 22061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 2207 includes a touch panel 22071 and at least one of other input devices 22072. The touch panel 22071 is also called a touch screen. The touch panel 22071 may include two parts: a touch detection device and a touch controller. Other input devices 22072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0323] The memory 2209 can be used to store software programs and various data. The memory 2209 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 2209 may include a volatile memory or a non-volatile memory, or the memory 2209 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 2209 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0324] The processor 2210 may include one or more processing units; in some embodiments, the processor 2210 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 2210.

[0325] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned display method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0326] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0327] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned display method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0328] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0329] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned display method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0330] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0331] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling an electronic device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0332] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.< / eot> < / en> < / cn> < / cn> < / en> < / sot> < / en> < / cn> < / sot> < / cn> < / en> < / sot> < / en> < / cn> < / sot> < / cn> < / en> < / sot> < / en> < / cn> < / sot> < / unk> < / unk> < / sot> < / eot> < / unk> < / cn> < / en> < / cn> < / en> < / cn> < / en> < / sot>

Claims

1. A display method, characterized in that: The method comprises: Displaying a voice interface, wherein the voice interface displays a sub-interface, wherein the sub-interface includes a first area and a second area, wherein the first area displays a conversion control; receiving a first input to the conversion control; In response to the first input, a first text is displayed in the second area according to input parameters of the first input, the first text including at least one of a recognized text and a translated text of the audio, the audio is acquired based on the voice interface, and the recognized text and the translated text are in different language types.

2. The method according to claim 1, characterized in that The step of displaying the first text in the second area according to the input parameter of the first input includes: Determining translation language information of the audio according to the input parameter of the first input, the translation language information indicating the language type of the recognized text and the language type of the translated text; According to the translation language information, the audio is voice processed using a first model to obtain data output by the first model, the data comprising: the recognized text obtained by voice recognition of the audio based on the first model and the translated text obtained by voice translation of the audio based on the first model, the voice processing comprising the voice recognition and the voice translation; The first text is displayed in the second area.

3. The method according to claim 2, characterized in that The voice interface is an interface of the first application in the call mode, and the first model is used to perform voice processing on the audio according to the translation language information to obtain data output by the first model, including: Inputting the first information and the second information into the first model for voice processing respectively, obtaining the first data and the second data output by the first model, wherein the first information includes the first audio of the local call location and the translation language information in the call mode, and the second information includes the second audio of the other party call location in the call mode; The data includes the first data and the second data, the first data includes the recognized text and the translated text of the first audio, and the second data includes the recognized text and the translated text of the second audio.

4. The method according to claim 2, characterized in that: The displaying the first text in the second area includes: Displaying the first text in the second area according to the speaker information of the audio; The data also includes the speaking object information, which is obtained by performing speech object separation on the audio based on the first model, and the speech processing also includes the speech object separation.

5. The method according to claim 2, characterized in that: The step of performing speech processing on the audio using a first model according to the translation language information to obtain data output by the first model includes: Input each third information into the first model in sequence for speech processing to obtain data output by the first model, wherein the third information includes the translation language information and an acquired speech data block; The data is output by the first model when the output decision result is determined to be a first decision result, the first decision result indicates that the audio meets the speech translation condition, and the audio is composed of the speech data blocks acquired sequentially.

6. The method according to claim 5, characterized in that The step of sequentially inputting each third information into the first model for speech processing to obtain data output by the first model includes: Inputting each of the speech data blocks into the first sub-model in sequence to extract speech features, thereby obtaining speech features of the audio; Inputting the speech feature into the second sub-model for feature mapping to obtain the first text feature of the audio; concatenating the first text feature and the second text feature to obtain a third text feature, wherein the second text feature is a text feature obtained by text encoding the input text; Inputting the third text feature into a third sub-model for natural language processing to obtain the data, wherein the natural language processing includes the speech recognition and the speech translation, and the data includes the recognized text and the translated text of the audio output by the third sub-model; Among them, the first model includes the first sub-model, the second sub-model and the third sub-model, and the input text includes the translation language information and the recognition text of the audio obtained by performing the natural language processing by the third sub-model.

7. The method according to claim 6, characterized in that The third sub-model further outputs speech object conversion information, wherein the speech object conversion information is used to indicate whether the speech object of the speech data block is converted; after inputting each of the speech data blocks into the first sub-model in sequence for speech feature extraction, the method further includes: Inputting the speech features into a fourth sub-model for feature extraction to obtain feature vectors of each of the speech data blocks, wherein the first model also includes the fourth sub-model; According to the speech object conversion information and the feature vectors of each speech data block, speech object separation is performed on each speech data block to obtain the speech object information of the audio.

8. The method according to claim 7, characterized in that The step of performing voice object separation on each voice data block according to the voice object conversion information and the feature vector of each voice data block to obtain the voice object information of the audio includes: In a case where the speaker conversion information is a first conversion label, feature clustering is performed on a feature vector of a speech data block of a first speaker to obtain a central feature vector, and the first conversion label indicates that the speaker of the speech data block has not been converted; In the case where the speaking object conversion information is a second conversion label, the central feature vector is compared with the feature vector of the speech data block of the second speaking object to obtain the speaking object information of the second speaking object, and the second conversion label indicates that the speaking object of the speech data block has been converted.

9. The method according to claim 8, characterized in that After inputting each of the speech data blocks into the first sub-model in sequence to extract speech features and obtaining the speech features of the audio, the method further includes at least one of the following: The speech feature of the audio and the second text feature are input into the fifth sub-model for output decision processing of the first model to obtain an output decision result, and the first model also includes the fifth sub-model; wherein the speech feature of the audio is obtained by splicing the speech features of each of the input speech data blocks when the speaking object conversion information output by the third sub-model is the first conversion label; the output decision processing includes: feature association of the speech feature of the audio and the second text feature to obtain a first probability, the first probability is used to indicate the probability of semantic completeness of the audio, and when the first probability is greater than or equal to a preset threshold, determining that the output decision result is the first decision result; When the speaking object conversion information output by the third sub-model is the second conversion label, the output decision result is determined to be the first decision result.

10. The method according to claim 1, characterized in that The voice interface simultaneously displays the interface of the target application and the sub-interface, and the display of the voice interface includes at least one of the following: On the interface of the target application, the sub-interface is displayed in a floating window manner, and a display setting control is also displayed in the first area; When a second input to the display setting control is received, the interface of the target application and the sub-interface are displayed in a split-screen manner.

11. The method according to claim 1, characterized in that: The sub-interface further includes a third area, and the third area further displays a function display control. After displaying the first text in the second area in response to the first input and according to the input parameter of the first input, the method further includes: upon receiving a third input to the function display control, displaying the function control in the third area; In case a fourth input to the function control is received, a control operation associated with the first text is performed.

12. The method according to claim 11, characterized in that The functional control includes at least one of the following: A first control, wherein the control operation is to start or stop calling a microphone to obtain audio corresponding to the first text; A second control, wherein the control operation is to take a screenshot of an interface of a target application, and the target application is associated with a program for displaying a sub-interface of the first text; A third control, wherein the control operation is to save the first text; A fourth control, wherein the control operation is to obtain and display the summary content of the first text; A fifth control, wherein the control operation is to obtain and display the speaker information of the audio corresponding to the first text; A sixth control, wherein the control operation is to edit the speaking object information; A seventh control, wherein the control operation is to obtain and broadcast the audio of the translated text, and the audio timbre of the translated text is the same as the audio timbre of the recognized text; An eighth control, wherein the control operation is to set fourth information, wherein the fourth information is used to trigger display of reminder information, and the reminder information is used to remind the content associated with the fourth information in the first text.

13. A display device, characterized in that: The device comprises: A first display module, used for displaying a voice interface, wherein the voice interface displays a sub-interface, wherein the sub-interface includes a first area and a second area, wherein the first area displays a conversion control; A receiving module, configured to receive a first input to the conversion control; A second display module is used to display a first text in the second area in response to the first input and according to the input parameters of the first input, wherein the first text includes at least one of a recognized text and a translated text of the audio, the audio is obtained based on the voice interface, and the recognized text and the translated text are in different language types.

14. The device according to claim 13, characterized in that The second display module comprises: A first determination submodule, configured to determine translation language information of the audio according to an input parameter of the first input, wherein the translation language information indicates a language type of the recognized text and a language type of the translated text; a speech processing submodule, configured to perform speech processing on the audio using a first model according to the translation language information, and obtain data output by the first model, wherein the data includes: the recognized text obtained by performing speech recognition on the audio based on the first model and the translated text obtained by performing speech translation on the audio based on the first model, wherein the speech processing includes the speech recognition and the speech translation; The display submodule is used to display the first text in the second area.

15. The device according to claim 14, characterized in that The speech processing submodule comprises: A second speech processing unit, used for sequentially inputting each third information into the first model for speech processing to obtain data output by the first model, wherein the third information includes the translation language information and an acquired speech data block; The data is output by the first model when the output decision result is determined to be a first decision result, the first decision result indicates that the audio meets the speech translation condition, and the audio is composed of the speech data blocks acquired sequentially.

16. The device according to claim 15, characterized in that The second speech processing unit is specifically used for: Inputting each of the speech data blocks into the first sub-model in sequence to extract speech features, thereby obtaining speech features of the audio; Inputting the speech feature into the second sub-model for feature mapping to obtain the first text feature of the audio; concatenating the first text feature and the second text feature to obtain a third text feature, wherein the second text feature is a text feature obtained by text encoding the input text; Inputting the third text feature into a third sub-model for natural language processing to obtain the data, wherein the natural language processing includes the speech recognition and the speech translation, and the data includes the recognized text and the translated text of the audio output by the third sub-model; Among them, the first model includes the first sub-model, the second sub-model and the third sub-model, and the input text includes the translation language information and the recognition text of the audio obtained by performing the natural language processing by the third sub-model.

17. The device according to claim 16, characterized in that The third sub-model further outputs speech object conversion information, wherein the speech object conversion information is used to indicate whether the speech object of the speech data block is converted; the speech processing sub-module further includes: A feature extraction unit, used for inputting the speech feature into the fifth sub-model for feature extraction to obtain a feature vector of each of the speech data blocks, wherein the first model also includes the fifth sub-model; The separation unit is used to perform speech object separation on each of the speech data blocks according to the speech object conversion information and the feature vector of each of the speech data blocks to obtain the speech object information of the audio.

18. The device according to claim 17, characterized in that The separation unit is specifically used for: In a case where the speaker conversion information is a first conversion label, feature clustering is performed on a feature vector of a speech data block of a first speaker to obtain a central feature vector, and the first conversion label indicates that the speaker of the speech data block has not been converted; In the case where the speaking object conversion information is a second conversion label, the central feature vector is compared with the feature vector of the speech data block of the second speaking object to obtain the speaking object information of the second speaking object, and the second conversion label indicates that the speaking object of the speech data block has been converted.

19. The device according to claim 18, characterized in that The speech processing submodule also includes: A first decision processing unit is used to input the speech feature of the audio and the second text feature into the fourth sub-model to perform output decision processing of the first model to obtain an output decision result, and the first model also includes the fourth sub-model; wherein the speech feature of the audio is obtained by splicing the speech features of each of the input speech data blocks when the speaking object conversion information output by the third sub-model is the first conversion label; the output decision processing includes: performing feature association on the speech feature of the audio and the second text feature to obtain a first probability, the first probability is used to indicate the probability of semantic completeness of the audio, and when the first probability is greater than or equal to a preset threshold, determining that the output decision result is the first decision result; The second decision processing unit is used to determine that the output decision result is the first decision result when the speaking object conversion information output by the third sub-model is the second conversion label.

20. The device according to claim 13, characterized in that The voice interface simultaneously displays the interface of the target application and the sub-interface, and the first display module is specifically used for: On the interface of the target application, the sub-interface is displayed in a floating window manner, and a display setting control is also displayed in the first area; When a second input to the display setting control is received, the interface of the target application and the sub-interface are displayed in a split-screen manner.

21. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the display method according to any one of claims 1 to 12 are implemented.

22. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by the processor, the steps of the display method according to any one of claims 1 to 12 are implemented.

Images