Voice processing method and electronic equipment

Abstract

The invention provides a voice processing method and electronic equipment, and relates to the technical field of audio processing. After the electronic equipment converts a voice file into a text, if a long-press operation is received, it is indicated that a user possibly wants to select speaking contents located in different segments at the same time, the electronic equipment can splice all displayed speaking information to obtain a spliced character string. Afterwards, the electronic device determines a target character area based on the initial position and the end position corresponding to the long press operation in combination with the spliced character string, and the target character area represents the selected character corresponding to the long press operation in the spliced character string. Afterwards, the electronic equipment displays the spliced character string in a plain text form, performs selection setting on the characters in the target character area, realizes cross-section selection of the speaking information, and can perform corresponding processing, such as sharing, searching, copying and other operations, according to the function control selected by the user, thereby providing rich services for the user and improving the user experience.

Description

Technical Field

[0001] The present application relates to the technical field of audio processing, and in particular to a voice processing method and electronic device. Background Art

[0002] With the development of electronic devices, the functions provided by electronic devices are becoming more and more abundant. For example, electronic devices provide a recording function. In daily work and study, many users will use the recording function of electronic devices. For example, users add recordings to notes.

[0003] At present, electronic devices generally provide basic services for voice data obtained by recording, such as services that support converting voice data into text, so that users can intuitively know the speaker's speech through text. However, when users have other needs for using voice data, since electronic devices do not provide corresponding services, the user's needs are not met. Therefore, electronic devices urgently need to provide richer services for voice data to meet user needs. Summary of the invention

[0004] In view of this, the present application provides a speech processing method and electronic device for realizing cross-segment merging of speech texts and providing richer services for voice data.

[0005] In a first aspect, the present application provides a voice processing method, wherein during recording, an electronic device displays a first interface of a first note in a note application, wherein the first interface includes a first recording card, and the first recording card displays a first control for triggering stopping recording.

[0006] After receiving the latest voice data, the electronic device converts the latest voice data into text to obtain the latest speech information corresponding to the latest voice data.

[0007] Afterwards, the electronic device adds the latest speech information to the first recording card. Afterwards, after receiving a trigger operation (such as a click operation) from the user on the above-mentioned first control, the electronic device responds to the trigger operation, stops recording, displays the second interface of the first note, and the recording card displayed on the second interface serves as the second recording card. The second recording card may include all speech information obtained during the recording and a second control. Generally speaking, the number of all speech information is multiple. The second control is used to trigger the electronic device to continue recording.

[0008] After receiving a long press operation input by the user, indicating that the user wants to select the content of the speech information, the electronic device updates the second recording card to a third recording card. In the third recording card, the target character area in the spliced ​​character string is selected, and the spliced ​​character is obtained by splicing multiple speech information, that is, the speech information is displayed in text form.

[0009] The target character area is determined based on the start position and the end position corresponding to the long press operation, and the target character area includes at least two speech information among the multiple speech information or information in different rows in one speech information among the multiple speech information.

[0010] The electronic device may display at least one function control, and the function control is used to trigger the electronic device to perform corresponding processing on the characters in the target character area.

[0011] In the present application, the note application provides a recording function, and the user can add a recording during the process of editing a note (such as the first note). Accordingly, during the recording, the electronic device converts the received voice data into speech information in real time and displays it, intuitively showing the speech content, so that the user can quickly know the speech content corresponding to the voice data. After that, when the recording is stopped, if the user wants to select the speech information, the user can enter a long press operation. After receiving the long press operation, the electronic device can obtain the starting position and the end position corresponding to the long press operation, and splice the characters in all the speech information to obtain the spliced ​​string, which is consistent with the content of the speech information. The electronic device displays the spliced ​​string to display the speech information in plain text, and selects and sets the target character area in the spliced ​​string. Since the spliced ​​string, that is, the spliced ​​speech information is plain text, the electronic device can select different speech information at the same time, such as selecting characters in multiple consecutive speech information at the same time, and selecting different types of information located in different rows in a speech information (such as selecting the speaker identifier and speech text at the same time), so as to realize the cross-segment selection of speech information. Afterwards, the electronic device can display function controls so that the user can select function controls according to usage requirements. The electronic device performs corresponding processing based on the function controls selected by the user, provides the user with rich services, meets the user's usage requirements, and improves the user experience.

[0012] The first control may be a control for pausing recording, and the second control may be a control for starting recording.

[0013] In a possible design, the electronic device may start splicing from the first character of all speech information displayed on the third recording card until the last character of all speech information is spliced, thereby completing the splicing of speech information and thus completing the format conversion of speech information.

[0014] In a possible design, the target character area includes characters corresponding to the offset character position and characters of the selected number of characters after the offset character position.

[0015] Among them, the process of determining the above-mentioned offset character position may include: the electronic device counts the number of characters corresponding to the starting position corresponding to the long press operation from the first character in the multiple speech information to obtain the offset character position, and the offset character position represents the position of the first selected character relative to the spliced ​​character string.

[0016] The number of selected characters is determined based on the starting position and the ending position corresponding to the long press operation.

[0017] In the present application, the position of the first selected character in the concatenated character string can be determined by utilizing the offset character position, thereby enabling accurate determination of the selected character, that is, accurate determination of the target character area.

[0018] In a possible design, the second recording card includes a sliding component, and the multiple speech information is carried by the sliding component; the sliding component includes multiple entries, the speech information corresponds to the item one by one, and each item carries the corresponding speech information. The item includes at least two text components, one of the at least two text components may include a speaker ID, and the other text component may include a speech text. The speech text and the speaker ID are generally located in different rows. Since multiple items cannot be selected at the same time, and the contents of different text components cannot be selected at the same time, the electronic device can replace the sliding component in the second recording card with a preset text component, that is, instead of using the sliding component to carry the speech information, the preset text component is used to display the spliced ​​speech information, that is, the preset text component is used to display the speech information in plain text form, so that the electronic device can select multiple speech information at the same time, or select different rows of information in the same speech information at the same time, so as to realize the cross-row (or cross-segment) selection of the speech information.

[0019] Optionally, the types of the text components in the at least two text components may be the same or different. The text components may include EditText and TextView. The preset speech text may be TextView.

[0020] In one possible design, after displaying at least one function control, when the function control stops displaying, the layout component of the speech information of the electronic device can be replaced by a sliding component from a preset text component, that is, the preset text component of the third recording card is replaced by a sliding component, and the sliding component carries multiple speech information, which makes it easier for the electronic device to respond to user operations and ensure the accuracy of operations.

[0021] In one possible design approach, the note-taking application also supports responding to click operations.

[0022] After receiving a click operation input by the user, in response to the click operation, if the click position corresponding to the click operation is in the speech text, it is determined whether it is in a playing state.

[0023] If the electronic device is not in the playing state, the electronic device enters the editing state.

[0024] If the electronic device is in a playing state, the electronic device plays the voice data corresponding to the first speaking time, wherein the first speaking time indicates the speaking time corresponding to the character corresponding to the clicked position; wherein the playing state indicates that the electronic device plays the voice data.

[0025] In an embodiment of the present application, when it is determined that the user has clicked on a speech text, the electronic device can determine whether the tablet computer is in a play state to determine the user's usage intention, that is, to determine whether the user wants to edit the clicked speech text, or to display the progress bar to move the play progress to the time corresponding to the clicked speech text. When the electronic device is not in a play state, the user is less likely to want to play voice data, but more likely to edit the speech text, so the electronic device can enter an edit state to enable the user to edit the clicked speech text. When the electronic device is in a play state, the user is more likely to want to play voice data, but less likely to edit the speech text, so the electronic device can play the voice data corresponding to the speech time corresponding to the clicked character, and achieve accurate playback of voice data without manual adjustment by the user.

[0026] In a possible design, the second recording card further includes a progress bar. If the electronic device is in a playing state, the electronic device moves the playing progress of the progress bar to the first speaking time, so that the user can intuitively know the playing status of the voice data.

[0027] In a possible design, the first speech time is obtained by superimposing the offset time on the speech start time corresponding to the first target speech text. The offset time is obtained based on the speech speed and the number of offset characters corresponding to the first target speech text, and the number of offset characters refers to the number of characters before the click position in the first target speech text. The first target speech text represents the speech text at the click position. Based on this, the accurate determination of the speech time can be displayed.

[0028] In a possible design, the above-mentioned speech information also includes a speaker identification, that is, the electronic device can also display the speaker identification.

[0029] When the click position corresponding to the above-mentioned click operation is in the speech text, it is determined whether it is in the playing state.

[0030] If the electronic device is not in the playing state, the user is less likely to want to play voice data, but is more likely to want to edit the speaker ID. The electronic device can enter the editing state and display the speaker modification control. The speaker modification control is used to trigger the electronic device to modify the speaker ID so that the user can use the speaker modification control to modify the speaker ID, thereby modifying the speech content and meeting the user's modification needs.

[0031] If the electronic device is in the playing state, the user is more likely to want to play voice data and less likely to want to edit the speaker identifier. The electronic device plays the voice data corresponding to the target speech start time; the target speech start time refers to the speech start time corresponding to the speaker identifier, that is, the speech start time corresponding to the speaker identifier clicked by the user, to achieve accurate playback of voice data.

[0032] In a possible design, the electronic device can also move the playback progress of the progress bar to the target speech start time, so that the user can intuitively know the playback status of the voice data.

[0033] In a second aspect, the present application provides an electronic device, comprising a display screen, a memory and one or more processors; the display screen, the memory and the processor are coupled; the display screen is used to display an image generated by the processor, and the memory is used to store computer program code, and the computer program code includes computer instructions; when the processor executes the computer instructions, the electronic device executes a speech processing method as described in any one of the above-mentioned first aspects.

[0034] In a third aspect, the present application provides a computer-readable storage medium, comprising computer instructions, which, when executed on an electronic device, enables the electronic device to execute a speech processing method as described in any one of the above-mentioned first aspects.

[0035] In a fourth aspect, the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes the speech processing method as described in any one of the above-mentioned first aspects.

[0036] It can be understood that the beneficial effects that can be achieved by the electronic device described in the second aspect, the computer-readable storage medium described in the third aspect, and the computer program product described in the fourth aspect provided above can be referred to the beneficial effects in the first aspect and any possible design method thereof, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1A A schematic diagram of an interface for recording notes provided in an embodiment of the present application;

[0038] Figure 1B A second schematic diagram of an interface for recording notes provided in an embodiment of the present application;

[0039] Figure 1C A third schematic diagram of an interface for recording notes provided in an embodiment of the present application;

[0040] Figure 2A A schematic diagram of an interface of a recording application provided in an embodiment of the present application;

[0041] Figure 2B A second schematic diagram of an interface of a recording application provided in an embodiment of the present application;

[0042] Figure 2C A third schematic diagram of an interface of a recording application provided in an embodiment of the present application;

[0043] Figure 2D A schematic diagram of an interface of a recording file provided in an embodiment of the present application;

[0044] Figure 3A A diagram of a call recording interface provided in an embodiment of the present application;

[0045] Figure 3B A second schematic diagram of a call recording interface provided in an embodiment of the present application;

[0046] Figure 3C A second schematic diagram of an interface of a recording file provided in an embodiment of the present application;

[0047] Figure 4A A schematic diagram of a speech-to-text interface provided in an embodiment of the present application;

[0048] Figure 4B A second schematic diagram of a speech-to-text interface provided in an embodiment of the present application;

[0049] Figure 4C A third schematic diagram of a speech-to-text interface provided in an embodiment of the present application;

[0050] Figure 5 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application;

[0051] Figure 6 A schematic diagram of a software framework of an electronic device provided in an embodiment of the present application;

[0052] Figure 7 A flowchart of a speech processing method provided in an embodiment of the present application is shown in FIG1 ;

[0053] Fig. 8A A fourth schematic diagram of a speech-to-text interface provided in an embodiment of the present application;

[0054] Figure 8B A speech-to-text interface provided in an embodiment of the present application Figure 5 ;

[0055] Figure 8C A speech-to-text interface provided in an embodiment of the present application Figure 6 ;

[0056] Fig.8D A speech-to-text interface provided in an embodiment of the present application Figure 7 ;

[0057] Fig. 8E Schematic diagram 8 of a speech-to-text interface provided in an embodiment of the present application;

[0058] Fig.8F Schematic diagram 9 of a speech-to-text interface provided in an embodiment of the present application;

[0059] Fig. 9A Schematic diagram 1 of a note application display interface provided in an embodiment of the present application;

[0060] Fig. 9B Schematic diagram 2 of a note application display interface provided in an embodiment of the present application;

[0061] Fig. 10A A schematic diagram of an interface layout provided in an embodiment of the present application;

[0062] Fig. 10B A second schematic diagram of an interface layout provided in an embodiment of the present application;

[0063] Fig. 10C A third schematic diagram of an interface layout provided in an embodiment of the present application;

[0064] Fig.11A An editing schematic diagram 1 provided for an embodiment of the present application;

[0065] Fig. 11B A second editing schematic diagram provided for an embodiment of the present application;

[0066] Fig.12 A schematic diagram of an editing process provided in an embodiment of the present application is shown in FIG1 ;

[0067] Fig.13 A second schematic diagram of an editing process provided in an embodiment of the present application;

[0068] Fig.14 A schematic diagram of a linkage playback provided in an embodiment of the present application;

[0069] Fig.15A Schematic diagram 3 of an editing process provided in an embodiment of the present application;

[0070] Fig. 15B Schematic diagram 4 of an editing process provided in an embodiment of the present application;

[0071] Fig. 15C A schematic diagram of an editing process provided in an embodiment of the present application Figure 5 ;

[0072] Fig.15D A schematic diagram of an editing process provided in an embodiment of the present application Figure 6 ;

[0073] Fig.15E A schematic diagram of an editing process provided in an embodiment of the present application Figure 7 ;

[0074] Fig.15F Schematic diagram 8 of an editing process provided in an embodiment of the present application;

[0075] Fig.16A A schematic diagram 1 of displaying speech information provided in an embodiment of the present application;

[0076] Fig. 16B A second schematic diagram of displaying speech information provided in an embodiment of the present application;

[0077] Fig.17A A long press schematic diagram 1 provided in an embodiment of the present application;

[0078] Fig. 17B A second long press schematic diagram provided in an embodiment of the present application;

[0079] Fig. 17C A selected schematic diagram provided for an embodiment of the present application;

[0080] Fig.17D A schematic diagram of a layout replacement provided in an embodiment of the present application;

[0081] Fig.17E A schematic diagram of a function control display provided in an embodiment of the present application;

[0082] Fig.18A A screening schematic diagram 1 provided for an embodiment of the present application;

[0083] Fig.18B A second screening schematic diagram provided for an embodiment of the present application;

[0084] Fig.18C A screening schematic diagram 3 provided for an embodiment of the present application;

[0085] Fig.19A Schematic diagram 1 of a text saving interface provided in an embodiment of the present application;

[0086] Fig.19B Schematic diagram 2 of a text saving interface provided in an embodiment of the present application;

[0087] Fig.19C Schematic diagram 3 of a text saving interface provided in an embodiment of the present application;

[0088] Fig.19D Schematic diagram 4 of a text saving interface provided in an embodiment of the present application;

[0089] Fig.19E Schematic diagram 1 of a text sharing interface provided in an embodiment of the present application;

[0090] Fig.19F Schematic diagram 2 of a text sharing interface provided in an embodiment of the present application;

[0091] Figure 19G Schematic diagram 3 of a text sharing interface provided in an embodiment of the present application;

[0092] Fig.19H A text saving interface provided in an embodiment of the present application Figure 5 ;

[0093] Fig. 20 A second flowchart of a speech processing method provided in an embodiment of the present application;

[0094] Fig.21 A schematic diagram of a speech processing process provided in an embodiment of the present application. DETAILED DESCRIPTION

[0095] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this embodiment, unless otherwise specified, "plurality" means two or more.

[0096] Electronic devices (such as tablet computers) generally provide a recording function. Users can use the recording function of the tablet computer to record. For example, the tablet computer can support the function of recording notes. When the user uses the note application of the tablet computer to record notes, he can add a recording to the note, that is, add a recording card. For example, the user clicks on Figure 1A The tablet computer starts recording in response to a click operation of the start recording control 11 on the first note interface 10, and displays the following Figure 1BThe tablet computer receives a click operation of the pause recording control 13 on the second note interface 12, indicating that the user wants to stop recording. The tablet computer can stop recording in response to the click operation of the pause recording control 13 and display the following Figure 1C The third note interface 14 is shown. Compared with the recording duration (i.e. 03:10) displayed on the first note interface 10, the recording duration (i.e. 04:10) displayed on the third note interface 14 is longer. The third note interface 14 may also include controls such as a recording playback control 15, which may be used to trigger the tablet computer to play the recording.

[0097] For another example, the tablet computer also includes a recording application, so that the user can record through the recording application. Figure 2A The tablet computer responds to the click operation of the recording application icon 20, starts the recording application, and displays the following Figure 2B The recording interface includes a start recording control 22, which can be used to trigger the tablet computer to start recording. Later, when the user wants to record, the user can click the start recording control 22. In response to the click operation of the start recording control 22, the tablet computer can start recording and display the following Figure 2C The interface 23 shown in the figure includes a stop recording control 24 and a recording time prompt information 25. The stop recording control 24 can be used to trigger the tablet computer to stop recording, and the recording time prompt information indicates the elapsed recording time. Later, when the user wants to stop recording, the user can click the stop recording control 24. In response to the click operation of the stop recording control 24, the tablet computer can stop recording and generate a corresponding voice file (such as Figure 2D The audio file shown in 1).

[0098] For another example, during a call, the tablet supports the call recording function, that is, the user can use the recording function provided by the tablet to record the call content. The user can click Figure 3A In response to the click operation of the start recording control 30, the tablet computer can start recording and display the following Figure 3B When the user wants to stop recording the call content, the user can click the stop recording control 31. In response to the click operation of the stop recording control 31, the tablet computer can stop recording and generate a corresponding voice file (such as Figure 3C Alternatively, when the call ends, the tablet computer can automatically end the recording and generate a corresponding recording file without the user having to manually stop the recording.

[0099] It should be noted that the above-mentioned start recording control 11 and the above-mentioned pause recording control 13 can be different controls, or they can be the same control. Specifically, when the tablet computer is recording, the control is in the recording-on state, and the control can be used as the pause recording control 13. After receiving the user's click operation on the pause recording control 13, the tablet computer can stop recording, and the control is in the recording-stop state, and the control can be used as the start recording control 11. Similarly, the above-mentioned start recording control 22 and the above-mentioned stop recording control 24 can be different controls, or they can be the same control. The above-mentioned start recording control 30 and the above-mentioned stop recording control 31 can also be different controls, or they can be the same control, and this application does not limit them.

[0100] In some embodiments, the electronic device also supports a service for converting voice files to text. Figure 4A The electronic device starts playing the voice file 3 and displays the following Figure 4B The interface shown in FIG. 4 provides a text conversion control 40, which is used to trigger the electronic device to convert the voice file (or recording file) into text. After the electronic device receives the trigger operation (such as a click operation) of the text conversion control 40, the electronic device can convert the voice file 3 into text. Afterwards, after the conversion is successful, the electronic device can display the following information: Figure 4C The interface shown in FIG. 1 shows the speaker identification (such as speaker 1, speaker 2) and the corresponding speech text, speech start time and other speech information. The speech text represents the speaker's speech content, which is obtained by converting the voice file (such as Figure 4C The speech start time corresponding to the speech text indicates the speech start time of the speech content (such as "The weather is very good today, shall we go to the park together?"). Figure 4C 00:00, 00:20, etc. as shown).

[0101] However, users may have other usage requirements, such as selecting speech information across lines (or paragraphs). For example, users want to process multiple consecutive speech information at the same time, such as users want to copy or share the above Figure 4C If the user wants to process different types of information located in different sections of a speech message, such as wanting to copy, share, or search the above Figure 4CThe speaker identifier and speech text in the speech information 43 shown, then you need to select the speaker identifier and speech text in the speech information 43 at the same time, so that the tablet computer can copy, share, or search the selected speaker identifier and speech text in the speech information 43.

[0102] Therefore, in order to meet more user needs, the electronic device provides a voice processing method. During recording, after receiving the latest voice data, the electronic device performs text conversion and displays the latest speech information corresponding to the latest voice data, wherein the electronic device also displays historical speech information, which is the speech information corresponding to the voice data received from the beginning of this recording to the receipt of the latest voice data. That is to say, the electronic device displays the speech information corresponding to all voice data obtained during recording. After receiving the long press operation input by the user, the electronic device obtains the starting position (such as the position where the user's finger falls) and the ending position (such as the position where the user's finger leaves) corresponding to the long press operation. Afterwards, the electronic device splices all the displayed speech information to obtain the spliced ​​string. Afterwards, the electronic device determines the target character area based on the starting position and the ending position corresponding to the long press operation, combined with the spliced ​​string, and the target character area represents the selected character corresponding to the long press operation.

[0103] Afterwards, the electronic device displays the concatenated character string in plain text form, and selects and sets the characters in the target character area. Since the concatenated character string, that is, the concatenated speech information is plain text, the electronic device can simultaneously select speech information located in different segments, such as selecting information in multiple consecutive speech information at the same time, and selecting different types of information located in different segments in one speech information at the same time, thereby realizing cross-segment selection of speech information.

[0104] Afterwards, the electronic device can display function controls so that the user can select function controls according to usage needs. The electronic device performs corresponding processing according to the function controls selected by the user, such as sharing, searching, copying and other operations, to provide users with rich services, meet users' usage needs and improve user experience.

[0105] Exemplarily, the electronic device in the embodiments of the present application may be a mobile phone, a tablet computer, a desktop, a laptop, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), an augmented reality (AR)\virtual reality (VR) device, and other electronic devices with a recording function. The embodiments of the present application do not impose any special restrictions on the specific form of the electronic device.

[0106] For example, Figure 5 1 shows a schematic diagram of the structure of the electronic device 100. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195.

[0107] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or separate some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0108] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0109] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0110] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0111] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0112] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present invention is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0113] The charging management module 140 is used to receive charging input from a charger. While the charging management module 140 is charging the battery 142 , it can also power the electronic device through the power management module 141 .

[0114] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0115] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0116] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0117] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0118] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0119] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, so that electronic device 100 can communicate with the network and other devices through wireless communication technology.

[0120] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0121] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0122] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0123] The ISP is used to process data fed back by the camera 193 .

[0124] The camera 193 is used to capture still images or videos. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0125] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0126] Video codecs are used to compress or decompress digital videos. The electronic device 100 may support one or more video codecs. Thus, the electronic device 100 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0127] NPU is a neural-network (NN) computing processor. It can quickly process input information by drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, and can also continuously self-learn.

[0128] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100 .

[0129] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0130] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0131] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0132] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The electronic device 100 can listen to music or listen to a hands-free call through the speaker 170A.

[0133] The receiver 170B, also called a "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 100 receives a call or voice message, the voice can be received by placing the receiver 170B close to the human ear.

[0134] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to microphone 170C to input the sound signal into microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify the sound source, realize directional recording function, etc.

[0135] The earphone interface 170D is used to connect a wired earphone and can be a USB interface 130 or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0136] Among them, the sensor module 180 may include a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, and the like.

[0137] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The electronic device 100 may receive key input and generate key signal input related to user settings and function control of the electronic device 100.

[0138] The motor 191 can generate a vibration prompt. The indicator 192 can be an indicator light, which can be used to indicate the charging status, power change, and can also be used to indicate messages, missed calls, notifications, etc.

[0139] The SIM card interface 195 is used to connect a SIM card.

[0140] The software system of the electronic device 200 can adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. TM Taking the system as an example, the software structure of the electronic device 200 is exemplified.

[0141] Figure 6 1 is a software structure block diagram of the electronic device 100 according to an embodiment of the present invention.

[0142] The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system library, and the kernel layer.

[0143] The application layer can include a series of application packages.

[0144] like Figure 6 As shown, the application package may include applications such as camera, note, call, recording, WLAN, short message, gallery, etc.

[0145] The note-taking application may provide an interface for user operation, on which a toolbar control panel, a sliding component (such as RecycleView) and a text component (such as TextView, EditTextView) may exist. The toolbar control panel may be a control for user operation, and the user may trigger the electronic device to perform relevant operations by triggering the relevant control. The text component may be arranged on the sliding component, and the text component may be used to display speech information.

[0146] Exemplarily, the note application may include a note recording module, a note speaker display module, a note data management module, and a text conversion engine.

[0147] The text-to-text engine is used to send the voice data or voice text obtained from the recording to the cloud, so that the cloud can perform text conversion on the voice data or voice text.

[0148] The recording module is used to obtain the voice data obtained by recording.

[0149] The note speaker display module is used to display the interface of the note application, such as a recording card, which may include the above-mentioned toolbar control panel, sliding component and text component. The toolbar control panel may be a tool control in the recording card, which is used to trigger the tablet computer to perform corresponding operations, such as recording, stopping recording, text conversion, displaying speaker status switching, exporting, sharing, filtering speakers, etc. The recording card may also include speech information, etc.

[0150] The note data management module is used to store the speech information in a preset storage location.

[0151] It should be understood that the above-mentioned note recording module, note speaker display module, note data management module, and text conversion engine may be internal implementations of the note application, wherein the text conversion engine is a software development kit (SDK).

[0152] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0153] like Figure 6 As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0154] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0155] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0156] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0157] The phone manager is used to provide communication functions of the electronic device 100, such as management of call status (including connecting, hanging up, etc.).

[0158] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0159] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.

[0160] Android Runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.

[0161] The core library consists of two parts: one is the function that needs to be called by the Java language, and the other is the Android core library.

[0162] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0163] The system library may include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0164] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0165] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0166] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0167] A 2D graphics engine is a drawing engine for 2D drawings.

[0168] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

[0169] Understandably, Figure 6 The layers in the structure shown and the components included in each layer do not constitute a specific limitation on the electronic device 100, i.e., a foldable device. In other embodiments of the present application, the structure may include more or fewer layers than shown in the figure, and each layer may include more or fewer components, which is not limited in the present application.

[0170] The present application provides a speech processing method. After starting recording, the electronic device can convert voice data into text in real time. After receiving the click operation of the user input on the speech information, the electronic device can determine the user's use intention according to the click position corresponding to the click operation and whether the electronic device is playing the recording, so as to enter the editing state according to the user's use intention to meet the user's editing needs for the speech information, or perform linkage playback of the recording to achieve accurate playback of the recording. After receiving the long press operation of the user input on different types of speech information located in different rows, all the speech information displayed on the interface is displayed in plain text, and the characters selected by the user in the speech information displayed in plain text are selected and set to meet the user's use needs of selecting speech information across segments, so that the electronic device can process the speech information of different rows selected at the same time, and provide rich text services. Among them, the voice data can be the recording in the recording note scenario introduced above, the recording in the recording application scenario, the recording in the call recording scenario (or called voice file), etc. Of course, the voice data can also be other recording data generated in one scene.

[0171] The following will take the electronic device as a tablet computer and the application scenario as a recording and note taking scenario, that is, a note taking application as an example, and describe in detail a speech processing method provided in an embodiment of the present application in conjunction with the accompanying drawings. Figure 7 As shown, the method may include steps S301-S323.

[0172] S301. During the recording process, the tablet computer receives the latest voice data.

[0173] S302: The tablet computer converts the latest voice data into text to obtain the latest speech information.

[0174] Among them, the above-mentioned latest speech information includes the latest speaker identification, and the latest speech text corresponding to the latest speaker identification, the latest speech start time and the latest speech start time. The latest speech text includes at least one character, which can be a text, punctuation mark, letter, etc. The latest speech start time indicates the time when the speaker starts speaking. For example, a speech information includes speaker 1, 00:40 and "Do you bring fruit?". Among them, speaker 1 is the speaker identification, 00:40 is the speech start time, and "Do you bring fruit?" is the speech text. Accordingly, the speech information indicates the speaker's speech at 00:40, and the speech content is "Do you bring fruit?"

[0175] The duration indicates the time during which the speaker corresponding to the speaker ID speaks continuously. The text content can also be called speech text.

[0176] In the embodiments of the present application, the above text conversion may include speech recognition and voiceprint recognition. During the process of editing a note (such as the first note), the tablet computer may record audio according to the user's usage requirements. During the recording, after the tablet computer receives the speech data of the speaker (such as the latest speech data), it performs speech recognition on the latest speech data to convert the speech into text and obtain the latest speech text. Moreover, in order to enable the user to more intuitively know the speech situations of different speakers, the tablet computer may perform voiceprint recognition on the speech data to distinguish the speech situations of different speakers, so as to obtain the latest speaker identifier corresponding to the latest speech text. Of course, the tablet computer may also request the server to perform speech recognition and voiceprint recognition on the above speech data, and the embodiments of the present application do not impose any restrictions on this.

[0177] In some embodiments, after obtaining the speech information, the tablet computer may save the speech information at a preset storage location (such as a location capable of storing data like a database (DB), etc.). Exemplarily, the storage format of a piece of speech information may be as shown in Table 1. Among them, Offset represents the speech start time, Content represents the speech text, Speaker represents the speaker identifier, and Duration represents the duration.

[0178] Table 1

[0179] Column Name type Chinese description Data Description Uuid TEXT The id value of the speech information key value, unique id Offset TEXT Time offset, i.e. the start time of the speech Content TEXT Speech text Speaker TEXT Speaker ID, such as the speaker's name Duration TEXT Duration

[0180] In some embodiments, the information included in the above-introduced latest speech information is only an example, and the content included in the latest speech information may be set according to actual requirements. For example, the latest speech information may include the speaker identifier and the speech text, but does not include the speech start time and the duration.

[0181] S303. The tablet computer displays the latest speech text on Interface 1. Among them, historical speech texts are also displayed on Interface 1.

[0182] Among them, the historical speech texts are obtained based on the speech data received before the above latest speech data.

[0183] In the embodiments of the present application, after obtaining the latest speech information, the tablet computer may display the latest speech text in the latest speech information on Interface 1 (or referred to as the first interface). Interface 1 displays all the speech texts obtained during the recording period, realizing real-time conversion and display of the speech content. Among them, the speech texts corresponding to different speaker identifiers are displayed in segments. For example, during the recording period, the tablet computer first receives the word "to" spoken by Speaker 1, and this "to" is the latest speech data, and then "to" is displayed on Interface 1 (as Fig. 8A shown). After that, the tablet computer receives the word "day" spoken by Speaker 1 and displays "day" on Interface 1 (as Figure 8B as shown). Here Fig. 8A the "today" shown can be regarded as the historical speech text, Figure 8B and the "day" shown can be regarded as the latest speech text. As the recording time increases, the tablet computer will sequentially convert the received voice data into text and display it on interface 1 (such as Figure 8C "It's a nice day today. Shall we go to the park together?"). After that, the tablet computer receives the latest voice data such as "Sure". The tablet computer performs text conversion on this latest voice data to obtain the latest speech text "Sure", and recognizes that the speaker identifier corresponding to the latest speech text is speaker 2. The tablet computer can display the latest speech text "Sure" in the next paragraph after the speech text of speaker 1 (such as Fig.8D as shown). Here, Fig.8D "It's a nice day today. Shall we go to the park together?" shown can be regarded as the historical speech text, and "Sure" is the latest speech text.

[0184] It should be understood that the example of the tablet computer introducing above that displays one character at a time is only an example of real-time display of speech text. The tablet computer can also display multiple characters at a time, and this application does not limit it.

[0185] In some embodiments, if the start speech time corresponding to the latest speech text differs greatly from the start speech time corresponding to the previous speech text, regardless of whether the speaker identifier corresponding to the latest speech text is the same as the speaker identifier corresponding to the previous speech text, the latest speech text may not be merged into the previous speech text. Of course, if the speaker identifier corresponding to the latest speech text is the same as the speaker identifier corresponding to the previous speech text, the tablet computer can directly merge the latest speech text into the previous speech text.

[0186] In some embodiments, the speech texts displayed on the above-mentioned interface 1 are arranged in the order of the start speech time from early to late. For example, as described above Fig.8D the start speech time corresponding to "It's a nice day today. Shall we go to the park together?" shown is earlier than the start speech time corresponding to "Sure".

[0187] In some embodiments, the above-mentioned interface 1 not only displays speech texts, but can also display the first information corresponding to the speech texts, and the first information includes the speaker identifier and / or the start speech time. For example, as Fig. 8E shown, interface 1 displays speech texts, speaker identifiers, and start speech times. Among them, Fig. 8E 00:00, 00:20, 00:40, 00:44 above are all start speech times, and speaker 1 and speaker 2 are both speaker identifiers.

[0188] Optionally, if the tablet computer is not in the speaker display state (i.e., the speaker control is not selected), the tablet computer may only display the speech text in the speech information, but not the first information in the speech information. If the tablet computer is in the speaker display state (i.e., the speaker control is selected), the tablet computer may not only display the speech text, but also display the speaker identification and / or the speech start time and other first information (such as the above Fig. 8E As shown, the tablet computer displays the speech text, speaker identification and speech start time).

[0189] In some embodiments, the tablet computer may not automatically display the speech content in real time, but after receiving the operation a input by the user, in response to the operation a, convert the voice data recorded before and / or after receiving the operation a into text. For example, the tablet computer may display the following Fig.8F The interface 50 shown in the figure includes a control 51, which can trigger the tablet computer to convert the voice data in the note into text. When the user wants to display the speech content corresponding to the voice data in text, the user can click the above Fig.8F The control 51 shown. Here, the click operation of the control 51 is the above-mentioned operation a.

[0190] Of course, the operation a described above is only an example, and the operation a can also be other types of operations. For example, the operation a can be a user inputting a first preset gesture on a relevant interface, such as inputting an S-shaped gesture on the above interface 50. The first preset gesture is used to trigger the tablet computer to convert voice data (or voice files) into text. The first preset gesture can be pre-configured in the tablet computer. In addition, the tablet computer can prompt the user on the relevant interface of the function triggered by the first preset gesture.

[0191] In the embodiment of the present application, when displaying speech information, the tablet computer differentiates and displays it according to different speakers, and the speech information of different speakers is located in different sections, which intuitively displays the speech status of different speakers.

[0192] S304: The tablet computer receives an operation to stop recording.

[0193] The stop recording operation can trigger the tablet computer to stop recording, that is, stop receiving voice data. Exemplarily, the stop recording operation can include the user stopping the recording control on the interface 1 (such as the above Fig. 8A Of course, the stop recording operation can also be other types of operations, as long as it can trigger the tablet computer to stop recording, and this application is not limited to it.

[0194] S305: In response to the stop recording operation, the tablet computer stops recording and displays interface 2. Interface 2 displays the speech text, play controls, and a progress bar.

[0195] The speech text displayed in interface 2 (or the second interface) includes all speech texts obtained during the current recording. The play control can trigger the tablet computer to play the voice data recorded during the recording. The progress bar can display the recording playback progress when the tablet computer is in the playback state. The playback state indicates that the tablet computer is playing the recording, that is, the state of playing the voice data recorded during the recording.

[0196] In some embodiments, when the tablet computer is not in the state of displaying the speaker, the interface 2 displays the speech text in the speech information, but does not display other information in the speech information, such as the speaker identification, the speech start time, etc. Fig. 9A As shown. Among them, Fig. 9A The interface shown (ie, interface 2) displays a play control 53 and a progress bar 54. In addition, the second interface may also include a start recording control.

[0197] When the tablet computer is in the state of displaying the speaker, the above interface 2 not only displays the speech text in the speech information, but also displays the first information in the speech information, where the first information includes the speaker identification and / or the speech start time. If the first information includes the speaker identification and the speech start time, the tablet computer displays Fig. 9B The interface 2 shown includes a play control 53, a progress bar 54, a speech text, a speech start time and a speaker identification.

[0198] S306: The tablet computer receives operation 1, where operation 1 is used to trigger switching to display speaker status.

[0199] In some embodiments, when the tablet computer is in the state of displaying a speaker, upon receiving a click operation of the user on the interface display of the tablet computer to stop displaying a speaker control, the tablet computer can, in response to the click operation, stop being in the state of displaying a speaker, that is, only display the speech text in the speech information, and cancel the display of information other than the speech text in the speech information, such as canceling the display of the first information and only displaying the speech text (such as the above Fig. 9A shown).

[0200] When the tablet computer is not in the display speaker state, upon receiving a user click operation on the display speaker control displayed on the interface of the tablet computer (such as the above interface 1 or interface 2), the tablet computer can switch to the display speaker state in response to the click operation, that is, display not only the speech text but also the first information (such as the above interface 1 or interface 2). Fig. 9BAs shown), the speaker status is switched. Here, the user's click operation on the display speaker control is operation 1.

[0201] It should be noted that the above-mentioned stop display speaker control and the above-mentioned display speaker control can be different controls or the same control. Specifically, the tablet computer only displays the speech text in the speech information, and the control is in the display speaker state. The control can be used as the stop display speaker control, as described above. Fig. 9A The control for displaying speakers shown is in a state of stopping displaying speakers (i.e., not selected), and the control can be used as a control for displaying speakers. The tablet computer displays the speech text and the first information of the speech information, and the control is in a state of displaying speakers (i.e., selected), and the control can be used as a control for stopping displaying speakers, as described above. Fig. 9B The Show Speaker control shown is in the Show Speaker state and can be used as a Stop Show Speaker control.

[0202] S307: In response to operation 1, the tablet computer displays interface 3. Interface 3 displays the speech text, the first information, the play control and the progress bar. The first information includes the speaker identification and / or the speech start time.

[0203] In some embodiments, when the tablet computer is in the display speaker state, when receiving a stop recording operation, the tablet computer does not need to receive operation 1 to display the above-mentioned interface 3, but can directly display interface 3 in response to the stop recording operation, that is, the above-mentioned interface 2 can be used as interface 3.

[0204] For the convenience of description, the above-mentioned first information is limited to the first information including the speaker identification and the speech start time. Accordingly, when the tablet computer is in the speaker display state, the displayed speech information includes the speech text, the speech start time and the speaker identification. However, it should be understood that the first information including the speaker identification and the speech start time is only limited for the convenience of description, and the first information only needs to include at least one of the speaker identification and the speech start time.

[0205] In some embodiments, the multiple speech information (i.e., speech text and first information) displayed in the above interface 3 can be carried by the RecycleView component. Among them, a speech information (i.e., speech text and first information) is an item (or called Adapter item) of the RecycleView component. The layout style of the item can be as follows: Fig. 10A As shown, two sub-layouts are included, one sub-layout is a linear layout control (LinearLayout), and the other sub-layout is an edit box (EditText).

[0206] The child controls (such as TextView) in LinearLayout will be arranged in a horizontal or vertical manner. The child controls in this application are arranged in a horizontal manner. The child controls are used to carry the first information. For example, one child control carries the speaker ID, and another child control carries the speech start time. The speech start time and the speaker ID are located in the same line (or the same paragraph) and are arranged in a horizontal manner (such as Fig. 10C As shown). Optionally, the speech start time and the speaker ID can be carried by other types of view components (such as TextView).

[0207] EditText carries the speech text (as mentioned above Fig. 10C As shown). Optionally, the speech text may also be carried by TextView.

[0208] In some embodiments, each item of the RecycleView component may further include a speaker avatar, which may be displayed on the left side of the speech start time. The speaker avatar may be carried by imgview, that is, the LinearLayout of the item may further include a child control corresponding to the speaker avatar, such as Fig. 10B shown.

[0209] S308: The tablet computer receives a click operation input by the user.

[0210] S309: In response to the above click operation, the tablet computer obtains a click position corresponding to the click operation.

[0211] In an embodiment of the present application, when the tablet computer receives a click operation of the user on the speech information on the interface 3, the tablet computer obtains the click position corresponding to the click operation, that is, obtains the position where the user's finger falls, so that the tablet computer can use the click position to determine the user's usage intention.

[0212] S310: When the click position is on the speech text, the tablet computer determines whether it is in a playing state.

[0213] Exemplarily, the play state indicates that the tablet computer is in a state of playing voice data. When the tablet computer is at a speech text when the click position is, it indicates that the user has clicked on the speech text, and the tablet computer can determine whether the tablet computer is in a play state to determine the user's usage intention. Simply put, the tablet computer determines whether the user wants to edit the clicked speech text or displays the time corresponding to the clicked speech text by determining whether the tablet computer is in a play state. Specifically, when the tablet computer is not in a play state, it indicates that the tablet computer is not currently playing a recording, and the user is more likely to want to edit the clicked speech text, and the tablet computer can execute S311. When the tablet computer is in a play state, it indicates that the tablet computer is currently playing a recording, and the user is more likely to want the tablet computer to play the voice data corresponding to the clicked speech text, and the tablet computer can execute S312.

[0214] It should be noted that the tablet computer may be externally connected to a mouse, so the above-mentioned click position may be the position where the mouse is clicked.

[0215] S311. The tablet computer enters the editing state.

[0216] In the embodiment of the present application, after the tablet computer enters the editing state, it can receive the user input for editing the clicked speech text (or referred to as the first target speech text). In response to the user's editing operation on the first target speech text, such as deleting, modifying, adding characters, etc., the first target speech text is processed accordingly. For example, when the tablet computer is not in the playing state, the display is as follows: Fig.11A The virtual keyboard shown in FIG. 1 is provided so that the user can use the virtual keyboard to edit the speech text. In addition, the tablet can be connected to an external keyboard (such as Fig. 11B The tablet computer can enter the editing state without displaying the virtual keyboard.

[0217] In some embodiments, the tablet computer entering the editing state may include the tablet computer moving the cursor to the click position (as described above). Fig.11A or Fig. 11B The position of the cursor shown is the position of the click operation input by the user), or moves to any character in the first target speech text, which is not limited by the present application.

[0218] In some embodiments, in response to the user's editing operation on the first target speech text, the tablet computer processes the first target speech text (or the original speech text) to obtain a processed speech text, and displays the processed speech text. The content of the first target speech text in the above-mentioned preset storage location is also the processed speech text. For example, assuming that speech text 1 is "OK, can I bring anything?", the user clicks on speech text 1 (such as Fig.12As shown in (a) below, this speech text 1 is the first target speech text. When the tablet computer responds to a click operation on the speech text 1 and is not in the playing state, the tablet computer displays a virtual keyboard (as shown in Fig.12 below in (b)). After that, the user deletes the character 'ke' in the speech text 1 through the virtual keyboard, that is, inputs a deletion operation (such as clicking the deletion key shown in Fig.12 below in (b)). The tablet computer responds to this deletion operation and deletes the character 'ke'. The processed speech text 1 then becomes 'yi, yong dai shen me dong xi ma?'. The tablet computer can display this processed speech text 1 instead of the original speech text 1 (as shown in Fig.12 below in (c)). Correspondingly, the tablet computer also deletes the 'ke' in the speech text 1 at the preset storage location, such as Fig.13 'ke yi, yong dai shen me dong xi ma?' shown in (a) below is the original speech text of the speech text 1. The deleted speech text 1 can be as shown in Fig.13 below in (b).

[0219] In some embodiments, the above preset storage location can be the target data list text as shown in the above Fig.13 in (a) or (b). The target data list file includes the above speech information, where Speaker represents the speaker identifier, Offset represents the speech time, and Content represents the speech content.

[0220] In some embodiments, the above database can store the above target data list file. Correspondingly, the tablet computer can directly query the target data list file from the database to obtain the speech information corresponding to the target speaker.

[0221] In other embodiments, the above database can store the save path corresponding to the target data list file. Correspondingly, the tablet computer queries the save path corresponding to the target data list file from the database for querying the target data list file using this save path.

[0222] In some embodiments, the above preset storage location (such as the above target data list) may further include a duration.

[0223] In some embodiments, if the speech text is carried by TextView, when the tablet computer is in the playing state, it can convert the TextView corresponding to the first target speech text into EditText so that the user can edit the speech text in the EditText.

[0224] S312. The tablet computer calculates the speech time corresponding to the clicked character based on the click position. The clicked character is the character corresponding to the click position.

[0225] S313: The tablet computer moves the playing progress of the progress bar to the speaking time, and plays the voice data corresponding to the speaking time.

[0226] In the embodiment of the present application, when the tablet computer is in a playback state, the voice data that the user wants to play is determined according to the position of the speech text clicked by the user (i.e., the click position). The tablet computer can use the click position and the duration corresponding to the speech text corresponding to the click position (i.e., the first target speech text) to determine the offset time of the click position relative to the speech start time corresponding to the first target speech text. Among them, the duration corresponding to the first target speech text is the duration in the speech information to which the first target speech text belongs, and the speech start time corresponding to the first target speech text is the speech start time in the speech information to which the first target speech text belongs.

[0227] Afterwards, the tablet computer determines the speech time (or first speech time) corresponding to the clicked position based on the speech start time and the offset time, that is, calculates the sum of the speech start time and the offset time, that is, determines the speech time corresponding to the clicked character (referred to as the clicked character). Afterwards, the tablet computer moves the playback progress of the progress bar to the position corresponding to the speech time, and plays the voice data corresponding to the speech time, that is, starts playing the voice data from the speech time, realizes the linkage playback of the recording and the speech text, and realizes the accurate playback of the voice data, so that the user can quickly hear the voice data he needs, and meets the user's voice playback needs.

[0228] In some embodiments, the specific calculation process of the above-mentioned offset time may include: the tablet computer may obtain the total number of all characters of the first target speech text. Then, the tablet computer may calculate the ratio of the total number of all characters to the duration corresponding to the first target speech text to obtain the speech speed. Then, the tablet computer may count the number of characters before the click position in the first target speech text to obtain the number of offset characters. Then, the tablet computer may calculate the product between the number of offset characters and the speech speed to obtain the offset time corresponding to the click position.

[0229] For example, if a user clicks Fig.14The "water" in "Do you bring any fruit?" shown in (a) above. The tablet computer is in the playback state, and the tablet computer calculates that the speaking time corresponding to the "water" is 00:40, and the duration is 4 seconds. Therefore, the speaking speed is 4 / 4=1 character / second, that is, 1 character is spoken per second. Afterwards, the number of characters before "water" in "Do you bring any fruit?" is 1, that is, the number of offset characters is 1. Correspondingly, the offset time is 1 character / second*1=1 second. Then, the speaking time corresponding to the click operation is 00:41, and the tablet computer moves the playback progress of the progress bar to the 00:41 position (such as Fig.14 As shown in (b) in the figure, and starts playing the voice data from 00:41, the user can quickly hear the voice data at 00:41 without the user manually moving the playback progress, thereby avoiding the user manually adjusting the playback progress resulting in inaccurate playback progress adjustment and the need for repeated adjustment, thereby achieving fast and accurate adjustment of the playback progress and meeting user needs.

[0230] In some embodiments, after calculating the speaking time, the tablet computer may not move the playing progress of the progress bar to the speaking time, but only play the voice data corresponding to the speaking time.

[0231] The above describes the case where the click position corresponding to the click operation is located at the speech text, that is, the user clicks the speech text. The following will continue to describe the case where the click position corresponding to the click operation is located at the first information, that is, the user clicks the first information.

[0232] S314: When the click position is at the first information, the tablet computer determines whether it is in a playing state.

[0233] In an embodiment of the present application, when the tablet computer is at a first information (such as a speaker identification) at the click position, it indicates that the user has clicked on the first information, and the tablet computer can determine whether the tablet computer is in a playing state to determine the user's usage intention. Simply put, the tablet computer determines whether the user wants to edit the clicked first information or displays the progress bar to move to the speech start time corresponding to the first information by determining whether the tablet computer is in a playing state. Specifically, when the tablet computer is not in a playing state, it indicates that the tablet computer is not currently playing a recording, and the user is more likely to want to edit the first information, and the tablet computer can execute S315. When the tablet computer is in a playing state, it indicates that the tablet computer is currently playing a recording, and the user is more likely to want the tablet computer to play the voice data corresponding to the first information, and the tablet computer can execute S316.

[0234] S315. The tablet computer displays a speaker modification control.

[0235] The speaker modification control is used to obtain a new speaker ID input by the user.

[0236] For example, generally speaking, the speech start time cannot be changed. Therefore, no matter the user clicks the speaker ID (or the original speaker ID) or the speech start time, the tablet computer, when not in the playback state, will assume that the user wants to modify the speaker ID, and the tablet computer can enter the editing state and display the speaker modification control (or rename control). Afterwards, the tablet computer can obtain the new speaker ID entered by the user in the speaker modification control, and modify the first target speaker ID (i.e., the speaker ID in the first information clicked by the user) to the new speaker ID.

[0237] Modifying the first target speaker identifier to the new speaker identifier may include modifying the displayed first target speaker identifier to the new speaker identifier, and / or modifying the first target speaker identifier in a preset storage location to the new speaker identifier.

[0238] For example, if Fig.15A As shown in FIG. 1 , the user clicks on the speaker identifier (i.e., speaker 1) corresponding to the speech text “The weather is great today, shall we go to the park together?” Afterwards, the tablet computer responds to the click operation on speaker 1 and displays the speaker modification controls (i.e., Fig. 15B ). Speaker 1 is the first target speaker ID. Then, the user enters Xiao Ming (such as Fig. 15C As shown in the figure), that is, a new speaker ID is input, and the tablet computer can change the speaker ID corresponding to the speech text "The weather is good today, shall we go to the park together?" to Xiao Ming (as shown in the figure). Fig.15D As shown), and the first target speaker identifier in the preset storage location is changed to Xiao Ming.

[0239] In some embodiments, when the user wants to modify the speaker ID in all speech information that is the same as the first target speaker ID to the new speaker ID, the user can click the Fig.15E In response to the click operation of the all-modify control 70, the tablet computer modifies the speaker identifiers in all speech information that are the same as the first target speaker identifier to the above-mentioned new speaker identifier, such as modifying the speaker identifier whose displayed value is the first target speech identifier to the new speaker identifier (such as Fig.15F As shown), and the first target speaker identifier in the preset storage location is modified to a new speaker identifier to achieve consistent modification of the information.

[0240] In the embodiment of the present application, since there is a probability of conversion error in speech-to-text conversion, such as the probability of text conversion error, the user can edit the speech text by clicking on the speech text when the tablet computer is not in the playing state, so as to correct the speech text and improve the accuracy of text conversion; speech-to-text conversion also has voiceprint recognition errors, such as recognizing the speech content of a speaker as the speech content of another person. The user can edit the speaker logo by clicking on the speaker logo or the speech start time when the tablet computer is in the playing state to modify the speaker logo, thereby realizing the correction of the speaker logo and improving the accuracy of text conversion. Of course, the user can also edit the speech information based on needs. The note application provides users with rich services to meet the user's usage needs and improve the user experience.

[0241] S316: The tablet computer moves the playing progress of the progress bar to the target speech start time, and plays the voice data corresponding to the target speech start time, wherein the target speech start time is the speech start time in the speech information to which the clicked position belongs.

[0242] In some embodiments, when the tablet computer is in the playback state, the tablet computer directly plays the voice data corresponding to the speech start time (i.e., the target speech time) in the speech information to which the first information clicked by the user belongs, that is, the voice data is played directly from the target speech start time, realizing the linkage playback of the recording and the speech text, and realizing the fast and accurate playback of the voice data, so that the user can quickly hear the voice data he needs, meeting the user's voice playback needs, without the user manually adjusting the playback progress, and improving the user experience. For example, when the user clicks as above Fig.14 In (a), the speaker 2 at 00:20 is a speaker identifier. The tablet computer is in a playing state, and the playing progress of the progress bar is directly moved to 00:20, and the voice data continues to be played from 00:20.

[0243] In some embodiments, when receiving a click operation, if the click position corresponding to the click operation is not in the preset valid area, the tablet computer may consider the click operation to be invalid, and the tablet computer may not respond to the click operation, that is, not process it. If the click position corresponding to the click operation is in the preset valid area, the tablet computer may consider the click operation to be valid, and the tablet computer may respond to the click operation and perform corresponding processing based on the click position and the playback state.

[0244] The above-mentioned preset effective area can be set according to requirements, for example, the preset effective area is a blank area on the recording card.

[0245] The above describes the process of the tablet computer processing based on the click operation after receiving the user's click operation on the speech information displayed on the tablet computer. The tablet computer can also select different types of information located in different segments displayed on the tablet computer based on the long press operation input by the user to achieve cross-segment selection of speech information. The following will continue to describe the process of the tablet computer processing based on this scenario operation.

[0246] S317: The tablet computer receives a long press operation input by the user.

[0247] S318. In response to the long press operation, the tablet computer obtains a start position and an end position corresponding to the long press operation.

[0248] The starting position indicates the position where the user's finger falls, and the ending position indicates the position where the user's finger is lifted. Exemplarily, the starting position and the ending position may be represented by (x, y) coordinates. Exemplarily, the user's finger falling may indicate that the user's finger falls on the display screen of the tablet computer, or the user's finger presses the mouse (such as the left mouse button). The user's finger lifting may indicate that the user's finger leaves the display screen of the tablet computer, or the user's finger leaves the mouse.

[0249] S319: The tablet computer concatenates characters from the first speech character displayed on interface 3 to the last speech character to obtain a concatenated character string. The first speech character refers to the first character in the first speech information displayed on interface 3. The last speech character refers to the last character in the last speech information displayed on interface 3.

[0250] Exemplarily, the above characters include words, punctuation marks (such as commas, question marks, periods, etc.) and spaces.

[0251] In the embodiment of the present application, the tablet computer concatenates the characters in all speech information displayed on the interface 3 to obtain a concatenated string, that is, to obtain concatenated speech information, which is in plain text form. Fig.16A as well as Fig. 16BAs shown, interface 3 displays 5 speech information. The tablet computer splices the displayed speech information one by one. First, the tablet computer splices the content of the first (or first) speech information to obtain a spliced ​​string, which is 00:00+space+speaker 1+the weather is very good today, shall we go to the park together? . Among them, the space refers to the control between the speech time and the speaker identifier. After that, the second speech information is spliced ​​to the spliced ​​string, and the new spliced ​​string is 00:00+space+speaker 1+the weather is very good today, shall we go to the park together? +00:20+space+speaker 2+yes, do I need to bring anything? . And so on, until the last character in the fifth speech information is spliced ​​and the splicing is stopped, so that all the speech information is spliced.

[0252] In some embodiments, in order to make the format of the spliced ​​character string correspond to the speech information displayed on interface 3, the tablet computer can also, during the splicing process, if the currently spliced ​​character and the previous character are in different lines, the tablet computer can splice a line break character (such as '\n') before the currently spliced ​​character, so that when the tablet computer displays the spliced ​​character string, the user cannot perceive the difference between the speech information displayed on interface 3, thereby achieving seamless switching display of the speech information.

[0253] S320: The tablet computer counts the number of characters starting from the first spoken character and ending at the starting position to obtain an offset character position, wherein the offset character position indicates the position of the first selected character relative to the concatenated character string.

[0254] In an embodiment of the present application, the tablet computer is in a state of displaying a speaker, and a speech information displayed by the tablet computer includes a speech text and a first message. The speech text is displayed after the first message, and can usually be displayed in the next line of the first message (or referred to as the next paragraph). The tablet computer determines the item to which the starting position corresponding to the received long press operation belongs, that is, determines the item to which it belongs, that is, determines the speech information to which the starting position belongs (or is described as the first speech information).

[0255] In the above-mentioned speech information splicing process, when the tablet computer splices to the first speech information, the tablet computer can continue splicing from the first character in the first speech information. Afterwards, when splicing to the character corresponding to the starting position corresponding to the long press operation, the tablet computer can count the number of spliced ​​characters, obtain the position of the first character selected by the user in all speech information, that is, obtain the offset character position. And the tablet computer can continue to splice the remaining characters, that is, the characters after the character corresponding to the starting position.

[0256] For example, the character corresponding to the starting position of the long - press operation (i.e., the selection operation) input by the user is "可" in the second speech message as shown in Fig.17A When the tablet computer splices the second speech message, the spliced string obtained is 00:00 + space + Speaker 1 + Is the weather nice today? Let's go to the park together?. Then, the tablet computer continues to splice the characters in the second speech message. When it splices to the character "可", the currently spliced string is 00:00 + space + Speaker 1 + Is the weather nice today? Let's go to the park together? + 00:20 + space + Speaker 2 + 可. The tablet computer counts the number of characters included in the currently spliced string, that is, the number of characters that have been spliced is 00:00 (5 characters) + space (1 character) + Speaker 1 (4 characters) + Is the weather nice today? Let's go to the park together? (17 characters) + 00:20 (5 characters) + space (1 character) + Speaker 2 (4 characters) + 可 (1 character) = 38, that is, the offset character position is 38.

[0257] S321. The tablet computer determines the number of selected characters based on the starting position and the ending position corresponding to the long - press operation.

[0258] Exemplarily, the tablet computer counts the number of characters in the area from the starting position to the ending position to obtain the number of selected characters. Specifically, the tablet computer determines the character corresponding to the starting position as the starting character and determines the character corresponding to the ending position as the ending character. Then, the tablet computer counts the number of characters between the starting character and the ending character to obtain the number of selected characters. Among them, the characters between the starting character and the ending character include the starting character, the ending character, and the characters after the starting character and before the ending character.

[0259] Continuing the above example, the character corresponding to the ending position of the long - press operation input by the user is "吗" in the third speech message as shown in Fig. 17B Correspondingly, the characters between the starting character and the ending character are "Sure. What should we bring? (11 characters) + 00:40 (5 characters) + space (1 character) + Speaker 1 (4 characters) + Bring fruits? (4 characters)", and the number of selected characters is 25.

[0260] S322. The tablet computer displays the above - spliced string and sets the target character area in the spliced string to the selected state. Among them, the target character area includes the character corresponding to the offset character position and the number of selected characters after the offset character position.

[0261] In an embodiment of the present application, the tablet computer displays the final spliced ​​string (that is, the spliced ​​string is displayed in the recording card), and all speech information is displayed in plain text, so that the tablet computer can select different segments of speech content. The tablet computer displays the target character area in the spliced ​​string according to the preset selection style, so that the characters in the target character area are in a selected state (as shown in 17C), that is, the target character area in the spliced ​​string in the recording card is selected. Among them, optionally, the preset selection style can be a highlighted display, and the color of the selected character becomes darker than when the character is not selected. Of course, the user can also set the selection style according to the needs, as long as it is different from the style when the character is not selected.

[0262] In some embodiments, the above-mentioned speech information is carried by RecycleView, and each speech information is an item. Multiple items cannot be selected at the same time, that is, the tablet computer cannot select multiple speech information across items at the same time. The item includes multiple view components, and the first information and the speech text in the speech information belong to different view controls in the item. Similarly, the view component cannot be selected across rows, that is, information located in different segments (or different types of information in different rows) in a speech information cannot be selected at the same time. Therefore, in order to provide rich text services, after receiving a long press operation, indicating that the user may have a need to select speech information across segments, the tablet computer can replace the RecycleView component with a TextView component (such as Fig.17D As shown), that is, all speech information (i.e., the concatenated string) is located in a TextView component. In other words, all speech information is displayed in the TextView in plain text, allowing the user to select multiple speech information at a time (such as selecting speech information across speakers, that is, selecting speech information from different speakers), or selecting different types of information in a speech information (or describing it as information in different views).

[0263] Among them, from the above introduction, it can be known that the view component where the first information is located can be TextView, and the view component where the speech text is located can be EditView or TextView. Optionally, the speaker ID and the start speech time in the first information can correspond to a TextView respectively, or the speaker ID and the start speech time can correspond to a TextView together, which is not limited by this application.

[0264] S323: The tablet computer displays at least one function control, each of the at least one function control is used to trigger the tablet computer to perform a corresponding processing operation on the characters in the target character area.

[0265] Exemplarily, the above-mentioned function control may be a sharing control, a copy control, a translation control, a search control, a video search control or an application search control, etc., such as a tablet computer displaying Fig.17E Functional controls displayed in the form of bubbles. Fig.17E The application 1 search shown indicates searching in application 1 in the tablet computer using the selected characters as keywords.

[0266] The sharing control is used to trigger the tablet computer to share the selected characters, such as sharing them with an application on the tablet computer.

[0267] The copy control is used to trigger the tablet computer to copy the selected characters, so that the user can paste the selected characters to other locations as needed without the user having to re-enter the selected characters.

[0268] The above-mentioned translation control is used to trigger the tablet computer to translate the selected characters into another voice.

[0269] The search control is used to trigger the tablet computer to search using the selected characters as keywords, such as searching for the selected characters in a browser application.

[0270] The above video control is used in a video application of a tablet computer to search for related videos using the selected characters as keywords.

[0271] The above application search control is used for a tablet computer to search for related content in a specific application in the tablet computer using the selected characters as keywords.

[0272] It should be understood that the functional controls listed above are only examples, and the functional controls may also be other types of controls, as long as they can trigger the tablet computer to process text, and this application does not limit them.

[0273] It should be noted that, when the above interface 3 displays the speaker's avatar, the tablet computer will no longer display the speaker's avatar when displaying the speech information in plain text.

[0274] In some embodiments, after the tablet computer displays the above function control, when the above function control stops displaying, the tablet computer can replace the layout component of the speech information from TextView to RecycleView, that is, continue to display the above function control. Fig. 10A or Fig. 10BThe layout style shown displays the speech information, so that the tablet computer can continue to perform corresponding processing based on the operation input by the user, such as editing the speech content (such as speech text, speaker identification) or realizing the linkage playback of recording and text based on the click operation, and filtering the displayed speech information. For example, the tablet computer can stop displaying the function control after receiving the user's trigger operation (such as a click operation) on a preset area (such as a blank area) on the interface 3. For another example, the tablet computer can stop displaying the function control after receiving the user's click operation on a certain function control, and in response to the click operation on the function control, the characters in the target character area, that is, the characters selected by the user, are processed accordingly. For another example, after the tablet computer displays the function control for a certain period of time, if it still does not receive the user's click operation on the function control, it can stop displaying the function control.

[0275] It should be understood that compared with TextView, the tablet computer can more accurately obtain the position of the user input operation (such as the click position, the start position and end position corresponding to the long press operation, etc.) by carrying the speech information through RecycleView, thereby improving the accuracy of the processing based on the operation. In addition, since each speech information corresponds to an entry when RecycleView carries the speech information, the tablet computer can filter the displayed entries, that is, it can filter the displayed speech information.

[0276] In some embodiments, the tablet computer may filter and display the tablet computer may receive operation b of the user on the fourth interface, and operation b is used to trigger the tablet computer to display the speech information corresponding to the target speaker identifier selected by the user. Afterwards, in response to operation b, the tablet computer may only display the speech information corresponding to the target speaker identifier, and no longer display the speech information corresponding to the speaker identifiers other than the target speaker identifier. For example, if the user clicks on Fig.18A In response to a click operation on the filter speaker control, the tablet computer displays the following Fig.18B After receiving the selection operation of selecting the speaker 1 option in the speaker option 80, the tablet computer displays the following Fig.18C The speech information corresponding to the speaker 1 shown is used to filter the speaker text. It should be understood that Fig.18C The speech information shown is the speech information corresponding to the filtered speaker 1. Fig.18A As shown in the speech text, since the speech information corresponding to speaker 2 is no longer displayed, the speech information of speaker 1 displayed on the tablet increases, and the speech information corresponding to speaker 1 arranged at the bottom moves upward to be directly displayed without moving the scroll bar (such as Fig.18C "See you later" as shown).

[0277] It should be noted that the above 18A to 18C This article describes the process of filtering the speech text of the speaker when the tablet is in the speaker display state, that is, the process of displaying the speech text corresponding to the target speaker ID selected by the user. The tablet also supports filtering the speaker when it is not in the speaker display state, and only displays the speech text corresponding to the target speaker ID.

[0278] In this application, the tablet computer supports the speaker filtering function. Users can select the speaker's speech information they want to display according to their needs, and Ping An Computer can display the desired speaker's speech information, meeting the user's need to view only some of the speaker's speech information.

[0279] In some embodiments, when a user wants to select multiple speech information of a speaker at the same time, he can first select the speaker ID corresponding to the speaker (i.e., the target speaker ID), and the tablet computer will respond by displaying all speech information corresponding to the target speaker ID, so that the speech information of the speaker becomes adjacent speech information. Afterwards, the user can input a long press operation on all speech information corresponding to the displayed target speaker ID as required, so that the tablet computer can respond to the long press operation and select multiple speech information of a speaker at the same time, thereby processing the multiple speech information of a speaker accordingly to meet the user's usage requirements.

[0280] In some embodiments, the above introduction is about the processing process of receiving click operations and long press operations when the tablet computer is in the state of displaying the speaker. The tablet computer can also receive click operations and long press operations when it is not in the state of displaying the speaker. The processing process is similar to the processing process of the click operation and the processing process of the long press operation introduced above, except that after receiving the click operation, there is no need to determine whether the user clicks on the speech text or the first information.

[0281] In some embodiments, the tablet computer supports the export function of voice data and speech information. The tablet computer receives operation c, and in response to operation c, the tablet computer exports the speech information currently displayed on the tablet computer.

[0282] It should be understood that when the tablet computer is not in the state of displaying the speaker, the speech text in the speech information currently displayed on the tablet computer, then the tablet computer exports the currently displayed speech text in response to operation c. When the tablet computer is in the state of displaying the speaker, the tablet computer currently displays the entire content of the speech information, then the tablet computer exports the entire currently displayed speech information in response to operation c. For example, it is assumed that the first information includes the speaker ID and the speech time. When the tablet computer is in the state of displaying the speaker, the speech information currently displayed on the tablet computer includes the speech text, the speaker ID and the speech time, and the content exported by the tablet computer includes the speech text, the speaker ID and the speech time. When the tablet computer is not in the state of displaying the speaker, the speech information currently displayed on the tablet computer includes the speech text, and the content exported by the tablet computer includes the speech text.

[0283] In addition, if the speech information currently displayed on the tablet computer is the speech information corresponding to some speakers, then the exported speech information is also the speech information corresponding to these speakers. If the speech information currently displayed on the tablet computer is the speech information corresponding to all speakers, then the exported speech information is also the speech information corresponding to all speakers, thereby meeting various export needs of users.

[0284] Optionally, the tablet computer can support exporting in multiple formats, that is, based on the first target format, the speech information currently displayed on the tablet computer can be saved, that is, the speech information currently displayed in the first target format is saved to a target location, which can be a default save location or a storage location selected by the user. Exemplarily, the first target format can be a document format or a picture format. The document format can include at least one of txt format, pdf format, doc format, docx format, and html.

[0285] In one example, the first target format may be a default format (or referred to as a default file format). Fig.19A The interface 90 shown in the figure includes a control 91, which is used to trigger the tablet computer to save the currently displayed speech information. In response to the user clicking the control 91, the tablet computer saves the currently displayed speech information in a default format (such as txt format) to a target location. In addition, after saving, the tablet computer can also display location prompt information (such as Fig.19B The location prompt information 92 shown in the figure reminds the user of the location where the currently displayed speech information is saved, so as to facilitate the user to view it.

[0286] In another example, the first target format may be a format selected by the user (i.e., a file format). The tablet computer may display at least one first format option. After receiving a user's selection operation for a target format option in the at least one format option, in response to the selection operation for the target format option, the tablet computer may save the currently displayed speech information in a format corresponding to the target format option (i.e., the first target format). Fig.19A When the control 91 in the control is clicked, the tablet computer displays the following Fig.19C The tablet computer can save the currently displayed speech information in doc format. The tablet computer can also display location prompt information (such as the location of the address, the address of the text file, the image file, the html file, and the docx file) after saving. Fig.19D The location prompt information 93 shown in the figure prompts the user where the currently displayed speech information is stored, so as to facilitate the user to view it.

[0287] In other embodiments, the tablet computer also supports sharing the speech information currently displayed on the tablet computer based on the second target format in response to the received operation d, such as the tablet computer sharing the currently displayed speech information in the second target format to the application on the tablet computer.

[0288] The second target format is similar to the first target format, and can be a default format or a format selected by a user. The second target format will be described in detail below with reference to two examples.

[0289] In one example, the second target format is a default format. Exemplarily, in response to a sharing operation input by a user (such as a click operation on a sharing control), the tablet computer displays an interface including a third control, where the third control is used to trigger the tablet computer to display a first candidate application. Afterwards, in response to the user's selection operation of a first target application in the first candidate application, the currently displayed speech information in the second target format is shared to the first target application. The first candidate application refers to an application that can receive files in the second target format. For example, the tablet computer responds to a user's selection operation of Fig.19E The click operation of the sharing control 94 shown in FIG. Fig.19F The interface 95 displays the format of the currently displayed speech information (such as txt format), the speech information, the first candidate application 96, etc. The first candidate application 96 represents an application that can receive files in txt format, so that the user can select an application for sharing according to needs.

[0290] In another example, the second target format is selected by the user. When the user wants to share the currently displayed speech information in a picture format, the user can select the picture format sharing option displayed on the tablet computer (such as the above Fig.19A In response to the user selecting the picture format sharing option, the following is displayed: Figure 19G The interface shown includes the currently displayed speech information 97 in the picture format and the second candidate application 98. The second candidate application 98 represents an application that can receive files in the picture format for the user to select to share.

[0291] In some embodiments, Fig.19H As shown, the tablet computer can display the currently displayed speech information in the first target format or the second target format on the interface displaying the recording card (such as speech information, various controls, etc.).

[0292] In some embodiments, the tablet computer may also export or share the speech information of all speakers or some speakers according to the user's choice. For example, in response to the above-mentioned control 91, the tablet computer may display an export speaker option, which includes an export all option and at least one export speaker option. The export all option is used to trigger the tablet computer to export the speech information of all speakers. The export speaker option corresponds to the speaker identifier one by one, and the export speaker option is used to trigger the tablet computer to export the speech information of the speaker corresponding to the speaker option. Similarly, the tablet computer may also export or share the speech information according to the user's choice to determine whether there is content other than the speech text, such as the speaker identifier and the speech time.

[0293] In some embodiments, the tablet computer can also share the currently displayed speech information with other devices in a default or user-selected file format.

[0294] It should be understood that the above-mentioned sharing and exporting operations can be performed after the tablet computer stops recording. During recording, the tablet computer generally does not perform exporting and sharing operations on the first voice file or the speech information corresponding to the first voice file. However, operations such as displaying speaker status switching, filtering speaker speech information, displaying all speaker speech information, selecting and editing speech information, and linked playback of voice data can be performed during recording or after stopping recording.

[0295] The above describes the functions of the electronic device supporting the conversion of voice into speech information, responding to the user's click operation on the speech information, and responding to the shadow tiger's selection operation on the speech information. Figure 6The software framework shown in the figure introduces the specific implementation process of the electronic device to realize the function of converting voice into speech information, responding to the user's click operation on the speech information, and responding to the user's selection operation on the speech information. Among them, converting voice into speech information may include Fig. 20 S1-S9 shown.

[0296] The implementation process of responding to the user's click operation on the speech information may include the following steps: Fig. 20 The implementation process of responding to the user's click operation on the speech information may include the following steps: Fig. 20 S18-S28 shown.

[0297] S1. The note recording module receives the latest voice data.

[0298] S2. The note recording module sends the latest voice data to the text-to-text engine.

[0299] S3. The text-to-text engine sends the latest voice data to the voice cloud module.

[0300] S4. The voice cloud module converts the latest voice data into text to obtain the latest speech information corresponding to the latest voice data.

[0301] S5. The voice cloud module sends the latest speech information to the text conversion engine.

[0302] Exemplarily, the note recording module sends the latest voice data to the cloud (such as the voice cloud module) through the text conversion engine. The cloud converts the latest voice data into text to obtain a text recognition result, which includes the latest speech information corresponding to the latest voice data.

[0303] In some embodiments, the data structure of the latest speech information in the above text recognition result may be {speaker identification, speech start time, duration, converted text content}.

[0304] S6. The text conversion engine sends the latest speech information to the note recording module.

[0305] S7. The note recording module displays the latest speech information on the interface 1 through the note speaker display module. The interface 1 also displays historical speech information.

[0306] Exemplarily, after receiving the text recognition result, the note recording module can extract the speech information in the text recognition result, and split and display the speech information through the note speaker display module. That is, the speech information corresponding to different speakers is displayed through different segments to avoid displaying the speech information corresponding to different speakers in the same item.

[0307] In addition, the note recording module can also save the extracted speech information to a preset storage location. Exemplarily, the note recording module can save the extracted speech information to a target data list file through the note data management module.

[0308] S8. The note speaker display module receives a user click operation on the stop recording control on interface 1.

[0309] S9, in response to a click operation on the stop recording control, the note speaker display module triggers the note recording module to stop recording and displays interface 2. Interface 2 displays speech information, playback controls and a progress bar.

[0310] The above introduces the process of converting voice data into text in real time during recording, and stopping recording. The following will continue to introduce the process of the tablet computer responding to the user's click operation on the speech information displayed on the tablet computer after stopping recording, and the process of the tablet computer responding to the user's long press operation on the speech information displayed on the tablet computer.

[0311] S10, the parent layout in the note speaker display module monitors the user's click event on the child layout corresponding to the parent layout. The parent layout is used to carry a speech information, and the child layout corresponding to the parent layout includes a first child layout and a second child layout, the first child layout is used to carry the speech text, and the second child layout is used to carry the first information.

[0312] The parent layout corresponds to an item. The first child layout can be the Fig. 10A or Fig. 10B The second sub-layout can be the EditText shown above Fig. 10A or Fig. 10B The LinearLayout shown in the figure. The parent layout can intercept the click event of the child element (or sub-layout) of the item. After the sub-layout listens to the click event, it does not process the click event, but the parent layout to which the sub-layout belongs processes the click event.

[0313] In some embodiments, each item corresponds to a mark bit, and in the click event of the item, a response is processed according to the mark bit. That is, if a user clicks on a certain speech information, the mark bit corresponding to the item corresponding to the speech information is updated from the first mark to the second mark, so that the item corresponding to the speech information responds to the click event. The first mark is different from the second mark, such as the first mark is 0 and the second mark is 1.

[0314] S11. The parent layout responds to a click event and determines whether it is in a playing state when the click position is in the first child layout.

[0315] For example, after the parent layout monitors the click event, it indicates that the user has clicked on the speech information, and the parent layout obtains the position where the user clicked, that is, obtains the click position corresponding to the click event. In the case where the click position is in the first child layout corresponding to the parent layout, it indicates that the click is located in the speech text carried by the first child layout, and the parent layout can determine the user's intention to use based on the playback status of the recording. The playback status of the recording can also be determined by the status of the playback control on interface 2.

[0316] S12: When the parent layout is not in the playing state, the parent layout enters the editing state.

[0317] S13. In the playback state, the parent layout calculates the speaking time based on the click position.

[0318] S14. The parent layout moves the playing progress of the progress bar on the interface 2 to the speaking time, and plays the voice data corresponding to the speaking time.

[0319] The above describes the case where the click position corresponding to the click operation is located at the speech text, that is, the user clicks the speech text. The following will continue to describe the case where the click position corresponding to the click operation is located at the first information, that is, the user clicks the first information.

[0320] S15. When the click position is in the second sub-layout, determine whether it is in a playing state.

[0321] S16. When not in playback state, the parent layout displays speaker modification controls.

[0322] S17: In the case of being in the playing state, the parent layout moves the playing progress of the progress bar on the interface 2 to the target speech start time, and plays the voice data corresponding to the target speech start time. The target speech start time is the speech start time in the speech information to which the click position belongs.

[0323] The above describes the process of the tablet receiving a user's click operation on the speech information displayed on the tablet and performing processing based on the click operation. The user can also perform a selection operation on at least one speech information displayed on the tablet, and the following will continue to describe the process of the tablet performing processing based on the selection operation.

[0324] S18. The note spokesperson display module detects a long press event on the first sub-layout.

[0325] S19. The note spokesperson display module intercepts the default function control corresponding to the first sub-layout.

[0326] S20. The note spokesperson display module obtains the location information of the first item belonging to the first sub-layout.

[0327] S21. The note speaker display module obtains the starting position and the ending position corresponding to the long press event.

[0328] Exemplarily, an OnTouchListener event listener is set for EditText. Both pressing and releasing the touchpad trigger the event. The position where the finger falls and the position where the finger leaves each time the EditText is clicked are recorded. When the click duration is long, the click operation is identified as a long-press event, thereby implementing the monitoring of the long-press event. Accordingly, after the user's finger falls to trigger the event, the position where the user's finger falls is recorded to obtain the starting position corresponding to the long-press event. Similarly, after the user's finger leaves the touchpad to trigger the event, the position where the user's finger leaves is recorded to obtain the ending position corresponding to the long-press event.

[0329] The position information of the first item indicates which speech information the first item is displayed in. The note speaker display module can obtain the position information of the first item in the onclick of the first item.

[0330] In some embodiments, a component corresponding to the first information, such as TextView, may also monitor long press events, such as setting a listener for click events for TextView to implement long press event monitoring.

[0331] S22, the note speaker display module concatenates the characters from the first speech character displayed on interface 2 to the last speech character to obtain a concatenated string. The first speech character refers to the first character in the first speech information displayed on interface 2. The last speech character refers to the last character in the last speech information displayed on interface 2.

[0332] S23. The note speaker display module counts the number of characters starting from the first speaking character and ending at the starting position to obtain the offset character position.

[0333] Exemplarily, the offset character position may represent the starting index of the currently selected character, that is, the position of the first selected character.

[0334] S24. The note spokesperson display module determines the number of selected characters based on the first item, the starting position and the ending position corresponding to the long press event.

[0335] S25. The note speaker display module replaces the RecycleView component with the TextView component.

[0336] S26. The note speaker display module displays the concatenated string in the above TextView component.

[0337] S27, the note speaker display module sets the target character area in the TextView component to a selected state, wherein the target character area includes characters corresponding to the offset character position and characters of the selected character number after the offset character position.

[0338] S28. The note speaker display module displays a function control, which is used to trigger the tablet computer to perform corresponding processing operations on the selected characters.

[0339] In some embodiments, the note-taking application also provides a toolbar control panel, which may be a tool control in the recording card. Fig. 17C The toolbar control panel 73 shown is used to trigger the tablet computer to perform corresponding operations, such as recording, stopping recording, text conversion, displaying speaker status switching, exporting, sharing, filtering speakers, etc. Fig. 17C The recording card 74) shown may also include speech information, etc. Fig.21 As shown, when the user clicks on the recording control, the toolbar control panel responds to the click operation of the recording control, triggering the note recording module to record to collect the corresponding voice data. Correspondingly, the tablet computer supports automatic real-time voice-to-text conversion, so the note recording module can send voice data to the voice-to-text service (or voice cloud module). After that, the voice-to-text service can send it to the note application after the conversion is successful. After obtaining the speech information, the note application can display it on the RecycleView component, and store the data through the data management module to realize the storage of the speech information.

[0340] Among them, the speech-to-text service can be as described above Fig.21 As shown, the voice-to-text service is located in the cloud, or the speech-to-text service is located locally on the tablet computer, and this application does not limit it.

[0341] It should be noted that the above Fig. 17C The tool controls included in the toolbar control panel 73 shown are merely examples and are not limited to them in the present application.

[0342] In some embodiments, the note speaker display module receives a request to save the control (as described above). Fig. 17C The click operation of the save control 76 shown can save the currently edited note through the note data management module, thereby saving the relevant data (such as speech information) of the recording card displayed on the current interface.

[0343] In some embodiments, the present application provides a computer-readable storage medium, including computer instructions, which, when executed on an electronic device, enable the electronic device to perform the speech processing method as described above.

[0344] In some embodiments, the present application provides a computer program product, which, when executed on an electronic device, enables the electronic device to execute the speech processing method as described above.

[0345] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0346] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0347] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0348] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0349] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0350] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application shall be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A speech processing method, characterized in that: include: When the electronic device displays the first interface of the first note in the note application, the electronic device converts the acquired latest voice data into corresponding latest speech information, the first interface includes a first recording card, the first recording card includes a first control; the first control is used to trigger the stop of recording; The electronic device displays the latest speech information in the first recording card; In response to the triggering operation of the first control, the electronic device displays a second interface of the first note, the second interface includes a second recording card; wherein the second recording card includes a plurality of speech information and a second control; the second control is used to trigger recording; the speech information includes at least a speech text; the second control is used to trigger recording; In response to a long press operation input by the user, the electronic device updates the second recording card to a third recording card, in which a target character area in the spliced ​​character string is selected; wherein the spliced ​​character is obtained by splicing the multiple speech information; the target character area is determined based on a starting position and an ending position corresponding to the long press operation, and the target character area includes at least two speech information among the multiple speech information or one speech information among the multiple speech information in different rows; The electronic device displays at least one function control, and the function control is used to trigger the electronic device to perform corresponding processing on the characters in the target character area.

2. The method according to claim 1, characterized in that The target character area includes characters corresponding to the offset character position and characters of the selected character number after the offset character position; The offset character position indicates the number of characters starting from the first character in the plurality of speech information and ending at the starting position corresponding to the long press operation; the offset character position indicates the position of the first selected character relative to the concatenated character string; The number of selected characters is determined based on the starting position and the ending position corresponding to the long press operation.

3. The method according to claim 1 or 2, characterized in that: The concatenated character string is concatenated from the first character in the plurality of speech information to the last character in the plurality of speech information.

4. The method according to any one of claims 1 to 3, characterized in that The second recording card includes a sliding component, and the plurality of speech information is carried by the sliding component; The electronic device updates the second recording card to a third recording card, including: The electronic device replaces the sliding component in the second recording card with a preset text component; The electronic device displays the spliced ​​character string in the preset text component to obtain the third recording card.

5. The method according to claim 4, characterized in that After displaying at least one functional control, the method further includes: When the function control stops displaying, the electronic device replaces the preset text component of the third recording card with the sliding component, and the sliding component carries the multiple speech information.

6. The method according to any one of claims 1 to 5, characterized in that The method further comprises: In response to a click operation input by a user, when the click position corresponding to the click operation is in the speech text, if the electronic device is not in a playing state, the electronic device enters an editing state; If the electronic device is in the playing state, the electronic device plays the voice data corresponding to the first speaking time; wherein the first speaking time indicates the speaking time corresponding to the character corresponding to the click position; The playing state indicates that the electronic device plays voice data.

7. The method according to claim 6, characterized in that The second recording card also includes a progress bar; If the electronic device is in the playing state, the method further includes: The electronic device moves the playing progress of the progress bar to the first speaking time.

8. The method according to claim 6 or 7, characterized in that: The first speech time is obtained by taking the speech start time corresponding to the first target speech text as a reference and superimposing the offset time; the offset time is obtained based on the speech speed corresponding to the first target speech text and the number of offset characters; the number of offset characters refers to the number of characters in the first target speech text that are before the click position; The first target speech text indicates the speech text at the click position.

9. The method according to any one of claims 1 to 5, characterized in that The speech information also includes a speaker identification; The method further comprises: In response to a click operation input by a user, when the click position corresponding to the click operation is at the speaker identifier, if the electronic device is not in a playing state, the electronic device displays a speaker modification control; the speaker modification control is used to trigger the electronic device to modify the speaker identifier; If the electronic device is in the playing state, the electronic device plays the voice data corresponding to the target speech start time; the target speech start time refers to the speech start time corresponding to the speaker identifier; The playing state indicates that the electronic device plays voice data.

10. An electronic device, characterized in that: The electronic device includes a display screen, a memory and one or more processors; the display screen, the memory and the processor are coupled; the display screen is used to display an image generated by the processor, and the memory is used to store computer program code, and the computer program code includes computer instructions; when the processor executes the computer instructions, the electronic device executes the method as described in any one of claims 1 to 9.

11. A computer-readable storage medium, characterized in that: The method comprises computer instructions, which, when executed on an electronic device, cause the electronic device to execute the method as claimed in any one of claims 1 to 9.

Images