Speech recognition method, electronic equipment and computer readable storage medium

Abstract

The invention provides a voice recognition method, electronic equipment and a computer readable storage medium. According to the method, the electronic equipment can recognize the speaking content in the audio while recording the audio. Moreover, no matter whether the audio features of the speakers are pre-stored in the electronic equipment or not, the electronic equipment can distinguish the speakers corresponding to each sentence of speaking content. Therefore, in some speaking scenes, the user can obtain the speaking content and the text record of the speaker through the electronic equipment without manual recording, so that the working efficiency of the user is improved.

Description

Technical Field

[0001] The present application relates to the field of terminals, and in particular to a speech recognition method, an electronic device and a computer-readable storage medium. Background Art

[0002] In some speaking scenarios, users usually need to record the speech of one or more speakers on an electronic device in a visual form such as text. Users sometimes record audio and store audio files through electronic devices, so that users can open the audio files through electronic devices to listen to the audio and then correct the recorded speech based on the audio. In order to ensure the integrity and accuracy of the recorded speech, users may need to listen to the audio sentence by sentence and repeatedly, which requires users to spend a lot of time recording the speech on electronic devices. Summary of the invention

[0003] The present application provides a speech recognition method, an electronic device, and a computer-readable storage medium. The electronic device can recognize and display the speech content in the audio while recording the audio. In addition, the electronic device can also distinguish the speaker of each speech content, so that the user can obtain a text record of the speech content and the speaker identification contained in the audio through the electronic device without manual recording.

[0004] In a first aspect, the present application provides a speech recognition method, which is applied to an electronic device, wherein the electronic device stores a first identifier and a first voiceprint feature of a first speaker, and the method includes: the electronic device receives a first operation of a user selecting a speech recognition method, and the speech recognition method includes a first recognition method and a second recognition method; the electronic device obtains a first audio segment, and the first audio segment includes a first sub-segment corresponding to a first speech content; when the first operation selects the first recognition method, the electronic device determines that the voiceprint feature of the first sub-segment matches the first voiceprint feature, and the electronic device displays the first speech content and the first identifier, and the first identifier indicates that the first speech content is spoken by the first speaker; when the first operation selects the second recognition method, the electronic device determines that the first speech content is spoken by a speaker, and the electronic device displays the first speech content and the second identifier, and the second identifier is used to identify the speaker of the first speech content.

[0005] The electronic device can recognize the spoken content in the audio and display the spoken content. And the electronic device can also identify the speaker based on the voice recognition method selected by the user. Among them, the electronic device can identify the registered speaker in the audio based on the voiceprint features, and can also identify the unregistered speaker based on the clustering method. Wherein registering the speaker means that the electronic device stores the speaker's identification and voiceprint features. In this way, when the user selects the voiceprint recognition method, the electronic device can recognize the speaker contained in the audio based on the voiceprint of the registered speaker. When the user selects the registration-free speaker recognition method, the electronic device can cluster the audio clips to distinguish the speakers in the audio clips. In this way, regardless of whether the user has registered the speaker in advance, the electronic device can provide the user with a speech recognition result containing the spoken content and its corresponding speaker.

[0006] In combination with the first aspect, in some embodiments, after the electronic device obtains the first audio segment, the method further includes: the electronic device divides the first audio segment into multiple sub-segments, the multiple sub-segments include the first sub-segment, and the semantics of the multiple sub-segments are different.

[0007] The first audio segment may be a valid audio segment. The electronic device may segment the valid audio segment based on semantics to obtain multiple sub-segments, and the semantics of the speech content of each sub-segment are different. In this way, the electronic device may increase the probability that each sub-segment contains only one speaker, thereby improving the accuracy of subsequent recognition of the speaker in the sub-segment.

[0008] In combination with the first aspect, in some embodiments, the first audio segment includes a second sub-segment, the second sub-segment includes a second speech content of a second speaker, and the second voiceprint feature of the second speaker is not stored in the electronic device. The method also includes: when the first operation selects the first identification method, the electronic device determines that the second speech content is spoken by a speaker, and the electronic device displays the second speech content and a third identifier, where the third identifier is used to identify the speaker of the second speech content.

[0009] Among them, the first identification method can be a voiceprint recognition method. When the electronic device identifies the speaker through the voiceprint, it may happen that the voiceprint features in some sub-segments cannot match the voiceprint features stored in the electronic device. This may be because the electronic device does not store the voiceprint features of the speaker in the sub-segment. The electronic device can use a clustering recognition method to determine the clustering of the sub-segment, and then distinguish the speaker of the speech content. Among them, the third identifier can be of a different type from the first identifier. For example, the third identifier can be "speaker A", "speaker B", etc., and the first identifier can be "Zhang San", "Li Si", etc.

[0010] In combination with the first aspect, in some embodiments, the first audio segment includes a second sub-segment, the second sub-segment includes a second speech content of a second speaker, and the electronic device stores a second voiceprint feature and a fourth identifier of the second speaker. The method also includes: when the first operation selects the first identification method, the electronic device determines that the voiceprint feature of the second sub-segment matches the second voiceprint feature, and the electronic device displays the second speech content and the fourth identifier, and the fourth identifier indicates that the second speech content is spoken by the second speaker.

[0011] Among them, the first identification method can be a voiceprint recognition method. The electronic device can extract the voiceprint feature from the second sub-segment, and then match the voiceprint feature with its stored voiceprint feature. Finally, the electronic device can determine that the voiceprint feature of the second sub-segment matches the second voiceprint feature, and then display the fourth identifier corresponding to the second voiceprint feature. In other words, the electronic device can mark the speaker of the speech content based on the stored voiceprint features and their corresponding identifiers, wherein the speaker identifier of the voiceprint feature of each speaker stored in the electronic device can be input by the user, so that the user can clearly understand which speaker said each speech content through the speaker identifier.

[0012] In combination with the first aspect, in some embodiments, after the electronic device displays the second speech content and the fourth identifier, the method further includes: the electronic device receives a second operation, the second operation is used to filter the first speaker; in response to the second operation, the electronic device hides the display of the second speech content and the fourth identifier.

[0013] That is, the electronic device can filter the speech content of one or more speakers in response to the user operation, and the electronic device can hide the speech content of the speakers that are not filtered and only display the speech content of the filtered speakers. In this way, the user can filter the speech content of the speaker that he is interested in from the speech content of many speakers to view, thereby improving the user experience.

[0014] In combination with the first aspect, in some embodiments, the first audio segment also includes a third sub-segment, the third sub-segment includes a third speech content, and the speaker of the third speech content is different from the speaker of the first speech content. The method also includes: when the second identification method is selected in the first operation, the electronic device determines that the third speech content is spoken by a speaker, and the electronic device displays the third speech content and a fifth identifier, and the fifth identifier is used to indicate that the speaker of the third speech content is different from the speaker of the first speech content.

[0015] Among them, the second identification method can be a cluster identification method, and the electronic device can determine that the speaker in the third speech content is different from the speaker of the first speech content through the cluster identification module, and then the electronic device can assign a speaker identifier different from the second identifier to the third speech content, so as to distinguish it from the speaker of the first speech content. That is to say, when the speech recognition method is a cluster identification method, the electronic device can assign different identifiers to different speakers to facilitate user understanding.

[0016] In combination with the first aspect, in some embodiments, the first audio segment also includes a fourth sub-segment, and the fourth sub-segment includes a fourth speech content. The method also includes: when the second identification method is selected in the first operation, the electronic device determines that the fourth speech content is spoken by a speaker, and the speaker of the fourth speech content is the same as the speaker of the first speech content, the electronic device displays the fourth speech content, and displays the second identifier again at the first position associated with the fourth speech content, and the second identifier displayed at the first position is used to indicate the speaker of the fourth speech content.

[0017] The electronic device can determine through the cluster recognition module that the speaker in the fourth speech content is the same as the speaker of the first speech content, and then the electronic device can assign a second identifier to the fourth speech content to indicate that the speaker of the first speech content and the fourth speech content is the same. That is, when the speech recognition method is a cluster recognition method, the electronic device can assign the same identifier to the same speaker, which is easy for the user to understand.

[0018] In combination with the first aspect, in some embodiments, before the electronic device receives a first operation of a user selecting a voice recognition method, the method also includes: the electronic device receives a second audio segment and the first identifier input by the user, the speaker of the second audio segment is the first speaker; the electronic device extracts a first voiceprint feature of the first speaker from the second audio segment; the electronic device stores the first voiceprint feature and the first identifier, wherein the first voiceprint feature and the first identifier are stored in association.

[0019] The electronic device can receive an audio clip and a speaker identifier input by a user to register a speaker, wherein the audio clip input by the user can be an audio clip containing only one speaker. Then the electronic device can extract the speaker's voiceprint features from the audio clip, and store the voiceprint features in association with the speaker. In this way, during the speech recognition process, the electronic device can identify the speaker of the audio clip to be identified based on the stored voiceprint features and the speaker identifier provided by the user, and the electronic device can use the pre-stored speaker identifier provided by the user to annotate the speech content of the audio clip to be identified, so that the user can accurately and clearly understand the speaker of the speech content.

[0020] In combination with the first aspect, in some embodiments, the method also includes: the electronic device collects a second audio segment through a microphone during the process of displaying the first speech content; and the electronic device recognizes the speech content and speaker of the second audio segment based on the voice recognition method selected by the first operation.

[0021] That is to say, the electronic device can perform real-time speech recognition while collecting audio. The electronic device can continue to collect audio while displaying the spoken content. Then the electronic device can continue to recognize the spoken content and the speaker in the newly collected audio. In this way, the user can obtain a real-time record of the spoken content and the corresponding speaker through the electronic device in the speaking scene, thereby improving the user experience.

[0022] In a second aspect, the present application provides an electronic device, which includes a storage module for storing a first identifier and a first voiceprint feature of a first speaker; a user interaction module for receiving a first operation of a user selecting a speech recognition method, the speech recognition method including a first recognition method and a second recognition method; an audio acquisition module, the audio acquisition module is used to obtain a first audio segment, the first audio segment includes a first sub-segment corresponding to a first speech content; a voiceprint recognition module is used to determine that the voiceprint feature of the first sub-segment matches the first voiceprint feature when the first operation selects the first recognition method; a display module is used to display the first speech content and the first identifier when the first operation selects the first recognition method, the first identifier indicating that the first speech content is spoken by the first speaker; a cluster recognition module is used to determine that the first speech content is spoken by a speaker based on the acoustic features of the first audio segment when the first operation selects the second recognition method; the display module is also used to display the first speech content and the second identifier when the first operation selects the second recognition method, the second identifier is used to identify the speaker of the first speech content.

[0023] In combination with the second aspect, in some embodiments, the electronic device also includes a speech recognition module and a speech segmentation module; the content recognition module is used to identify the speech content contained in the first audio segment; the speech segmentation module is used to divide the first audio segment into multiple sub-segments based on the speech content contained in the first audio segment, the multiple sub-segments include the first sub-segment, and the semantics of the multiple sub-segments are different.

[0024] In combination with the second aspect, in some embodiments, the first audio segment includes a second sub-segment, the second sub-segment includes a second speech content of a second speaker, and the second voiceprint feature of the second speaker is not stored in the storage module; when the second recognition method is selected in the first operation, the cluster recognition module is also used to determine that the second speech content is spoken by a speaker; the display module is also used to display the second speech content and a third identifier, and the third identifier is used to identify the speaker of the second speech content.

[0025] In combination with the second aspect, in some embodiments, the first audio segment includes a second sub-segment, the second sub-segment includes a second speech content of a second speaker, and the storage module stores a second voiceprint feature and a fourth identifier of the second speaker; when the first operation selects the first recognition method, the voiceprint recognition module is also used to determine whether the voiceprint feature of the second sub-segment matches the second voiceprint feature; the display module is also used to display the second speech content and the fourth identifier, and the fourth identifier indicates that the second speech content is spoken by the second speaker.

[0026] In combination with the second aspect, in some embodiments, the first audio segment also includes a third sub-segment, the third sub-segment includes a third speech content, and the speaker of the third speech content is different from the speaker of the first speech content; when the first operation selects the second identification method, the cluster identification module is also used to determine that the third speech content is spoken by a speaker; the display module is also used to display the third speech content and a fifth identifier, and the fifth identifier is used to indicate that the speaker of the third speech content is different from the speaker of the first speech content.

[0027] In combination with the second aspect, in some embodiments, the first audio segment also includes a fourth sub-segment, and the fourth sub-segment includes a fourth speech content; when the second identification method is selected in the first operation, the cluster identification module is also used to determine that the fourth speech content is spoken by a speaker, and the speaker of the fourth speech content is the same as the speaker of the first speech content; the display module is also used to display the fourth speech content, and display the second identifier again at the first position associated with the fourth speech content, and the second identifier displayed at the first position is used to indicate the speaker of the fourth speech content.

[0028] In combination with the second aspect, in some embodiments, the electronic device also includes a voiceprint registration module, and the voiceprint recognition module is used to receive a second audio segment and a first identifier input by a user, and the speaker of the second audio segment is the first speaker; the voiceprint registration module is also used to send the second audio segment to the voiceprint recognition module; the voiceprint recognition module is also used to extract the first voiceprint feature of the first speaker from the second audio segment, and send the first voiceprint feature to the voiceprint registration module; the voiceprint registration module is also used to receive the first voiceprint feature, and send the first voiceprint feature and the first identifier to the storage module; the storage module is also used to store the first voiceprint feature and the first identifier, wherein the first voiceprint feature and the first identifier are stored in association.

[0029] In a third aspect, the present application provides an electronic device, comprising a display screen, a memory, and a processor coupled to the memory; the display screen is used to display an interface, the memory stores a computer program, and when the processor executes the above computer program, the electronic device implements any one of the methods described in the above first aspect.

[0030] In a fourth aspect, the present application provides a computer-readable storage medium, which stores a computer program or computer instructions, and the aforementioned computer program or computer instructions are executed by a processor to implement any method described in the first aspect above.

[0031] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product is executed by a processor, the method described in any one of the above-mentioned first aspects will be implemented.

[0032] In the sixth aspect, an embodiment of the present application provides a chip, which includes a processor and a memory, wherein the memory is used to store computer programs or computer instructions, and the processor is used to execute the computer programs or computer instructions stored in the memory, so that the chip executes any method described in the first aspect above.

[0033] The solutions provided in the second to sixth aspects are used to implement or cooperate with the corresponding methods provided in the first aspect, and therefore can achieve the same or corresponding beneficial effects as the corresponding methods in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0035] Figure 2 is a software structure block diagram of an electronic device 100 provided in an embodiment of the present application;

[0036] Figure 3A It is a structural block diagram of an AI framework engine of an electronic device 100 provided in an embodiment of the present application;

[0037] Figure 3B is a schematic diagram of a flow chart of speech recognition performed by the electronic device 100 provided in an embodiment of the present application;

[0038] Figure 4A-4W is a schematic diagram of user interaction involved in the speech recognition method provided in an embodiment of the present application;

[0039] Figure 5 It is a flowchart diagram of a speech recognition method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to be used as limitations to the present application. As used in the specification and appended claims of the present application, the singular expressions "one", "a kind of", "said", "above", "the" and "this" are intended to also include plural expressions, unless there is a clear indication to the contrary in the context. It should also be understood that the term "and / or" used in the present application refers to and includes any or all possible combinations of one or more listed items.

[0041] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more.

[0042] In some speaking scenarios (such as meeting scenarios), users usually need to record the speech content of one or more speakers in a visual form such as text. Among them, the speech content may include the words, words, phrases, sentences spoken by the speaker, and the tone of the speaker's speech, etc. During the speaker's speech, the user usually records the speech content in real time in the form of text or the like to the electronic device. Sometimes, in order to ensure the integrity and accuracy of the speech content they record, the user will record the speaker's speech and then store the recorded audio in the electronic device. In this way, the user can open the audio file through the electronic device at any time to listen to the audio and modify or supplement the speech content they have recorded. In this process, the user may need to listen to the audio repeatedly and sentence by sentence, which requires the user to spend a lot of time recording the speech content on the electronic device.

[0043] In order to save the time spent by users in recording spoken content, the embodiments of the present application propose a speech recognition method, an electronic device, and a computer-readable storage medium. In the speech recognition method, the electronic device can also recognize the spoken content contained in the audio while recording, and then the electronic device can display the spoken content contained in the audio. In addition, the electronic device can also distinguish the speakers in the audio, so that the user can understand which speaker said each sentence, each word, etc. in the audio. In other words, the electronic device can generate and display a speech record containing the spoken content and the speaker identification in real time when the speaker is speaking, so that the user can modify the speech record immediately, thereby improving the user's work efficiency.

[0044] The following first introduces an exemplary electronic device 100 provided in an embodiment of the present application.

[0045] Figure 1It is a schematic diagram of the structure of the electronic device 100 provided in an embodiment of the present application.

[0046] The following is a detailed description of the embodiment using the electronic device 100 as an example. It should be understood that the electronic device 100 may have more than Figure 1 More or fewer components may be shown, two or more components may be combined, or there may be a different configuration of components. Figure 1 The various components shown in the EMBODIMENTS 2000 may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.

[0047] The electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a display screen 190, etc. The sensor module 180 may include a pressure sensor 180A, a touch sensor 180B, etc.

[0048] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0049] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0050] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0051] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc., and the components in the electronic device 100 may be connected through the above interfaces.

[0052] The charging management module 140 is used to receive charging input from a charger, where the charger can be a wireless charger or a wired charger.

[0053] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to provide power to the processor 110, the internal memory 121, the external memory, the display screen 190 and the like.

[0054] The modem processor may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 190. In some embodiments, the modem processor may be an independent device.

[0055] The electronic device 100 implements the display function through a GPU, a display screen 190, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 190 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0056] The display screen 190 is used to display images, videos, etc. The display screen 190 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 190, where N is a positive integer greater than 1.

[0057] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0058] NPU is a neural-network (NN) computing processor. It can quickly process input information by drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, and can also continuously self-learn.

[0059] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0060] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, applications required for at least one function (such as face recognition function, fingerprint recognition function, mobile payment function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as face information template data, fingerprint information template, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0061] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0062] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0063] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The electronic device 100 can listen to music or listen to a hands-free call through the speaker 170A.

[0064] The receiver 170B, also called a "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 100 receives a call or voice message, the voice can be received by placing the receiver 170B close to the human ear.

[0065] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to microphone 170C to input the sound signal into microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify the sound source, realize directional recording function, etc.

[0066] The earphone interface 170D is used to connect a wired earphone and can be a USB interface 130 or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0067] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 190. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. A capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure based on the change in capacitance. When a touch operation acts on the display screen 190, the electronic device 100 detects the intensity of the touch operation based on the pressure sensor 180A. The electronic device 100 can also calculate the position of the touch based on the detection signal of the pressure sensor 180A.

[0068] The touch sensor 180B is also called a "touch panel". The touch sensor 180B can be arranged on the display screen 190, and the touch sensor 180B and the display screen 190 form a touch screen, also called a "touch screen". The touch sensor 180B is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 190. In other embodiments, the touch sensor 180B can also be arranged on the surface of the electronic device 100, which is different from the position of the display screen 190.

[0069] In some embodiments, the electronic device can receive audio containing speech content through the microphone 170C, and the audio module 170 can convert the audio data in the form of analog signals output by the microphone 170C into digital signals. The DSP can send the audio data to the NPU after resampling, denoising, and other processing of the audio in the form of digital signals output by the audio module 170. In addition, the NPU can recognize the speech content in the audio, that is, convert the speech content contained in the audio into visual speech content (for example, in the form of text, pattern, etc.).

[0070] In some embodiments, after the NPU converts the speech content in the audio into speech content in text form, it can also segment the audio data based on the text. For example, the NPU can divide the audio into multiple segments based on the semantics of the speech content, each audio segment can correspond to a complete phrase or sentence, and then the NPU can identify the speaker corresponding to each audio segment.

[0071] In some embodiments, the NPU can identify the speaker corresponding to each audio segment based on a clustering algorithm. In other embodiments, the NPU can identify the speaker corresponding to each audio segment based on a voiceprint. A voiceprint is a biometric feature composed of multiple dimensional features such as wavelength, frequency, and intensity, which is used to uniquely identify a speaker. A voiceprint may include a mathematical vector obtained by embedding multiple dimensional features. Optionally, the electronic device may store the voiceprints of one or more speakers in the internal memory 121. For example, the electronic device may store the voiceprints in a read-only memory (ROM).

[0072] The electronic device 100 can also register the speaker. The above-mentioned registration of the speaker means that the electronic device extracts the voiceprint based on the audio data of the speaker to be registered, and then stores the voiceprint of the speaker to be registered in the memory or cloud server. The electronic device can receive the voice audio of the speaker to be registered through the microphone 170C, and the audio data can be sent to the NPU after being processed by the audio module 170 and DSP, etc., and the NPU extracts the voiceprint of the speaker to be registered from the audio data. Alternatively, the electronic device can also send the audio data of the speaker to be registered stored in the external memory or cloud to the NPU, and then the NPU can extract the voiceprint of the speaker to be registered from the audio data. Optionally, the NPU can extract the voiceprint of the speaker to be registered based on the D-Vector or X-Vector method. The embodiment of the present application does not limit the method for the NPU to extract the voiceprint of the speaker to be registered from the audio data.

[0073] Figure 2 It is a software structure block diagram of an electronic device 100 provided in an embodiment of the present application.

[0074] The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the system of the electronic device 100 may include five layers, namely, an application layer, an application framework layer, a runtime and system library, a kernel layer, and an artificial intelligence (AI) framework engine.

[0075] The application layer can include a series of application packages. Figure 2 As shown, the application package may include application programs (also referred to as applications) such as notes and a recorder.

[0076] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions. For example, the application framework layer may include an audio system and a view system.

[0077] The audio system is responsible for the comprehensive management of the transmission and control of audio streams. The audio system can provide an interface for upper-layer applications to call audio control. An audio stream is continuous audio data transmitted using streaming technology. Streaming technology can be either real-time streaming or progressive streaming.

[0078] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. The display interface can be composed of one or more views. For example, a view can include a view that displays the speech content in a note application.

[0079] Not limited to this, the application framework layer may also include a content provider ( Figure 2 ) and the window manager ( Figure 2 The content provider is used to store and retrieve data and make the data accessible to the application. The above data may include video, images, audio, calls made and received, browsing notes, phone books, etc. The resource manager provides various resources for the application, such as localized strings, icons, pictures, layout files, video files, etc.

[0080] The runtime includes the core library and the virtual machine. The runtime is responsible for the scheduling and management of the system.

[0081] The core library consists of two parts: one part is the function that the programming language (for example, Java language) needs to call, and the other part is the core library of the system.

[0082] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the programming files (e.g., java files) of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0083] The system library may include multiple functional modules, such as audio stream processing library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), surface manager, media library, etc.

[0084] The audio stream processing library is used to implement the transmission and control of the audio stream. For example, after receiving a user operation to enable the audio recording function, the note-taking application will call the interface of the audio system. The audio system can call the audio stream processing library to transmit the request to enable audio recording to the corresponding module, thereby implementing the audio recording function.

[0085] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0086] A 2D graphics engine is a drawing engine for 2D drawings.

[0087] The surface manager is used to manage the display subsystem and provides the fusion of two-dimensional (2D) and three-dimensional (3D) layers for multiple applications.

[0088] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0089] The kernel layer is a layer between hardware and software. The kernel layer may include a display driver and an audio driver. The display driver may be used to drive the display screen 190 to display images, and the audio driver may be used to drive the microphone 170C to record voice audio.

[0090] The AI ​​framework engine can be used to process artificial intelligence tasks. Among them, the AI ​​framework engine may include a content recognition module, a speech segmentation module, a voiceprint registration module, a voiceprint recognition module, and a cluster recognition module. The application can call the software modules in the AI ​​framework engine to implement the corresponding functions. For the introduction of each module in the AI ​​framework engine, please refer to the introduction of the subsequent embodiments, which will not be expanded here.

[0091] Figure 2 The software modules in the software structure diagram are only examples and are not intended to limit the embodiments of the present application. Figure 2 More or fewer software modules may be shown, two or more software modules may be combined, and different software module configurations may be possible.

[0092] For example, in some embodiments, the electronic device 100 may further include a storage module, a user interaction module, an audio acquisition module, and a display module. The storage module is used to store a speaker identifier and its corresponding voiceprint feature. The user interaction module is used to receive user operations and determine the user's operation content. The audio acquisition module is used to collect audio streams through a microphone. The display module is used to display a user interface.

[0093] The following introduces the structure of an AI framework engine provided in an embodiment of the present application.

[0094] Figure 3A It is a structural block diagram of an AI framework engine of an electronic device 100 provided in an embodiment of the present application.

[0095] like Figure 3A As shown, the AI ​​framework engine may include a content recognition module, a speech segmentation module, a voiceprint registration module, a voiceprint recognition module and a cluster recognition module.

[0096] The electronic device 100 first obtains an audio stream, and divides the audio stream to obtain valid audio segments. Among them, valid audio refers to audio that may contain human voices. The audio stream is continuous audio data transmitted using streaming technology. The electronic device 100 can filter out valid audio in the audio stream by calculating the short-time energy value or short-time zero-crossing rate of the audio segments in the audio stream. For example, the electronic device 100 can calculate whether the short-time energy value in each window in the audio stream exceeds a threshold value through a sliding window. When the short-time energy value in a window in the audio stream is lower than a preset threshold value, the electronic device can select the starting time point of the window to cut the audio stream. In this way, the electronic device 100 filters out a valid audio segment, and the short-time energy value in each window in the valid audio segment is greater than the preset threshold value. Not limited to this method, the electronic device 100 can also filter out valid audio segments by other methods, and the embodiments of the present application are not limited to this.

[0097] The content recognition module can be used to identify the speech content contained in the audio. The electronic device 100 can input the valid audio clips screened from the audio stream into the content recognition module. One of the valid audio clips can refer to an audio clip with coherent sound. Here, the sound coherence can refer to the fact that the human voice part in the audio is continuously present or appears, and the human voice part is not interrupted or the duration of the interruption does not exceed the preset duration, that is, the pause duration of the speaker's voice in the audio does not exceed the preset duration. The electronic device 100 can determine whether the sound in the audio stream is coherent by the short-time energy value or the short-time zero-crossing rate. Each valid audio clip is composed of one or more frames of audio, and the electronic device 100 can use streaming technology to input the data of one or more frames of audio in the audio clip into the content recognition module. The electronic device 100 can use every 20 milliseconds of audio as a frame of audio, not limited to 20 milliseconds, and a frame of audio can also be audio of other time lengths, for example, a frame of audio data can also be 10 milliseconds of audio, or 5 milliseconds of audio, etc., which is not limited in the embodiment of the present application. The electronic device 100 may process each audio frame frame by frame through the content recognition module to recognize the speech text content in each audio frame.

[0098] Exemplarily, the content recognition module may include an encoder, an embedding layer, a language prediction model, and a joint neural network (Joint NN) model. Among them, the encoder can encode the audio data to obtain an acoustic coding sequence (or acoustic coding features). The embedding layer can receive the text that has been recognized from the current valid audio segment, and then encode the text to obtain a text coding sequence (or text coding features). Then the language prediction model can predict the subsequent text based on the text coding sequence, and then generate a new text coding sequence. The joint neural network model can splice the text coding sequence and the acoustic coding sequence to form a new coding sequence, and then the joint neural network model can obtain a probability distribution matrix based on the spliced ​​coding sequence and then output the text of the speech content contained in the audio segment, wherein the probability distribution matrix can be a four-dimensional tensor, which is used to represent the probability of the audio input into the content recognition module corresponding to different characters and words in the vocabulary.

[0099] Taking the content recognition module's recognition of Chinese as an example, after the electronic device 100 cuts out a valid audio segment containing human voices, it can call the content recognition module to process the data of the valid audio segment. The first-frame audio data in the data of the valid audio segment may contain the pronunciation of the character "我" (wo), and the second-frame audio data may contain the pronunciation of the character "是" ("shi"). The content recognition module first uses an encoder and a joint neural network model to recognize the spoken content contained in the first-frame audio data of the valid audio segment, and outputs the text of the spoken content contained in the first-frame audio data, which is "我". Then, the content recognition module uses the encoder to process the second-frame audio data to obtain the acoustic encoding sequence of the second-frame audio data. At the same time, the content recognition module can input the text of the spoken content "我" recognized from the first-frame audio data into the embedding layer. The embedding layer will generate a text encoding sequence based on the text "我", and then the language prediction model can predict the subsequent text based on this text encoding sequence to generate a new text encoding sequence, which can be used to strengthen the possible text. For example, after "我", "是" may follow. The language prediction model can strengthen "是" through the new text encoding sequence. The joint neural network model can splice the text encoding sequence and the acoustic encoding sequence to obtain a probability distribution matrix. The pronunciation "shi" of the second-frame audio data can be recognized as "四", "是", "事", etc. Since the language prediction model strengthens "是" based on "我", the joint neural network model can determine that the probability of the spoken content contained in the second-frame audio data being the character "是" is higher, and thus recognize the spoken content corresponding to the second-frame audio data as "是". In this way, the accuracy of the content recognition module in recognizing the text of the spoken content is improved.

[0100] The method for the content recognition module to recognize the content in other languages from speech can refer to the above method for recognizing Chinese content from speech, which will not be elaborated here. Not limited to the above method, the content recognition module can also use other methods to recognize the spoken content contained in the audio segment. The embodiments of the present application do not limit the method for the content recognition module to recognize the spoken content contained in the audio segment and then output the spoken content in text form.

[0101] The electronic device can input the speech content text of the valid audio segment into the speech segmentation module. Among them, the content recognition module can input some intermediate results into the speech segmentation module in the process of identifying the speech content of each valid audio segment. The above intermediate results refer to the speech content text output when the content recognition module has not yet completely identified all the speech content of the audio segment. For example, the speech content of an audio segment includes: I plan to go to the beach, and the intermediate results recognized by the content recognition module may include "I", "I plan to", "I plan to go", then the electronic device 100 can input it into the speech segmentation module when the above intermediate results are recognized. And / or, the content recognition module can input all the speech content texts contained in each valid audio segment into the speech segmentation module after identifying all the speech content of each valid audio segment. For example, the content recognition module inputs "I plan to go to the beach" into the speech segmentation module after identifying all the speech content of the audio segment "I plan to go to the beach". The speech segmentation module can receive the speech content text of the valid audio segment and the valid audio segment, and then cut the valid audio segment based on the semantic features of the text and the audio features in the audio, and divide the valid audio segment into one or more sub-segments.

[0102] In some embodiments, the speech segmentation module may include a multimodal speech segmentation model. The multimodal speech segmentation model may receive the speech content text processed by the electronic device 100 and recognized by the content recognition module from the valid audio segment. The multimodal speech segmentation model may obtain audio features from the valid audio segment, obtain semantic features from the speech content text, and then segment the valid audio segment based on the audio features and semantic features, thereby outputting one or more audio sub-segments. The multimodal speech segmentation model may cut the audio when the duration of the speech pause in the audio exceeds the preset duration, or when a semantically complete sentence ends. The multimodal speech segmentation model may be a neural network model, and the multimodal speech segmentation model may include one or more of the following: a convolutional neural network (CNN) model, a recurrent neural network (RNN) model, a long short-term memory network (LSTM) model, a deep neural network (DNN) model, a Transformer model, and the like.

[0103] Among the one or more valid audio segments obtained by the electronic device 100 after initially segmenting the audio stream, some audio segments may be relatively long. For example, some valid audio segments may include speech of a single speaker for a relatively long time. If the electronic device directly inputs an entire valid audio segment into a subsequent voiceprint recognition module, it may result in poor recognition efficiency. Alternatively, some valid audio segments may include speech of multiple speakers speaking alternately. If a valid audio segment containing speech of multiple speakers is input into a subsequent voiceprint recognition module or a cluster recognition module, it may affect the accuracy of speaker recognition. Therefore, the speech segmentation module can further segment the audio segment to obtain one or more audio sub-segments.

[0104] For example, the speech content of the valid audio segment input by the electronic device to the speech segmentation module may be "If it is sunny tomorrow, how about we go play football? Okay, I think it's okay." Here, "If it is sunny tomorrow, how about we go play football?" may be the speech content of the first speaker, and "Okay, I think it's okay" may be the speech content of the second speaker. The electronic device 100 can use the speech segmentation module to segment the valid audio segment into two audio sub-segments, the speech content of the first audio sub-segment is "If it is sunny tomorrow, how about we go play football?", and the speech content of the second audio sub-segment is "Okay, I think it's okay". Then the speech segmentation module can input the data of the two audio sub-segments into the subsequent voiceprint recognition module or cluster recognition module for processing in sequence. Since the electronic device 100 uses the speech segmentation module for segmentation, the probability of only containing a single speaker in the audio sub-segment can be increased, which can improve the efficiency and accuracy of the subsequent voiceprint recognition module or cluster recognition module in identifying the speaker.

[0105] The voiceprint recognition module can receive the audio sub-segment output by the speech segmentation module, and then extract the voiceprint contained in the audio sub-segment, and then identify the speaker based on the voiceprint. Among them, the voiceprint is a biometric feature composed of multiple dimensional features such as the wavelength, frequency, and intensity of the sound, which is used to uniquely identify the speaker. The voiceprint can include a mathematical vector obtained by vectorizing the above multiple dimensional features. The voiceprint recognition module can include an audio segmentation module, a voiceprint feature extraction module, a speaker classification module, and an activation function.

[0106] The audio segmentation module can segment the audio sub-segments segmented by the speech segmentation module according to a preset time length to obtain multiple shorter audio segments. The voiceprint feature extraction module can receive the data of the multiple audio short segments after segmentation, and then extract the voiceprint features. The voiceprint feature extraction module can input the data of each audio short segment into the DNN model, and then obtain multiple feature vectors. Furthermore, the voiceprint feature extraction module can calculate the average value of the above multiple feature vectors, and output the calculation result to the speaker classification module, which can also be called the voiceprint feature of the audio sub-segment. The speaker classification module can match the voiceprint features of the audio sub-segment with the voiceprint features stored in the electronic device based on the voiceprint features of the audio sub-segment, where each voiceprint feature can correspond to a speaker. Then, the speaker classification module can input the matching result of the voiceprint feature of the audio sub-segment and the voiceprint features of one or more speakers stored in the electronic device into the activation function (for example, it can be a softmax function). The activation function can normalize the above matching results to obtain the probability of matching the voiceprint feature of the audio sub-segment with each voiceprint feature stored in the electronic device. Finally, the voiceprint recognition module can output the identification of the speaker corresponding to the voiceprint feature with the highest probability. The method of identifying speakers based on voiceprints by the above-mentioned voiceprint recognition module can also be called a voiceprint recognition method based on D-Vector. It is not limited to the voiceprint recognition method based on D-Vector. The voiceprint recognition module can also extract the voiceprint of the audio segment by other methods to identify the speaker. The embodiment of the present application does not limit the method of extracting the voiceprint of the audio segment by the voiceprint recognition module and then identifying the speaker based on the voiceprint.

[0107] Among them, the voiceprint features stored in the electronic device can be obtained after the voiceprint registration module registers the speaker. The voiceprint registration module can input the speaker identification and the audio segment to be identified of the speaker into the voiceprint recognition module. After the audio segmentation module segments the audio segment to be identified into multiple short audio segments, the data of the multiple short audio segments are input into the voiceprint feature extraction module. The voiceprint feature extraction module can extract the voiceprint of the entire audio segment based on the data of the multiple short audio segments. Then, the voiceprint registration module can store the voiceprint extracted based on the audio segment to be identified and the speaker identification corresponding to the audio segment to be identified in the memory, so that the voiceprint recognition module can identify the speaker based on the voiceprint stored in the memory. Here, each speaker identification can correspond to the voiceprint of the speaker one by one, wherein the speaker identification can be input by the user, for example, the speaker identification can be "Zhang San", "Li Si", and so on.

[0108] The cluster recognition module may include an audio segmentation module, a number recognition module and a clustering module. The audio segmentation module is used to segment the audio sub-segments segmented by the speech segmentation module according to a preset time length to obtain multiple shorter audio short segments. The time length of the audio short segments segmented by the audio segmentation module in the cluster recognition module may be different from the time length of the audio short segments segmented by the audio segmentation module in the voiceprint recognition module. The number recognition module can identify the number of speakers contained in the audio sub-segments based on the segmented audio short segments, wherein the number recognition module can extract audio features in the audio short segments, calculate the similarity of each audio short segment with the audio features of other audio short segments in the audio sub-segment, and then determine the number of speakers contained in the audio sub-segment. The above-mentioned audio features may be, for example, mel-frequency cepstral coefficients (MFCC) or voiceprint features, which are not limited in the embodiments of the present application. Then, the number of speakers contained in the data of the entire audio sub-segment output by the speech segmentation module can be output by the number of speakers recognized by the number of speakers recognized by the number of speakers recognized by the number of speakers recognized by the number of speakers recognized by the number of speakers recognized by the number of speakers recognized by the audio segmentation module, and the clustering module can construct a similarity matrix and a Laplace matrix, and cluster the data of the audio sub-segments based on the Laplace matrix. Finally, the clustering module can output one or more clusters (also referred to as categories), each cluster being used to indicate that the data of one or more audio sub-segments is generated by the same speaker. For example, when the number of speakers recognized by the number of speakers recognizes that only one person speaks in the audio sub-segment, the electronic device 100 can cluster the short audio segments contained in the audio sub-segment into the same cluster. The inclusion of multiple speakers in the same audio sub-segment can mean: multiple speakers speak alternately in the audio sub-segment, and / or multiple speakers speak simultaneously in the audio sub-segment. The clustering recognition module can use different speaker role identifiers to distinguish each cluster, and the result output by the clustering recognition module can include the identifier of the audio sub-segment corresponding to each cluster and the speaker role identifier corresponding to each cluster. For example, the cluster recognition module divides five audio clips in an audio sub-segment into two clusters, the first cluster corresponds to the first audio clip to the second audio clip, and the second cluster corresponds to the third audio clip to the fifth audio clip. Then the result output by the cluster recognition module can be used to indicate that the speakers corresponding to the first audio clip and the second audio clip are "role A", and the speakers corresponding to the third audio clip to the fifth audio clip are "role B". Here, the speaker roles output by the cluster recognition module can only distinguish different speakers, but are not associated with specific speakers.That is to say, the same speaker role can be used for different speakers. For example, in one speech recognition result, "role A" is used to identify the speech content of speaker A, and in another speech recognition result, "role A" can be used to identify the speech content of speaker B. The speaker identifier output by the voiceprint recognition module can be associated with a specific speaker, and the user can determine a unique speaker based on the speaker identifier. For example, the electronic device can store the identifier "Zhang San" of speaker A and the voiceprint features of speaker A. Then, when subsequent electronic devices use the voiceprint recognition module to identify the speaker, as long as the audio clip contains the speech content of speaker A, the electronic device uses "Zhang San" to identify the speech content of speaker A. In this way, as long as the user sees the speaker identifier "Zhang San" in the speech recognition result, he knows that the speech content corresponding to the speaker identifier is spoken by speaker A.

[0109] In some embodiments. The clustering module can store the clustering results of the audio sub-segments when clustering each audio sub-segment output by the speech segmentation module, so that the clustering module can determine whether there is audio short segment data that can be classified into the previously classified cluster when clustering subsequent audio sub-segments. For example, the clustering module obtains two clusters after clustering the first audio sub-segment, and the two clusters are distinguished by "Role A" and "Role B" respectively. When the clustering module clusters the second audio sub-segment, it can determine whether there is an audio short segment in the second audio sub-segment that can be merged into the cluster corresponding to "Role A" or "Role B".

[0110] That is to say, both the voiceprint recognition module and the cluster recognition module can distinguish the speaker. Among them, the voiceprint recognition module needs to register the speaker in advance when identifying the speaker. Here, registering the speaker means extracting the voiceprint of a speaker from the audio data of a certain speaker, and then storing the voiceprint of the speaker in a memory (such as a ROM) or a cloud server. The voiceprint recognition module can only recognize the voiceprint of the registered speaker. The cluster recognition module can distinguish the role of the speaker without the speaker registering in advance. The electronic device can input the audio sub-segment output by the speech segmentation module into different recognition modules to meet the different needs of users. For example, the electronic device can input the audio sub-segment to be identified into the voiceprint recognition module to identify the registered speaker in the audio segment. Alternatively, the electronic device can input the audio sub-segment to be identified into the cluster recognition module to distinguish different speakers in the audio sub-segment without registering the speaker. In this way, regardless of whether the user registers the speaker, the electronic device can distinguish the speaker of each speaking content for the user while identifying the speaking content in the audio stream, thereby improving the user experience. In which, when the speaker corresponding to the voice in the audio stream has been registered, the electronic device 100 provides the speaker identification of the speaker when registered in the voice recognition content, so that the user can clearly distinguish which registered speaker spoke the speech content in the audio stream.

[0111] Figure 3A The method in which the cluster identification module clusters the audio sub-segments to distinguish the speaker roles is only an example, and the cluster identification module can also use other clustering algorithms to distinguish speakers. At the same time, the cluster identification module can also include more or fewer modules than shown in the figure, or combine certain modules, which is not limited in the embodiments of the present application. In some embodiments, the cluster identification module may not include a number recognition module.

[0112] Figure 3B It is a flowchart of the electronic device 100 performing voice recognition provided by an embodiment of the present application. Figure 3B The rectangles represent the audio streams. Figure 3B As shown, the electronic device 100 performs speech recognition mainly including the following steps:

[0113] 1. Electronic devices collect audio through microphones.

[0114] The electronic device 100 can collect audio through a microphone, and then the electronic device 100 can generate audio data in the form of a digital signal based on the audio in the form of an analog signal collected by the microphone. Multiple audio data are transmitted using a streaming transmission technology to form an audio stream.

[0115] 2. Electronic devices use endpoint detection to truncate the audio stream.

[0116] The electronic device 100 can cut off the audio segment without human voice in the audio stream through endpoint detection. The electronic device 100 can determine whether there is human voice by whether the short-time energy value of the audio segment in each time window in the audio stream exceeds a threshold. Then the valid audio segment can be an audio segment in which the short-time energy value of the audio data in each time window is greater than a preset threshold.

[0117] 3. The electronic device calls the content recognition module to recognize the spoken content in the valid audio segment.

[0118] The electronic device 100 can call the content recognition module in the artificial intelligence framework engine to recognize the spoken content in the valid audio segment.

[0119] 4. The electronic device calls the voice segmentation module to segment the valid audio segments.

[0120] Among them, the speech segmentation module can segment the valid audio segments based on semantic features and audio features.

[0121] For example, after the electronic device 100 calls the content recognition module to recognize two sentences from a valid audio segment: sentence 1 and sentence 2, the electronic device 100 can input the speech content 1, speech content 2, and the valid audio segments corresponding to the speech content 1 and speech content 2 into the speech segmentation module. Then the speech segmentation module can segment two audio sub-segments from the valid audio segment: audio sub-segment 1 and audio sub-segment 2. The speech content of audio sub-segment 1 is speech content 1, and the speech content of audio sub-segment 2 is speech content 2.

[0122] 5. The electronic device calls the voiceprint recognition module to identify the speaker of the audio sub-segment.

[0123] In some embodiments, the electronic device 100 may call the voiceprint recognition module to recognize the voiceprint of the audio sub-segment segmented by the speech segmentation module, and then identify the speaker. For example, if the electronic device 100 recognizes the voiceprint of speaker A from audio sub-segment 1, it can be determined that the speaker of audio sub-segment 1 is speaker A, that is, speech content 1 is spoken by speaker A; similarly, if the electronic device 100 recognizes the voiceprint of speaker B from audio sub-segment 2, it can be determined that the speaker of audio sub-segment 2 is speaker B, that is, speech content 2 is spoken by speaker B.

[0124] 5'. The electronic device calls the clustering recognition module to distinguish the speakers of the audio sub-segments.

[0125] In some other embodiments, the electronic device 100 may also call the clustering recognition module to recognize the voiceprint of the audio sub-segment segmented by the voice segmentation module, thereby distinguishing the speaker. For example, the electronic device 100 may divide the audio sub-segment 1 into multiple short audio segments according to a fixed length through the clustering recognition module, and then extract the audio features of each short audio segment, and then cluster the short audio segments based on the audio features of each short audio segment. Finally, the electronic device may cluster the multiple short audio segments in the audio sub-segment 1 into category 1; the electronic device 100 may cluster the audio data in the audio sub-segment 2, and then cluster the multiple short audio segments in the audio sub-segment 2 into category 2. The electronic device may distinguish two categories with different identifiers. For example, the electronic device may use "speaker A" to identify category 1 and "speaker B" to identify category 2, where "speaker A" and "speaker B" represent two different speakers respectively. This indicates that the speech content 1 in the audio sub-segment 1 is spoken by speaker A, and the speech content 2 in the audio sub-segment 2 is spoken by speaker B.

[0126] In some embodiments, one or more of the content recognition module, speech segmentation module, voiceprint recognition module, voiceprint registration module, and cluster recognition module in the AI ​​framework engine may also be deployed in a cloud server. The electronic device 100 may cut the valid audio in the audio stream and send the cut audio data to the cloud server, and the cloud server may identify the speech content and the corresponding speaker contained in the audio data, and then the cloud server may return the recognition result to the electronic device 100. The method for the cloud server to identify the speech content and the corresponding speaker contained in the audio data may refer to the method for the electronic device to identify the speech content and the corresponding speaker contained in the audio data, which will not be repeated here.

[0127] The following introduces the application scenarios of the speech recognition method provided in the embodiments of the present application.

[0128] Figure 4A-4W It is a schematic diagram of user interaction (user interface, UI) involved in the speech recognition method provided in an embodiment of the present application.

[0129] Here we first introduce a scenario in which an electronic device performs speech recognition through a clustering recognition module when the speaker is not registered.

[0130] like Figure 4A As shown, Figure 4AThe home screen interface 400 on the electronic device 100 is exemplarily shown. The home screen interface 400 may include desktop icons of one or more applications, which may include a desktop icon 401 of a note application. The note application is used to create and edit notes. The notes may include a title and content, and the content of the notes may include but is not limited to: text, pictures, audio, etc. The electronic device may store the data of the notes created by the user in a memory.

[0131] The electronic device 100 may detect a user operation, such as a click operation, on the desktop icon 401. In response to the operation, the electronic device 100 may display a Figure 4B The main interface 410 of the note application is shown. The main interface 410 of the note application may include a note display area 411 and a note adding button 412.

[0132] The note display area 411 may include one or more note list items, including a note list item 411A. The note list item 411A may correspond to a note, and provide a user with an entry for editing a note title or content.

[0133] For example, Figure 4B As shown, the note list item 411A may display the title of the note (such as "Welcome to Notes"), the time when the note was created (such as "10 minutes ago"), and the preview content of the note (such as "Here you can create notes;") to help users quickly understand the relevant information of the note.

[0134] The note adding button 412 is used to create a new note.

[0135] The electronic device 100 may receive an operation of the user clicking the note adding button 412. In response to the operation, the electronic device 100 may create a new note and display the following information: Figure 4C The note editing interface 420 is shown. The note editing interface 420 is used to edit the title or content of the note. The note editing interface 420 may include: a title editing box 421, a note content display area 422, a toolbar 423, a soft keyboard 424, a save button 425 and a return button 426.

[0136] The title edit box 421 can display the title of the current note. Figure 4C The default title of the note ("Title") may be displayed in the title edit box 421.

[0137] The note content display area 422 is used to display the content of the current note.

[0138] The tool bar 423 may include one or more tool buttons, each of which is used to implement different functions. The tool bar 423 may include a recording button 423A, which is used to start the voice recognition function.

[0139] The soft keyboard 424 is used to edit the title text. The soft keyboard 424 may include a zoom button 424A, wherein the zoom button 424A is used to hide the soft keyboard 424 in the note editing interface 420.

[0140] The save button 425 is used to store the data of the current note, which may include information such as the note title, content, creation time, etc.

[0141] The return button 426 is used to return to the main interface 410 of the note application.

[0142] The electronic device 100 may receive an operation of editing the title text by the user through the soft keyboard 424, and in response to the operation, the electronic device 100 may synchronously display the text entered by the user in the title editing box 421. Figure 4D As shown, the electronic device 100 changes the text displayed in the title editing box 421 from "Title" to "Lunch Location". Then, the electronic device 100 can receive the operation of the user clicking the zoom button 424A, such as Figure 4E As shown, in response to the above operation, the electronic device can hide the displayed soft keyboard 424.

[0143] Furthermore, the electronic device 100 can enable the voice recognition function through the recording button 423A in the toolbar 423. Figure 4E As shown, the electronic device 100 can receive an operation of the user clicking the recording button 423A, and in response to the operation, the electronic device 100 can display the following Figure 4F Menu bar 430 is shown.

[0144] The menu bar 430 may include multiple options for selecting a voice recognition mode, or changing parameters during voice recognition, etc. The multiple options may include an "only recognize voice content" option, a "registration-free speaker distinction" option, a "voiceprint recognition speaker" option, and a "voiceprint management" option. The "only recognize voice content" option can be used to instruct the electronic device to only recognize the speech content in the audio, and not to recognize the speaker. The "registration-free speaker distinction" option is used to instruct the electronic device to distinguish speakers in a manner that does not require speaker registration, and the electronic device 100 can use a clustering recognition module to distinguish speakers in the audio. The "voiceprint recognition speaker" option is used to instruct the electronic device to recognize speakers based on voiceprints, and the electronic device 100 can use a voiceprint recognition module to recognize speakers in the audio. The "voiceprint management" option is used to provide users with an entry to manage voiceprints.

[0145] The electronic device may receive an operation of the user clicking the "registration-free speaker identification" option in the menu bar 430. In response to the operation, the electronic device 100 may collect audio through the microphone and display the audio. Figure 4G The voice recognition box 431 is used to display the voice recognition result. The voice recognition box 431 may include a total audio duration 431A, a waveform display area 431B, a speech content display area 431C, and a stop button 431D.

[0146] Among them, the total audio duration 431A is used to display the time length of the audio collected by the electronic device in the current note.

[0147] The waveform display area 431B is used to display the waveform of the audio collected by the electronic device.

[0148] The speech content display area 431C is used to display the speech content recognized by the electronic device from the audio, and the speech content display area 431C can use different identifiers to distinguish the speakers.

[0149] The stop button 431D is used to stop recording audio.

[0150] The electronic device 100 can obtain an audio stream through a microphone, and then transmit the audio data in the audio stream to the content recognition module. At the same time, the electronic device 100 performs endpoint detection on the audio stream. When the electronic device 100 detects that a certain frame of audio data in the audio stream does not contain a human voice, the audio stream can be truncated.

[0151] like Figure 4H As shown, as the electronic device 100 collects audio through the microphone, the total audio duration is Figure 4G 00:00:00 becomes 00:00:13, which indicates that the electronic device 100 has collected 13 seconds of audio through the microphone. At the same time, the waveform display area 431B can display the waveform of the 13 seconds of audio. The note application can call the artificial intelligence framework engine when the microphone collects audio, and then the artificial intelligence framework engine can process the audio stream in real time and feed back the results of speech recognition to the note application.

[0152] The speech content display area 431C can display the results of the artificial intelligence framework engine recognizing the audio stream in real time. Figure 4H In the illustrated embodiment, a speech recognition record may be displayed in the speech content display area 431C, and the speech recognition record may include an identity identifier ("Recognizing..."), speech content ("Where are we going to eat lunch today? How about grilled fish, or hot pot is also fine.") and the start time ("00:00:03") of the above speech content appearing in the audio.

[0153] like Fig. 4I As shown, the total duration of the audio recorded by the electronic device 100 becomes 00:00:18. Fig. 4I The user interface displayed by the electronic device 100 in the illustrated embodiment can be regarded as Figure 4H In the illustrated embodiment, the electronic device 100 obtains the audio after collecting 5 seconds of audio. The speech content display area 431C may include three voice recognition records, wherein the first voice recognition record may include an identity identifier ("spokesperson A"), speech content ("Where are we going to eat lunch today?") and the start time and end time ("00:00:03-00:00:07") of the above speech content appearing in the audio. This voice recognition record is obtained after the electronic device recognizes the voice from the third second to the seventh second. The second voice recognition record in the speech content display area 431C may include: an identity identifier ("spokesperson B"), speech content ("How about going to eat grilled fish, or go to eat hot pot.") and the start time and end time ("00:00:07-00:00:13") of the above speech content appearing in the audio. The third voice recognition record may include an identity identifier ("identifying..."), speech content ("Let's go to eat hot pot.") and the start time ("00:00:15") of the above speech content appearing in the audio. Here the first and second voice recognition records can be regarded as Figure 4H The final recognition result of the speech recognition record in the speech content display area 431C is shown. When the electronic device 100 displays the speech recognition record in the speech content display area 431C, the speech content of different speakers can be segmented, so as to avoid the appearance of long paragraphs of text in the speech content display area 431C that affects the reading effect.

[0154] Here, the electronic device 100 is Figure 4G-Figure 4I The process of speech recognition in the scenario shown is described below. Figure 4GIn the illustrated embodiment, the electronic device starts the microphone to obtain audio, and then the electronic device 100 can determine that the audio from 00:00:00 to 00:00:03 does not contain human voice through an endpoint detection method (for example, by calculating whether the short-time energy value of the audio exceeds a threshold), and the electronic device 100 can not input the audio data into the content recognition module during the time period from 00:00:00 to 00:00:03 of the audio stream. When the time when the electronic device records the audio reaches 00:00:03, the electronic device 100 detects that a human voice appears in the audio stream, and then the electronic device 100 can start to input the audio data in the audio stream into the content recognition module. The content recognition module can process the audio data in the audio stream frame by frame. For example, here, taking the speech content of a frame of audio data as "today" as an example, the content recognition module first recognizes "today" from the audio stream, and the electronic device can display "today" in the speech content display area 431C. Since the speech content in the valid audio segment has not been completely recognized, the electronic device 100 can display the identity as "identifying". Then the electronic device 100 can input the next frame of audio data into the content recognition module, and input "today" into the embedding layer at the same time. The content recognition module can process the next frame of audio data through the encoder, and use the embedding layer and the language prediction model to process "we", and then the content recognition module can recognize the next frame of audio data. It should be noted that the speech content that can be contained in a frame of audio data here is only for the convenience of understanding. In fact, the content recognition module can identify "today" from multiple frames of audio data. The content recognition module can process the audio data in the audio stream frame by frame until it determines that the truncation position of the audio stream is reached. Finally, the content recognition module can recognize "Where are we going to eat at noon today? How about grilled fish, or hot pot." Although "Where shall we go for lunch today?" and "How about grilled fish, or hot pot?" are spoken by two speakers, the two sentences may overlap partially, resulting in a very short interval. The electronic device does not separate the two sentences in the preliminary endpoint detection until the sentence "Or we can go for hot pot." ends and the electronic device 100 detects that the audio does not contain the speaker. At this time, the electronic device 100 can cut off the position (i.e., 00:00:13) detected in the audio stream that does not contain the speaker. The electronic device can input the speech content text "Where shall we go for lunch today? How about grilled fish, or hot pot?" and the audio segment from 00:00:03 to 00:00:13 recognized from 00:00:03 to 00:00:13 into the speech segmentation module. The audio segment from 00:00:03 to 00:00:13 can also be called a valid audio segment.Then, the speech segmentation module divides 00:00:03-00:00:13 into two audio sub-segments based on semantic features and audio features, one of which is the audio from 00:00:03 to 00:00:07, and the speech content contained in the audio is: "Where are we going to eat lunch today?", and the other is the audio from 00:00:07 to 00:00:13, and the speech content contained in the audio is "How about eating grilled fish, or hot pot is also OK." In this way, the speech segmentation module divides the long speech spoken by the two speakers. Then, the speech segmentation module can input the two audio sub-segments into the clustering recognition module in turn. The clustering recognition module can cluster the two audio sub-segments into different clusters, and the electronic device can use "speaker A" to identify the first audio sub-segment and use "speaker B" to identify the second audio sub-segment. In this way, the electronic device 100 can. Figure 4H The speech recognition record shown contains two speakers split into Fig. 4I The first and second speech recognition records are shown.

[0155] After the electronic device 100 collects another 2 seconds of audio, the electronic device 100 can determine that the audio stream does not contain human voice after 00:00:18, and the electronic device 100 will truncate the audio stream at 00:00:18. Then the clustering recognition module can distinguish the speakers in the audio sub-segment of 00:00:15-00:00:18, and finally the clustering recognition module can determine that the audio sub-segment of 00:00:15-00:00:18 and the audio sub-segment of 00:00:03-00:00:07 can be clustered into the same cluster, which indicates that the two audio sub-segments contain the voice content of the same speaker, and then the electronic device 100 can also use "Speaker A" to identify the audio sub-segment of 00:00:15-00:00:18. Figure 4J As shown, the total duration of the audio recorded by the electronic device 100 becomes 00:00:20, and the third voice recognition record here may be in the state of recognition completion. For example, the third voice recognition record may include: identity identification ("Speaker A"), speech content ("Then let's go eat hot pot.") and the start time and end time of the speech content in the audio ("00:00:15-00:00:18").

[0156] like Figure 4J As shown, the electronic device 100 receives the operation of the user clicking the stop button 431D. In response to the operation, the electronic device 100 can stop recording audio and stop inputting the audio stream to the artificial intelligence framework engine for voice recognition. Figure 4KThe play button 431E and the slider 431F are shown, and the stop button 431D is transformed into a record button 431G. The play button 431E is used to play the recorded audio, the slider 431F and the slider thereon are used to check and adjust the progress of the audio playback, and the record button 431G can be used to instruct the electronic device to continue recording the audio and recognize the speech content in the audio. Figure 4K As shown, the electronic device 100 can receive a single click operation of the user on the save button 425. In response to the operation, the electronic device 100 can store the note title and note content in the note editing interface 420, wherein the note content includes both the audio of the current note recorded by the electronic device 100 in the note editing interface 420 and the speech content and identity identification recognized by the electronic device 100 while recording the audio. Figure 4L As shown, the electronic device 100 may receive an operation of the user clicking the return button 426, and in response to the operation, the electronic device 100 may return to the Figure 4M The main interface 410 of the note application is shown. Among them, the note display area 411 will add a note list item 411B, and the note list item 411B is used to provide a view of the content of the "lunch location" note (i.e. Figure 4C-4L In the embodiment shown, the electronic device may display an entry for notes created and edited by the electronic device. When the user clicks on the note list item 411B, the electronic device may display the following information: Figure 4M The note editing interface 420 shown in FIG. 4 shows that the note list item 411B may display the title of the note (“lunch location”) and a partial preview of the note (eg, a spoken content text “where are we going to eat at noon today”).

[0157] The following describes a scenario in which an electronic device performs speech recognition on a registered speaker through a voiceprint recognition module.

[0158] like Figure 4N As shown, the electronic device 100 can receive an operation in which a user clicks a note adding button 412 in the main interface 410 of the note application. In response to the operation, the electronic device 100 can create a new note and display Fig.4O The note editing interface 420 is shown. The electronic device 100 may receive and edit the note title of the note to “Meeting Time” in response to the user operation, and then the electronic device 100 may display “Meeting Time” in the title editing box 421 .

[0159] like Fig.4O As shown, the electronic device 100 may receive an operation of the user clicking the recording button 423A, and in response to the operation, the electronic device 100 may display a Figure 4P The menu bar 430C shown in FIG. 4 is shown in FIG. 4 . For an introduction to the menu bar 430C, please refer to Figure 4FThe introduction of the illustrated embodiment will not be repeated here. The electronic device 100 may receive an operation of a user clicking the "Voiceprint Management" option in the menu bar 430C. In response to the operation, the electronic device 100 may display a voiceprint management interface 440, which is used to manage voiceprints. The voiceprint management interface 440 may include a speaker registration button 441 and a voiceprint list 442. The speaker registration button 441 is used to register the speaker's voiceprint. The voiceprint list 442 can be used to view and edit voiceprints. The voiceprint list 442 may include one or more voiceprint options, each of which corresponds to a speaker's voiceprint, and is used to provide the user with an entry to edit the speaker's voiceprint. The user can edit the voiceprint of the speaker corresponding to the voiceprint option through the voiceprint option, for example, re-extract the voiceprint, delete the voiceprint, etc. Exemplarily, the voiceprint list 442 may include an option of "Zhang San" and an option of "Li Si", which indicates that the current electronic device 100 stores voiceprints of two speakers, one of which corresponds to the speaker's identifier "Zhang San" and the other corresponds to the speaker's identifier "Li Si".

[0160] like Figure 4Q As shown, the electronic device 100 can receive a single click operation of the user on the speaker registration button 441. In response to the operation, the electronic device 100 can register the voiceprint of the speaker. The electronic device 100 can send the audio data of a speaker and the identification of the speaker to the voiceprint registration module. The voiceprint registration module can extract the voiceprint from the audio data through the audio segmentation module and the voiceprint feature extraction module, and then the voiceprint registration module can store the speaker identification and the voiceprint of the speaker in the memory. Accordingly, the electronic device 100 can display the voiceprint options corresponding to the newly registered speaker in the voiceprint list 442, such as Figure 4R As shown, a newly registered speaker option, namely, the option "Wang Wu", is added to the voiceprint list 442. This indicates that the electronic device 100 has added a voiceprint of a speaker, and the identification of the speaker is "Wang Wu".

[0161] The electronic device 100 may receive an operation in which the user clicks the return button 426. In response to the operation, the electronic device 100 may exit the voiceprint management interface 440 and redisplay the voiceprint management interface 440. Figure 4S The note editing interface 420 is shown. Figure 4S As shown, the electronic device 100 can receive and respond to the user's click on the "voiceprint recognition speaker" option in the menu bar 430C, and then the electronic device 100 can start recording audio and display the following information: Figure 4TThe voice recognition box 431 shown. At the same time, the electronic device 100 can process the audio stream collected by the microphone through the artificial intelligence framework engine, wherein the electronic device 100 can extract the voiceprint from the audio stream through the voiceprint recognition module in the artificial intelligence framework engine, and match the extracted voiceprint with the pre-stored voiceprint. Furthermore, the electronic device 100 can display the result of the voice recognition on the display screen. Optionally, when the voiceprint of any speaker is not stored in the electronic device 100, the electronic device 100 can receive the operation of the user clicking the "Voiceprint Identification Speaker" option in the menu bar 430C. In response to the operation, the electronic device 100 can prompt the user that the speaker's voiceprint needs to be registered on the electronic device before the electronic device can identify the speaker using voiceprint recognition.

[0162] Figure 4U The electronic device 100 is Figure 4T It is obtained after collecting 13 seconds of audio in the state shown. Among them, the speech content display area 431C can display the result of speech recognition performed by the electronic device 100 based on the 13 seconds of audio. Two speech recognition records can be displayed in the speech content display area 431C, wherein the first speech recognition record may include: speaker identification ("Zhang San"), speech content ("What time does the meeting start today?") and the start time and end time of the speech content in the audio ("00:00:02-00:00:05"). The second speech recognition record may include: speaker identification ("Wang Wu"), speech content ("Well, it should be two o'clock in the afternoon.") and the start time and end time of the speech content in the audio ("00:00:07-00:00:10"). As shown Figure 4V As shown, the electronic device 100 can receive an operation of the user clicking the stop button 431D, and then the electronic device 100 can stop recording the audio and display the following Figure 4W The play button 431E and the slide bar 431F are shown. The introduction of the play button 431E and the slide bar 431F can refer to the above-mentioned embodiment, which will not be repeated here. The electronic device can store the recorded audio and the recognition result displayed in the speech content display area 431C in the memory so that the user can check it at any time.

[0163] In some embodiments, the electronic device 100 may also receive an operation by a user to filter the speech content of one or more speakers. In response to the operation, the electronic device 100 may hide the speech content and identification of other speakers. Figure 4WTaking the embodiment shown as an example, the electronic device 100 can receive an operation by a user to filter the speech content of "Zhang San". In response to the operation, the electronic device 100 can hide the speech recognition records other than "Zhang San" (for example, the speech recognition record of "Wang Wu") and only display the speech recognition record of Zhang San. In this way, the user can only view the speech content of the filtered speaker, thereby improving the user experience.

[0164] In some embodiments, when the electronic device uses the voiceprint recognition module to identify the speaker in response to the user selecting the "voiceprint recognition speaker" option, it may happen that the voiceprints of some audio sub-segments cannot match the voiceprints of the speakers stored in the electronic device or cloud server. This may be because the voiceprint of the speaker in the audio sub-segment is not stored in the electronic device or cloud server. The electronic device 100 can call the clustering recognition module to cluster the audio features in the audio sub-segment, and then distinguish the speakers in the audio sub-segment. Finally, the electronic device 100 can use the speaker role identification (such as "speaker A", "speaker B") to identify the speaker in the audio sub-segment. In this way, even if an unregistered speaker appears in the audio when the user selects voiceprint recognition speaker, the electronic device 100 can distinguish the unregistered speakers in the audio.

[0165] Compared to Figure 4F-4L The embodiment shown, Figure 4P-Figure 4U In the illustrated embodiment, the speaker identification recognized by the electronic device 100 can point to a specific speaker, because the speaker's voiceprint is stored in the memory of the electronic device 100 or in the cloud server. Figure 4P-Figure 4U Compared with the embodiment shown Figure 4F-4L The illustrated embodiment has a more accurate recognition result.

[0166] When the electronic device 100 performs speech recognition on a registered speaker, the method by which the electronic device processes the audio stream through the content recognition module and the speech segmentation module can refer to the method by which the electronic device performs speech recognition on an unregistered speaker in the aforementioned embodiment, and will not be repeated here. When the electronic device 100 performs speech recognition on a registered speaker, the electronic device 100 can use the content recognition module and the speech segmentation module to process the audio stream, and then call the voiceprint recognition module to recognize the voiceprint in the audio sub-segment, and then determine the speaker of the audio sub-segment.

[0167] That is, when the user has not registered the speaker, he can choose the speaker role recognition method without registration, and the electronic device 100 will distinguish different speaker roles through the clustering recognition module; when the user has registered the speaker, the speaker can be identified by voiceprint recognition, so that the electronic device can identify the specific speaker through the voiceprint recognition module. In this way, regardless of whether the user has registered the speaker, the electronic device can distinguish different speakers during voice recognition, thereby improving the user experience.

[0168] Figure 5 It is a flowchart diagram of a speech recognition method provided in an embodiment of the present application.

[0169] like Figure 5 As shown, the speech recognition method may include but is not limited to the following steps:

[0170] S501: An electronic device receives a first operation of a user selecting a voice recognition mode, where the voice recognition mode includes a first recognition mode and a second recognition mode.

[0171] by Figure 4F Taking the note editing interface 420 shown as an example, the first operation may be an operation in which the user clicks the "recognize voice content only" option, the "distinguish speakers by registration-free" option, or the "recognize speakers by voiceprint" option in the menu bar 430. The first recognition method may refer to the electronic device using the voiceprint recognition method to recognize the speaker in the voice. When the "recognize speakers by voiceprint" option is selected in the first operation, it indicates that the user has selected the voiceprint recognition method to recognize the speaker, and the electronic device may call the voiceprint recognition module to recognize the speaker; the second recognition method may refer to the electronic device using the cluster recognition method to recognize the speaker in the voice. When the "distinguish speakers by registration-free" option is selected in the first operation, it indicates that the user has selected the cluster recognition method to recognize the speaker, and the electronic device may call the cluster recognition module to recognize the speaker.

[0172] S502: The electronic device obtains a first audio segment, where the first audio segment includes a first sub-segment corresponding to a first speech content.

[0173] The first audio segment may be a valid audio segment, the first audio segment may include a first sub-segment, and the speech content of the first sub-segment is the first speech content. Figure 4H-Figure 4ITaking the illustrated embodiment as an example, the first audio segment may be a valid audio segment from 00:00:03 to 00:00:13 in the audio stream. A valid audio segment may refer to an audio segment with coherent sound. Here, coherent sound may refer to the continuous presence or appearance of the human voice part in the audio, without interruption or the duration of the interruption does not exceed a preset duration, that is, the pause duration of the speaker's voice in the audio does not exceed a preset duration. The electronic device 100 may determine whether the sound in the audio stream is coherent by calculating the short-time energy value or short-time zero-crossing rate of the audio in the sliding window. The first sub-segment may be the audio segment corresponding to 00:00:03-00:00:07, where the first speech content may be, for example, "Where are we going to eat at noon today?".

[0174] S503: The electronic device determines the voice recognition method selected by the first operation.

[0175] The electronic device may determine the speech recognition mode based on the user's first operation. The method for the electronic device to determine the speech recognition mode selected by the first operation may refer to the introduction in step S501, which will not be repeated here. The electronic device may call the content recognition module to recognize the speech content of the first audio segment, and then the electronic device may input the first audio segment and the speech content of the first audio segment into the speech segmentation module, and then the speech segmentation module may segment the first audio segment based on the semantic text features of the speech content and the audio features of the first audio segment to obtain the first sub-segment. Then the electronic device may input the first sub-segment into the voiceprint recognition module or the cluster recognition module based on the first operation.

[0176] When the electronic device determines that the first operation selects the first identification method, step S504 may be executed. When the electronic device determines that the first operation selects the second identification method, step S505 may be executed.

[0177] S504: The electronic device determines that the voiceprint feature of the first sub-segment matches the first voiceprint feature, and displays the first speech content and the first identifier, where the first identifier indicates that the first speech content is spoken by the first speaker.

[0178] When the first operation selects the first recognition mode, the electronic device may call the voiceprint recognition module to process the first sub-segment. The voiceprint recognition module may determine that the voiceprint feature of the first sub-segment matches the first voiceprint feature, and then the electronic device may display the first speech content and the first identifier. The electronic device may pre-store the first identifier and the first voiceprint feature of the first speaker, so that the speaker can be matched based on the stored first voiceprint feature.

[0179] by Figure 4RTaking the embodiment shown as an example, the electronic device 100 stores the voiceprints of Zhang San, Li Si, and Wang Wu. Here, the first voiceprint feature may refer to the voiceprint feature of Zhang San, and the first speaker may refer to Zhang San. In this way, after the voiceprint recognition module obtains the audio sub-segment, it can extract the voiceprint feature in the audio sub-segment and match it with the voiceprints of Zhang San, Li Si, and Wang Wu. Figure 4U As shown, the first sub-segment may be the audio sub-segment from 00:00:02 to 00:00:05 in the audio stream. The audio sub-segment from 00:00:02 to 00:00:05 may be obtained by the speech segmentation module after segmenting the valid audio segment based on the first speech content ("What time does the meeting start today?"). The electronic device may display the first speech content and the first identifier of the first speaker (Zhang San) (i.e. Figure 4U The "Zhang San" logo in the figure).

[0180] S505: The electronic device determines that the first speech content is spoken by a speaker, and displays the first speech content and a second identifier, wherein the second identifier is used to identify the speaker of the first speech content.

[0181] When the second identification method is selected in the first operation, the electronic device can call the cluster identification module to process the first sub-segment. The cluster identification module can determine that the first speech content is spoken by a speaker, and then the cluster identification module can display the first speech content and the second identifier, and the second identifier is used to identify the speaker of the first speech content.

[0182] by Fig. 4I Taking the illustrated embodiment as an example, the first sub-segment may be, for example, the audio sub-segment from 00:00:03 to 00:00:07 in the audio stream. The audio sub-segment may be obtained after the speech segmentation module segments the valid audio segment. Here, the first speech content may be, for example, "Where are we going to eat today?". Then the clustering recognition module may determine that the audio sub-segment contains only one speaker, and then the clustering recognition module may cluster the audio features of each short audio segment in the audio sub-segment into the same cluster (or class), and assign a second identifier to the class. Here, the second identifier may be, for example, "Speaker A".

[0183] That is to say, the electronic device can provide the user with an entrance for pre-registered and unregistered speakers while recognizing the spoken content. In this way, regardless of whether the user registers the speaker, the electronic device can distinguish the speaker of each spoken content and display the speaker distinction result on the display screen. When the user selects a speech recognition method that requires pre-registration of the speaker, the electronic device can provide the user with a more accurate speaker identification (such as the speaker's name), so that the user can determine which speaker is speaking each sentence based on the speaker identification.

[0184] In some embodiments, after the electronic device obtains the first audio segment, the method further includes: the electronic device divides the first audio segment into multiple sub-segments, the multiple sub-segments include the first sub-segment, and the semantics of the multiple sub-segments are different. Among them, the electronic device can segment the first audio segment through a speech segmentation module to obtain multiple sub-segments. In an embodiment of the present application, the sub-segments may also be referred to as audio sub-segments. Among them, the speech segmentation module can segment the first audio segment based on the text semantic features of the speech content contained in the first audio segment and the audio features of the first audio segment, and finally the speech segmentation module can segment out multiple sub-segments with different semantics. Figure 4H-Figure 4I Taking the shown embodiment as an example, the first audio segment may include "Where are we going to eat lunch today? How about grilled fish, or hot pot is also fine", then the speech segmentation module can segment it into two sub-segments, where the speech content of the first sub-segment is "Where are we going to eat lunch today?", and the speech content of the other sub-segment can be "How about grilled fish, or hot pot is also fine." The semantics of the speech content of the two sub-segments are different.

[0185] In some embodiments, the first audio segment includes a second sub-segment, the second sub-segment includes a second speech content of a second speaker, and the second voiceprint feature of the second speaker is not stored in the electronic device. The method also includes: when the second recognition method is selected in the first operation, the electronic device determines that the second speech content is spoken by a speaker, and the electronic device displays the second speech content and a third identifier, where the third identifier is used to identify the speaker of the second speech content.

[0186] When the first identification method (i.e., voiceprint identification of the speaker) is selected, the electronic device calls the voiceprint identification module to determine whether the voiceprint in each sub-segment matches the pre-stored voiceprint feature. When the second voiceprint feature of the second speaker is not stored in the electronic device, the electronic device cannot identify the second speaker by calling the voiceprint identification module. The electronic device can call the cluster identification module to process the second sub-segment, and then use the role identifier (such as speaker A) to distinguish the speaker of the second sub-segment. In this way, when the user selects voiceprint recognition, the electronic device can also distinguish the speech content of the unregistered speaker.

[0187] In some embodiments, the first audio segment includes a second sub-segment, the second sub-segment includes a second speech content of a second speaker, and the electronic device stores a second voiceprint feature and a fourth identifier of the second speaker. The method also includes: when the first operation selects the first recognition method, the electronic device determines that the voiceprint feature of the second sub-segment matches the second voiceprint feature, and the electronic device displays the second speech content and the fourth identifier, and the fourth identifier indicates that the second speech content is spoken by the second speaker.

[0188] by Figure 4U Taking the illustrated embodiment as an example, the first sub-segment may be an audio sub-segment of 00:00:02-00:00:05, and the second sub-segment may be an audio sub-segment of 00:00:07-00:00:10. The electronic device stores the voiceprint features of Zhang San and Wang Wu, as well as the identifiers of Zhang San and Wang Wu. Here, Zhang San may be the first speaker, the voiceprint feature of Zhang San is the first voiceprint feature, and Zhang San's name "Zhang San" is the first identifier; Wang Wu may be the second speaker, the voiceprint feature of Wang Wu may be called the second voiceprint feature, and Wang Wu's name "Wang Wu" may be called the fourth identifier. If the first operation selects voiceprint recognition, the voiceprint recognition module may recognize that the voiceprint of the first sub-segment matches the first voiceprint feature of Zhang San, and then the electronic device may display the first identifier of Zhang San next to the first speech content. Similarly, the voiceprint recognition module may recognize that the voiceprint of the second sub-segment matches the second voiceprint feature of Wang Wu, and then the electronic device may display the fourth identifier of Wang Wu next to the second speech content.

[0189] In some embodiments, after the electronic device displays the second speech content and the fourth identifier, the method further includes: the electronic device receives a second operation, the second operation is used to filter the first speaker; in response to the second operation, the electronic device hides the second speech content and the fourth identifier.

[0190] The second operation may be an operation of filtering the speech content of a certain speaker. In response to this operation, the electronic device may hide the speaker's identification and speech content and only display the speech content of the filtered speaker.

[0191] In some embodiments, the first audio clip also includes a third sub-clip, the third sub-clip includes a third speech content, and the speaker of the third speech content is different from the speaker of the first speech content. The method also includes: when the second identification method is selected in the first operation, the electronic device determines that the third speech content is spoken by a speaker, and the electronic device displays the third speech content and a fifth identifier, and the fifth identifier is used to indicate that the speaker of the third speech content is different from the speaker of the first speech content.

[0192] That is to say, when the second identification method is selected in the first operation, the electronic device can call the cluster identification module to determine the speaker in the third sub-segment. The cluster identification module determines that the third sub-segment contains a speaker, and after clustering the audio features of the short audio segments in the third sub-segment, the cluster identification module can determine that the audio features contained in the third sub-segment and the audio features contained in the first sub-segment do not belong to the same category, and then the electronic device can use the fifth identifier to identify the speaker of the third sub-segment, wherein the fifth identifier is different from the second identifier, for example, the fifth identifier can be "speaker B", and the second identifier can be "speaker A", both of which are used to indicate that the speaker of the third speech content is different from the speaker of the first speech content.

[0193] In some embodiments, the first audio clip also includes a fourth sub-clip, the fourth sub-clip includes a fourth speech content, and the method also includes: when the second identification method is selected in the first operation, the electronic device determines that the fourth speech content is spoken by a speaker, and the speaker of the fourth speech content is the same as the speaker of the first speech content, the electronic device displays the fourth speech content, and displays the second identifier again at the first position associated with the fourth speech content, and the second identifier displayed at the first position is used to indicate the speaker of the fourth speech content.

[0194] That is to say, when the second identification method is selected in the first operation, the electronic device determines through the cluster identification module that the speaker contained in the fourth sub-segment is the same as the speaker of the first speech content, that is, the audio features of the audio clip contained in the fourth sub-segment can be classified into the same category as the audio features of the audio clip contained in the first sub-segment. In this way, the electronic device can assign the same identifier as the first sub-segment (that is, the second identifier) ​​to the fourth sub-segment, indicating that the speaker contained in the fourth sub-segment is the same as the first sub-segment. Among them, the electronic device can display the second identifier again at the first position, and displaying the second identifier again here can mean that the electronic device also displays the second identifier at the first position on the basis of displaying the second identifier at the second position. The second identifier displayed at the second position is used to indicate the speaker of the first speech content, and the second identifier displayed at the first position is used to indicate the speaker of the fourth speech content. The first position is different from the second position.

[0195] In some embodiments, before the electronic device receives a first operation in which a user selects a voice recognition method, the method further includes: the electronic device receives a second audio segment and a first identifier input by the user, the speaker of the second audio segment being a first speaker; the electronic device extracts a first voiceprint feature of the first speaker from the second audio segment; the electronic device stores the first voiceprint feature and the first identifier, wherein the first voiceprint feature and the first identifier are stored in association.

[0196] The electronic device may receive an operation by a user to register a speaker (eg Figure 4Q The operation of clicking the speaker registration button 441 shown in the figure) can then receive the second audio segment and the first mark input by the user in the voiceprint registration module of the electronic device. The voiceprint registration module can input the second audio segment and the first mark into the voiceprint recognition module, and the voiceprint recognition module can extract the first voiceprint feature of the first speaker from the second audio segment. Then the electronic device can store the first voiceprint feature and the first mark. Figure 4Q The electronic device shown receives the operation of the user clicking the speaker registration button 441, and then the electronic device can receive the second audio segment and the first tag input by the user. Here, the first tag can be, for example, "Wang Wu". In this way, the electronic device can extract the first voiceprint feature of the speaker "Wang Wu" in the second audio segment, and store the first voiceprint feature in association with "Wang Wu". When the electronic device determines that the voiceprint feature of the audio sub-segment matches the first voiceprint feature when identifying the audio sub-segment, it can use "Wang Wu" to mark the speech content of the audio sub-segment.

[0197] In some embodiments, the method also includes: the electronic device collects a second audio segment through a microphone while displaying the first speech content; and the electronic device recognizes the speech content and speaker of the second audio segment based on the voice recognition method selected by the first operation.

[0198] That is to say, the electronic device performs voice recognition in real time while recording audio. The electronic device can also collect the second audio clip while displaying the first speech content. Then, after displaying the first speech content, the electronic device can recognize the speech content and speaker of the second audio clip.

[0199] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0200] As used in the above embodiments, the term "when..." may be interpreted to mean "if..." or "after..." or "in response to determining..." or "in response to detecting...", depending on the context. Similarly, the phrases "upon determining..." or "if (the stated condition or event) is detected" may be interpreted to mean "if determining..." or "in response to determining..." or "upon detecting (the stated condition or event)" or "in response to detecting (the stated condition or event)", depending on the context.

[0201] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media integration. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk), etc.

[0202] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.

Claims

1. A speech recognition method, the method being applied to an electronic device, characterized in that: The electronic device stores a first identifier and a first voiceprint feature of a first speaker, and the method includes: The electronic device receives a first operation of a user selecting a voice recognition mode, wherein the voice recognition mode includes a first recognition mode and a second recognition mode; The electronic device acquires a first audio segment, where the first audio segment includes a first sub-segment corresponding to a first speech content; In a case where the first operation selects the first identification mode, the electronic device determines that the voiceprint feature of the first sub-segment matches the first voiceprint feature, and the electronic device displays the first speech content and the first identifier, wherein the first identifier indicates that the first speech content is spoken by the first speaker; When the second identification method is selected in the first operation, the electronic device determines that the first speech content is spoken by a speaker, and the electronic device displays the first speech content and a second identifier, where the second identifier is used to identify the speaker of the first speech content.

2. The method according to claim 1, characterized in that: After the electronic device acquires the first audio segment, the method further includes: The electronic device divides the first audio segment into a plurality of sub-segments, wherein the plurality of sub-segments include the first sub-segment, and the plurality of sub-segments have different semantics.

3. The method according to claim 1 or 2, characterized in that: The first audio segment includes a second sub-segment, the second sub-segment includes a second speech content of a second speaker, the electronic device does not store a second voiceprint feature of the second speaker, and the method further includes: When the first operation selects the first identification method, the electronic device determines that the second speech content is spoken by a speaker, and the electronic device displays the second speech content and a third identifier, where the third identifier is used to identify the speaker of the second speech content.

4. The method according to claim 1 or 2, characterized in that: The first audio segment includes a second sub-segment, the second sub-segment includes a second speech content of a second speaker, the electronic device stores a second voiceprint feature and a fourth identifier of the second speaker, and the method further includes: When the first operation selects the first recognition method, the electronic device determines that the voiceprint feature of the second sub-segment matches the second voiceprint feature, and the electronic device displays the second speech content and the fourth identifier, wherein the fourth identifier indicates that the second speech content is spoken by the second speaker.

5. The method according to claim 4, characterized in that After the electronic device displays the second speech content and the fourth identifier, the method further includes: The electronic device receives a second operation, where the second operation is used to screen the first speaker; In response to the second operation, the electronic device hides and displays the second utterance content and the fourth identifier.

6. The method according to any one of claims 1 to 5, characterized in that The first audio segment also includes a third sub-segment, the third sub-segment includes a third speech content, and the speaker of the third speech content is different from the speaker of the first speech content. The method further includes: When the second identification method is selected in the first operation, the electronic device determines that the third speech content is spoken by a speaker, and the electronic device displays the third speech content and a fifth identifier, wherein the fifth identifier is used to indicate that the speaker of the third speech content is different from the speaker of the first speech content.

7. The method according to any one of claims 1 to 6, characterized in that The first audio segment also includes a fourth sub-segment, and the fourth sub-segment includes fourth speech content. The method further includes: In the case where the second identification method is selected in the first operation, the electronic device determines that the fourth speech content is spoken by a speaker, and the speaker of the fourth speech content is the same as the speaker of the first speech content, the electronic device displays the fourth speech content, and displays the second identifier again at the first position associated with the fourth speech content, and the second identifier displayed at the first position is used to indicate the speaker of the fourth speech content.

8. The method according to any one of claims 1 to 7, characterized in that Before the electronic device receives a first operation of a user selecting a voice recognition mode, the method further includes: The electronic device receives a second audio segment and the first identifier input by a user, wherein the speaker of the second audio segment is the first speaker; The electronic device extracts the first voiceprint feature of the first speaker from the second audio segment; The electronic device stores the first voiceprint feature and the first identifier, wherein the first voiceprint feature and the first identifier are stored in association with each other.

9. The method according to any one of claims 1 to 8, characterized in that The method further comprises: The electronic device collects a second audio clip through a microphone during the process of displaying the first speech content; The electronic device recognizes the speech content and speaker of the second audio segment based on the speech recognition method selected by the first operation.

10. An electronic device, characterized in that: The electronic device comprises: A storage module, used to store a first identifier and a first voiceprint feature of a first speaker; A user interaction module, configured to receive a first operation of a user selecting a voice recognition mode, wherein the voice recognition mode includes a first recognition mode and a second recognition mode; An audio acquisition module, the audio acquisition module is used to acquire a first audio segment, the first audio segment includes a first sub-segment corresponding to a first speech content; a voiceprint recognition module, configured to determine, when the first operation selects the first recognition mode, that the voiceprint feature of the first sub-segment matches the first voiceprint feature; A display module, configured to display the first speech content and the first identifier when the first operation selects the first recognition mode, wherein the first identifier indicates that the first speech content is spoken by the first speaker; a cluster identification module, configured to determine, when the second identification mode is selected in the first operation, that the first speech content is spoken by one speaker based on acoustic features of the first audio segment; The display module is further configured to display the first speech content and a second identifier when the second identification mode is selected in the first operation, wherein the second identifier is used to identify a speaker of the first speech content.

11. The electronic device according to claim 10, characterized in that: The electronic device also includes a speech recognition module and a speech segmentation module; the content recognition module is used to identify the speech content contained in the first audio segment; the speech segmentation module is used to divide the first audio segment into multiple sub-segments based on the speech content contained in the first audio segment, the multiple sub-segments include the first sub-segment, and the semantics of the multiple sub-segments are different.

12. The electronic device according to claim 10 or 11, characterized in that: The first audio segment includes a second sub-segment, the second sub-segment includes a second speech content of a second speaker, and the storage module does not store a second voiceprint feature of the second speaker; When the second identification method is selected in the first operation, the cluster identification module is also used to determine that the second speech content is spoken by a speaker; the display module is also used to display the second speech content and a third identifier, and the third identifier is used to identify the speaker of the second speech content.

13. The electronic device according to claim 10 or 11, characterized in that: The first audio segment includes a second sub-segment, the second sub-segment includes a second speech content of a second speaker, and the storage module stores a second voiceprint feature and a fourth identifier of the second speaker; When the first operation selects the first recognition method, the voiceprint recognition module is also used to determine whether the voiceprint feature of the second sub-segment matches the second voiceprint feature; the display module is also used to display the second speech content and the fourth identifier, and the fourth identifier indicates that the second speech content is spoken by the second speaker.

14. The electronic device according to any one of claims 10 to 13, characterized in that: The first audio segment also includes a third sub-segment, the third sub-segment includes a third speech content, and the speaker of the third speech content is different from the speaker of the first speech content; When the first operation selects the second identification method, the cluster identification module is also used to determine that the third speech content is spoken by one speaker; the display module is also used to display the third speech content and a fifth identifier, and the fifth identifier is used to indicate that the speaker of the third speech content is different from the speaker of the first speech content.

15. The electronic device according to any one of claims 10 to 14, characterized in that: The first audio segment also includes a fourth sub-segment, and the fourth sub-segment includes a fourth speech content; In the case where the second identification method is selected in the first operation, the cluster identification module is further used to determine that the fourth speech content is spoken by a speaker, and the speaker of the fourth speech content is the same as the speaker of the first speech content; the display module is further used to display the fourth speech content and display the second identifier again at the first position associated with the fourth speech content, and the second identifier displayed at the first position is used to indicate the speaker of the fourth speech content.

16. The electronic device according to any one of claims 10 to 15, characterized in that: The electronic device further includes a voiceprint registration module, wherein the voiceprint recognition module is used to receive a second audio segment and the first identifier input by a user, wherein the speaker of the second audio segment is the first speaker; The voiceprint registration module is further used to send the second audio segment to the voiceprint recognition module; The voiceprint recognition module is further used to extract the first voiceprint feature of the first speaker from the second audio segment, and send the first voiceprint feature to the voiceprint registration module; The voiceprint registration module is further used to receive the first voiceprint feature, and send the first voiceprint feature and the first identifier to the storage module; The storage module is further used to store the first voiceprint feature and the first identifier, wherein the first voiceprint feature and the first identifier are stored in association.

17. An electronic device, characterized in that: The electronic device comprises: a display screen, a memory, and a processor coupled to the memory; the display screen is used to display a user interface, the memory stores a computer program, and when the processor executes the computer program, the electronic device implements the method as described in any one of claims 1 to 9.

18. A computer-readable storage medium storing computer instructions, characterized in that: When the computer instructions are executed on an electronic device, the electronic device is caused to execute the method as claimed in any one of claims 1 to 9.

Images