Audio processing method and electronic device

By adopting end-to-end solutions and semantic information correction technology with fusion training in AI conference minutes, the problem of low robustness of audio processing algorithms in AI conference minutes is solved, and more accurate speaker identity identification and content segmentation are achieved, improving the ability to improve the The robustness of the entire audio processing system.

CN118447853BActive Publication Date: 2024-12-06SHANGHAI GLORY SMART TECH DEV CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202311171185.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-09-11
Publication Date
2024-12-06
Estimated Expiration
2043-09-11

AI Technical Summary

Technical Problem

In AI conference minutes, due to factors such as noise, reverb, voiceprint drift, and voice overlap, the audio processing algorithm is less robust, resulting in inaccurate identification of speakers and content segmentation.

Method used

Using an end-to-end scheme based on fusion training, the speaker's identity identification and content extraction are performed through the acoustic and vocalprint feature extraction of speech signals, combined with semantic information, and using speech activity detection and change point segmentation to enhance the speaker's identity characteristics, and Through historical speech content and emotional analysis correction results, the robustness of the audio processing algorithm is improved.

Benefits of technology

It significantly improves the accuracy of speaker identity identification and the robustness of audio processing algorithms, reduces the problems of inaccurate segmentation clustering and overflow, and improves the processing ability for multi-speaker scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118447853B_ABST
    Figure CN118447853B_ABST
Patent Text Reader

Abstract

The present application provides an audio processing method and electronic device. The method comprises: inputting the separated speech signal corresponding to the target audio into a trained speaker and content extraction model, and the speaker and content extraction model outputs the target speech content and at least two candidate speakers corresponding to the separated speech signal; and determining the target speaker corresponding to the target speech content from at least two candidate speakers according to the historical speech content and the target speech content of the at least two candidate speakers. In this way, by adopting an end-to-end solution based on fusion training and assisting in correcting the speaker identification results with semantic information, the robustness of speaker identity identification clustering is improved, thereby improving the robustness of the entire audio processing algorithm.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of terminal equipment, and in particular to an audio processing method and electronic equipment. Background Art

[0002] AI (Artificial Intelligence) meeting minutes provide intelligent meeting minutes for meetings in various industries, realizing the conversion of speeches in office meetings, government reports, press releases, academic lectures and other activities into text for screen display, and recording by role to make meetings smoother.

[0003] The goal of AI meeting minutes is to clearly express "who when spoke what" in the voice stream. Affected by noise, reverberation, voiceprint drift, voice overlap, etc., the robustness of AI meeting minutes in related technologies is relatively low. Summary of the invention

[0004] In order to solve the above technical problems, the present application provides an audio processing method and an electronic device, which can improve the robustness of the audio processing algorithm.

[0005] In the first aspect, the present application provides an audio processing method, the method comprising: inputting a separated speech signal corresponding to the target audio into a trained speaker and content extraction model, and the speaker and content extraction model outputs the target speech content and at least two candidate speakers corresponding to the separated speech signal; and determining the target speaker corresponding to the target speech content from the at least two candidate speakers according to the historical speech content and the target speech content of the at least two candidate speakers. In this way, by adopting an end-to-end solution based on fusion training and assisting in correcting the speaker identification results with semantic information, the robustness of speaker identity identification clustering is improved, thereby improving the robustness of the entire audio processing algorithm.

[0006] According to the first aspect, the speaker and content extraction model processes the separated speech signal including: extracting target acoustic features from the separated speech signal; extracting target voiceprint features from the separated speech signal; enhancing the target voiceprint features based on the target acoustic features to obtain enhanced target identity features; inputting the target acoustic features and the enhanced target identity features into the trained joint training model, and the joint training model outputs the target speech content and at least two candidate speakers corresponding to the separated speech signal. In this way, the speaker and speech content are obtained through identity feature enhancement and end-to-end joint training, which improves the robustness of identity recognition and thus improves the robustness of the entire algorithm.

[0007] According to the first aspect, the processing process of the separated speech signal by the speaker and content extraction model includes: extracting target acoustic features from the separated speech signal; inputting the target acoustic features into a trained acoustic feature encoding model, and obtaining first high-dimensional information by high-dimensional spatial encoding of the target acoustic features output by the acoustic feature encoding model; extracting target voiceprint features from the separated speech signal; inputting the target voiceprint features into a trained identity feature encoding model, and obtaining second high-dimensional information by high-dimensional spatial encoding of the target voiceprint features output by the identity feature encoding model; based on the first high-dimensional information, enhancing the second high-dimensional information to obtain enhanced target identity features; inputting the target acoustic features and the enhanced target identity features into a trained joint training model, and the joint training model outputs the target speech content and at least two candidate speakers corresponding to the separated speech signal.

[0008] According to the first aspect, based on the historical speech content and the target speech content of at least two candidate speakers, a target speaker corresponding to the target speech content is determined from at least two candidate speakers, including: for each candidate speaker, determining the probability score that the target speech content and the previous speech content of the candidate speaker belong to the same person; and determining the candidate speaker with the largest probability score among the at least two candidate speakers as the target speaker corresponding to the target speech content. In this way, the speaker's intention can be understood through the probability score that the target speech content and the previous speech content of the candidate speaker belong to the same person, so that the speaker can be corrected based on the understanding of the speaker's intention, which can greatly improve the robustness of the audio processing algorithm.

[0009] According to the first aspect, determining the probability score that the target speech content and the previous speech content of the candidate speaker belong to the same person includes: obtaining the correlation value of the first text corresponding to the target speech content and the second text corresponding to the previous speech content of the candidate speaker, and determining the correlation value as the probability score that the target speech content and the previous speech content of the candidate speaker belong to the same person. In this way, using text modeling technology, the probability score that the target speech content and the previous speech content of the candidate speaker belong to the same person can be obtained, which provides a basis for correcting the speaker based on the understanding of the speaker's intention, and can greatly improve the robustness of the audio processing algorithm.

[0010] According to the first aspect, determining the probability score that the target speech content and the previous speech content of the candidate speaker belong to the same person includes: extracting the first language feature corresponding to the target speech content; extracting the second language feature corresponding to the previous speech content of the candidate speaker; determining the similarity value between the first language feature and the second language feature, and determining the similarity value as the probability score that the target speech content and the previous speech content of the candidate speaker belong to the same person. In this way, using the language feature extraction technology, the probability score that the target speech content and the previous speech content of the candidate speaker belong to the same person can be obtained, which provides a basis for correcting the speaker based on the understanding of the speaker's intention, and can greatly improve the robustness of the audio processing algorithm.

[0011] According to the first aspect, the speech features included in the first language feature and the second language feature include at least one of the following features: word frequency feature, length feature, and pause feature.

[0012] According to the first aspect, determining the probability score that the target speech content and the previous speech content of the candidate speaker belong to the same person includes: performing sentiment analysis on the target speech content to obtain a first sentiment state; performing sentiment analysis on the previous speech content of the candidate speaker to obtain a second sentiment state; determining a similarity value between the first sentiment state and the second sentiment state, and determining the similarity value as the probability score that the target speech content and the previous speech content of the candidate speaker belong to the same person. In this way, using sentiment analysis technology, the probability score that the target speech content and the previous speech content of the candidate speaker belong to the same person can be obtained, which provides a basis for correcting the speaker based on the understanding of the speaker's intention, and can greatly improve the robustness of the audio processing algorithm.

[0013] According to the first aspect, after determining the target speaker corresponding to the target speech content from at least two candidate speakers based on the historical speech content of at least two candidate speakers and the target speech content, the method further includes: storing the target speech content as the previous speech content of the target speaker. In this way, the previous speech content of the target speaker can be updated in a timely manner, so that a more accurate comparison basis is provided when determining which speaker the target speech content belongs to again in the future.

[0014] According to the first aspect, before the separated speech signal corresponding to the target audio is input into the trained speaker and content extraction model, and the speaker and content extraction model outputs the target speech content and at least two candidate speakers corresponding to the separated speech signal, the method further includes: performing speech activity detection on the target audio to obtain the speech signal to be separated; detecting the speaker change point in the speech signal to be separated; and dividing the speech signal to be separated into multiple separated speech signals according to the detected speaker change point. In this way, the input information of the speaker and content extraction model can be obtained, so as to determine the target speech content and the candidate speakers according to the input information.

[0015] In a second aspect, the present application provides an electronic device, comprising: a memory and a processor, wherein the memory is coupled to the processor; the memory stores program instructions, and when the program instructions are executed by the processor, the electronic device executes any one of the audio processing methods of the first aspect.

[0016] In a third aspect, the present application provides a computer-readable storage medium, comprising a computer program. When the computer program is executed on an electronic device, the electronic device executes the audio processing method of any one of the aforementioned first aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a schematic structural diagram of an illustrative electronic device 100;

[0018] Figure 2 1 is a software structure block diagram of an electronic device 100 according to an embodiment of the present application;

[0019] Figure 3 This is an example diagram of the audio processing flow in the AI ​​meeting minutes;

[0020] Figure 4 An example flow chart of an audio processing method is shown as an example;

[0021] Figure 5 Another example flow chart of an audio processing method is shown as an example;

[0022] Figure 6 is another example flow chart of an audio processing method shown exemplarily;

[0023] Figure 7 The figure is an example diagram of a process of extracting speakers and content. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0025] The term "and / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0026] The terms "first" and "second" in the description and claims of the embodiments of the present application are used to distinguish different objects rather than to describe a specific order of objects. For example, a first target object and a second target object are used to distinguish different target objects rather than to describe a specific order of target objects.

[0027] This embodiment provides an audio processing method, which can be applied to electronic devices such as tablets and PCs (Personal Computers).

[0028] In this embodiment, the structure of the electronic device can be as follows Figure 1 This structure is only exemplary and is not intended to limit the structure of the electronic device. In other embodiments, the electronic device may also adopt other structures.

[0029] Figure 1 FIG. 1 is a schematic diagram of the structure of an electronic device 100 exemplarily shown. It should be understood that Figure 1 The illustrated electronic device 100 is merely one example of an electronic device, and the electronic device 100 may have more or fewer components than those shown in the figure, may combine two or more components, or may have a different configuration of components. Figure 1 The various components shown in the EMBODIMENTS 2000 may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.

[0030] See also Figure 1The electronic device 100 may include: a processor 110, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, an indicator 192, a camera 193, etc.

[0031] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0032] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0033] A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory.

[0034] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0035] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0036] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0037] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.

[0038] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0039] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0040] Video codecs are used to compress or decompress digital videos. The electronic device 100 may support one or more video codecs. Thus, the electronic device 100 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0041] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0042] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0043] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0044] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0045] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0046] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0047] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The electronic device 100 can listen to music or listen to a hands-free call through the speaker 170A.

[0048] The receiver 170B, also called a "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 100 receives a call or voice message, the voice can be received by placing the receiver 170B close to the human ear.

[0049] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to microphone 170C to input the sound signal into microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify the sound source, realize directional recording function, etc.

[0050] The earphone interface 170D is used to connect a wired earphone and can be a USB interface 130 or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0051] Figure 2 1 is a block diagram of the software structure of the electronic device 100 according to an embodiment of the present application.

[0052] The layered architecture of the electronic device 100 divides the software into several layers, each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system may include an application layer, an application framework layer, a system library, and a kernel layer.

[0053] The application layer can include a series of application packages.

[0054] like Figure 2 As shown, the application package may include camera, gallery, map, WLAN, music, short message, call, navigation, Bluetooth, video and other applications. Of course, these applications are only exemplary. In other embodiments, the application layer may include Figure 2 Applications not shown in the figure may also not include Figure 2 One or more applications shown in .

[0055] like Figure 2 As shown, the application package may also include an audio processing module, which is used to execute the audio processing method of the embodiment of the present application.

[0056] It should be noted that the audio processing module can be used as a separate application (APP) or as a new functional module in an existing application.

[0057] like Figure 2 As shown, the application framework layer may include a window manager, a resource manager, a content provider, a view system, etc.

[0058] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0059] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0060] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0061] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0062] Android Runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.

[0063] The core library consists of two parts: one is the function that needs to be called by the Java language, and the other is the Android core library.

[0064] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0065] The system library may include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0066] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0067] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0068] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0069] A 2D graphics engine is a drawing engine for 2D (two-dimensional) drawing.

[0070] The kernel layer is the layer between hardware and software.

[0071] like Figure 2 As shown, the kernel layer may include display driver, Wi-Fi driver, audio driver, sensor driver, Bluetooth driver, etc.

[0072] Understandably, Figure 2 The layers in the software structure shown and the components included in each layer do not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer layers than shown, and each layer may include more or fewer components, which is not limited in the present application.

[0073] AI meeting minutes are mainly realized by speaker segmentation clustering (Speaker Diarization, SD for short) + speech recognition technology. The audio processing method of each embodiment of the present application can be used in AI meeting minutes. Of course, in addition to the AI ​​meeting minutes scenario, the audio processing method of each embodiment of the present application can also be used in other scenarios.

[0074] The following uses the AI ​​meeting minutes scenario as an example to illustrate the audio processing method of this application.

[0075] Figure 3 This is an example diagram of the audio processing flow in the AI ​​conference minutes. Figure 3 In this embodiment, in the audio processing flow, automatic speech recognition (ASR) and speaker diarization (SD) are performed on the audio obtained in the meeting. Through automatic speech recognition, multiple speech blocks (word1, word2, word3, word4) of the audio are obtained. Through speaker segmentation and clustering, multiple speakers (sp1, sp2) corresponding to the audio are obtained, and then the multiple speech blocks (word1, word2, word3, word4) and multiple speakers (sp1, sp2) are reordered (Reconciliation), and finally the speakers and corresponding speech content (such as Figure 3 in the rounded rectangle in the lower right corner of the image).

[0076] Figure 4 FIG. 1 is a flowchart of an exemplary audio processing method. Figure 4 In this embodiment, the process of the audio processing method may include the following steps:

[0077] S401: Perform voice activity detection on the audio.

[0078] The voice activity detection module is used to detect which frames contain voice.

[0079] S402: Perform change point detection or speech segmentation.

[0080] In this step, the change point detection module is used to detect the change point of the speaker. In this step, the speech segment corresponding to the audio can also be directly divided into blocks.

[0081] S403, extracting features from each segmented speech.

[0082] S404: Perform speaker clustering.

[0083] Through clustering technology, we can find out who said each speech segment. The main purpose of speaker clustering here is to cluster the segmented speech blocks and cluster the speech blocks together according to the speakers. Commonly used methods include IAC, AHC, k-means, spectral clustering, etc. Clustering can be used to obtain the speaker identity result.

[0084] S405: Send the clustered speech signal to a speech recognition system, and output text content corresponding to the speech signal.

[0085] In this embodiment, the speaker identity and the text content corresponding to the voice signal are output respectively through different processing processes.

[0086] In the above embodiment, the robustness of the audio processing algorithm is relatively low due to the influence of noise, reverberation, voiceprint drift, speech overlap, etc. For example, due to the influence of voiceprint drift, the segmentation and clustering of the above embodiment is inaccurate and overflow is prone to occur. For another example, the reverberation and noise in the environment will seriously affect the robustness of the above embodiment.

[0087] This paper proposes a comprehensive judgment technology that integrates semantic understanding, speech recognition and speaker segmentation and clustering to achieve a more robust and accurate algorithm.

[0088] Figure 5 FIG. 1 is another example flow chart of an audio processing method. Figure 5 In this embodiment, the process of the audio processing method may include the following steps:

[0089] S501, inputting the separated speech signal corresponding to the target audio into a trained speaker and content extraction model, and the speaker and content extraction model outputs the target speech content and at least two candidate speakers corresponding to the separated speech signal.

[0090] S502: Determine a target speaker corresponding to the target speech content from at least two candidate speakers according to the historical speech contents of at least two candidate speakers and the target speech content.

[0091] In this embodiment, the speaker and the speech content are trained end-to-end and the speaker and the speech content are output together. In addition, in step S502, the speaker identification result output in step S501 is corrected according to the semantic information, which can improve the robustness of the speaker identification clustering, thereby making the entire audio processing algorithm more accurate.

[0092] This solution proposes an end-to-end fusion training solution based on the characteristics of AI meeting minutes, and uses semantic information to correct the speaker recognition results to improve the robustness of speaker identity recognition clustering.

[0093] based on Figure 5 Based on the principles of the embodiments shown in the figure, this article provides an audio processing flow, such as Figure 6 As shown, this process can be applied to AI meeting minutes.

[0094] Figure 6 FIG. 1 is another example flow chart of an audio processing method. Figure 6 In this embodiment, the process of the audio processing method may include the following steps:

[0095] S601: Perform voice activity detection on the audio.

[0096] In one example, voice activity detection can be performed using a voice activity detection module, the input of which is audio x, and generally a voice activity detection (VAD) algorithm (such as a threshold judgment method, a neural network (NN) classification method) can be used to delete the silent segment in x and retain the speech segment. The output of the module is a speech signal s to be separated.

[0097] S602, performing change point detection.

[0098] In one example, the change point detection can be performed by a change point detection module, the input of which is the speech signal s to be separated. In this step, the speech signal s to be separated needs to be segmented. The goal of this module is to find the speaker change point and segment the speech signal s to be separated.

[0099] In this embodiment, the change point detection method may adopt any of the following methods (of course not limited to the following):

[0100] 1. Use a sliding window to slide on the speech signal s to be separated, extract features from the signal in the sliding window and calculate the score. When the score changes by more than a certain threshold, it is considered that the speaker identity has changed, and the current block si is updated to si+1.

[0101] 2. Use the NN algorithm to classify each point (it is necessary to extract the first n frames and the last n frames, extract the static features of each frame, and then aggregate them through the NN network, and finally output the binary classification results). The classification output is whether the current point is a change point, and then divide and separate the speech signal s through these change points.

[0102] The output of the change point detection module is the divided time domain signals s1, s2, s3...sn.

[0103] S603, extracting the speaker and content.

[0104] This step can be performed by the speaker and content extraction module, and the input of the speaker and content extraction module is each speech signal si. Here, this embodiment adopts an end-to-end network modeling method to realize the extraction of speaker identity and content. The specific implementation scheme can be as follows: Figure 7 shown.

[0105] Figure 7 This is a flowchart showing an example of extracting speakers and content. Figure 7 In this embodiment, the process of extracting the speaker and content may include the following steps:

[0106] For the input speech block signal si, firstly, the acoustic features are extracted (for example, the acoustic features can be Mel Frequency Cepstrum Coefficient (MFCC) or fbank (FilterBank) features, etc.), and then the extracted acoustic features are sent to the acoustic feature encoder (the acoustic feature encoder can be regarded as an acoustic feature encoding model, and the acoustic feature encoder can be implemented by a neural network NN network, for example, the NN network can be a convolutional neural network (CNN), a recurrent neural network (RNN), a long short-term memory network (LSTM), a conformer or any combination of these networks, etc.), and the high-dimensional information embedding representation representing the acoustic features is extracted, that is, the high-dimensional information of the acoustic features. In other embodiments, extracting the acoustic features and encoding the acoustic features can also be implemented in one step.

[0107] At the same time, after extracting the voiceprint features of the input speech block signal si (for example, the voiceprint features may be MFCC features or FBANK features, which may be used as an identity feature), the voiceprint features are sent to an identity feature encoder (for example, the identity feature encoder may be regarded as an identity feature encoding model, which may be implemented using a neural network) to extract the high-dimensional information embedding representation representing the identity features, i.e., the identity feature high-dimensional information. In other embodiments, extracting the voiceprint features and encoding the voiceprint features may also be implemented in one step.

[0108] Then, the identity feature is enhanced, that is, the extracted high-dimensional information of the identity feature and the high-dimensional information of the acoustic feature are fused and enhanced using the attention mechanism technology, and the identity feature is enhanced based on the acoustic feature to obtain the enhanced high-dimensional information of the identity feature, making the identity feature more robust and more distinguishable. The fusion enhancement of the extracted high-dimensional information of the identity feature and the high-dimensional information of the acoustic feature using the attention mechanism technology means that different attention levels are assigned to the high-dimensional information of the identity feature and the high-dimensional information of the acoustic feature. The attention level of the high-dimensional information of the identity feature and the attention level of the high-dimensional information of the acoustic feature are Figure 7 The network shown is determined during training. Figure 7 After the network training is completed, the attention of the high-dimensional information of the identity feature and the high-dimensional information of the acoustic feature are determined. Figure 7In the application process of the network training shown in the figure, the attention degree of the high-dimensional information of the identity feature and the attention degree of the high-dimensional information of the acoustic feature are both determined. For a detailed description of the attention mechanism technology, please refer to the related technology, which will not be repeated here.

[0109] Next, the high-dimensional information of acoustic features and the enhanced high-dimensional information of identity features are input into the joint training network, and the speakers and content are jointly extracted to obtain the speaker identity information (the top N speakers with the highest scores, where N is a natural number, for example, N can be 3) and the speech content (for example, the speech content can be the text result ti) and output them. The output content is N candidate speakers and the recognized text result ti.

[0110] Figure 7 The illustrated process for extracting speakers and content is performed by a speaker and content extraction model.

[0111] Figure 7 In the illustrated embodiment, identity feature enhancement based on the attention mechanism and end-to-end joint training are used to obtain the speaker and speech content, thereby improving the robustness of identity recognition and further improving the robustness of the entire algorithm.

[0112] It should be noted that in step S603, the speaker and the speech content are output by one model, that is, the speaker and the speech content are output by the speaker and content extraction model together, rather than outputting the speaker and the speech content through different processing processes respectively. In this way, the problem of inaccurate segmentation and clustering and overflow can be reduced, and the accuracy of speaker identification for each speech segment can be improved, and the robustness can be improved.

[0113] S604, performing natural language processing.

[0114] In this step, the recognized text result ti and the previous text information t(1,i-1), t(2,i-1)...t(N,i-1) corresponding to the N candidate speakers obtained from the speaker text dictionary are respectively sent to the natural speech processing module to determine whether ti and t(n,i-1) (n ranges from 1 to N) are scored f1, f2...fN spoken by the same person.

[0115] The natural speech processing process may include:

[0116] For each candidate speaker, determine the probability score that the recognized text result ti and the text corresponding to the previous sentence of the candidate speaker belong to the same person;

[0117] The candidate speaker with the largest possibility score among all candidate speakers is determined as the target speaker corresponding to the target speech content.

[0118] The likelihood score of the natural speech processing in this step can be obtained by any of the following methods, but is certainly not limited to the following methods:

[0119] (1) Text modeling technology (comparing the relevance of two texts)

[0120] Obtain a correlation value between text result ti and text t(n,i-1) of the previous sentence of candidate speaker n, and determine the correlation value as a probability score that text result ti and t(n,i-1) belong to the same person.

[0121] For example, the probability score f1 that the recognized text result ti and the previous text information t(1, i-1) corresponding to the first candidate speaker belong to the same person, the correlation score f1 between the recognized text result ti and the previous text information t(2, i-1) corresponding to the second candidate speaker, and the correlation score fN between the recognized text result ti and the previous text information t(N, i-1) corresponding to the Nth candidate speaker are calculated respectively. Then find the maximum value fmax among f1, f2, ... fN, and determine the candidate speaker corresponding to the maximum value fmax as the speaker of the text result ti. For example, if the maximum value fmax is f5, then the fifth candidate speaker is the speaker of the text result ti.

[0122] (2) Language feature extraction technology (language features such as word frequency, length, and pauses)

[0123] Extract the first language feature corresponding to the text result ti;

[0124] Extract the second language features corresponding to the previous sentence t(n,i-1) of the candidate speaker;

[0125] A similarity value between the first language feature and the second language feature is determined, and the similarity value is determined as a probability score that the text results ti and t(n, i-1) belong to the same person.

[0126] The speech features included in the first language feature and the second language feature include at least one of the following features: word frequency feature, length feature, and pause feature.

[0127] (3) Sentiment analysis technology (emotional state)

[0128] Perform sentiment analysis on the text result ti to obtain the first sentiment state;

[0129] Perform sentiment analysis on the last sentence t(n,i-1) of the candidate speaker to obtain the second sentiment state;

[0130] A similarity value between the first emotional state and the second emotional state is determined, and the similarity value is determined as a probability score that the text results ti and t(n,i-1) belong to the same person.

[0131] The text t(n,i-1) of the previous speech content of candidate speaker n can be read from the speaker text dictionary. The speaker text dictionary is used to store the text t(n,i-1) of the previous speech content of candidate speaker n.

[0132] S605: Perform speaker correction based on the result of natural speech processing.

[0133] The speaker corresponding to the highest score output in natural speech processing is selected as the speaker of the current speech block signal si.

[0134] In steps S604 and S605 of this embodiment, the speaker is corrected based on the understanding of the speaker's intention (the speaker's intention is a kind of semantic information), which can greatly improve the robustness of the audio processing algorithm.

[0135] S606, output of speaker and content.

[0136] That is, the corrected speakerid (speaker identifier) ​​and content information (ie, text result ti) are output.

[0137] S607, storing the speaker and content outputted in step S606 into the speaker text dictionary.

[0138] That is, the speakerid and the text content ti of the most recent sentence are stored (stored for the first time), or the text content of the most recent sentence of the speakerid is updated to the text result ti.

[0139] As the speaker and content are output, the text content of the most recent sentence corresponding to the speaker in the speaker text dictionary is also updated.

[0140] The audio processing method of this embodiment adopts an end-to-end solution based on fusion training, and uses semantic information to correct the speaker recognition results, thereby improving the robustness of speaker identity recognition clustering, and further improving the robustness of the entire audio processing algorithm.

[0141] An embodiment of the present application also provides an electronic device, which includes a memory and a processor, wherein the memory is coupled to the processor, and the memory stores program instructions. When the program instructions are executed by the processor, the electronic device implements the audio processing method executed by the aforementioned electronic device.

[0142] It is understandable that, in order to realize the above functions, the electronic device includes hardware and / or software modules corresponding to the execution of each function. In combination with the algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to be beyond the scope of the present application.

[0143] This embodiment further provides a computer storage medium, in which computer instructions are stored. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the audio processing method in the above-mentioned embodiment.

[0144] This embodiment further provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute the above-mentioned related steps to implement the audio processing method in the above-mentioned embodiment.

[0145] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer execution instructions, and when the device is running, the processor can execute the computer execution instructions stored in the memory so that the chip executes the audio processing method in the above-mentioned method embodiments.

[0146] Among them, the electronic device, computer storage medium, computer program product or chip provided in this embodiment is used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above and will not be repeated here.

[0147] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0148] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0149] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0150] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0151] Any content of each embodiment of the present application, as well as any content of the same embodiment, can be freely combined. Any combination of the above content is within the scope of the present application.

[0152] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0153] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

[0154] The steps of the method or algorithm described in conjunction with the disclosure of the embodiments of the present application can be implemented in a hardware manner, or can be implemented by a processor executing a software instruction. The software instruction can be composed of corresponding software modules, and the software module can be stored in a random access memory (Random Access Memory, RAM), a flash memory, a read-only memory (Read Only Memory, ROM), an erasable programmable read-only memory (Erasable Programmable ROM, EPROM), an electrically erasable programmable read-only memory (Electrically EPROM, EEPROM), a register, a hard disk, a mobile hard disk, a read-only compact disk (CD-ROM) or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and can write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0155] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented with hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein the communication media include any media that facilitates the transmission of a computer program from one place to another. The storage medium can be any available medium that a general or special-purpose computer can access.

[0156] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. An audio processing method, It is characterized in that Applied to electronic equipment, the method comprises: Perform voice activity detection on the target audio to obtain the voice signal to be separated; Detecting a speaker change point in the speech signal to be separated; According to the detected speaker change point, the speech signal to be separated is divided into a plurality of separated speech signals; Inputting the separated speech signal corresponding to the target audio into a trained speaker and content extraction model, and the speaker and content extraction model outputs the target speech content and at least two candidate speakers corresponding to the separated speech signal; According to the historical speech contents of the at least two candidate speakers and the target speech content, a target speaker corresponding to the target speech content is determined from the at least two candidate speakers.

2. The method according to claim 1, It is characterized in that The speaker and content extraction model processes the separated speech signal including: extracting target acoustic features from the separated speech signal; Extracting target voiceprint features from the separated speech signal; Based on the target acoustic feature, the target voiceprint feature is enhanced to obtain an enhanced target identity feature; The target acoustic features and the enhanced target identity features are input into a trained joint training model, and the joint training model outputs the target speech content and at least two candidate speakers corresponding to the separated speech signal.

3. The method according to claim 1, It is characterized in that The speaker and content extraction model processes the separated speech signal including: extracting target acoustic features from the separated speech signal; Inputting the target acoustic feature into a trained acoustic feature encoding model, and obtaining first high-dimensional information by performing high-dimensional spatial encoding on the target acoustic feature output by the acoustic feature encoding model; Extracting target voiceprint features from the separated speech signal; Inputting the target voiceprint feature into a trained identity feature encoding model, and performing high-dimensional spatial encoding of the target voiceprint feature output by the identity feature encoding model to obtain second high-dimensional information; Based on the first high-dimensional information, the second high-dimensional information is enhanced to obtain enhanced target identity features; The target acoustic features and the enhanced target identity features are input into a trained joint training model, and the joint training model outputs the target speech content and at least two candidate speakers corresponding to the separated speech signal.

4. The method according to claim 1, It is characterized in that Determining a target speaker corresponding to the target speech content from the at least two candidate speakers according to the historical speech content of the at least two candidate speakers and the target speech content includes: For each candidate speaker, determining a probability score of the target speech content and the previous speech content of the candidate speaker belonging to the same person; The candidate speaker with the largest possibility score among the at least two candidate speakers is determined as the target speaker corresponding to the target speech content.

5. The method according to claim 4, It is characterized in that Determining the probability score that the target speech content and the previous speech content of the candidate speaker belong to the same person includes: Obtain a correlation value between a first text corresponding to the target speech content and a second text corresponding to the previous speech content of the candidate speaker, and determine the correlation value as a probability score that the target speech content and the previous speech content of the candidate speaker belong to the same person.

6. The method according to claim 4, It is characterized in that Determining the probability score that the target speech content and the previous speech content of the candidate speaker belong to the same person includes: Extracting a first language feature corresponding to the target speech content; Extracting a second language feature corresponding to the last sentence spoken by the candidate speaker; A similarity value between the first language feature and the second language feature is determined, and the similarity value is determined as a probability score that the target speech content and the previous speech content of the candidate speaker belong to the same person.

7. The method according to claim 6, It is characterized in that The speech features included in the first language feature and the second language feature include at least one of the following features: word frequency feature, length feature, and pause feature.

8. The method according to claim 4, It is characterized in that Determining the probability score that the target speech content and the previous speech content of the candidate speaker belong to the same person includes: Performing sentiment analysis on the target speech content to obtain a first sentiment state; Performing sentiment analysis on the last sentence spoken by the candidate speaker to obtain a second sentiment state; A similarity value between the first emotional state and the second emotional state is determined, and the similarity value is determined as a probability score that the target speech content and the previous speech content of the candidate speaker belong to the same person.

9. The method according to claim 1, It is characterized in that After determining a target speaker corresponding to the target speech content from the at least two candidate speakers according to the historical speech content of the at least two candidate speakers and the target speech content, the method further includes: The target speech content is stored as the previous speech content of the target speaker.

10. An electronic device, It is characterized in that include: a memory and a processor, the memory being coupled to the processor; The memory stores program instructions, and when the program instructions are executed by the processor, the electronic device executes the audio processing method according to any one of claims 1 to 9.

11. A computer-readable storage medium comprising a computer program, It is characterized in that When the computer program is executed on an electronic device, the electronic device is enabled to execute the audio processing method according to any one of claims 1 to 9.