Audio analysis method, apparatus, device, medium, and program product

Abstract

This disclosure provides an audio analysis method, apparatus, device, medium, and program product, relating to the fields of artificial intelligence and speech processing technology. The method includes: in response to acquiring audio to be analyzed, performing speech detection on the audio to be analyzed to obtain multiple speech segments containing speech information in the audio; for each speech segment, analyzing the spectral energy distribution in the speech segment to obtain voiceprint features representing the energy density of the speech segment; based on the voiceprint features and the voiceprint features of related segments, performing temporal difference calculation on the speech segment to obtain enhancement features of the speech segment; performing feature matching on the embedding features of the speech segment based on preset embeddings in a preset voiceprint library to determine the target embedding corresponding to the embedding features, the embedding features being determined based on the enhancement features; and generating analysis results for the audio to be analyzed based on the target embedding.

Description

Technical Field

[0001] This disclosure relates to the fields of artificial intelligence and speech processing technology, and more specifically, to an audio analysis method, apparatus, device, medium, and program product. Background Technology

[0002] With the continuous development of speech processing technology, related technologies are now able to perform basic analysis on audio data and extract voiceprint features, thereby enabling the parsing of audio in conversational scenarios to distinguish different speakers in the audio.

[0003] In the process of realizing this disclosure, it was found that the related technology has at least the following problems: in the conversation scene, in addition to pure human voice, there is also environmental noise. This environmental noise will interfere with human voice, thereby reducing the ability of extracted voiceprint features to represent and affecting the accuracy of audio analysis. Summary of the Invention

[0004] In view of this, the present disclosure provides an audio analysis method, apparatus, device, medium, and program product.

[0005] One aspect of this disclosure provides an audio analysis method, comprising: in response to acquiring an audio to be analyzed, performing speech detection on the audio to be analyzed to obtain multiple speech segments containing speech information in the audio to be analyzed; for each speech segment, analyzing the spectral energy distribution in the speech segment to obtain a voiceprint feature representing the energy density of the speech segment; performing temporal difference calculation on the speech segment based on the voiceprint feature and the voiceprint features of related segments associated with the speech segment to obtain an enhancement feature of the speech segment; performing feature matching on the embedding features of the speech segment based on a preset embedding in a preset voiceprint library to determine a target embedding corresponding to the embedding feature, wherein the embedding feature is determined based on the enhancement feature; and generating an analysis result of the audio to be analyzed based on the target embedding.

[0006] According to embodiments of this disclosure, for each speech segment, the spectral energy distribution in the speech segment is analyzed to obtain voiceprint features representing the energy density of the speech segment. This includes: dividing the multiple speech segments into multiple speech frames based on the confidence levels of each speech segment to obtain the speech to be extracted using a sliding window, wherein the size of the sliding window is an integer multiple of the size of the speech segment; performing frequency mapping on the speech to be extracted to obtain the spectral energy distribution of the speech to be extracted; analyzing the spectral energy distribution to obtain the static features of the speech to be extracted, and using the static features as the voiceprint features of the multiple speech segments included in the speech to be extracted.

[0007] According to embodiments of this disclosure, based on the confidence levels of multiple speech frames included in multiple speech segments, a sliding window is used to divide the multiple speech segments to obtain the speech to be extracted. This includes: for each speech segment, determining the confidence variance of the speech segment based on the confidence levels of the multiple speech frames included in the speech segment; if the confidence variance is greater than a confidence threshold, setting the size of the sliding window to a first size; if the confidence variance is less than or equal to the confidence threshold, setting the size of the sliding window to a second size, where the second size is greater than the first size; and dividing the multiple speech segments using the sliding window based on the size of the sliding window and the size of the speech segments to obtain multiple speech samples to be extracted.

[0008] According to embodiments of this disclosure, the enhancement features include voiceprint features, first-order difference features, and second-order difference features. Based on the voiceprint features and the voiceprint features of related segments associated with the speech segment, temporal difference calculation is performed on the speech segment to obtain the enhancement features of the speech segment. This includes: determining multiple related segments from the multiple speech segments whose distance from the speech segment is less than or equal to the preset half-length based on the temporal relationship between multiple speech segments and a preset half-length; determining the first-order difference features of the speech segment based on the voiceprint features of the speech segment and the voiceprint features of the multiple related segments; determining the second-order difference features of the speech segment based on the first-order difference features of the speech segment and the voiceprint features of the multiple related segments; and concatenating the voiceprint features, first-order difference features, and second-order difference features to obtain the enhancement features.

[0009] According to embodiments of this disclosure, the embedding features are determined by: rearranging the dimensions of the enhancement features to obtain multi-channel dimensional rearranged features, wherein the multiple channels of the rearranged features correspond one-to-one with the voiceprint features, first-order difference features and second-order difference features; and masking the rearranged features using attention weights to obtain the embedding features.

[0010] According to embodiments of this disclosure, in response to acquiring the audio to be analyzed, speech detection is performed on the audio to be analyzed to obtain multiple speech segments containing speech information in the audio to be analyzed, including: splitting the audio to be analyzed according to a preset granularity to obtain multiple audio frames with the same preset granularity; performing speech detection on each of the multiple audio frames to obtain the confidence level of each of the multiple audio frames containing speech information; and determining multiple speech segments based on the confidence level of each of the multiple audio frames.

[0011] According to embodiments of this disclosure, determining multiple speech segments based on the confidence levels of multiple audio frames includes: classifying the multiple audio frames based on their respective confidence levels to obtain multiple speech frames to be evaluated and multiple noise frames, wherein the confidence level of the speech frames to be evaluated is greater than or equal to a noise threshold, and the confidence level of the noise frames is less than the noise threshold; segmenting the audio to be analyzed using the noise frames to obtain multiple speech frame groups composed of the speech frames to be evaluated; for each speech frame group, determining the speech frames to be evaluated in the speech frame group as speech frames if the number of speech frames to be evaluated in the speech frame group is greater than or equal to a number threshold; and splicing the multiple speech frames according to a preset splicing number to obtain a speech segment, wherein the number of speech frames in the speech segment is the preset splicing number.

[0012] According to embodiments of this disclosure, feature matching is performed on the embedding features of a speech segment based on preset embeddings in a preset voiceprint library to determine the target embedding corresponding to the embedding features. This includes: calculating the similarity between multiple preset embeddings in the preset voiceprint library and the embedding features respectively; for each preset embedding, if the similarity is greater than or equal to a first similarity threshold, determining the preset embedding as the target embedding; determining an update embedding based on the embedding features and the target embedding, and updating the target embedding using the update embedding; if the similarity is less than a second similarity threshold, determining that there is no target embedding corresponding to the embedding features in the preset voiceprint library, and adding the embedding features to the preset voiceprint library, wherein the second similarity threshold is less than the first similarity threshold; if the similarity is less than the first similarity threshold and greater than or equal to the second similarity threshold, calculating the update similarity between the embedding features and the update embedding; and if the update similarity is greater than the first similarity threshold, determining the preset embedding as the target embedding, and updating the preset embedding using the update embedding.

[0013] According to embodiments of this disclosure, a preset voiceprint library is used to store preset embeddings. The preset embeddings include an object identifier and the speech features of the voice-producing object corresponding to the object identifier. The object identifier is used to distinguish different voice-producing objects.

[0014] Another aspect of this disclosure provides an audio analysis apparatus, comprising: an audio detection module for performing speech detection on the audio to be analyzed in response to acquiring the audio to be analyzed, thereby obtaining multiple speech segments containing speech information in the audio to be analyzed; an energy analysis module for analyzing the spectral energy distribution in each speech segment to obtain voiceprint features representing the energy density of the speech segment; a feature difference module for performing temporal difference calculation on the speech segment based on the voiceprint features and the voiceprint features of related segments associated with the speech segment, thereby obtaining enhancement features of the speech segment; a feature matching module for performing feature matching on the embedding features of the speech segment based on preset embeddings in a preset voiceprint library, thereby determining a target embedding corresponding to the embedding features, wherein the embedding features are determined based on the enhancement features; and a result generation module for generating analysis results of the audio to be analyzed based on the target embedding.

[0015] Another aspect of this disclosure provides an electronic device comprising: one or more processors; and a memory for storing one or more programs, wherein, when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to perform the method as described above.

[0016] Another aspect of this disclosure provides a computer-readable storage medium storing computer-executable instructions that, when executed, are used to implement the method described above.

[0017] Another aspect of this disclosure provides a computer program product including computer-executable instructions that, when executed, are used to implement the method described above.

[0018] According to embodiments of this disclosure, speech detection can accurately extract speech segments containing speech information from the audio to be analyzed, while removing segments lacking speech information, thus reducing the amount of information that needs to be processed in subsequent processes and eliminating noise interference. By analyzing the spectral energy distribution to generate voiceprint features that characterize speech energy density, and combining this with temporal correlation segments to perform temporal difference calculations, enhanced features can be constructed by fusing static features with temporal dynamic change information. This enriches the feature expression dimensions, improves the anti-interference ability and temporal representation ability of the enhanced features, and facilitates subsequent generation of embedded features based on the enhanced features. Furthermore, feature matching with a preset voiceprint database is performed to determine the target embedding, resulting in the analysis results of the audio to be analyzed. Attached Figure Description

[0019] The above and other objects, features, and advantages of this disclosure will become clearer from the following description of embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0020] Figure 1The illustrations depict application scenarios of audio analysis methods, apparatus, devices, media, and program products according to embodiments of the present disclosure.

[0021] Figure 2 A flowchart illustrating an audio analysis method according to an embodiment of the present disclosure is shown schematically;

[0022] Figure 3 The illustration shows a schematic diagram of dimensional rearrangement of enhanced features according to an embodiment of the present disclosure;

[0023] Figure 4 This schematic diagram illustrates parallel processing based on a preset embedding according to an embodiment of the present disclosure;

[0024] Figure 5 A block diagram schematically illustrates an audio analysis apparatus according to an embodiment of the present disclosure; and

[0025] Figure 6 A block diagram of an electronic device suitable for implementing an audio analysis method according to an embodiment of the present disclosure is shown schematically. Detailed Implementation

[0026] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0027] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0028] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0029] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0030] In the embodiments disclosed herein, the collection, updating, analysis, processing, use, transmission, provision, disclosure, and storage of data (e.g., including but not limited to user personal information) comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. In particular, necessary measures have been taken to prevent unauthorized access to user personal information data and to safeguard user personal information security, network security, and national security.

[0031] In the embodiments disclosed herein, user authorization or consent is obtained before acquiring or collecting user personal information.

[0032] Related technologies typically extract only static features of speech information. In interactive scenarios with a large time span, the voice of the same speaker will change with time and environment. In this case, the feature robustness of static features is insufficient, resulting in low recognition accuracy.

[0033] Embodiments of this disclosure provide an audio analysis method, including: in response to acquiring audio to be analyzed, performing speech detection on the audio to be analyzed to obtain multiple speech segments containing speech information in the audio to be analyzed; for each speech segment, analyzing the spectral energy distribution in the speech segment to obtain voiceprint features representing the energy density of the speech segment; performing temporal difference calculation on the speech segment based on the voiceprint features and the voiceprint features of related segments associated with the speech segment to obtain enhancement features of the speech segment; performing feature matching on the embedding features of the speech segment based on preset embeddings in a preset voiceprint library to determine the target embedding corresponding to the embedding features, wherein the embedding features are determined according to the enhancement features; and generating an analysis result of the audio to be analyzed based on the target embedding.

[0034] Figure 1 The illustration shows an application scenario of an audio analysis method, apparatus, device, medium, and program product according to embodiments of the present disclosure.

[0035] It is important to note that Figure 1 The examples shown are merely examples of system architectures that can be applied to the embodiments of this disclosure, in order to help those skilled in the art understand the technical content of this disclosure, but do not mean that the embodiments of this disclosure cannot be used in other devices, systems, environments or scenarios.

[0036] like Figure 1 As shown, the system architecture 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing communication links between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0037] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, and / or social media platform software, etc. (for example only).

[0038] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0039] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0040] It should be noted that the information display method provided in this embodiment can generally be executed by server 105. Correspondingly, the information display device provided in this embodiment can generally be located in server 105. The information display method provided in this embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the information display device provided in this embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Alternatively, the information display method provided in this embodiment can also be executed by the first terminal device 101, the second terminal device 102, or the third terminal device 103, or by other terminal devices different from the first terminal device 101, the second terminal device 102, or the third terminal device 103. Accordingly, the information display device provided in this embodiment may also be disposed in the first terminal device 101, the second terminal device 102 or the third terminal device 103, or disposed in other terminal devices different from the first terminal device 101, the second terminal device 102 or the third terminal device 103.

[0041] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0042] Figure 2 A flowchart illustrating an audio analysis method according to an embodiment of the present disclosure is shown schematically.

[0043] like Figure 2 As shown, the method includes operations S210 to S250.

[0044] In operation S210, in response to acquiring the audio to be analyzed, speech detection is performed on the audio to be analyzed to obtain multiple speech segments containing speech information in the audio to be analyzed.

[0045] In operation S220, for each speech segment, the spectral energy distribution in the speech segment is analyzed to obtain the voiceprint features used to represent the energy density of the speech segment.

[0046] In operation S230, based on the voiceprint features and the voiceprint features of related segments, temporal difference calculation is performed on the speech segment to obtain the enhanced features of the speech segment.

[0047] In operation S240, feature matching is performed on the embedding features of the speech segment based on the preset embedding in the preset voiceprint library to determine the target embedding corresponding to the embedding features.

[0048] In operation S250, analysis results are generated based on target embedding to generate the audio to be analyzed.

[0049] The audio to be analyzed can be obtained either as an audio file or through streaming. The audio to be analyzed can include the speech information of multiple speakers.

[0050] Typically, the speech information in the audio to be analyzed will not cover the entire audio file. For example, in the case of a meeting recording, pauses during continuous speaking by the same speaker, or changes in the speaker's position, will result in the absence of speech information in the audio at those moments.

[0051] Therefore, speech detection can be performed on the audio to be analyzed to identify speech segments containing speech information, and only these speech segments can be processed and analyzed in subsequent processes, without having to process audio segments that do not contain speech information.

[0052] For example, the audio to be analyzed can be segmented into frames to obtain multiple audio frames. Speech detection can then be performed on each audio frame sequentially to determine whether speech information exists in each audio frame. Finally, consecutive audio frames containing speech information can be spliced ​​together to obtain a speech segment.

[0053] Specifically, a Voice Activity Detection (VAD) algorithm can be used to detect audio frames of the audio to be analyzed in order to determine whether the audio frames contain speech information.

[0054] The spectral energy distribution of a speech segment can be used to represent the proportion or distribution of energy at different frequencies. The spectral energy distribution in a speech segment can be analyzed using methods such as spectrum extraction to obtain voiceprint features that represent the energy density of the speech segment.

[0055] Voiceprint features represent the static distribution of energy in a speech segment. Furthermore, the dynamic features of a speech segment can be analyzed to enhance the static features, thereby improving the representational power of the enhanced features.

[0056] Other speech segments that are temporally adjacent to a speech segment are considered as related segments. Based on the speaker characteristics of the speech segment and the speaker characteristics of its related segments, a difference calculation can be performed on the speech segment in the time domain to obtain dynamic features characterizing the temporal changes of the speech segment. Enhancement features of the speech segment are then derived based on these dynamic and static speaker characteristics.

[0057] The embedding features of a speech segment can be determined based on the enhancement features. The enhancement features are used to characterize the short-time spectral energy distribution of the speech segment and the dynamic changes in the energy distribution. The embedding features are higher-order mappings of the enhancement features and can be used to distinguish different vocal subjects.

[0058] The preset voiceprint database can store multiple preset embeddings. These preset embeddings are then matched with the embedding features of the speech segments. If a match fails, the preset voiceprint database can be expanded based on the embedding features, and the speech object corresponding to the embedding features can be marked. This ensures that when the same speech object is processed again in a subsequent session, it can be successfully matched and recognized based on the preset voiceprint database.

[0059] If a match is successful, the target embedding corresponding to the embedding feature can be determined from multiple preset embeddings based on the matching result. The analysis result for the audio to be analyzed can then be determined based on the target embedding. The analysis result may include the actual speaker of the speech segment, the time of the most recent speech by that speaker, etc.

[0060] In the embodiments of this disclosure, different voice-speaking objects can be assigned corresponding object numbers. Therefore, it is not necessary to collect personal information of the voice-speaking objects, but multiple voice-speaking objects can be distinguished by object numbers.

[0061] According to embodiments of this disclosure, speech detection can accurately extract speech segments containing speech information from the audio to be analyzed, while removing segments lacking speech information, thus reducing the amount of information that needs to be processed in subsequent processes and eliminating noise interference. By analyzing the spectral energy distribution to generate voiceprint features that characterize speech energy density, and combining this with temporal correlation segments to perform temporal difference calculations, enhanced features can be constructed by fusing static features with temporal dynamic change information. This enriches the feature expression dimensions, improves the anti-interference ability and temporal representation ability of the enhanced features, and facilitates subsequent generation of embedded features based on the enhanced features. Furthermore, feature matching with a preset voiceprint database is performed to determine the target embedding, resulting in the analysis results of the audio to be analyzed.

[0062] According to embodiments of this disclosure, in response to acquiring the audio to be analyzed, speech detection is performed on the audio to be analyzed to obtain multiple speech segments containing speech information in the audio to be analyzed, including: splitting the audio to be analyzed according to a preset granularity to obtain multiple audio frames with the same preset granularity; performing speech detection on each of the multiple audio frames to obtain the confidence level of each of the multiple audio frames containing speech information; and determining multiple speech segments based on the confidence level of each of the multiple audio frames.

[0063] Speech frames are obtained by splitting the audio to be analyzed according to a preset granularity. In one example, the audio to be analyzed includes multiple initial frames, and the granularity of the initial frames is larger than the preset division granularity. In this case, the initial frames can be further divided according to the preset division granularity to obtain multiple speech frames. Preferably, the preset granularity can be 5ms, that is, the audio to be analyzed is divided into multiple audio frames with a length of 5ms.

[0064] Using VAD to detect speech in a speech frame, a confidence level indicating the presence of speech information in the current speech frame can be output. The higher the confidence level, the greater the probability that the current speech frame contains speech information.

[0065] According to embodiments of this disclosure, a preset speech determination threshold can be used to compare the confidence levels of multiple audio frames. Audio frames with confidence levels greater than the speech determination threshold are identified as audio frames containing speech information. Furthermore, a speech segment is determined based on the temporal relationship between the multiple audio frames and the multiple audio frames containing speech information.

[0066] According to embodiments of this disclosure, audio is uniformly segmented using a fixed preset granularity. Each audio frame is then used as the smallest unit to detect and output the confidence level of speech presence. Based on the quantified confidence level, speech and non-speech elements in the audio to be analyzed are distinguished. Frame-by-frame evaluation of multiple audio frames accurately locates the start and end boundaries of speech, avoiding issues such as missed speech segments and residual noise caused by coarse segmentation. This achieves accurate initial screening of effective speech segments, ensuring that subsequent processing only focuses on the features of effective speech content, thus reducing the amount of data processing.

[0067] According to embodiments of this disclosure, the audio to be analyzed can be standardized to obtain standardized audio, for example, uniformly converted to a 16kHz mono, 16-bit quantization format. Then, the standardized audio is split into multiple audio frames according to a preset granularity. These audio frames are then preprocessed to reduce spectral leakage and improve detection accuracy. The preprocessing of the audio frames may include pre-emphasis and Hamming windowing.

[0068] According to embodiments of this disclosure, the speech determination threshold can be set to 0.5. Audio frames with a confidence level greater than 0.5 are considered to contain speech information. However, since the preset granularity of audio frames is usually small, the stability of speech information in a single audio frame is often weakened, and the features of speech information are not fully represented in a single audio frame. Therefore, the confidence level of speech detection output for a single speech frame may be low. Using the speech determination threshold to compare and judge audio frames one by one can easily lead to missed detections.

[0069] According to embodiments of this disclosure, determining multiple speech segments based on the confidence levels of multiple audio frames includes: classifying the multiple audio frames based on their respective confidence levels to obtain multiple speech frames to be evaluated and multiple noise frames, wherein the confidence level of the speech frames to be evaluated is greater than or equal to a noise threshold, and the confidence level of the noise frames is less than the noise threshold; segmenting the audio to be analyzed using the noise frames to obtain multiple speech frame groups composed of the speech frames to be evaluated; for each speech frame group, determining the speech frames to be evaluated in the speech frame group as speech frames if the number of speech frames to be evaluated in the speech frame group is greater than or equal to a number threshold; and splicing the multiple speech frames according to a preset splicing number to obtain a speech segment, wherein the number of speech frames in the speech segment is the preset splicing number.

[0070] A noise threshold lower than the speech determination threshold can be selected, and multiple audio frames can be classified using the same judgment method as the speech determination threshold, resulting in multiple speech frames to be evaluated and multiple noise frames.

[0071] Once the noise frames are identified, they can be used as segmentation points to segment the audio to be analyzed, resulting in multiple groups of speech frames composed of consecutive speech frames to be evaluated.

[0072] For example, if the speech judgment threshold is set to 0.5, the noise threshold can be set to 0.45. After classifying multiple audio frames using the noise threshold, more audio frames can be classified into speech frames to be evaluated, thereby reducing the number of missed detections of speech information due to excessively high threshold settings.

[0073] Conversely, lowering the threshold may misclassify audio frames that originally had no speech information as speech frames to be evaluated. For example, for audio frames with a confidence level of 0.45 to 0.49, a threshold of 0.5 would classify them as noise frames, while a threshold of 0.45 would classify them as speech frames to be evaluated, thus increasing the false detection rate.

[0074] Based on the characteristics of speech information and the length of audio frames, the minimum number of audio frames included in a normal speech activity can be determined, and this number is used as a quantity threshold. Multiple speech frame groups are then further evaluated using this threshold. If the number of speech frames to be evaluated in a speech frame group is greater than or equal to the quantity threshold, the speech frames to be evaluated in that speech frame group are identified as speech frames.

[0075] For example, the duration of a short syllable or a single utterance in normal communication is usually no less than 15ms, while the length of an audio frame is 5ms. This means that at least three audio frames are needed to form a short syllable. Therefore, the quantity threshold can be set to 3.

[0076] By using a quantity threshold to evaluate speech frame groups, speech frames that appear less than 3 times consecutively can be filtered out, thereby reducing misjudgments of single or double frames due to occasional interference and thus lowering the false detection rate.

[0077] According to embodiments of this disclosure, the speech frames can be denoised first, and then the denoised speech frames can be spliced ​​together according to a preset number of splices to obtain a speech segment.

[0078] The noise reduction process may include performing a short-time Fourier transform on the speech frame to convert the speech frame from the time domain to the frequency domain, obtaining the frequency domain representation signal of the speech frame, then using a noise reduction model for the frequency domain signal to process the frequency domain representation signal to obtain the noise-reduced frequency domain representation signal, then using an inverse short-time Fourier transform to convert the noise-reduced frequency domain representation signal back from the frequency domain to the time domain, and finally superimposing the overlapping parts between multiple speech frames to obtain the noise-reduced speech frame.

[0079] Preferably, the preset number of splicing can be 10. Splicing 10 speech frames together to obtain a speech segment can avoid insufficient speech information in a speech segment and can also provide a data foundation for subsequent feature extraction.

[0080] According to embodiments of this disclosure, by performing dual screening using noise threshold and quantity threshold, it is possible to accurately distinguish between speech frames to be evaluated and noise frames. By segmenting independent speech intervals using noise frames, it is possible to filter out audio segments with invalid noise. Therefore, it is possible to filter out the interference of noise such as ambient background noise, transient plosives, and intermittent noise, ensuring the integrity and effectiveness of speech segments, thereby reducing the interference of noise on voiceprint feature extraction and feature matching.

[0081] According to embodiments of this disclosure, for each speech segment, the spectral energy distribution in the speech segment is analyzed to obtain voiceprint features representing the energy density of the speech segment. This includes: dividing the multiple speech segments into multiple speech frames based on the confidence levels of each speech segment to obtain the speech to be extracted using a sliding window, wherein the size of the sliding window is an integer multiple of the size of the speech segment; performing frequency mapping on the speech to be extracted to obtain the spectral energy distribution of the speech to be extracted; analyzing the spectral energy distribution to obtain the static features of the speech to be extracted, and using the static features as the voiceprint features of the multiple speech segments included in the speech to be extracted.

[0082] Based on the confidence scores of multiple speech frames within a speech segment, a judgment result can be obtained indicating whether there is a change in the speaker within that speech segment. According to the judgment result, multiple speech segments are divided using a sliding window to obtain the speech to be extracted. Since the size of the sliding window is an integer multiple of the size of the speech segment, each speech to be extracted includes multiple speech segments.

[0083] The speech to be extracted is frequency-mapped to obtain its frequency domain features. These features can then be processed using Mel filtering to obtain the spectral energy distribution of the speech to be extracted.

[0084] In one embodiment, before frequency mapping of the speech to be extracted, preprocessing such as pre-emphasis and Hamming windowing can be performed on the speech to be extracted. Pre-emphasis solves the problem of insufficient high-frequency energy in the speech to be extracted, and Hamming windowing solves the problem of boundary abrupt changes in the speech to be extracted, thereby making the spectral energy distribution more balanced, facilitating the extraction of speech information corresponding to high-frequency features, and reducing potential environmental noise in the speech to be extracted.

[0085] By performing a logarithmic operation on the spectral energy distribution, the static features of the speech to be extracted can be obtained, and these static features can be used as the static features of multiple speech segments in the speech to be extracted.

[0086] According to embodiments of this disclosure, based on the confidence distribution of speech frames, speech segments are batch-divided using sliding windows of integer multiples in size. The complete spectral energy distribution is then restored through frequency mapping. Static features are extracted by parsing the spectral information and reused as voiceprint features for multiple speech segments. Batch processing reduces redundant computation and improves feature extraction efficiency. Furthermore, preserving the details of the speech spectral energy distribution through frequency mapping ensures the integrity and accuracy of voiceprint features, providing high-quality foundational features for subsequent feature enhancement and matching.

[0087] According to embodiments of this disclosure, based on the confidence levels of multiple speech frames included in multiple speech segments, a sliding window is used to divide the multiple speech segments to obtain the speech to be extracted. This includes: for each speech segment, determining the confidence variance of the speech segment based on the confidence levels of the multiple speech frames included in the speech segment; if the confidence variance is greater than a confidence threshold, setting the size of the sliding window to a first size; if the confidence variance is less than or equal to the confidence threshold, setting the size of the sliding window to a second size, where the second size is greater than the first size; and dividing the multiple speech segments using the sliding window based on the size of the sliding window and the size of the speech segments to obtain multiple speech samples to be extracted.

[0088] When the confidence variance exceeds the confidence threshold, it indicates that the speech information of multiple speech frames in the speech segment fluctuates, suggesting a change in the speaker. Therefore, the sliding window can be set to a smaller first size to improve temporal resolution, achieve a faster response, and facilitate accurate differentiation of changes in the speaker. Preferably, the first size can be set to four times the size of the speech segment; in the case of a 50ms speech segment, the first size can be set to 200ms.

[0089] When the confidence variance is less than the confidence threshold, it indicates that the speech information of multiple speech frames in the speech segment is relatively stable and there is no switching of the speaker. Therefore, the sliding window can be set to a larger second size to ensure feature stability and improve processing efficiency. Preferably, the second size can be set to 10 times the size of the speech segment. If the speech segment is 50ms, the second size can be set to 500ms.

[0090] The speech segments are divided based on the size of the sliding window and the size of the speech segments. Since each speech segment is 50ms in size, the sliding window step size can be set to 50ms. After each sliding window moves forward, the size of the sliding window is determined based on the confidence variance of the first speech segment covered by the current sliding window, and the speech segments are then divided accordingly.

[0091] If the size of the sliding window remains unchanged before and after one slide, the speech segment can be divided directly based on the first or second size of the current sliding window. However, if the size of the sliding window changes before and after one slide, the size of the sliding window can be adjusted according to the preset size step.

[0092] For example, if the sliding window size is 200ms, and it remains 200ms after one slide, a 200ms sliding window can be directly used to segment the speech segment. If the sliding window size is 500ms, and it becomes 200ms after one slide, it can be determined that the direction of change is decreasing. Therefore, the sliding window size is adjusted according to the preset size adjustment step of 50ms and the direction of change, i.e., a 450ms sliding window is used to segment the speech segment. After the next slide, if the sliding window size is still 200ms, the size can be further adjusted to obtain a 400ms sliding window for segmenting the speech segment.

[0093] According to embodiments of this disclosure, the sliding window size is adaptively adjusted based on the confidence variance of speech segments. When confidence fluctuates greatly and the speech state is unstable, a small window size is used to achieve fine segmentation and preserve subtle local features. When the confidence distribution is stable, a large window size is used to merge redundant segments and improve computational efficiency. By using an adaptive window to segment the speech to be extracted, both the fineness of feature extraction and overall computational efficiency can be balanced, improving the scenario adaptability and robustness of audio analysis.

[0094] According to embodiments of this disclosure, the enhancement features include voiceprint features, first-order difference features, and second-order difference features. Based on the voiceprint features and the voiceprint features of related segments associated with the speech segment, temporal difference calculation is performed on the speech segment to obtain the enhancement features of the speech segment. This includes: determining multiple related segments from the multiple speech segments whose distance from the speech segment is less than or equal to the preset half-length based on the temporal relationship between multiple speech segments and a preset half-length; determining the first-order difference features of the speech segment based on the voiceprint features of the speech segment and the voiceprint features of the multiple related segments; determining the second-order difference features of the speech segment based on the first-order difference features of the speech segment and the voiceprint features of the multiple related segments; and concatenating the voiceprint features, first-order difference features, and second-order difference features to obtain the enhancement features.

[0095] Temporal relationships can be used to represent the sequential order and distance between multiple speech segments. Preset half-lengths can be used to restrict the selection of speech segments, that is, the distance between related segments and speech segments must be less than or equal to the preset half-length.

[0096] Preferably, the preset half-length can be 2, that is, for a speech segment, the two speech segments before it and the two speech segments after it are regarded as related segments. For example, if the speech segment currently being studied is the t-th speech segment f(t) in the audio to be analyzed, f(t-2), f(t-1), f(t+1), and f(t+2) can be regarded as related segments.

[0097] The first-order difference feature of a speech segment can be obtained by weighting and summing the voiceprint features of multiple related segments according to the triangular block weights. Specifically, the triangular block weights are distributed as [0.2, 0.4, 0, 0.4, 0.2], and the first-order difference feature Δf(t) of the t-th speech segment can be determined by equation (1):

[0098] (1)

[0099] Where w(n) is the weight of the triangular block, and f(t+n) represents the voiceprint feature of the t+nth speech segment.

[0100] Based on the individual voiceprint features of multiple related segments, the first-order difference features of each segment can be determined. Based on the first-order difference features of the speech segment and the first-order difference features of the multiple related segments, the second-order difference features of the speech segment can be determined. The calculation of the second-order difference features is similar to that of the first-order difference features, as shown in equation (2):

[0101] (2)

[0102] Where △△f(t) represents the second-order difference feature of the t-th speech segment, and △f(t+n) represents the first-order difference feature of the (t+n)-th speech segment.

[0103] The first-order difference features, second-order difference features, and voiceprint features have the same dimension. The voiceprint features of the t-th speech segment, the first-order difference features of the speech segment, and the second-order difference features of the speech segment can be concatenated to obtain the enhanced features.

[0104] According to embodiments of this disclosure, neighboring related segments are selected based on temporal distance constraints. First-order and second-order difference features are calculated sequentially, and the original speaker features and the two-level temporal difference features are concatenated into a composite enhanced feature. The first-order difference can characterize short-term temporal changes in speech, while the second-order difference can capture deep dynamic change patterns, compensating for the deficiency that a single static speaker feature cannot describe the temporal evolution of speech. Multi-feature fusion significantly improves feature discrimination and weakens the negative impact of environmental noise and speech rate fluctuations.

[0105] According to embodiments of this disclosure, the embedding features are determined by: rearranging the dimensions of the enhancement features to obtain multi-channel dimensional rearranged features, wherein the multiple channels of the rearranged features correspond one-to-one with the voiceprint features, first-order difference features and second-order difference features; and masking the rearranged features using attention weights to obtain the embedding features.

[0106] Enhanced features are obtained by concatenating voiceprint features, first-order difference features, and second-order difference features end-to-end. Therefore, the information of these features is not aligned in the frequency dimension in the enhanced features. Multi-channel dimensional rearranged features can be obtained by rearranging the dimensions of the enhanced features.

[0107] Figure 3 The illustration shows a schematic diagram of dimensional rearrangement of enhanced features according to an embodiment of the present disclosure.

[0108] like Figure 3 As shown, voiceprint feature 311, first-order difference feature 312, and second-order difference feature 313 are all 1*40 dimensional feature vectors, representing the feature distribution within a 40-frequency range. Enhancement feature 320 is a 1*120 dimensional feature vector obtained by concatenating the above features end-to-end. Although enhancement feature 320 includes the above features, it does not fully utilize the relationship between the above features and the frequency range.

[0109] Therefore, the 1*120 dimension enhancement feature 320 can be rearranged in dimensions to obtain the 3*40 dimension rearranged feature 330. The rearranged feature is equivalent to aligning the voiceprint feature 311, the first-order difference feature 312 and the second-order difference feature 313 in 40 frequency ranges respectively.

[0110] The rearranged feature 330 includes three row-level channels, which correspond to the voiceprint feature 311, the first-order difference feature 312, and the second-order difference feature 313, respectively.

[0111] The rearranged feature 330 includes 40 column vectors, each corresponding to a frequency range. Taking column vector 340 as an example, column vector 340 represents the voiceprint feature components, first-order difference feature components, and second-order difference feature components within the frequency range corresponding to that column vector.

[0112] According to embodiments of this disclosure, a lightweight model based on a depthwise separable convolutional architecture can be used to process rearranged features. By implementing inference and feature processing through a lightweight model, the efficiency of feature extraction can be improved and the real-time performance of audio processing can be enhanced.

[0113] Specifically, the network layers and core parameters of the lightweight model can be set as follows: the initial convolutional layer is a 3×3 depthwise separable convolution with a stride of 1, outputting 32-channel feature pre-extraction and dimension mapping inverted residual blocks. The number of channels in the four residual blocks are 32, 64, 128, and 256 respectively. The depthwise separable convolution is used to extract the core features of the voiceprint at multiple scales. The global average pooling layer is used to compress the compressed feature map to a 256-dimensional fixed-length vector and remove the time dimension from the features. The fully connected output layer is used to unify the feature length, that is, to further compress the 256-dimensional vector output by the global average pooling layer to 128 dimensions. The linear activation layer is used to generate the final standardized embedding features.

[0114] In a linear activation layer, the rearranged features can be masked using attention weights to obtain the embedded features. These attention weights can be determined during training; they are used to determine the contribution of each feature value at each position in the rearranged features to the embedded features.

[0115] According to embodiments of this disclosure, the embedded features can also be quantized and optimized. For example, after masking the rearranged features, the data type of the resulting embedded features is FP32, which has high precision but also relatively large data volume. A pre-trained quantization optimization model can be used to process the embedded features to adjust the data type of the embedded features to INT8. Although this data type will lose some precision compared to FP32, the data volume can usually be reduced by 3-4 times.

[0116] The quantization optimization model can be trained using a pre-set dataset. For the same vocal object, different optimization strategies are used to optimize its FP32 type embedding features to obtain multiple INT8 type embedding features. These multiple INT8 type embedding features are then used for classification to determine the optimization strategy with the least accuracy loss.

[0117] According to embodiments of this disclosure, the enhanced features after multi-type fusion are dimensionally rearranged to construct a multi-channel feature structure that corresponds one-to-one with different feature types, thereby achieving decoupling of differentiated features. Attention weight masks are used to automatically filter key effective features and suppress redundant and invalid information, strengthening the core voiceprint representation, reducing feature dimensional redundancy, making the generated embedded features more distinctive, and significantly improving the accuracy of subsequent voiceprint matching.

[0118] According to embodiments of this disclosure, feature matching is performed on the embedding features of a speech segment based on preset embeddings in a preset voiceprint library to determine the target embedding corresponding to the embedding features. This includes: calculating the similarity between multiple preset embeddings in the preset voiceprint library and the embedding features respectively; for each preset embedding, if the similarity is greater than or equal to a first similarity threshold, determining the preset embedding as the target embedding; determining an update embedding based on the embedding features and the target embedding, and updating the target embedding using the update embedding; if the similarity is less than a second similarity threshold, determining that there is no target embedding corresponding to the embedding features in the preset voiceprint library, and adding the embedding features to the preset voiceprint library, wherein the second similarity threshold is less than the first similarity threshold; if the similarity is less than the first similarity threshold and greater than or equal to the second similarity threshold, calculating the update similarity between the embedding features and the update embedding; and if the update similarity is greater than the first similarity threshold, determining the preset embedding as the target embedding, and updating the preset embedding using the update embedding.

[0119] The preset voiceprint library can be used to store preset embeddings. The preset embeddings include object identifiers and the speech features of the corresponding voice object. The object identifiers are used to distinguish different voice objects.

[0120] By embedding predefined identifiers into the inherent speech features of the corresponding voice-producing objects, accurate differentiation between different voice-producing objects is achieved through unique object identifiers. The standardized data structure within the database ensures accurate object localization and efficient retrieval during feature matching, providing clear data support for audio classification and voiceprint tracing in scenarios with multiple voice-producing objects.

[0121] If a preset embedding exists in the preset voiceprint database, the similarity between the preset embedding and the embedded feature in the preset voiceprint database is calculated respectively. The similarity between the preset embedding and the embedded feature can be determined by the cosine similarity between the feature vector corresponding to the speech feature of the preset embedding and the feature vector corresponding to the embedded feature.

[0122] According to embodiments of this disclosure, the number of threads can be determined based on the number of preset embeddings in a preset voiceprint library, and one thread can be allocated to each preset embedding for parallel similarity calculation and subsequent threshold comparison, thereby improving the efficiency of audio analysis by increasing parallelism.

[0123] Figure 4 The illustration shows a schematic diagram of parallel processing based on a preset embedding according to an embodiment of the present disclosure.

[0124] like Figure 4 As shown, after determining the embedding features, the preset voiceprint library is detected. If it is determined that the current preset voiceprint library includes three preset embeddings (embedded 1, embedded 2 and embedded 3), then three threads are allocated to process embedded 1, embedded 2 and embedded 3 respectively.

[0125] Taking the second thread as an example, the second thread corresponds to embedding 2. Therefore, the second thread only needs to calculate the similarity between embedding 2 and the embedded features, compare this similarity with the first similarity threshold and the second similarity threshold, and determine different processing strategies based on the relationship between the similarity and the first and second similarity thresholds. In this embodiment, the first similarity threshold can be 0.85, and the second similarity threshold can be 0.7.

[0126] If the similarity is greater than the first similarity threshold, the preset embedding corresponding to that similarity can be determined as the target embedding.

[0127] Since the sound characteristics of the sounding object and the environmental noise change over time, the updated embedding can be calculated based on the embedded features and the target embedding, and the updated embedding can be used to update the target embedding to ensure that the target embedding stored in the preset voiceprint library can reflect the characteristics of the current sounding object.

[0128] Update Embedded It can be determined by equation (3):

[0129] (3)

[0130] in, Represents embedded features, This indicates the target embedding.

[0131] If the similarity is less than the first similarity threshold and greater than the second similarity threshold, the updated embedding can be calculated using Equation (3) based on the preset embedding and the embedding feature corresponding to the similarity, and the updated similarity between the updated embedding and the embedding feature can be calculated. If the updated similarity is greater than the first similarity threshold, it can be determined that the updated embedding matches the embedding feature, the preset embedding is determined as the target embedding, and the updated embedding is used to update the preset embedding.

[0132] If the similarity is less than the second similarity threshold, wait for the comparison results of other threads. If it is determined that the similarity calculated by multiple threads is less than the second similarity threshold, it means that there is no preset embedding in the preset voiceprint library that matches the embedded feature. Therefore, an independent voice object can be assigned to the embedded feature, and a new object identifier can be assigned to it. Based on the new object identifier and the embedded feature, a new preset embedding is constructed and added to the preset voiceprint library.

[0133] If no preset embedding exists in the preset voiceprint library, an independent voice object can be directly assigned to the embedding feature, and a new object identifier can be assigned to it. Based on the new object identifier and the embedding feature, a new preset embedding can be constructed and added to the preset voiceprint library.

[0134] According to embodiments of this disclosure, a voiceprint database is managed using a dynamic update method through a first similarity threshold and a second similarity threshold. The first similarity threshold enables accurate matching and real-time iterative updates of the target embedding, adapting to slow changes in the timbre of the voice source. The second similarity threshold identifies new voice features and automatically adds them to the database, expanding the coverage of the voiceprint database. The intermediate range between the first and second similarity thresholds is optimized through secondary similarity checks, achieving progressive parameter optimization. This effectively avoids misjudgments and missed detections caused by a single rigid threshold, achieving a balance between accurate voiceprint matching and the dynamic iteration capability of the voiceprint database.

[0135] According to embodiments of this disclosure, analysis sub-results for speech segments can be generated based on the object identifier embedded in the target. These sub-results can characterize the phonological entity of the current speech segment. Based on the analysis results of multiple speech segments, analysis results for the audio to be analyzed can be obtained. These analysis results can characterize the phonological entities at different times within the audio to be analyzed.

[0136] According to embodiments of this disclosure, the preset embedding may further include the time of the most recent speech activity of the speech object corresponding to the object identifier. Therefore, the analysis sub-result may also include the time of the most recent speech activity of the speech object in the current speech segment before the current speech activity.

[0137] The preset storage structure embedded in the preset voiceprint library can be represented as {"speaker_X":{"embedding":128-dimensional_vector, "last_active_time":time_stap}}. Here, speaker_X represents the object identifier, which can be a system-level unique string or represented using speaker_1, speaker_2, etc.; embedding represents the key of the embedded feature; 128-dimensional_vector represents the value of the embedded feature; last_active_time represents the key of the time of the most recent voice activity; and time_stap represents the value of the time of the most recent voice activity, expressed as a timestamp.

[0138] Based on the time of the most recent voice activity of the voice object in the preset embedding and the current time, the duration of inactivity of the voice object can be determined. If the duration of inactivity reaches a preset time threshold, the preset embedding corresponding to the voice object is deleted from the preset voiceprint library to free up the memory of the preset voiceprint library.

[0139] Figure 5 A block diagram of an audio analysis apparatus according to an embodiment of the present disclosure is shown schematically.

[0140] like Figure 5 As shown, the audio analysis device 500 includes an audio detection module 510, an energy analysis module 520, a feature difference module 530, a feature matching module 540, and a result generation module 550.

[0141] The audio detection module 510 is used to perform speech detection on the audio to be analyzed in response to the acquisition of the audio to be analyzed, and to obtain multiple speech segments containing speech information in the audio to be analyzed.

[0142] The energy analysis module 520 is used to analyze the spectral energy distribution in each speech segment to obtain the voiceprint features that represent the energy density of the speech segment.

[0143] The feature difference module 530 is used to perform temporal difference calculation on the speech segment based on the voiceprint features and the voiceprint features of the relevant segments associated with the speech segment, so as to obtain the enhanced features of the speech segment.

[0144] The feature matching module 540 is used to perform feature matching on the embedding features of a speech segment based on the preset embeddings in the preset speaker database, and to determine the target embedding corresponding to the embedding features. The embedding features are determined based on the enhancement features.

[0145] The results generation module 550 is used to generate analysis results for the audio to be analyzed based on the target embedding.

[0146] According to embodiments of this disclosure, the energy analysis module 520 includes a speech segmentation submodule, a speech mapping submodule, and an energy analysis submodule.

[0147] The speech segmentation submodule is used to divide multiple speech segments into segments based on the confidence of the multiple speech frames included in each speech segment, using a sliding window to obtain the speech to be extracted. The size of the sliding window is an integer multiple of the size of the speech segment.

[0148] The speech mapping submodule is used to perform frequency mapping on the speech to be extracted, thereby obtaining the spectral energy distribution of the speech.

[0149] The energy analysis submodule is used to analyze the spectral energy distribution to obtain the static features of the speech to be extracted, and use the static features as the voiceprint features of multiple speech segments included in the speech to be extracted.

[0150] According to embodiments of this disclosure, the speech segmentation submodule includes a variance calculation unit, a first size determination unit, a second size determination unit, and a speech segmentation unit.

[0151] The variance calculation unit is used to determine the confidence variance of each speech segment based on the confidence scores of the multiple speech segments included in the speech segment.

[0152] The first size determination unit is used to set the size of the sliding window to the first size when the confidence variance is greater than the confidence threshold.

[0153] The second size determination unit is used to set the size of the sliding window to a second size, which is larger than the first size, when the confidence variance is less than or equal to the confidence threshold.

[0154] The speech segmentation unit is used to divide multiple speech segments into multiple speech segments based on the size of the sliding window and the size of the speech segments, thereby obtaining multiple speech segments to be extracted.

[0155] According to embodiments of this disclosure, the feature difference module 530 includes a related segment determination submodule, a first feature determination submodule, a second feature determination submodule, and a feature splicing submodule.

[0156] The related segment determination submodule is used to determine multiple related segments from multiple speech segments based on the temporal relationship between multiple speech segments and a preset half-length, and the distance between the speech segments is less than or equal to the preset half-length.

[0157] The first feature determination submodule is used to determine the first-order difference features of a speech segment based on the voiceprint features of the speech segment and the voiceprint features of multiple related segments.

[0158] The second feature determination submodule is used to determine the second-order difference features of a speech segment based on the first-order difference features of the speech segment and the individual voiceprint features of multiple related segments.

[0159] The feature splicing submodule is used to splice voiceprint features, first-order difference features, and second-order difference features to obtain enhanced features.

[0160] According to embodiments of this disclosure, the audio analysis device 500 further includes a feature rearrangement module and a feature masking module.

[0161] The feature rearrangement module is used to rearrange the dimensions of the enhanced features to obtain multi-channel dimensional rearranged features. The multiple channels of the rearranged features correspond one-to-one with the voiceprint features, first-order difference features and second-order difference features.

[0162] The feature masking module is used to mask the rearranged features using attention weights to obtain the embedded features.

[0163] According to embodiments of this disclosure, the audio detection module 510 includes an audio splitting submodule, a speech detection submodule, and a speech determination submodule.

[0164] The audio splitting submodule is used to split the audio to be analyzed according to a preset granularity, resulting in multiple audio frames with the same preset granularity.

[0165] The speech detection submodule is used to perform speech detection on multiple audio frames separately and obtain the confidence level of the speech information present in each audio frame.

[0166] The speech determination submodule is used to determine multiple speech segments based on the confidence levels of each of the multiple audio frames.

[0167] According to embodiments of this disclosure, the speech determination submodule includes an audio classification unit, an audio segmentation unit, a speech determination unit, and a speech splicing unit.

[0168] An audio classification unit is used to classify multiple audio frames based on their respective confidence levels, resulting in multiple speech frames to be evaluated and multiple noise frames. The confidence level of the speech frames to be evaluated is greater than or equal to the noise threshold, while the confidence level of the noise frames is less than the noise threshold.

[0169] The audio segmentation unit is used to segment the audio to be analyzed using noise frames, resulting in multiple speech frame groups composed of speech frames to be evaluated.

[0170] The speech determination unit is used to determine the speech frames to be evaluated in each speech frame group as speech frames if the number of speech frames to be evaluated in the speech frame group is greater than or equal to a number threshold.

[0171] The speech splicing unit is used to splice multiple speech frames according to a preset splicing number to obtain a speech segment. The number of speech frames in the speech segment is the preset splicing number.

[0172] According to embodiments of this disclosure, the feature matching module 540 includes a similarity determination submodule, a first comparison submodule, a first embedding update submodule, an embedding addition submodule, a second comparison submodule, and a second embedding update submodule.

[0173] The similarity determination submodule is used to calculate the similarity between multiple preset embeddings and embedded features in the preset voiceprint library.

[0174] The first comparison submodule is used to determine the preset embedding as the target embedding for each preset embedding if the similarity is greater than or equal to the first similarity threshold.

[0175] The first embedding update submodule is used to determine the update embedding based on the embedding features and the target embedding, so as to update the target embedding using the update embedding.

[0176] An embedded new sub-module is used to determine that there is no target embedding corresponding to the embedded feature in the preset voiceprint library when the similarity is less than the second similarity threshold, and to add the embedded feature to the preset voiceprint library. The second similarity threshold is less than the first similarity threshold.

[0177] The second comparison submodule is used to calculate the update similarity between the embedded features and the updated embedding when the similarity is less than the first similarity threshold and greater than or equal to the second similarity threshold.

[0178] The second embedding update submodule is used to determine the preset embedding as the target embedding when the update similarity is greater than the first similarity threshold, and to update the preset embedding using the update embedding.

[0179] Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure, or at least part of the functions of any one or more of them, can be implemented in one module. Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be implemented by dividing them into multiple modules. Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be at least partially implemented as hardware circuitry, such as a Field-Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a System-on-Chip, a System-on-a-Substrate, a System-on-Package, an Application-Specific Integrated Circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be at least partially implemented as computer program modules, which, when run, can perform corresponding functions.

[0180] For example, any and more of the audio detection module 510, energy analysis module 520, feature difference module 530, feature matching module 540, and result generation module 550 can be combined into one module / unit / subunit, or any one of these modules / units / subunits can be split into multiple modules / units / subunits. Alternatively, at least some of the functionality of one or more of these modules / units / subunits can be combined with at least some of the functionality of other modules / units / subunits and implemented in one module / unit / subunit. According to embodiments of this disclosure, at least one of the audio detection module 510, energy analysis module 520, feature difference module 530, feature matching module 540, and result generation module 550 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable method of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three methods. Alternatively, at least one of the audio detection module 510, energy analysis module 520, feature difference module 530, feature matching module 540, and result generation module 550 can be at least partially implemented as a computer program module, which, when run, can perform corresponding functions.

[0181] It should be noted that the audio analysis device part in the embodiments of this disclosure corresponds to the audio analysis method part in the embodiments of this disclosure. For a detailed description of the audio analysis device part, please refer to the audio analysis method part, which will not be repeated here.

[0182] Figure 6 A block diagram of an electronic device suitable for implementing an audio analysis method according to an embodiment of the present disclosure is shown schematically. Figure 6 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0183] like Figure 6 As shown, an electronic device 600 according to an embodiment of this disclosure includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage portion 608 into a random access memory (RAM) 603. The processor 601 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include onboard memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this disclosure.

[0184] RAM 603 stores various programs and data required for the operation of electronic device 600. Processor 601, ROM 602, and RAM 603 are interconnected via bus 604. Processor 601 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 602 and / or RAM 603. It should be noted that the programs may also be stored in one or more memories other than ROM 602 and RAM 603. Processor 601 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0185] According to embodiments of this disclosure, the electronic device 600 may further include an input / output (I / O) interface 605, which is also connected to a bus 604. The electronic device 600 may also include one or more of the following components connected to the input / output (I / O) interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the input / output (I / O) interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 610 as needed so that computer programs read from it can be installed into the storage section 608 as needed.

[0186] According to embodiments of this disclosure, the method flow according to embodiments of this disclosure can be implemented as a computer software program. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via communication section 609, and / or installed from removable medium 611. When the computer program is executed by processor 601, it performs the functions defined in the system of embodiments of this disclosure. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0187] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0188] According to embodiments of this disclosure, the computer-readable storage medium can be a non-volatile computer-readable storage medium. Examples include, but are not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0189] For example, according to embodiments of this disclosure, a computer-readable storage medium may include the ROM 602 and / or RAM 603 described above and / or one or more memories other than ROM 602 and RAM 603.

[0190] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods provided in the embodiments of this disclosure. When the computer program product is run on an electronic device, the program code is used to enable the electronic device to implement the methods provided in the embodiments of this disclosure.

[0191] When the computer program is executed by the processor 601, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0192] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via the communication section 609, and / or installed from the removable medium 611. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0193] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on a user's computing device, partially on a user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0194] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions. Those skilled in the art will understand that the features described in the various embodiments of the present disclosure can be combined and / or combined in various ways, even if such combinations are not explicitly described in the present disclosure. In particular, the features described in the various embodiments of this disclosure may be combined and / or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0195] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. An audio analysis method, comprising: In response to acquiring the audio to be analyzed, speech detection is performed on the audio to be analyzed to obtain multiple speech segments containing speech information in the audio to be analyzed; For each of the speech segments, the spectral energy distribution in the speech segment is analyzed to obtain voiceprint features that represent the energy density of the speech segment; Based on the voiceprint features and the voiceprint features of related segments associated with the speech segment, temporal difference calculation is performed on the speech segment to obtain the enhanced features of the speech segment; Based on the preset embeddings in the preset voiceprint library, feature matching is performed on the embedding features of the speech segment to determine the target embedding corresponding to the embedding features, wherein the embedding features are determined according to the enhancement features; as well as Based on the target embedding, analysis results for the audio to be analyzed are generated.

2. The method according to claim 1, wherein, For each of the speech segments, the spectral energy distribution in the speech segment is analyzed to obtain voiceprint features representing the energy density of the speech segment, including: Based on the confidence of multiple speech frames included in each of the multiple speech segments, the multiple speech segments are divided using a sliding window to obtain the speech to be extracted, wherein the size of the sliding window is an integer multiple of the size of the speech segment; The speech to be extracted is frequency-mapped to obtain the spectral energy distribution of the speech; and The spectral energy distribution is analyzed to obtain the static features of the speech to be extracted, and the static features are used as the voiceprint features of the multiple speech segments included in the speech to be extracted.

3. The method according to claim 2, wherein, The step of dividing the multiple speech segments into segments based on the confidence levels of the multiple speech frames included in each of the multiple speech segments, and obtaining the speech to be extracted using a sliding window, includes: For each of the aforementioned speech segments, the confidence variance of the speech segment is determined based on the confidence scores of the multiple speech segments included in the speech segment. If the confidence variance is greater than the confidence threshold, the size of the sliding window is set to a first size; If the confidence variance is less than or equal to the confidence threshold, the size of the sliding window is set to a second size, which is larger than the first size; and Based on the size of the sliding window and the size of the speech segment, the sliding window is used to divide the multiple speech segments to obtain multiple speech segments to be extracted.

4. The method according to claim 2, wherein, The enhancement features include the voiceprint features, first-order difference features, and second-order difference features; The step of performing temporal difference calculation on the speech segment based on the voiceprint features and the voiceprint features of related segments associated with the speech segment to obtain the enhanced features of the speech segment includes: Based on the temporal relationship between the multiple speech segments and a preset half-length, a number of related segments with a distance less than or equal to the preset half-length are determined from the multiple speech segments; Based on the voiceprint features of the speech segment and the voiceprint features of each of the multiple related segments, the first-order difference features of the speech segment are determined. Based on the first-order difference features of the speech segment and the respective speaker characteristics of multiple related segments, the second-order difference features of the speech segment are determined; and The enhanced feature is obtained by concatenating the voiceprint feature, the first-order difference feature, and the second-order difference feature.

5. The method according to claim 4, wherein, The embedding features are determined in the following way: The enhanced features are rearranged dimensionally to obtain multi-channel rearranged features, wherein multiple channels of the rearranged features correspond one-to-one with the voiceprint features, the first-order difference features, and the second-order difference features; and The rearranged features are masked using attention weights to obtain the embedded features.

6. The method according to claim 1, wherein, In response to acquiring the audio to be analyzed, speech detection is performed on the audio to be analyzed to obtain multiple speech segments containing speech information in the audio to be analyzed, including: The audio to be analyzed is split according to a preset granularity to obtain multiple audio frames with the same preset granularity. Speech detection is performed on each of the multiple audio frames to obtain the confidence level of the presence of speech information in each of the multiple audio frames; and Multiple speech segments are determined based on the confidence levels of the multiple audio frames.

7. The method according to claim 6, wherein, The step of determining multiple speech segments based on the confidence levels of each of the multiple audio frames includes: Based on the confidence scores of each of the multiple audio frames, the multiple audio frames are classified to obtain multiple speech frames to be evaluated and multiple noise frames, wherein the confidence scores of the speech frames to be evaluated are greater than or equal to the noise threshold, and the confidence scores of the noise frames are less than the noise threshold. The audio to be analyzed is segmented using the noise frames to obtain multiple groups of speech frames composed of the speech frames to be evaluated; For each of the said speech frame groups, if the number of speech frames to be evaluated in the speech frame group is greater than or equal to a number threshold, the speech frames to be evaluated in the speech frame group are determined as speech frames; and Multiple audio frames are spliced ​​together according to a preset splicing number to obtain the audio segment, wherein the number of audio frames in the audio segment is the preset splicing number.

8. The method according to claim 1, wherein, The step of performing feature matching on the embedding features of the speech segment based on the preset embeddings in the preset voiceprint database to determine the target embedding corresponding to the embedding features includes: Calculate the similarity between multiple preset embeddings in the preset voiceprint library and the embedded features respectively; For each of the preset embeddings, if the similarity is greater than or equal to a first similarity threshold, the preset embedding is determined as the target embedding; Based on the embedding features and the target embedding, an updated embedding is determined so as to update the target embedding using the updated embedding; If the similarity is less than the second similarity threshold, it is determined that there is no target embedding corresponding to the embedding feature in the preset voiceprint library, and the embedding feature is added to the preset voiceprint library, wherein the second similarity threshold is less than the first similarity threshold; If the similarity is less than a first similarity threshold and greater than or equal to a second similarity threshold, calculate the updated similarity between the embedded feature and the updated embedding; and If the updated similarity is greater than the first similarity threshold, the preset embedding is determined as the target embedding, and the updated embedding is used to update the preset embedding.

9. The method according to claim 8, wherein, The preset voiceprint library is used to store the preset embedding, which includes an object identifier and the speech features of the voice object corresponding to the object identifier. The object identifier is used to distinguish different voice objects.

10. An audio analysis device, comprising: The audio detection module is used to perform speech detection on the audio to be analyzed in response to acquiring the audio to be analyzed, and to obtain multiple speech segments containing speech information in the audio to be analyzed; An energy analysis module is used to analyze the spectral energy distribution in each speech segment to obtain voiceprint features that represent the energy density of the speech segment. The feature difference module is used to perform temporal difference calculation on the speech segment based on the voiceprint features and the voiceprint features of related segments associated with the speech segment, so as to obtain the enhanced features of the speech segment. The feature matching module is used to perform feature matching on the embedding features of the speech segment based on the preset embedding in the preset voiceprint library, and determine the target embedding corresponding to the embedding features, wherein the embedding features are determined according to the enhancement features; as well as The result generation module is used to generate analysis results for the audio to be analyzed based on the target embedding.

11. An electronic device, comprising: One or more processors; Memory, used to store one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1 to 9.

12. A computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the method of any one of claims 1 to 9.

13. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 9.

Images