Short-time audio speaker recognition method, apparatus, device and storage medium

Abstract

This application relates to a method, apparatus, device, and storage medium for short-time audio speaker recognition. The method includes: acquiring speech features of all speech frames in a short-time audio stream, filtering all speech features to generate an effective speech feature map; sequentially fusing the effective speech feature map with local features from different frequency bands several times to obtain local fused features for different frequency bands; performing staged global fusion based on the local fused features of each frequency band except for the low-frequency band to obtain a global fused feature; inferring a speech vector based on the global fused feature, and comparing the speech vector with a registered vector in a registered sound database to identify the speaker's identity. This method improves the accuracy of short-time audio speaker recognition without compromising the core architecture of the model, while maintaining lightweight design and deployment compatibility.

Description

Technical Field

[0001] This application relates to the field of speaker recognition technology, and in particular to a short-time audio speaker recognition method, apparatus, device, and storage medium. Background Technology

[0002] Existing speaker recognition models based on TDNN and Res2Net architectures (D-TDNN, CAM++, ERes2Net, EResNetV2) achieve effective fusion of multi-scale features through local and global feature fusion mechanisms. The core design of these models revolves around capturing features of long-duration speech (over 3 seconds). However, in short-duration audio scenarios (around 1 second), they struggle to meet identity verification requirements (such as identity verification upon wake-up from smart speakers, wake-up authentication from smart wearable devices, and rapid identity verification in vehicles) due to issues such as sparse available speaker information, high proportion of invalid frame interference, and easy loss of temporal details. Summary of the Invention

[0003] Therefore, it is necessary to provide a speaker recognition solution that is adapted to 1-second short audio clips, which can improve the accuracy of speaker recognition in short audio clips without destroying the core architecture of the original model, while maintaining lightweight design and deployment compatibility.

[0004] In a first aspect of this application, a method for short-time audio speaker recognition is provided. The method includes: The speech features of all speech frames in a short audio segment are obtained, and all speech features are filtered to generate an effective speech feature map. The effective speech feature map is sequentially fused with local features of different frequency bands several times to obtain local fused features of different frequency bands; Global fusion is performed in stages based on the local fusion features of each frequency band except the low frequency band, to obtain global fusion features; Speech vectors are inferred based on global fusion features, and the speech vectors are compared with the registered vectors in the registered sound library to identify the speaker's identity.

[0005] Preferably, the step of filtering all speech features to generate an effective speech feature map includes: Calculate the frame energy of the speech features of each speech frame, and normalize each frame energy respectively; Calculate the spectral flatness of the speech features for each speech frame; The normalized frame energy and spectral flatness corresponding to the speech features of the same speech frame are weighted and summed to obtain the quality score of the speech features of the corresponding speech frame. Retain speech features with quality scores greater than or equal to a preset threshold to generate effective speech feature maps.

[0006] Preferred options also include: When the proportion of retained speech features is less than the second preset threshold, the average of the neighboring retained speech features is used to fill the gap until the number of retained speech features reaches 60% of the total number of speech features, and an effective speech feature map is generated.

[0007] Preferably, the step of sequentially fusing local features of different frequency bands into the effective speech feature map to obtain local fused features of different frequency bands includes: The effective speech feature map is fused with local features in the low-frequency band to obtain local fused features in the low-frequency band; The local fusion features of the low-frequency band are fused with the local features of the mid-frequency band to obtain the local fusion features of the mid-frequency band. The local fusion features of the mid-frequency band are fused with the local features of the mid-high frequency band to obtain the local fusion features of the mid-high frequency band. The local fusion features of the mid-to-high frequency band are fused with the local features of the high frequency band to obtain the local fusion features of the high frequency band.

[0008] Preferably, the local feature fusion process for any frequency band includes: The channel dimension of the local fusion feature from the previous step is doubled by a 1×1 convolution, and the expanded local fusion feature is divided into two subsets according to the expanded channel dimension; the effective speech feature map is the local fusion feature from the initial step. For any subset, slide the window across the speech frames of the subset and divide it into several micro-segments. For each micro-segment, use a 1×3 depthwise separable convolution to extract the local features of the micro-segment. The local features of the micro-segments are processed through global average pooling, multilayer perceptron, and mapping function to obtain the corresponding micro-segment attention weights; The local features of the micro-segments are weighted by the micro-segment attention weights to obtain the weighted features of the micro-segments; The weighted features of all micro-segments in the aggregate subset are used to obtain the corresponding weighted feature subset. The two weighted feature subsets are concatenated along the channel dimension and then input into the attention feature fusion module to obtain the dynamic fusion weights; The fused features are obtained by summing two weighted feature subsets based on the dynamic fusion weights. The current local fusion features are obtained by adding the results of 1×1 convolution and batch normalization to the fused features.

[0009] Preferably, the step of performing phased global fusion based on the local fusion features of each frequency band except the low-frequency band to obtain global fusion features includes: Align the local fusion features of the mid-frequency band with the local fusion features of the mid-high frequency band using 3×3 convolution, considering both the frequency band dimension and the channel dimension. The frequency band dimension of the local fusion features in the high-frequency band is aligned with the frequency band dimension of the local fusion features in the mid-to-high-frequency band by using 3×3 convolution. The aligned local fusion features of the mid-frequency band and the local fusion features of the mid-high frequency band are input into the attention feature fusion module to obtain the preliminary global fusion features. The temporal and channel dimensions of the initial global fusion features and the local fusion features of the high-frequency band are aligned using 3×3 convolution. The aligned preliminary global fusion features and the aligned high-frequency local fusion features are input into the attention feature fusion module to obtain intermediate global fusion features; The global fusion feature is obtained by aligning the frequency band dimension of the intermediate global fusion feature with the frequency band dimension of the original high-frequency local fusion feature through a 1×1 convolution.

[0010] Preferably, the step of inferring a speech vector based on global fusion features and comparing the speech vector with the registered vectors in the registered sound database to identify the speaker's identity includes: The mean and standard deviation of the global fusion features are calculated in the time dimension, and the mean and standard deviation are concatenated to obtain a fixed-length representation. The fixed-length representation is passed through the first fully connected layer to obtain preliminary embedded features; The initial embedded features are passed through a second fully connected layer to obtain the speech vector; The speech vectors are L2 normalized, and the cosine similarity between the L2 normalized speech vectors and each registered vector in the registered sound library is calculated. Each cosine similarity is compared with the recognition threshold. When any cosine similarity is greater than or equal to the recognition threshold, the speaker is determined to be a registered user to which the corresponding registration vector belongs; otherwise, the speaker is determined to be an unregistered user.

[0011] In a second aspect of this application, a short-time audio speaker recognition device is provided. The device includes: The filtering module is used to obtain the speech features of all speech frames in a short audio clip, filter all speech features, and generate an effective speech feature map. The local fusion module sequentially fuses the effective speech feature map through several local feature fusions of different frequency bands to obtain local fused features of different frequency bands. The global fusion module performs phased global fusion based on the local fusion features of each frequency band except the low-frequency band, to obtain global fusion features; The recognition module infers speech vectors based on global fusion features and compares the speech vectors with the registered vectors in the registered sound library to identify the speaker's identity.

[0012] In a third aspect of this application, an electronic device is provided. The electronic device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect of this application.

[0013] In a fourth aspect of this application, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the method described in the first aspect of this application.

[0014] Beneficial effects: This method improves the accuracy of speaker recognition in short audio segments without compromising the core architecture of the model, while maintaining lightweight design and deployment compatibility. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a flowchart of the short-time audio speaker recognition method in the embodiments of this application.

[0017] Figure 2 This is a flowchart of local feature fusion for any frequency band in the embodiments of this application.

[0018] Figure 3 This is a flowchart illustrating the phased global fusion process in this application embodiment.

[0019] Figure 4 This is a block diagram of the short-time audio speaker recognition device in the embodiments of this application.

[0020] Figure 5 This is a schematic diagram of the structure of the terminal device or server in the embodiments of this application. Detailed Implementation

[0021] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, the specific embodiments of this application are described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of this application. However, this application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of this application. Therefore, this application is not limited to the specific embodiments disclosed below.

[0022] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0023] like Figure 1 As shown, this embodiment provides a short-time audio speaker recognition method, including: S101: Obtain the speech features of all speech frames in the short audio, filter all speech features, and generate an effective speech feature map.

[0024] Due to severe interference from invalid frames—which are prone to contain silence, sudden noise, or non-speech frames within a 1-second audio clip—existing models directly input all frames into the feature fusion module. Invalid frames dilute the effective speaker features, leading to a decrease in embedding discriminativeness. To address this, this embodiment provides the following lightweight frame quality assessment (FQA) unit filtering process. Based on frame energy and spectral flatness, it filters effective frames, retaining only those with strong discriminative power for subsequent modules, thus reducing interference from invalid frames.

[0025] Specifically, the step of filtering all speech features (a total of 80) to generate a valid speech feature map includes: Calculate the frame energy of the speech features of each speech frame, and normalize each frame energy respectively; Calculate the spectral flatness of the speech features for each speech frame. Spectral flatness is used to reflect the speech / noise properties of the speech features. The normalized frame energy and spectral flatness corresponding to the speech features of the same speech frame are weighted and summed to obtain the quality score of the speech features of the corresponding speech frame. Retain speech features with a quality score greater than or equal to a preset threshold (e.g., 0.6) (the number is usually 50-70) to generate an effective speech feature map.

[0026] Furthermore, the formula for calculating the quality score of speech features is: ; ; ; ; in, Indicates the first Quality scores of speech features for each speech frame Indicates the first Normalized frame energy of speech features of a speech frame Indicates the first Spectral flatness of speech features in a speech frame This indicates taking the maximum value. This indicates taking the minimum value. Indicates the first Frame energy of speech features of a speech frame. This represents the minimum frame energy. Corresponding to the -40dB silence threshold, Indicates the maximum frame energy. This indicates the number of frequency points (taken as 80, i.e., the frequency band dimension is 80). Indicates the first The speech features of the speech frame in the th speech frame Power at each frequency point Represents a very small number (take 10). -8 ).

[0027] Optionally, S102 also includes: When the proportion of retained speech features is less than the second preset threshold (e.g., 50%), the average of the neighboring retained speech features is used for padding (e.g., if frame t is an invalid frame, use t). (The speech features of frames 1 and t+1 are filled with the mean value) to avoid the feature being too short and causing subsequent fusion failure, until the number of retained speech features reaches 60% of the total number of speech features, and an effective speech feature map is generated.

[0028] The above screening process has no additional learnable parameters, relies solely on statistical calculations, has a latency increase of ≤5ms, and is lightweight and easy to deploy.

[0029] S102: The effective speech feature map is sequentially fused with local features of different frequency bands several times to obtain local fused features of different frequency bands.

[0030] Existing technologies suffer from insufficient micro-segment feature capture: key speaker discrimination information (such as fundamental frequency fluctuations and formant details) is concentrated in the 20-40ms micro-segment. Existing BLFF modules only use channel splitting and convolutional fusion to capture features, without designing a focusing mechanism for the micro-segment scale, which easily leads to the loss of instantaneous discrimination information. To address this, this embodiment provides the following local fusion process, optimizing the original BLFF module by adding micro-segment convolutional windows and micro-segment attention to focus on key speaker features at the 20-40ms micro-segment scale, thereby improving the discriminative power of local features.

[0031] Specifically, the step of sequentially fusing local features of different frequency bands into the effective speech feature map to obtain local fused features of different frequency bands includes: The effective speech feature map is fused with local features in the low-frequency band to obtain local fused features in the low-frequency band; The local fusion features of the low-frequency band are fused with the local features of the mid-frequency band to obtain the local fusion features of the mid-frequency band. The local fusion features of the mid-frequency band are fused with the local features of the mid-high frequency band to obtain the local fusion features of the mid-high frequency band. The local fusion features of the mid-to-high frequency band are fused with the local features of the high frequency band to obtain the local fusion features of the high frequency band.

[0032] The four stages of local fusion are described in Table 1; Table 1. Introduction to Local Fusion from Low Frequency to High Frequency Bands

[0033] like Figure 2 As shown, the local feature fusion process for any frequency band includes: S201: Double the channel dimension of the local fusion features from the previous step using a 1×1 convolution, and divide the expanded local fusion features into two subsets based on the expanded channel dimension (the channel dimension of each subset is half of the original). The effective speech feature map is the local fusion feature from the initial step; the channel dimension expansion formula is: ; in, This represents the expanded local fusion features. Represents a 1×1 convolution. This represents the local fusion features from the previous step. This indicates batch normalization.

[0034] S202: For any subset, slide the window across the speech frames of the subset according to the window size (e.g., L=3) and divide it into several micro-segments, each micro-segment denoted as . , , , They represent the first The first micro-segment , , The speech features of each speech frame are extracted. When the window slides to the edge of the subset, the positions corresponding to empty frames in the micro-segment are filled with zeros. For each micro-segment, a 1×3 depthwise separable convolution is used to extract the local features of the micro-segment. The calculation formula is as follows: ; in, Indicates the first Local features of a micro-segment This indicates a 1×3 depthwise separable convolution (extracting micro-segment local features only in the time dimension). Indicates the first Each micro-segment retains its original feature information through residual connections.

[0035] S203: The local features of the micro-segments are processed through global average pooling, a multilayer perceptron, and a mapping function to obtain the corresponding micro-segment attention weights. The calculation formula is as follows: ; in, Indicates the first Micro-segment attention weights corresponding to the speech features of each speech frame. This represents the sigmoid function. This represents a small multilayer perceptron (containing one 64-dimensional hidden layer with SiLU activation function). Indicates global average pooling. Indicates the first The speech features corresponding to the first speech frame Local features of a micro-segment.

[0036] S204: Based on the local features of the micro-segments weighted by the micro-segment attention weights, the weighted features of the micro-segments are obtained, and the calculation formula is as follows: ; in, Indicates the first The weighted features of each micro-segment.

[0037] S205: The weighted features of all micro-segments in the aggregate subset are used to obtain the corresponding weighted feature subset, calculated as follows: ; ; in, This represents the weighted feature subset corresponding to the first subset. This represents the weighted feature subset corresponding to the second subset. Indicates the number of micro-segments. Indicates the first subset The segment-weighted features of each segment Indicates the second subset The weighted features of each micro-segment.

[0038] S206: The two weighted feature subsets are concatenated along the channel dimension and then input into the attention feature fusion module to obtain the dynamic fusion weights, calculated as follows: ; in, Indicates dynamic fusion weights. Represents the hyperbolic tangent function. Indicates batch normalization, , All are 1×1 convolutions (channel compression ratio of 4). This represents the SiLU activation function. This indicates a splicing operation.

[0039] S207: Based on the dynamic fusion weighted summation of two weighted feature subsets, the fused feature is obtained, and the calculation formula is: ; in, This indicates the fusion feature.

[0040] S208: The current local fusion feature is obtained by adding the results of batch normalization to the fused features after passing each feature through a 1×1 convolution. The calculation formula is as follows: ; in, This indicates the current local fusion features. This represents a 1×1 convolution.

[0041] By adding a 1×3 depthwise separable convolution and a small multilayer perceptron, key features of micro-segments can be accurately captured, while maintaining compatibility with the forward computation process of the original BLFF module.

[0042] S103: Perform phased global fusion based on the local fusion features of each frequency band except the low frequency band to obtain global fusion features.

[0043] Existing technologies suffer from poor global fusion adaptability: existing BDFF modules only fuse Stage 3-4 features and use 3×3 downsampling convolutions with a stride of 2, which further compresses the frame resolution of 1 second of audio to approximately 6 frames. Excessive loss of temporal details leads to distortion of global feature representation. To address this, this embodiment improves the BDFF module by extending the fusion stage (from Stage 3-4 to Stage 2-3-4) and reducing the downsampling rate (changing the step size from 2 to 1), thus preserving more temporal details and enhancing the integrity of global feature representation.

[0044] Specifically, such as Figure 3 As shown, the phased global fusion based on the local fusion features of each frequency band except the low-frequency band yields global fusion features, including: The dimensions of the local fusion features in the mid-frequency band, mid-high frequency band, and high frequency band are respectively , , .

[0045] S301: Align the frequency band dimension and channel dimension of the local fusion features in the mid-frequency band with the local fusion features in the mid-high frequency band by using a 3×3 convolution with a stride of 1. Specifically, expand the channel dimension of the local fusion features in the mid-frequency band from 128 to 256 and downsample the frequency band dimension of the local fusion features in the mid-frequency band from 20 to 10. The frequency band dimension of the local fusion features in the high-frequency band is aligned with that in the mid-to-high-frequency band by using a 3×3 convolution with a stride of 1, that is, the frequency band dimension 5 of the local fusion features in the high-frequency band is upsampled to 10.

[0046] S302: The local fusion features of the aligned mid-frequency band (dimension: Local fusion features of the mid-to-high frequency band (dimension: The input is fed into the attention feature fusion module to obtain preliminary global fusion features; Specifically, in the attention feature fusion module, bilinear interpolation is first used to upsample the time dimension of the local fusion features in the mid-to-high frequency bands to [a specific value]. Then, the fusion weights are calculated, and the dimensions of the preliminary global fusion features are obtained. .

[0047] S303: Align the temporal and channel dimensions of the initial global fusion features with the local fusion features of the high-frequency bands using a 3×3 convolution with a stride of 1. This involves downsampling the temporal dimension of the initial global fusion features to... This expands the channel dimension of the initial global fusion features to 512.

[0048] S304: The aligned preliminary global fusion features (dimension: ...) ) and the local fusion features of the aligned high-frequency bands (dimension ) The input is fed into the attention feature fusion module to obtain intermediate global fused features with dimensions of [dimensionality missing]. ; S305: Align the frequency band dimension of the intermediate global fusion feature with the frequency band dimension of the original high-frequency local fusion feature using a 1×1 convolution to obtain the global fusion feature. This involves downsampling the time dimension of the intermediate global fusion feature from 10 to 5, resulting in a final global fusion feature with the following dimension: .

[0049] By extending global fusion and using downsampling convolution with a stride of 1, approximately three times the temporal details are preserved (the original BDFF module only preserves 6 frames, while the improved version preserves 18-20 frames), making the global features more complete.

[0050] S104: Based on global fusion features, infer the speech vector and compare the speech vector with the registered vector in the registered sound library to identify the speaker's identity.

[0051] Specifically, the steps include: The mean and standard deviation of the global fusion features are calculated over time, and the mean and standard deviation are concatenated to obtain a fixed-length representation. The calculation formula is as follows: ; ; ; in, Indicates a fixed length representation. , Indicates the number of channels (e.g., 512). Indicates the number of frequency bands (usually 5~10); This represents the mean of the globally fused features over time. This represents the standard deviation of the globally fused features over time. It represents the time dimension (usually 16~20 frames). This represents the element corresponding to the t-th speech frame in the global fusion features. , This indicates global fusion features.

[0052] The fixed-length representation is passed through the first fully connected layer to obtain the preliminary embedded features, calculated as follows: ; in, This indicates the initial embedded features. Represents the ReLU activation function. This represents the weights of the first fully connected layer. This indicates the bias of the first fully connected layer.

[0053] The initial embedded features are passed through a second fully connected layer to obtain the speech vector, calculated as follows: ; in, Represents speech vectors, Indicates the weights of the second fully connected layer. This indicates the bias of the second fully connected layer.

[0054] Optionally, both the first fully connected layer and the second fully connected layer have batch normalization.

[0055] The speech vectors are L2 normalized to improve comparison stability, and the cosine similarity between the L2 normalized speech vectors and each registered vector in the registered sound database is calculated using the following formula: ; ; in, This indicates that the L2-normalized speech vector is compared with the first speech vector in the registered sound database. Cosine similarity between the registered vectors This represents the L2-normalized speech vector. Indicates the number of registered sound libraries One registration vector, This represents the L2 norm. The registered sound library stores the set of registration vectors of the user's voice during registration.

[0056] Each cosine similarity is compared with the recognition threshold. When any cosine similarity is greater than or equal to the recognition threshold, the speaker is determined to be a registered user to which the corresponding registration vector belongs; otherwise, the speaker is determined to be an unregistered user.

[0057] Existing models use 3-second speech slices as training samples. These models tend to learn long-term contextual dependencies but underfit sparse features within 1 second. Furthermore, they lack dedicated data augmentation for short segments, resulting in poor generalization in short-term validation scenarios involving different content (such as different phrases or numbers). To address this, this embodiment adjusts the training sample slice length (0.8-1.2 seconds), introduces a "micro-segment shuffling perturbation" augmentation strategy, fine-tunes the AAM-Softmax loss parameters, and forces the model to learn short-term sparse features, thereby improving generalization ability.

[0058] Specifically, short-term adaptation training strategies include: Training sample cropping: From each audio segment in the training set (such as VoxCeleb, cn-celeb), a 0.8-1.2 second segment is randomly cropped (instead of the original 3 seconds) to ensure that the training samples match the inference scenario; If the length of a speech segment is less than 0.8 seconds, a "repeated splicing" strategy (no more than 2 times) is used to generate training samples to avoid morpheme distortion (e.g., splicing a 0.6-second speech segment into a 1.2-second segment). Data augmentation: Basic enhancements: MUSAN noise superposition (SNR=0-15dB), RIR reverberation, 0.9 / 1.1x speed perturbation; Micro-segment shuffling perturbation: Randomly shuffle the temporal order of 10% of the 30ms micro-segments (3 frames) (e.g., t1→t3→t2) to simulate the morpheme fluctuations in short-term speech and enhance the robustness of the model to micro-segment features; Fine-tuning of loss parameters: The core structure of AAM-Softmax loss (angle margin + scaling factor) is adopted, and the angle margin is set to 0.4 to avoid overfitting of sparse features in 1 second. With a scaling factor of 32, the total loss formula is: ,in The L2 regularization term (weight decay of 1e-4) is used to prevent the model from overfitting. Optimizer configuration: SGD optimizer is used, with cosine annealing learning rate scheduling.

[0059] By optimizing the training data and loss parameters, the model is forced to learn short-term sparse features at the 1-second level.

[0060] The short-time audio speaker recognition method provided in this embodiment has the following beneficial effects: This method improves the accuracy of speaker recognition in short audio by four layers: effective frame selection, micro-segment feature enhancement, global fusion adaptation, and short-time training optimization, without destroying the core architecture of the original model, while maintaining lightweight design and deployment compatibility.

[0061] like Figure 4 As shown, this embodiment provides a short-time audio speaker recognition device, including: The filtering module 401 is used to obtain the speech features of all speech frames in the short audio and filter all speech features to generate an effective speech feature map. The local fusion module 402 sequentially fuses the effective speech feature map through several local feature fusions of different frequency bands to obtain local fused features of different frequency bands. The global fusion module 403 performs phased global fusion based on the local fusion features of each frequency band except the low frequency band, to obtain global fusion features; The recognition module 404 infers a speech vector based on global fusion features and compares the speech vector with the registered vector in the registered sound library to identify the speaker's identity.

[0062] Figure 5 A schematic diagram of a terminal device or server suitable for implementing embodiments of this application is shown.

[0063] like Figure 5 As shown, the terminal device or server includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 502 or a program loaded from storage section 508 into random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of the terminal device or server. The CPU 501, ROM 502, and RAM 503 are interconnected via bus 504. An input / output (I / O) interface 505 is also connected to bus 504.

[0064] The following components are connected to I / O interface 505: an input section 506 including a keyboard, mouse, etc.; an output section 507 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to I / O interface 505 as needed. A removable medium 511, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 510 as needed so that computer programs read from it can be installed into storage section 508 as needed.

[0065] Specifically, according to embodiments of this application, the above method flow steps can be implemented as a computer software program. For example, embodiments of this application include a computer program product comprising a computer program carried on a machine-readable medium, the computer program containing program code for performing the methods shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via communication section 509, and / or installed from removable medium 511. When the computer program is executed by central processing unit (CPU) 501, it performs the functions defined in the system of this application.

[0066] It should be noted that the computer-readable medium shown in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0067] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0068] The units or modules described in the embodiments of this application can be implemented in software or hardware. The described units or modules can also be located in a processor. The names of these units or modules do not, in certain circumstances, constitute a limitation on the unit or module itself.

[0069] In another aspect, this application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable storage medium stores one or more programs that, when used by one or more processors, execute the methods described in this application.

[0070] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0071] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for short-time audio speaker recognition, characterized in that, include: The speech features of all speech frames in a short audio segment are obtained, and all speech features are filtered to generate an effective speech feature map. The effective speech feature map is sequentially fused with local features of different frequency bands several times to obtain local fused features of different frequency bands; Global fusion is performed in stages based on the local fusion features of each frequency band except the low frequency band, to obtain global fusion features; Speech vectors are inferred based on global fusion features, and the speech vectors are compared with the registered vectors in the registered sound library to identify the speaker's identity.

2. The method according to claim 1, characterized in that, The step of filtering all speech features to generate an effective speech feature map includes: Calculate the frame energy of the speech features of each speech frame, and normalize each frame energy respectively; Calculate the spectral flatness of the speech features for each speech frame; The normalized frame energy and spectral flatness corresponding to the speech features of the same speech frame are weighted and summed to obtain the quality score of the speech features of the corresponding speech frame. Retain speech features with quality scores greater than or equal to a preset threshold to generate effective speech feature maps.

3. The method according to claim 2, characterized in that, Also includes: When the proportion of retained speech features is less than the second preset threshold, the average of the neighboring retained speech features is used to fill the gap until the number of retained speech features reaches 60% of the total number of speech features, and an effective speech feature map is generated.

4. The method according to claim 1, characterized in that, The step of sequentially fusing local features of different frequency bands into the effective speech feature map to obtain local fused features of different frequency bands includes: The effective speech feature map is fused with local features in the low-frequency band to obtain local fused features in the low-frequency band; The local fusion features of the low-frequency band are fused with the local features of the mid-frequency band to obtain the local fusion features of the mid-frequency band. The local fusion features of the mid-frequency band are fused with the local features of the mid-high frequency band to obtain the local fusion features of the mid-high frequency band. The local fusion features of the mid-to-high frequency band are fused with the local features of the high frequency band to obtain the local fusion features of the high frequency band.

5. The method according to claim 4, characterized in that, The local feature fusion process for any frequency band includes: The channel dimension of the local fusion feature from the previous step is doubled by a 1×1 convolution, and the expanded local fusion feature is divided into two subsets according to the expanded channel dimension; the effective speech feature map is the local fusion feature from the initial step. For any subset, slide the window across the speech frames of the subset and divide it into several micro-segments. For each micro-segment, use a 1×3 depthwise separable convolution to extract the local features of the micro-segment. The local features of the micro-segments are processed through global average pooling, multilayer perceptron, and mapping function to obtain the corresponding micro-segment attention weights; The local features of the micro-segments are weighted by the micro-segment attention weights to obtain the weighted features of the micro-segments; The weighted features of all micro-segments in the aggregate subset are used to obtain the corresponding weighted feature subset. The two weighted feature subsets are concatenated along the channel dimension and then input into the attention feature fusion module to obtain the dynamic fusion weights; The fused features are obtained by summing two weighted feature subsets based on the dynamic fusion weights. The current local fusion features are obtained by adding the results of 1×1 convolution and batch normalization to the fused features.

6. The method according to claim 4, characterized in that, The phased global fusion based on the local fusion features of each frequency band except the low-frequency band yields global fusion features, including: Align the local fusion features of the mid-frequency band with the local fusion features of the mid-high frequency band using 3×3 convolution, considering both the frequency band dimension and the channel dimension. The frequency band dimension of the local fusion features in the high-frequency band is aligned with the frequency band dimension of the local fusion features in the mid-to-high-frequency band by using 3×3 convolution. The aligned local fusion features of the mid-frequency band and the local fusion features of the mid-high frequency band are input into the attention feature fusion module to obtain the preliminary global fusion features. The temporal and channel dimensions of the initial global fusion features and the local fusion features of the high-frequency band are aligned using 3×3 convolution. The aligned preliminary global fusion features and the aligned high-frequency local fusion features are input into the attention feature fusion module to obtain intermediate global fusion features; The global fusion feature is obtained by aligning the frequency band dimension of the intermediate global fusion feature with the frequency band dimension of the original high-frequency local fusion feature through a 1×1 convolution.

7. The method according to claim 1, characterized in that, The process of inferring speech vectors based on global fusion features and comparing these speech vectors with registered vectors in a registered sound database to identify the speaker's identity includes: The mean and standard deviation of the global fusion features are calculated in the time dimension, and the mean and standard deviation are concatenated to obtain a fixed-length representation. The fixed-length representation is passed through the first fully connected layer to obtain preliminary embedded features; The initial embedded features are passed through a second fully connected layer to obtain the speech vector; The speech vectors are L2 normalized, and the cosine similarity between the L2 normalized speech vectors and each registered vector in the registered sound library is calculated. Each cosine similarity is compared with the recognition threshold. When any cosine similarity is greater than or equal to the recognition threshold, the speaker is determined to be a registered user to which the corresponding registration vector belongs; otherwise, the speaker is determined to be an unregistered user.

8. A short-time audio speaker recognition device, characterized in that, include: The filtering module is used to obtain the speech features of all speech frames in a short audio clip, filter all speech features, and generate an effective speech feature map. The local fusion module sequentially fuses the effective speech feature map through several local feature fusions of different frequency bands to obtain local fused features of different frequency bands. The global fusion module performs phased global fusion based on the local fusion features of each frequency band except the low-frequency band, to obtain global fusion features; The recognition module infers speech vectors based on global fusion features and compares the speech vectors with the registered vectors in the registered sound library to identify the speaker's identity.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the method as described in any one of claims 1 to 7.

Images