Target object classification method, system, electronic device, and storage medium

Abstract

The application provides a target object classification method, system, electronic equipment and storage medium, and belongs to the technical field of artificial intelligence. The target object classification method comprises the following steps: obtaining an original speech signal of a target object; inputting the original speech signal into a preset target object classification model; wherein the target object classification model comprises a time domain feature extractor and a Transformer model, and the time domain feature extractor and the Transformer model constitute an end-to-end structure; performing feature extraction processing on the original speech signal based on the time domain feature extractor to obtain a mel filter bank feature signal; performing spectral enhancement and down-sampling processing on the mel filter bank feature signal to obtain a preprocessed speech signal; performing feature extraction processing on the preprocessed speech signal based on the Transformer model to obtain a classification result of the target object. The application realizes extraction of speaker discrimination features from an original speech input, improves the capacity and recognition ability of speaker features, and can effectively classify target objects.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and particularly relates to a target object classification method and system, electronic equipment and a storage medium. BACKGROUND

[0002] Speaker Verification (SV) is a binary classification task, which aims to verify whether an unknown speech belongs to the identity it claims. In the research of SV, a deep neural network is usually used to extract the features of the speaker as the target object. Attention is a powerful method to obtain more discriminative speech-level features. The SV model and method in the related art cannot extract speaker discriminative features from the original speech input, which limits the capacity and recognition ability of the speaker features, and cannot effectively classify the target object. SUMMARY

[0003] The main purpose of the embodiments of the present application is to provide a target object classification method and system, electronic equipment and a storage medium, which can extract speaker discriminative features from the original speech input, improve the capacity and recognition ability of the speaker features, and effectively classify the target object.

[0004] To achieve the above purpose, the first aspect of the embodiments of the present application provides a target object classification method, which comprises:

[0005] obtaining an original speech signal of a target object;

[0006] inputting the original speech signal into a preset target object classification model; wherein the target object classification model comprises a time domain feature extractor and a Transformer model, and the time domain feature extractor and the Transformer model constitute an end-to-end structure;

[0007] performing feature extraction processing on the original speech signal based on the time domain feature extractor to obtain a mel filter bank feature signal;

[0008] performing spectral enhancement and down-sampling processing on the mel filter bank feature signal to obtain a preprocessed speech signal, performing feature extraction processing on the preprocessed speech signal based on the Transformer model to obtain a classification result of the target object.

[0009] In some embodiments of the present application, the time domain feature extractor is constructed by the following steps:

[0010] obtaining a pre-emphasis module, a framing module, a windowing module, a short-time Fourier transform module and a mel filter module;

[0011] The pre-emphasis module, the frame segmentation module, the windowing module, and the short-time Fourier transform module are sequentially combined into a convolutional layer, and the Mel filter module is combined into a fully connected layer.

[0012] The temporal feature extractor is obtained based on the convolutional layer and the fully connected layer.

[0013] In some embodiments of the present invention, the temporal feature extractor is trained through the following steps:

[0014] Acquire sample speech signals from multiple raw samples;

[0015] The sample speech signal is input into the temporal feature extractor to obtain the sample Mel filter bank feature signal. The temporal feature loss value is obtained based on the sample Mel filter bank feature signal, and the parameters of the temporal feature extractor are optimized and updated based on the temporal feature loss value.

[0016] In some embodiments of the present invention, the Transformer model includes multiple Transformer groups, and the Transformer model is trained through the following steps:

[0017] Obtain training samples, which include multiple preprocessed audio data;

[0018] The preprocessed audio data is input into the Transformer model to obtain the frame-level features output by each Transformer group; wherein, the frame-level features include sentence features and speech features corresponding to multiple speech frames;

[0019] The Transformer model loss value is determined based on the distance between the sentence features and each of the speech features, and the parameters of the Transformer model are optimized and updated based on the Transformer model loss value.

[0020] In some embodiments of the present invention, the speech feature is the first frame of the frame-level feature, and the speech feature is the remaining frames of the frame-level feature excluding the first frame. Determining the Transformer model loss value based on the distance between the sentence feature and each of the speech features includes:

[0021] Construct the classification loss function for the Transformer model;

[0022] Construct the diffusion loss function of the Transformer model, wherein the diffusion loss function represents the sum of distances between the first frame and each of the remaining frames;

[0023] The model loss function used during Transformer model training is obtained based on the difference between the classification loss function and the diffusion loss function, and the Transformer model loss value is obtained based on the model loss function.

[0024] In some embodiments of the present invention, the classification loss function for constructing the Transformer model includes:

[0025] Obtain the scaling factor, hyperparameters, and natural constant;

[0026] The first parameter is the exponent obtained by taking the natural constant as the base and based on the scaling factor, the phase angle of the frame-level feature, and the hyperparameter.

[0027] The exponent, based on the natural constant and the phase angle of the scaling factor and the frame-level feature, is used as the second parameter.

[0028] The additional angle margin loss function is calculated based on the first parameter and the second parameter, and the obtained additional angle margin loss function is used as the classification loss function.

[0029] In some embodiments of the present invention, the diffusion loss function for constructing the Transformer model includes:

[0030] Obtain the KL divergence between the first frame and each of the remaining frames;

[0031] The diffusion loss function is obtained by summing the multiple KL divergences within the obtained Transformer model.

[0032] To achieve the above objectives, a second aspect of the present invention provides a target object classification system, the system comprising:

[0033] The signal receiving module is used to acquire the original voice signal of the target object;

[0034] A target object classification model is used to receive the original speech signal. The target object classification model includes a temporal feature extractor and a Transformer model, and the temporal feature extractor and the Transformer model constitute an end-to-end structure.

[0035] The time-domain feature extractor is used to perform feature extraction processing on the original speech signal to obtain the Mel filter bank feature signal;

[0036] The target object classification model is also used to perform spectral enhancement and downsampling processing on the feature signals of the Mel filter bank to obtain a preprocessed speech signal and input it into the Transformer model;

[0037] The Transformer model is used to perform feature extraction on the preprocessed speech signal to obtain the classification result of the target object.

[0038] To achieve the above objectives, a third aspect of the present invention provides an electronic device, the electronic device including a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for implementing communication between the processor and the memory, wherein the program, when executed by the processor, implements the method described in the first aspect above.

[0039] To achieve the above objectives, a fourth aspect of the present invention provides a storage medium, which is a computer-readable storage medium for computer-readable storage, wherein the storage medium stores one or more programs that can be executed by one or more processors to implement the method described in the first aspect.

[0040] The target object classification method, system, electronic device, and storage medium proposed in this invention embodiment can be applied in a target object classification system. By using the target object classification method, this invention embodiment obtains the original speech signal of the target object and inputs the original speech signal into a preset target object classification model for processing. The target object classification model includes a temporal feature extractor and a Transformer model, which constitute an end-to-end structure. Therefore, the target object classification model is an end-to-end SV model. In this model, the temporal feature extractor performs feature extraction processing on the original speech signal to obtain the Mel filter bank feature signal. The Mel filter bank feature signal is then subjected to spectral enhancement and downsampling processing to obtain a preprocessed speech signal. The preprocessed speech signal is then subjected to feature extraction processing based on the Transformer model to obtain the classification result of the target object. In this invention embodiment, the original speech signal can be directly obtained and processed through the temporal feature extractor, and the speech classification result is output through the Transformer model. Therefore, this invention realizes the extraction of speaker discrimination features from the original speech input, improves the capacity and recognition ability of speaker features, and can effectively classify target objects.

[0041] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the description, claims and drawings. Attached Figure Description

[0042] The accompanying drawings are provided to further understand the technical solutions of the present invention and constitute a part of the specification. They are used together with the embodiments of the present invention to explain the technical solutions of the present invention, and do not constitute a limitation on the technical solutions of the present invention.

[0043] Figure 1 This is a flowchart of the target object classification method provided in the embodiments of the present invention;

[0044] Figure 2 This is a schematic diagram of the SV model provided in an embodiment of the present invention;

[0045] Figure 3 This is a flowchart illustrating the construction process of the temporal feature extractor provided in this embodiment of the invention;

[0046] Figure 4 This is a framework diagram of the temporal feature extractor provided in an embodiment of the present invention;

[0047] Figure 5 This is a flowchart of the training process of the temporal feature extractor provided in an embodiment of the present invention;

[0048] Figure 6 This is a flowchart of the training process of the Transformer model provided in an embodiment of the present invention;

[0049] Figure 7 yes Figure 6 The flowchart of step S403 in the process;

[0050] Figure 8 This is a schematic diagram of the SV model prior to the embodiments of the present invention;

[0051] Figure 9 yes Figure 7 The flowchart of step S501 in the process;

[0052] Figure 10 yes Figure 7 The flowchart of step S502 in the document;

[0053] Figure 11 This is a schematic diagram of the target object classification system provided in an embodiment of the present invention;

[0054] Figure 12 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0056] It should be noted that although functional modules are divided in the system diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the system or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0057] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein is for the purpose of describing embodiments of the invention only and is not intended to limit the invention.

[0058] First, let's analyze some of the terms used in the embodiments of this invention:

[0059] Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0060] The Transformer model is a fully self-attention-based neural network model that computes input and output representations using a self-attention mechanism. The Transformer model is a framework built upon attention mechanisms, and its overall architecture can be divided into an input layer, an encoding layer, a decoding layer, and an output layer. If the Transformer model inputs an image feature matrix into the encoding layer, after performing self-attention computation on the image feature matrix, the encoding layer sends the self-attention-computed image feature matrix to the feedforward neural network of that layer. This feedforward neural network then sends the image feature matrix to the next encoding layer, repeating the process until all multiple encoding layers have been executed. Finally, residual blocks are used to decode the features, and a convolutional layer is used to obtain the feature extraction result.

[0061] Embedding: Embedding is a vector representation, also called a feature or feature vector. It refers to representing an object with a low-dimensional vector. The property of embedding vectors is that vectors that are close in distance correspond to objects with similar meanings. Essentially, embedding is a mapping from semantic space to vector space, while preserving the semantic relationships of the original samples in the vector space as much as possible. For example, two semantically similar words are also likely to be located close in the vector space. Embedding can encode objects with low-dimensional vectors while retaining their meaning. It is commonly used in machine learning to improve efficiency during machine learning model building.

[0062] In Segmentation Verification (SV) research, there are generally two types of models: statistical models and neural network models. As the scale of labeled data continues to expand, neural network models are used to extract speaker feature vectors for speaker verification, driven by advanced models such as CNNs and RNNs.

[0063] However, more powerful deep neural networks are still needed to better extract speaker features. Attention mechanisms are a powerful approach that provides a way to obtain more discriminative utterance-level features. The Transformer-based SV method enables the model to learn frame-level features, thus more accurately representing speaker characteristics.

[0064] Based on this, embodiments of the present invention provide a target object classification method, system, electronic device, and storage medium, which aim to extract speaker discrimination features from the original speech input, improve the capacity and recognition ability of speaker features, and effectively classify target objects.

[0065] The target object classification method, system, electronic device, and storage medium provided in the embodiments of the present invention are specifically described through the following embodiments. First, the target object classification method in the embodiments of the present invention is described.

[0066] The embodiments of this invention can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that utilize digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0067] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0068] The target object classification method provided in this invention relates to the field of artificial intelligence technology. This method can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments of this invention, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the target object classification method, but is not limited to the above forms.

[0069] Embodiments of the present invention can be used in numerous general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. Embodiments of the present invention can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. Embodiments of the present invention can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0070] Figure 1 This is an optional flowchart of the target object classification method provided in the embodiments of the present invention. Figure 1 The target object classification method may include, but is not limited to, steps S101 to S107.

[0071] Step S101: Obtain the original speech signal of the target object.

[0072] It should be noted that in the embodiments of the present invention, the original voice signal of the target object can be directly obtained for subsequent processing by the target object classification method in the embodiments of the present invention. The original voice signal includes the voice information of multiple objects, and the voice of the speaker needs to be separated through speaker verification. The speaker is the target object described in the embodiments of the present invention.

[0073] Step S102: Input the original speech signal into the preset target object classification model; wherein, the target object classification model includes a temporal feature extractor and a Transformer model, and the temporal feature extractor and the Transformer model constitute an end-to-end structure.

[0074] It should be noted that the embodiments of the present invention provide a target object classification model, such as... Figure 2 As shown, the target object classification model is the aforementioned SV model, used for speaker verification to output speaker classification results. The proposed target object classification model sets up a time-domain feature extractor (TDFE) and a Transformer model. The setting of the time-domain feature extractor and the Transformer model makes the target object classification model an end-to-end model, as detailed in the following steps.

[0075] Step S103: Perform feature extraction processing on the original speech signal based on the time-domain feature extractor to obtain the Mel filter bank feature signal.

[0076] It should be noted that in this embodiment of the invention, the original speech signal is processed by a preset temporal feature extractor to obtain the Mel filter bank feature signal. The temporal feature extractor is a pre-set neural network model used to process the original speech signal to obtain the Mel filter bank feature signal. In the prior art, the SV model often takes a Mel-Fbank feature or MFCC feature as input. Here, we take the Mel-Fbank feature as an example, which is the Mel filter bank feature signal. However, the original speech signal is not a Mel-Fbank feature, so the Mel-Fbank feature needs to be transformed. In this embodiment of the invention, with the addition of the temporal feature extractor, compared with the prior art SV model, the original speech signal can be directly input for speaker classification. Moreover, the temporal feature extractor is a neural network model learned from the original data distribution, which makes all operations possible on the processor through the original temporal input, which is more suitable for subsequent processing of the neural network and improves processing efficiency and speed.

[0077] Step S104: Perform spectral enhancement and downsampling on the feature signal of the Mel filter bank to obtain a preprocessed speech signal. Then, perform feature extraction on the preprocessed speech signal based on the Transformer model to obtain the classification result of the target object.

[0078] It should be noted that, in the target object classification model of this invention, the Mel filter bank feature signal output by the temporal feature extractor is also preprocessed. The preprocessing process includes spectral enhancement and downsampling of the Mel filter bank feature signal. The preprocessed signal is the preprocessed speech signal. The preprocessed speech signal meets the conditions for input into the Transformer model. Subsequently, the preprocessed speech signal is input into the Transformer model, and the output is the speaker classification loss. The classification result of the target object in this invention is the speaker classification loss, and finally the speaker classification is completed. It can be understood that, in this invention, by directly inputting the original speech signal, and setting the temporal feature extractor and Transformer model in the SV model to form an end-to-end structure, the speaker classification loss can be output. This invention can extract the features of the original speech input more accurately and effectively, and achieve a more accurate speaker recognition capability.

[0079] It is understood that the preprocessing process in the embodiments of the present invention may also include other preprocessing steps. The preprocessing process is a step that all SV models based on the Transformer model have. In one embodiment, the preprocessing also includes position encoding of the signal. Specifically, after the original speech signal is input to the temporal feature extractor, the mel-fbank feature is output. The spectral enhancement module (Spec Aug) in the SV model performs spectral enhancement processing on the mel-fbank feature to enhance the data and improve the robustness of the model. Subsequently, the sub-sampling module performs downsampling processing on the spectrally enhanced signal. The position encoding module (Positional Encoding) in the preprocessing process performs position encoding processing on the downsampled signal. This position encoding is supplemented. Then, the obtained preprocessed speech signal is input into the Transformer model. After the Transformer model processes the signal, the speech will be segmented into frame-by-frame vectors. It is understood that the SV model also has a classification module at the end. The classification module is used to classify the signal extracted by the Transformer model and finally outputs the speaker classification loss. The speaker classification loss is the classification result of the target object described in the above embodiments.

[0080] Reference Figure 3As shown, in some embodiments of the present invention, the temporal feature extractor is constructed through the following steps, which may include, but are not limited to, steps S201 to S203.

[0081] Step S201: Obtain the pre-emphasis module, frame segmentation module, windowing module, short-time Fourier transform module, and Mel filter module.

[0082] In step S202, the pre-emphasis module, the framing module, the windowing module, and the short-time Fourier transform module are sequentially combined into a convolutional layer, and the Mel filter module is combined into a fully connected layer.

[0083] Step S203: Obtain the temporal feature extractor based on the convolutional layer and the fully connected layer.

[0084] It should be noted that the temporal feature extractor mentioned in this embodiment of the invention is a pre-set neural network architecture extractor. The temporal feature extractor is a front-end module that directly extracts features from the original speech signal, which is the original waveform input. When constructing a temporal feature extractor, a standard mel-fbank feature extractor generally consists of the following modules: a pre-emphasis module, a framing module, a windowing module, a short-time Fourier transform (STFT) module, an energy spectrum (ES) module (not shown in the figure), and a Mel-filter module. The framework of the temporal feature extractor in this embodiment of the invention is as follows: Figure 4 As shown, all these parts are combined into a black-box operation of a convolutional layer and a fully connected layer. The pre-emphasis module, the framing module, the windowing module, and the short-time Fourier transform module are sequentially combined into a convolutional layer, and the Mel filter module is combined into a fully connected layer. It can be understood that the general steps for obtaining Mel-Fbank features from speech signals in this embodiment of the invention are inspired by human perceptual biology. The features extracted by the proposed TDFE are learned from the distribution of real data and are more suitable for subsequent processing by neural networks.

[0085] Specifically, since the temporal feature extractor in this embodiment of the invention is learned from the distribution of real data, and a neural network model is used to replace the original operation on the original speech signal that originally needed to be performed externally, the temporal feature extractor of the neural network model is directly added to the SV model, which helps to form an end-to-end system. Moreover, due to the addition of the neural network model, the speed of data training is also significantly improved. The use of the temporal feature extractor can extract the features of the original speech input more accurately and effectively, further improving the processing capability of the original speech signal, so that the SV model in this embodiment of the invention has greater capacity and resolution.

[0086] Reference Figure 5As shown, in some embodiments of the present invention, the temporal feature extractor is trained through the following steps, which may include, but are not limited to, steps S301 to S302.

[0087] Step S301: Obtain sample speech signals from multiple original samples.

[0088] Step S302: Input the sample speech signal into the time-domain feature extractor to obtain the sample Mel filter bank feature signal, obtain the time-domain feature loss value based on the sample Mel filter bank feature signal, and optimize and update the parameters of the time-domain feature extractor based on the time-domain feature loss value.

[0089] It should be noted that the temporal feature extractor in this embodiment of the invention is learned from the distribution of real data, making it more suitable for subsequent processing by the neural network. Specifically, during the model training phase, sample speech signals from multiple original samples can be obtained. The original samples are samples from multiple real data sets, and the sample speech signals are the sample set of the original speech signals corresponding to those in the above embodiment. Subsequently, the sample speech signals are input into the temporal feature extractor to obtain sample Mel filter bank feature signals. The temporal feature loss value is obtained based on the sample Mel filter bank feature signals, and the parameters of the temporal feature extractor are optimized and updated based on the temporal feature loss value. The temporal feature extractor proposed in this embodiment of the invention is learnable and consists of a convolutional layer and a fully connected layer. The input is the original speech signal in the temporal domain, and the output is a feature similar to a Mel-Fbank. The purpose is to make the entire system an end-to-end model. Feature extraction through the neural network can be performed on the processor, accelerating the forward propagation speed and model training speed. At the same time, the features extracted by the neural network are learned from the distribution of real data, making them more suitable for subsequent processing by the neural network.

[0090] The Transformer model consists of multiple Transformer groups, also called Transformer layers, as shown in the reference. Figure 6 As shown, in some embodiments of the present invention, the Transformer model is trained through the following steps, which may include, but are not limited to, steps S401 to S403.

[0091] Step S401: Obtain training samples, which include multiple preprocessed audio data.

[0092] Step S402: Input the preprocessed audio data into the Transformer model to obtain the frame-level features output by each Transformer group; wherein, the frame-level features include sentence features and speech features corresponding to multiple speech frames.

[0093] Step S403: Determine the Transformer model loss value based on the distance between sentence features and each speech feature, and optimize and update the Transformer model parameters based on the Transformer model loss value.

[0094] It should be noted that, as Figure 2 As shown, the Transformer model contains multiple Transformer Blocks, which are connected sequentially to form a layered connection. During the training phase of the Transformer model, training samples can be obtained. These training samples include multiple preprocessed audio data, similar to the preprocessed speech signals in the above embodiment. The preprocessed audio data is then input into the Transformer model to obtain the frame-level features output by each Transformer Block. These frame-level features include sentence features and speech features corresponding to multiple speech frames, such as... Figure 2 As shown, the obtained frame-level features are the embeddings. There are multiple embeddings. Sentence features are the utterance-level embeddings in the embeddings. The utterance-level features summarize information from other frame-level embeddings through a self-attention mechanism. The speech features are the aforementioned frame-level embeddings. In this embodiment of the invention, the Transformer model loss value is determined based on the distance between the sentence features and each speech feature. The parameters of the Transformer model are optimized and updated based on the Transformer model loss value, thereby enhancing the speaker-related information in the utterance-level embeddings and weakening the speaker-related information in other frame-level embeddings, which helps to obtain the classification results of the target object.

[0095] It should be noted that each transformer group includes sequentially adjacent attention layers, summation and normalization layers, feedforward layers, and add & normalization layers. In the embedding layer of the transformer group, the current input is embedded to obtain multiple feature vectors. In the attention layer, P input vectors are obtained from the previous transformer group. Using any first input vector among the P input vectors as the center, based on the correlation between each input vector within a preset attention window and the first input vector, an intermediate vector corresponding to the first input vector is obtained. This process determines P intermediate vectors corresponding to the P input vectors. In the pooling layer, the P intermediate vectors are merged into Q output vectors, where multiple output vectors obtained from the last transformer layer in at least one transformer layer are used as feature representations of the current input. In the embedding layer, the current input (which can be text input, such as a paragraph or a sentence; the text can be Chinese / English or other languages) is embedded to obtain multiple feature vectors. After obtaining the current input, the embedding layer can embed each word in the current input to obtain feature vectors for each word. The feature vectors described here are the embeddings in the above embodiment.

[0096] Specifically, the first frame contains speech features at the frame level, and the remaining frames (also called other frames) contain speech features at the frame level. (See reference...) Figure 7 As shown, in some embodiments of the present invention, step S403 in the target object classification method may include, but is not limited to, steps S501 to S503.

[0097] Step S501: Construct the classification loss function for the Transformer model.

[0098] Step S502: Construct the diffusion loss function of the Transformer model. The diffusion loss function represents the sum of distances between the first frame and all other frames.

[0099] Step S503: Obtain the model loss function used during Transformer model training based on the difference between the classification loss function and the diffusion loss function, and obtain the Transformer model loss value based on the model loss function.

[0100] It should be noted that the sentence features are the first frame in the frame-level features, that is, the utterance-level embedding is the first frame feature of each Transformer layer, while the speech features are the remaining frames of the frame-level features excluding the first frame. The model loss function used during Transformer model training is obtained based on the classification loss function and the difffluence loss function. The classification loss is a cross-entropy function, and the difffluence loss function represents the sum of distances between the first frame and all the remaining frames. The difffluence loss function can enhance the relevant information of the speaker in the utterance-level embedding. In this embodiment of the invention, the model loss function used during Transformer model training is obtained based on the difference between the classification loss function and the difffluence loss function, and the Transformer model loss value is obtained based on the model loss function.

[0101] It should be noted that the first frame in this embodiment is the final embedding used. The first frame embedding is added extra and can learn information from the entire sentence. Each subsequent frame embedding corresponds to a segment of the original signal. The input and purpose of the first frame signal are different from the signals of the other frames, so the more different the first frame is from the others, the better. Here, the sum of the distances between the first frame embedding and the embeddings of other frames is calculated, and this sum is maximized during learning to make the first frame embedding different from the others. Therefore, to enhance the speaker information in the first frame, the distance between the first frame and the other frames needs to be increased. This requires minimizing the model's loss function. By subtracting a diffusion loss function from the classification loss function, the distance between the first frame and the other frames is increased, thereby enhancing the speaker information in the first frame and weakening the speaker information in the other frames. This reduces the waste of frame-level information in the output characteristics and improves the capacity and recognition ability of speaker features.

[0102] Understandably, in SV-based models, similar to Transformers applied to natural language processing (such as BERT) and computer vision (such as ViT), ... Figure 8As shown, an extra learnable speaker embedding is added to the sequence as the first frame of the model, which is the speech feature of the frame-level features in this embodiment of the invention. The speech feature is the first frame in the frame-level features, and the final hidden state corresponding to this frame is used as the utterance-specific embedding for speaker classification. Information in other frames is ignored. However, speaker-related information not only exists in the embedding of the first frame, but also in the other frames. In order to improve the recognition ability of speaker features, the ignored information needs to be further aggregated to form a better utterance-level embedding.

[0103] The SV model in the embodiments of the present invention, such as Figure 2 As shown, a diffusion loss function is added to the model loss function. The diffusion loss function is the sum of the distances between the first frame embedding (the first dot in the figure) and the other frame embeddings (all other dots except the first dot) in each layer. This enhances the speaker-related information in the first frame and weakens the speaker-related information in the remaining frames. Because this loss function utilizes all the features of the entire sentence, it is more suitable for capturing the speaker's features in the sentence.

[0104] Reference Figure 9 As shown, in some embodiments of the present invention, step S501 in the target object classification method may include, but is not limited to, steps S601 to S604.

[0105] Step S601: Obtain the scaling factor, hyperparameters, and natural constants.

[0106] Step S602: The first parameter is obtained by using the natural constant as the base and the exponent obtained based on the scaling factor, the phase angle of the frame-level feature, and the hyperparameter.

[0107] Step S603: The exponent, which is based on the natural constant and the phase angle of the scaling factor and frame-level features, is used as the second parameter.

[0108] Step S604: Calculate the additional angle margin loss function based on the first parameter and the second parameter, and use the obtained additional angle margin loss function as the classification loss function.

[0109] It should be noted that the model loss function in this embodiment of the invention consists of two parts: a classification loss function and a diffusion loss function. The classification loss function is a cross-entropy function. In this embodiment, the AAM-Softmax loss function is used as the classification loss function. The scaling factor, hyperparameters, and natural constant are obtained, where τ is a scaling factor to prevent gradients from becoming too small during training, m is a hyperparameter, and e is a natural constant, resulting in the classification loss function L. C L C As shown below:

[0110]

[0111]

[0112]

[0113] Wherein, formula (1) is the classification loss function in the embodiment of the present invention, φ n The first parameter is an exponent Ψ, derived from the scaling factor, the phase angle of the frame-level features, and the hyperparameters. n The second parameter is an exponent derived from the scaling factor and the phase angle of the frame-level features.

[0114] It should be noted that τ in the above formula applies L2 regularization to the sample features and then scales them. m is a hyperparameter with a default value of 0.5. Under the premise of meeting the requirements of this embodiment, the value of m can be adjusted according to actual needs, and no specific restrictions are imposed. Furthermore, in the above formula, N is the number of samples selected in one training iteration (BatchSize), n is the class number, and l represents the l-th Transformer group. It can be understood that BatchSize is a hyperparameter used to define the number of samples to be processed before updating the internal model parameters; that is, it controls the number of training samples before updating the model's internal parameters. The training dataset can be divided into one or more batches. When all training samples are used to create a batch, the learning algorithm is called batch gradient descent; when the batch size is one sample size, the learning algorithm is called stochastic gradient descent; when the batch size exceeds one sample size but is smaller than the size of the training dataset, the learning algorithm is called mini-batch gradient descent. Batch Size refers to the number of samples processed before updating the model.

[0115] Reference Figure 10 As shown, in some embodiments of the present invention, step S502 in the target object classification method may include, but is not limited to, steps S701 to S702.

[0116] Step S701: Obtain the KL divergence between the first frame and each of the remaining frames.

[0117] Step S702: Sum the multiple KL divergences in the obtained Transformer model to obtain the diffusion loss function.

[0118] It should be noted that the model loss function in this embodiment of the invention consists of two parts: a classification loss function and a diffusion loss function. The diffusion loss function uses Kullback-Leibler divergence, which is widely used in machine learning and deep learning, to represent distance. Specifically, the diffusion loss function obtains the Kullback-Leibler divergence between the first frame and all other frames. Kullback divergence is a measure of the asymmetry of the difference between two probability distributions, and it can measure the distance between two random distributions. When two random distributions are the same, their Kullback divergence is zero. When the difference between two random distributions increases, their Kullback divergence also increases. Therefore, Kullback divergence can be used to compare the similarity of texts. Subsequently, the multiple Kullback divergences obtained in the Transformer model are summed to obtain the diffusion loss function L. D L D As shown below:

[0119]

[0120] Among them, V li Let L represent the i-th frame of layer l, L represent the total number of frame-level features, T represent the number of layers in the Transformer group, and V represent the remaining frames. l0 Representing the first frame, KL() is used to represent the Kullback-Leibler divergence, therefore KL(V l0 ||V li ) is the KL divergence between the first frame and some other frame.

[0121] In summary, the model loss function in this embodiment of the invention is obtained based on the difference between the classification loss function and the diffusion loss function. Therefore, the model loss function L is obtained, and the formula corresponding to L is as follows:

[0122] L = L C -L D (5)

[0123] It is understood that the target object classification method proposed in this embodiment of the invention is an improved Transformer-based temporal speaker verification method. It utilizes a temporal feature extractor and a diffusion loss function to capture speaker features, giving it greater capacity and discriminative power. Compared with other existing schemes, the SV model in this invention has outstanding performance and advantages. Compared with the standard mel-fbank extractor, the SV model in this embodiment uses a temporal feature extractor to extract features of the original speech input more accurately and effectively. Furthermore, the new loss function based on the diffusion loss function enhances speaker-related information in utterance-level embeddings while weakening speaker-related information in other frame-level embeddings. The diffusion loss function can also be flexibly embedded into other similar structures. Through these improvements, the proposed SV model achieves more accurate speaker recognition capabilities and ultimately outputs target object classification results.

[0124] Please see Figure 11 This invention also provides a target object classification system that can implement the target object classification method described in the above embodiments. The target object classification system includes:

[0125] The signal receiving module 1101 is used to acquire the original voice signal of the target object.

[0126] The target object classification model 1110 is used to receive the original speech signal. The target object classification model includes a temporal feature extractor 1102 and a Transformer model 1103. The temporal feature extractor and the Transformer model constitute an end-to-end structure.

[0127] The time-domain feature extractor 1102 is used to perform feature extraction processing on the original speech signal to obtain the Mel filter bank feature signal.

[0128] The target object classification model 1110 is also used to perform spectral enhancement and downsampling on the feature signals of the Mel filter bank to obtain a preprocessed speech signal and input it into the Transformer model 1103.

[0129] Transformer model 1103 is used to extract features from preprocessed speech signals to obtain classification results of target objects.

[0130] It should be noted that the target object classification system can execute the target object classification method in any of the above embodiments. By using the target object classification method, the embodiments of the present invention obtain the original speech signal of the target object, input the original speech signal into a preset target object classification model for processing. The target object classification model includes a temporal feature extractor and a Transformer model. The temporal feature extractor and the Transformer model constitute an end-to-end structure. Therefore, the target object classification model is an end-to-end SV model. In this model, the original speech signal is processed by feature extraction based on the temporal feature extractor to obtain the Mel filter bank feature signal. The Mel filter bank feature signal is then subjected to spectral enhancement and downsampling to obtain a preprocessed speech signal. The preprocessed speech signal is then processed by feature extraction based on the Transformer model to obtain the classification result of the target object. In the embodiments of the present invention, the original speech signal can be directly obtained and processed through the temporal feature extractor, and the speech classification result is output through the Transformer model. Therefore, the present invention realizes the extraction of speaker discrimination features from the original speech input, improves the capacity and recognition ability of speaker features, and can effectively classify target objects.

[0131] In addition, the signal receiving module 1101 can be part of the target object classification model 1110 or connected before the target object classification model 1110. The target object classification system can also be equipped with a classification module, which is used to classify the signal extracted by the Transformer model and finally output the speaker classification loss, that is, the classification result of the target object.

[0132] The specific implementation of this target object classification system is basically the same as the specific implementation of the target object classification method described above, and will not be repeated here.

[0133] This invention also provides an electronic device, comprising: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for communication between the processor and the memory. When the program is executed by the processor, it implements the aforementioned target object classification method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0134] Please see Figure 12 , Figure 12 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0135] The processor 1201 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention.

[0136] The memory 1202 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1202 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1202 and is called and executed by the processor 1201 to execute the target object classification method of the embodiments of this invention.

[0137] The input / output interface 1203 is used to implement information input and output;

[0138] The communication interface 1204 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0139] Bus 1205 transmits information between various components of the device (e.g., processor 1201, memory 1202, input / output interface 1203, and communication interface 1204);

[0140] The processor 1201, memory 1202, input / output interface 1203 and communication interface 1204 are connected to each other within the device via bus 1205.

[0141] This invention also provides a storage medium, which is a computer-readable storage medium for computer-readable storage. The storage medium stores one or more programs, which can be executed by one or more processors to implement the above-described target object classification method.

[0142] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0143] The embodiments described in this invention are for the purpose of more clearly illustrating the technical solutions of the embodiments of this invention, and do not constitute a limitation on the technical solutions provided by the embodiments of this invention. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this invention are also applicable to similar technical problems.

[0144] It will be understood by those skilled in the art that Figure 1 , Figure 3 , Figure 5 , Figure 6 , Figure 7 , Figure 9 and Figure 10 The technical solutions shown do not constitute a limitation on the embodiments of the present invention, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0145] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0146] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0147] The terms "first," "second," "third," "fourth," etc. (if present) in the specification and accompanying drawings of the embodiments of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0148] It should be understood that in the embodiments of the present invention, "at least one (item)" refers to one or more, and "more than one" refers to two or more. "And / or" is used to describe the relationship between associated objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following associated objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0149] In the embodiments provided by this invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between systems or units may be electrical, mechanical, or other forms.

[0150] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0151] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0152] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of the present invention, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0153] The preferred embodiments of the present invention have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and spirit of the present invention should be within the scope of the claims of the present invention.

Claims

1. A method for classifying target objects, characterized in that, The method includes: Acquire the raw speech signal of the target object; The original speech signal is input into a preset target object classification model; wherein, the target object classification model includes a temporal feature extractor and a Transformer model, and the temporal feature extractor and the Transformer model constitute an end-to-end structure; Based on the time-domain feature extractor, feature extraction processing is performed on the original speech signal to obtain the Mel filter bank feature signal; The feature signal of the Mel filter bank is subjected to spectral enhancement and downsampling to obtain a preprocessed speech signal. Based on the Transformer model, feature extraction is performed on the preprocessed speech signal to obtain the classification result of the target object. The Transformer model comprises multiple Transformer groups, and is trained through the following steps: Obtain training samples, which include multiple preprocessed audio data; The preprocessed audio data is input into the Transformer model to obtain the frame-level features output by each Transformer group; wherein, the frame-level features include sentence features and speech features corresponding to multiple speech frames, the sentence features are the first frame of the frame-level features, and the speech features are the remaining frames of the frame-level features excluding the first frame. The embedding of the first frame is added extra and used to learn the information of the entire sentence, and the embeddings of the remaining frames correspond to a segment of the original signal. Construct the classification loss function for the Transformer model; Construct the diffusion loss function of the Transformer model, wherein the diffusion loss function represents the sum of distances between the first frame and each of the remaining frames; The difference between the classification loss function and the diffusion loss function is used to train the Transformer model. The model loss function is defined, and the loss value of the Transformer model is obtained based on the model loss function. The parameters of the Transformer model are optimized and updated based on the loss value of the Transformer model.

2. The target object classification method according to claim 1, characterized in that, The temporal feature extractor is constructed through the following steps: The pre-emphasis module, frame segmentation module, windowing module, short-time Fourier transform module, and Mel filter module are acquired. The pre-emphasis module, the frame segmentation module, the windowing module, and the short-time Fourier transform module are sequentially combined into a convolutional layer, and the Mel filter module is combined into a fully connected layer. The temporal feature extractor is obtained based on the convolutional layer and the fully connected layer.

3. The target object classification method according to claim 1 or 2, characterized in that, The temporal feature extractor is trained through the following steps: Acquire sample speech signals from multiple raw samples; The sample speech signal is input into the temporal feature extractor to obtain the sample Mel filter bank feature signal. The temporal feature loss value is obtained based on the sample Mel filter bank feature signal, and the parameters of the temporal feature extractor are optimized and updated based on the temporal feature loss value.

4. The target object classification method according to claim 1, characterized in that, The classification loss function used to construct the Transformer model includes: Obtain the scaling factor, hyperparameters, and natural constant; The first parameter is the exponent obtained by taking the natural constant as the base and based on the scaling factor, the phase angle of the frame-level feature, and the hyperparameter. The exponent, based on the natural constant and the phase angle of the scaling factor and the frame-level feature, is used as the second parameter. The additional angle margin loss function is calculated based on the first parameter and the second parameter, and the obtained additional angle margin loss function is used as the classification loss function.

5. The target object classification method according to claim 1, characterized in that, The diffusion loss function used to construct the Transformer model includes: Obtain the KL divergence between the first frame and each of the remaining frames; The diffusion loss function is obtained by summing the multiple KL divergences within the obtained Transformer model.

6. A target object classification system, characterized in that, The system includes: The signal receiving module is used to acquire the original voice signal of the target object; A target object classification model is used to receive the original speech signal. The target object classification model includes a temporal feature extractor and a Transformer model, and the temporal feature extractor and the Transformer model constitute an end-to-end structure. The time-domain feature extractor is used to perform feature extraction processing on the original speech signal to obtain the Mel filter bank feature signal; The target object classification model is also used to perform spectral enhancement and downsampling processing on the feature signals of the Mel filter bank to obtain a preprocessed speech signal and input it into the Transformer model; The Transformer model is used to perform feature extraction on the preprocessed speech signal to obtain the classification result of the target object; The Transformer model comprises multiple Transformer groups, and is trained through the following steps: Obtain training samples, which include multiple preprocessed audio data; The preprocessed audio data is input into the Transformer model to obtain the frame-level features output by each Transformer group; wherein, the frame-level features include sentence features and speech features corresponding to multiple speech frames, the sentence features are the first frame of the frame-level features, and the speech features are the remaining frames of the frame-level features excluding the first frame. The embedding of the first frame is added extra and used to learn the information of the entire sentence, and the embeddings of the remaining frames correspond to a segment of the original signal. Construct the classification loss function for the Transformer model; Construct the diffusion loss function of the Transformer model, wherein the diffusion loss function represents the sum of distances between the first frame and each of the remaining frames; The difference between the classification loss function and the diffusion loss function is used to train the Transformer model. The model loss function is defined, and the loss value of the Transformer model is obtained based on the model loss function. The parameters of the Transformer model are optimized and updated based on the loss value of the Transformer model.

7. An electronic device, characterized in that, The electronic device includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for enabling communication between the processor and the memory. When the program is executed by the processor, it implements the steps of the target object classification method as described in any one of claims 1 to 5.

8. A storage medium, said storage medium being a computer-readable storage medium for computer-readable storage, characterized in that, The storage medium stores one or more programs, which can be executed by one or more processors to implement the steps of the target object classification method according to any one of claims 1 to 5.

Images