Large model-based multi-feature voice identification method, apparatus and device, and medium
By extracting and fusing the semantic and acoustic features of speech, generating speech fusion features, and combining them with decision-making models, the problem of identifying deep fake speech in the financial and medical fields is solved, and the identification accuracy is improved.
Patent Information
- Application Number
- CN202510714402.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-29
- Publication Date
- 2025-09-05
AI Technical Summary
Deep fake voice technology has led to identity impersonation, the spread of false information, and security threats in the financial and medical fields. Existing voice authentication systems find it difficult to accurately distinguish genuine and fake voice information.
The semantic features and speech features of the initial speech information are extracted through the preset semantic feature model and speech feature model, and speech fusion features are generated, which are then combined with the preset speech identification decision model for identification.
The system improves the accuracy of voice authentication in different languages, speaking styles and background environments, and is suitable for the fields of financial technology, healthcare and elderly care.
Smart Images
Figure CN120600030A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of speech semantics technology, and in particular to a multi-feature speech identification method, apparatus, device and medium based on a large model. Background Art
[0002] With the rapid development of technology, the application of deepfake voice technology in the financial and medical fields has brought unprecedented challenges. Deepfake voices—fake voices generated by tampering with, splicing, or completely synthesizing real voices—have been widely used for identity impersonation, dissemination of false information, financial fraud, and other illegal activities, posing a serious threat to social security and stability. The impact of deepfake voices is particularly profound in finance and healthcare, two fields that involve sensitive personal information and critical decision-making.
[0003] In the financial sector, attackers can use deepfake voices to mimic someone else's voice, gain their trust, and then conduct financial operations such as transfers and loans, resulting in financial losses for individuals and businesses. For example, they could forge voice commands from company executives to deceive financial personnel into transferring large sums of money. Many financial institutions rely on voice recognition technology for identity verification. Deepfake voices can bypass these security mechanisms, allowing attackers to illegally access account information and modify passwords, posing a serious threat to the security of the financial system.
[0004] In the medical field, attackers can forge patient voice recordings and tamper with critical medical information such as medical history and diagnosis results to obtain illegal insurance claims. This not only harms the interests of insurance institutions but also wastes medical resources. If forged voice information is used to influence doctors' diagnostic decisions, patients may receive incorrect treatment plans, delay their condition, and have serious consequences. Therefore, improving the accuracy of voice authentication systems in business fields such as fintech, healthcare, and elderly care has become a pressing technical problem. Summary of the Invention
[0005] This application provides a large-model-based multi-feature speech identification method, device, equipment and medium to improve the accuracy of speech authentication systems in identifying speech information in business fields such as financial technology, medical health and elderly care.
[0006] In a first aspect, the present application provides a multi-feature speech identification method based on a large model, the method comprising:
[0007] Extracting initial semantic features of initial speech information through a preset semantic feature model;
[0008] Extracting initial speech features of the initial speech information by using a preset speech feature model;
[0009] generating a speech fusion feature according to the initial semantic feature and the initial speech feature;
[0010] Based on a preset speech identification decision model and the speech fusion features, a speech identification result of the initial speech information is generated.
[0011] In a second aspect, the present application further provides a multi-feature speech identification device based on a large model, the device comprising:
[0012] An initial semantic feature generation module, used to extract initial semantic features of initial speech information through a preset semantic feature model;
[0013] An initial speech feature extraction module, configured to extract initial speech features of the initial speech information using a preset speech feature model;
[0014] A speech fusion feature generation module, configured to generate speech fusion features based on the initial semantic features and the initial speech features;
[0015] The speech identification result generating module is used to generate the speech identification result of the initial speech information based on a preset speech identification decision model and the speech fusion feature.
[0016] In a third aspect, the present application also provides a computer device comprising a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program and implement the multi-feature speech identification method based on a large model as described above when executing the computer program.
[0017] In a fourth aspect, the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the processor implements the multi-feature speech identification method based on a large model as described above.
[0018] The present application discloses a multi-feature speech identification method, apparatus, equipment and medium based on a large model. The multi-feature speech identification method based on a large model includes extracting the initial semantic features of the initial speech information through a preset semantic feature model; extracting the initial speech features of the initial speech information through a preset speech feature model; generating speech fusion features based on the initial semantic features and the initial speech features; and generating a speech identification result of the initial speech information based on a preset speech identification decision model and the speech fusion features. Through the above-mentioned method, the present application fuses the semantic features and the speech features to generate speech fusion features, comprehensively considering the semantic information and acoustic information of the speech. The speech identification results generated based on the preset speech identification decision model and the speech fusion features can adapt to speech data of different languages, different speaking styles and different background environments, and improve the accuracy of the speech authentication system in identifying speech information in business fields such as financial technology, medical health and elderly care. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.
[0020] Figure 1 This is a schematic flow chart of a multi-feature speech identification method based on a large model provided in the first embodiment of the present application;
[0021] Figure 2 This is a schematic flow chart of a multi-feature speech identification method based on a large model provided in the second embodiment of the present application;
[0022] Figure 3 A schematic block diagram of a multi-feature speech identification device based on a large model provided in an embodiment of the present application;
[0023] Figure 4 A schematic block diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0024] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.
[0025] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, combined, or partially merged, so the actual execution order may vary depending on the actual situation.
[0026] It should be understood that the terms used in this specification are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in this specification and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms unless the context clearly indicates otherwise.
[0027] It will also be understood that the term "and / or" used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.
[0028] The embodiments of the present application provide a large-scale model-based multi-feature speech identification method, apparatus, device, and medium. Specifically, the large-scale model-based multi-feature speech identification method can be applied to a speech authentication system, fusing semantic features with speech features to generate speech fusion features. Taking into account both the semantic and acoustic information of speech, the speech authentication results generated based on a preset speech authentication decision model and the speech fusion features can adapt to speech data in different languages, different speaking styles, and different background environments. This improves the accuracy of speech authentication systems in identifying speech information in business areas such as financial technology, healthcare, and elderly care.
[0029] The following describes some embodiments of the present application in detail with reference to the accompanying drawings. In the absence of conflict, the following embodiments and features therein may be combined with each other.
[0030] See also Figure 1 , Figure 1 This is a schematic flow chart of a large-model-based multi-feature speech identification method provided in the first embodiment of this application. This large-model-based multi-feature speech identification method can be applied to speech authentication systems in areas such as financial technology, healthcare, and elderly care, improving the accuracy of speech authentication systems in identifying speech information.
[0031] like Figure 1 As shown, the multi-feature speech identification method based on a large model specifically includes steps S10 to S40.
[0032] Step S10: extracting initial semantic features of the initial speech information through a preset semantic feature model;
[0033] Specifically, for the fintech business field, this embodiment can select a pre-trained semantic feature extraction model suitable for text analysis in the financial field, such as the BERT model (Bidirectional Encoder Representations from Transformers, pre-trained language model) fine-tuned on financial text data, transcribe the preprocessed speech data into text, use the selected model to encode the text, extract the initial semantic feature vector of each speech segment, and capture key semantic information such as financial terms and trading instructions.
[0034] In the field of medical care, health care, and elderly care, voice data such as medical consultation recordings, doctor's ward rounds, and patient medical history descriptions are collected to ensure that the data covers different medical scenarios, such as outpatient, inpatient, and rehabilitation. The voice data is transcribed into text and the transcribed text is cleaned, including removing noise words and standardizing medical terminology, to improve the quality of semantic feature extraction.
[0035] The preprocessed text is fed into a pre-set semantic feature extraction model, which encodes the text and generates a semantic feature vector output by the model. This vector captures semantic information such as key medical terms, symptom descriptions, and treatment plans. The extracted initial semantic features are analyzed to identify healthcare-related keywords and phrases, such as "hypertension," "diabetes," and "electrocardiogram." These initial semantic features are then applied to healthcare scenarios, such as auxiliary diagnosis systems, intelligent medical record analysis, and medical voice assistants.
[0036] Step S20: extracting initial speech features of the initial speech information using a preset speech feature model;
[0037] Specifically, in the healthcare and elderly care sectors, the initial speech information can be medical consultation audio recordings, including conversations such as patients describing their symptoms and doctors asking about their medical history. This ensures coverage of different symptom descriptions, different vocal emotions (such as pain and anxiety), and background environments (such as consulting rooms and wards). The preprocessed speech data is transcribed into text, encoded using the selected model, and the initial semantic feature vector of each speech segment is extracted to capture key semantic information such as symptom descriptions and treatment plans.
[0038] Select a pre-trained semantic feature extraction model suitable for medical text analysis, such as the BERT model fine-tuned on medical text data, to extract initial speech features, including pitch, timbre, and speaking rate. Pitch can be used to analyze emotional fluctuations during speech, such as pain, which can cause a higher pitch; timbre can be used to determine whether a patient has vocal cord disease; and speaking rate can be used to assess a patient's psychological state, such as anxiety, which can lead to faster speech.
[0039] Step S30: generating speech fusion features based on the initial semantic features and the initial speech features;
[0040] Specifically, a fusion network consisting of a multi-layer perceptron and an attention mechanism is constructed to learn the complex nonlinear relationship between the initial semantic features and the initial speech features, and dynamically adjust the weights of different features to highlight important feature information.
[0041] When training the fusion network, a labeled speech dataset (which can be a medical speech dataset or a financial speech dataset) is obtained in advance and used to train the feature fusion network. The labeled content includes information such as the diagnosis results corresponding to the speech and the severity of the disease. This data is used to train the fusion network, and the network parameters are adjusted through the backpropagation algorithm to minimize the error between the predicted results and the true labels. The performance indicators on the validation set, such as accuracy, recall rate, F1 value, etc., are monitored to prevent overfitting.
[0042] The time-axis aligned and normalized initial semantic features and initial speech features are input into the trained fusion network. After being processed by the network, the fused speech fusion feature vector is output.
[0043] Step S40: Generate a speech identification result of the initial speech information based on a preset speech identification decision model and the speech fusion feature.
[0044] Specifically, this embodiment is described using the medical, health and elderly care fields as an example. It should be noted that this embodiment can also be applied to other business fields, such as the financial technology business field, and is not limited here.
[0045] Collect a large amount of voice data in the medical field, including real medical consultation records, patients' voices describing their symptoms, doctors' voices during the diagnosis process, and some known forged medical voice samples (such as false medical statements used for fraud). These voice data are labeled to record their authenticity (real or forged), key medical information in the voice (such as disease type, treatment plan, etc.) and corresponding patient background information (such as age, gender, etc.).
[0046] For each annotated speech sample, the initial semantic features and initial speech features are extracted according to the method described above, and speech fusion features are generated. This will serve as the input features for model training.
[0047] The speech data to be identified undergoes preprocessing, including voice activity detection, noise removal, and audio normalization. Initial semantic and initial speech features are then extracted, and speech fusion features are generated using the same methodology, ensuring their dimensions and format are consistent with those used to train the model. These fusion features are then fed into the pre-set speech identification decision model. The model outputs a speech identification result based on the association of the patterns and features learned during training with real or fabricated speech.
[0048] This embodiment discloses a multi-feature speech identification method based on a large model, the multi-feature speech identification method based on a large model includes extracting initial semantic features of initial speech information through a preset semantic feature model; extracting initial speech features of the initial speech information through a preset speech feature model; generating speech fusion features based on the initial semantic features and the initial speech features; and generating speech identification results of the initial speech information based on a preset speech identification decision model and the speech fusion features. Through the above-mentioned method, the present application fuses semantic features and speech features to generate speech fusion features, comprehensively considering the semantic information and acoustic information of the speech. The speech identification results generated based on the preset speech identification decision model and the speech fusion features can adapt to speech data of different languages, different speaking styles, and different background environments, and improve the accuracy of speech authentication systems in identifying speech information in business fields such as financial technology, medical care, and elderly care.
[0049] See also Figure 2 , Figure 2 This is a schematic flow chart of a large-model-based multi-feature speech identification method provided in the second embodiment of the present application. This large-model-based multi-feature speech identification method can be applied to a speech authentication system to extract accurate frame-level time-frequency features, frame-level pitch features, and frame-level timbre features through WavLM. ResNet deeply characterizes frame-level features, mines deep-level information, and solves the problem of inconsistent dimensions and distributions between different features, making feature fusion more accurate. In business fields such as financial technology, medical health, and elderly care, this method improves the accuracy of speech authentication systems in identifying speech information.
[0050] based on Figure 1 The embodiment shown, this embodiment Figure 2 As shown, step S20 includes steps S201 to S204.
[0051] Step S201: generating an initial speech waveform according to the initial speech information;
[0052] In a specific embodiment, the collected speech data is converted into an initial speech waveform using an audio processing tool or programming library. A speech waveform is a representation of the amplitude variation of a speech signal on a time axis.
[0053] Step S202: performing frame-level feature extraction processing on the initial speech waveform using the WavLM to generate frame-level time-frequency features, frame-level pitch features, and frame-level timbre features;
[0054] Specifically, a WavLM model, trained with a large amount of speech data and capable of extracting frame-level features of speech, is loaded. The preprocessed speech waveform is fed into the WavLM model, which then outputs the time-frequency, pitch, and timbre features of each frame. These features reflect the frequency variations, pitch, and timbre of the speech signal at each time point.
[0055] Step S203: performing deep characterization on the frame-level time-frequency features, the frame-level pitch features, and the frame-level timbre features through a residual network (ResNet), so as to map the frame-level time-frequency features, the frame-level pitch features, and the frame-level timbre features to the same feature space;
[0056] Specifically, a deep learning framework is used to build a ResNet model. The ResNet model contains multiple residual blocks. Each residual block consists of a convolutional layer, a batch normalization layer, and a ReLU activation function. The appropriate network depth and convolution kernel size are set to adapt to the frame-level feature dimension of the input.
[0057] The frame-level time-frequency features, pitch features, and timbre features extracted by WavLM are fed into the constructed ResNet model as input. The ResNet model uses its residual connections and convolution operations to deeply represent the input features, capture the complex relationships and hierarchical structures between features, and map different types of features into the same feature space, making the features comparable and fusible.
[0058] Step S204: Fusing the frame-level time-frequency features, the frame-level pitch features, and the frame-level timbre features in the same feature space to generate the initial speech features.
[0059] Specifically, the time-frequency features, pitch features, and timbre features processed by ResNet are aligned on the time axis. If the lengths of the feature sequences are inconsistent, they can be adjusted by interpolation or truncation.
[0060] In the same feature space, the aligned frame-level time-frequency features, pitch features, and timbre features are fused. For example, a weighted summation method can be used to perform weighted fusion of the three features based on the weights obtained from the actual application scenario or model training.
[0061] The fused feature sequence is used as the initial speech feature for subsequent speech analysis tasks, such as speech recognition, speech synthesis, or speech authentication. The initial speech feature integrates the time-frequency, pitch, and timbre information of the speech and can more comprehensively describe the characteristics of the speech signal.
[0062] This embodiment discloses a multi-feature speech identification method based on a large model, which includes generating an initial speech waveform according to the initial speech information; performing frame-level feature extraction processing on the initial speech waveform through the WavLM to generate frame-level time-frequency features, frame-level pitch features and frame-level timbre features; performing deep representation of the frame-level time-frequency features, the frame-level pitch features and the frame-level timbre features through the residual network ResNet to map the frame-level time-frequency features, the frame-level pitch features and the frame-level timbre features to the same feature space; and fusing the frame-level time-frequency features, the frame-level pitch features and the frame-level timbre features in the same feature space to generate the initial speech features. Through the above method, this application uses WavLM to extract accurate frame-level time-frequency features, frame-level pitch features and frame-level timbre features, and ResNet deeply represents the frame-level features, mines deep-level information and solves the problem of inconsistent dimensions and distributions between different features, making feature fusion more accurate. In business areas such as financial technology, medical health and elderly care, it improves the accuracy of voice authentication systems in identifying voice information.
[0063] based on Figure 2 In the embodiment shown, in this embodiment, step S30 includes:
[0064] generating an attention score matrix based on a cross-attention mechanism, the initial semantic features, and the initial speech features;
[0065] Determining a correlation coefficient between the initial semantic feature and the initial speech feature according to the attention score matrix;
[0066] The fusion weight of the initial semantic feature and the initial speech feature is determined according to the correlation coefficient, and the initial semantic feature and the initial speech feature are fused according to the fusion weight to generate the speech fusion feature.
[0067] Specifically, a cross-attention model is constructed using a deep learning framework. The cross-attention model consists of three components: query, key, and value, which represent the initial semantic features and initial speech features, respectively. The initial semantic features are used as the query, and the initial speech features are used as the key and value. The initial semantic features and initial speech features are mapped to the same hidden layer dimension through a linear transformation (fully connected layer).
[0068] Based on the normalized attention weight matrix, the correlation coefficient matrix between the initial semantic features and the initial speech features is calculated. The correlation coefficient can be calculated using the Pearson correlation coefficient formula, which is used to measure the correlation between the two features at different locations. Based on the correlation coefficient matrix, the fusion weight matrix of the initial semantic features and the initial speech features is determined. The fusion weight can be expressed as the normalized value of the correlation coefficient, ensuring that the sum of the weights is 1. The fused feature vector is used as the speech fusion feature for subsequent medical speech analysis tasks such as auxiliary diagnosis, speech authentication, or sentiment analysis. The speech fusion feature integrates semantic and acoustic information and can more comprehensively represent the characteristics of medical speech data.
[0069] In a specific embodiment, the fusion weight of the initial semantic feature and the initial speech feature is determined according to the correlation coefficient, and the initial semantic feature and the initial speech feature are fused according to the fusion weight to generate the speech fusion feature, including:
[0070] The initial semantic features and the initial speech features are aligned based on a dynamic time warping algorithm.
[0071] Specifically, a distance matrix is created to store the distances between feature points in the initial semantic feature sequence and the initial speech feature sequence. The number of rows and columns of the matrix corresponds to the lengths of the initial semantic feature sequence and the initial speech feature sequence, respectively.
[0072] For each pair of feature points in the initial semantic feature sequence and the initial speech feature sequence, the distance between them is calculated, and the distance matrix is filled according to the rules of the dynamic time warping algorithm. Starting from the lower right corner of the distance matrix, backtrack to find the path with the smallest cumulative distance. This path represents the best matching relationship between the initial semantic feature sequence and the initial speech feature sequence.
[0073] The initial semantic features and the initial speech features are aligned according to the path to obtain an aligned feature sequence, and the aligned initial semantic features and the initial speech features are fused to generate a speech fusion feature.
[0074] based on Figure 2 In the embodiment shown, in this embodiment, step S40 includes:
[0075] performing layer normalization processing on each of the speech fusion features to determine the correlation between the speech fusion features;
[0076] Based on the correlation, each of the speech fusion features is projected and transformed through the fully connected layer and the channel mixing layer of the gMLP model to generate a transformed speech fusion feature;
[0077] The speech identification result is generated by fusing features of the output layer of the gMLP with the transformed speech.
[0078] Specifically, methods such as the Pearson correlation coefficient or cosine similarity are used to measure the similarity or correlation between features. The elements of the correlation matrix represent the correlation coefficients between different speech fusion features. The values typically range from -1 to 1, with larger absolute values indicating stronger correlation. The normalized speech fusion features are input to the fully connected layer of the gMLP model for projection transformation. The number of neurons in the fully connected layer can be set based on actual needs and is typically slightly larger than the input feature dimension to capture more characteristic patterns.
[0079] The features transformed by the fully connected layer are input to the channel mixing layer. The channel mixing layer further mixes the feature information by performing linear transformation on each feature channel and summing them up, highlighting important feature channels and suppressing unimportant feature channels. It integrates different aspects of speech fusion features to form a more discriminative feature representation.
[0080] The output features of the channel mixing layer are input to the output layer of the gMLP model. The number of neurons in the output layer is determined by the specific speech identification task, and the speech identification result is generated based on the output layer's output. A threshold (such as 0.5) can be set and the output probability can be compared with the threshold to obtain the final speech identification judgment (authentic or fake).
[0081] In a specific embodiment, after step S40, the following steps are included:
[0082] Calculating a confidence score for the speech identification result, wherein the confidence score is an uncertainty estimate or an entropy of an output probability distribution of the speech identification decision model;
[0083] When the confidence score is lower than a preset confidence threshold, a repeated evaluation mechanism is triggered, and a performance evaluation of the speech identification decision model is performed based on the repeated evaluation mechanism.
[0084] Based on any of the above embodiments, in this embodiment, step S10 includes:
[0085] Extracting each initial voice keyword of the initial voice information through the preset semantic feature model;
[0086] The initial speech keyword features corresponding to the initial speech keywords are fused to generate the initial semantic features.
[0087] Specifically, for each initial speech keyword, a feature vector is extracted using a preset semantic feature model. The keyword can be converted into a vector representation of a fixed dimension using word embedding technology.
[0088] The feature vectors of all initial speech keywords are merged. If feature concatenation is used, the feature vectors are connected end to end to form a long vector. If feature weighted averaging is used, the feature vectors are weighted and summed, and the weight can be determined based on the importance of the keyword.
[0089] See also Figure 3 , Figure 3 The embodiment of the present application provides a schematic block diagram of a large-model-based multi-feature speech identification device, which is used to perform the aforementioned large-model-based multi-feature speech identification method. The large-model-based multi-feature speech identification device can be configured on a server.
[0090] like Figure 3 As shown, the multi-feature speech identification device 400 based on the large model includes:
[0091] An initial semantic feature generating module 410 is configured to extract initial semantic features of initial speech information using a preset semantic feature model;
[0092] An initial speech feature extraction module 420 is configured to extract initial speech features of the initial speech information using a preset speech feature model;
[0093] A speech fusion feature generating module 430 is configured to generate speech fusion features based on the initial semantic features and the initial speech features;
[0094] The speech identification result generating module 440 is configured to generate a speech identification result of the initial speech information based on a preset speech identification decision model and the speech fusion feature.
[0095] Furthermore, the initial speech feature extraction module 420 includes:
[0096] an initial speech waveform generating unit, configured to generate an initial speech waveform according to the initial speech information;
[0097] a frame-level feature extraction unit, configured to perform frame-level feature extraction processing on the initial speech waveform using the WavLM to generate frame-level time-frequency features, frame-level pitch features, and frame-level timbre features;
[0098] a deep representation unit, configured to perform deep representation on the frame-level time-frequency features, the frame-level pitch features, and the frame-level timbre features through a residual network (ResNet), so as to map the frame-level time-frequency features, the frame-level pitch features, and the frame-level timbre features to the same feature space;
[0099] The initial speech feature generating unit is configured to fuse the frame-level time-frequency features, the frame-level pitch features and the frame-level timbre features in the same feature space to generate the initial speech features.
[0100] Furthermore, the speech fusion feature generation module 430 includes:
[0101] an attention score matrix generating unit, configured to generate an attention score matrix based on a cross-attention mechanism, the initial semantic features, and the initial speech features;
[0102] a correlation coefficient determining unit, configured to determine a correlation coefficient between the initial semantic feature and the initial speech feature according to the attention score matrix;
[0103] The speech fusion feature generating unit is used to determine the fusion weight of the initial semantic feature and the initial speech feature according to the correlation coefficient, and fuse the initial semantic feature and the initial speech feature according to the fusion weight to generate the speech fusion feature.
[0104] Furthermore, the speech fusion feature generation module 430 includes:
[0105] A feature alignment processing unit is used to align the initial semantic features and the initial speech features based on a dynamic time warping algorithm.
[0106] Furthermore, the speech identification result generating module 440 includes:
[0107] a correlation determination unit, configured to perform layer normalization processing on each of the speech fusion features to determine the correlation between the speech fusion features;
[0108] a transformed speech fusion feature generating unit, configured to project each of the speech fusion features through the fully connected layer and the channel mixing layer of the gMLP model based on the correlation to generate a transformed speech fusion feature;
[0109] The speech identification result generating unit is used to generate the speech identification result by fusing features of the transformed speech with the output layer of the gMLP.
[0110] Furthermore, the multi-feature speech identification device 400 based on the large model further includes:
[0111] a confidence score calculation module, configured to calculate a confidence score of the speech identification result, wherein the confidence score is an uncertainty estimate of the speech identification decision model or an entropy of an output probability distribution;
[0112] The performance evaluation module is used to trigger a repeated evaluation mechanism when the confidence score is lower than a preset confidence threshold, and perform performance evaluation on the speech identification decision model based on the repeated evaluation mechanism.
[0113] Furthermore, the initial semantic feature generation module 410 includes:
[0114] an initial speech keyword extraction unit, configured to extract each initial speech keyword of the initial speech information by using the preset semantic feature model;
[0115] The initial semantic feature generating unit is used to fuse the initial voice keyword features corresponding to the initial voice keywords to generate the initial semantic feature.
[0116] It should be noted that those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0117] The above-mentioned device can be realized in the form of a computer program. The computer program can be used in Figure 4 Runs on the computer equipment shown.
[0118] See also Figure 4 , Figure 4 1 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device may be a server.
[0119] See Figure 4 The computer device includes a processor, a memory, and a network interface connected through a system bus, wherein the memory may include a non-volatile storage medium and an internal memory.
[0120] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any multi-feature speech identification method based on a large model.
[0121] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.
[0122] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any multi-feature speech identification method based on a large model.
[0123] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 4The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.
[0124] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.
[0125] In one embodiment, the processor is configured to execute a computer program stored in the memory to implement the following steps:
[0126] Extracting initial semantic features of initial speech information through a preset semantic feature model;
[0127] Extracting initial speech features of the initial speech information by using a preset speech feature model;
[0128] generating a speech fusion feature according to the initial semantic feature and the initial speech feature;
[0129] Based on a preset speech identification decision model and the speech fusion features, a speech identification result of the initial speech information is generated.
[0130] In one embodiment, the initial speech features of the initial speech information are extracted by a preset speech feature model to achieve:
[0131] generating an initial speech waveform according to the initial speech information;
[0132] Performing frame-level feature extraction processing on the initial speech waveform by the WavLM to generate frame-level time-frequency features, frame-level pitch features, and frame-level timbre features;
[0133] Performing deep characterization on the frame-level time-frequency features, the frame-level pitch features, and the frame-level timbre features through a residual network (ResNet) to map the frame-level time-frequency features, the frame-level pitch features, and the frame-level timbre features to the same feature space;
[0134] The frame-level time-frequency features, the frame-level pitch features, and the frame-level timbre features are fused in the same feature space to generate the initial speech features.
[0135] In one embodiment, a speech fusion feature is generated based on the initial semantic feature and the initial speech feature to achieve:
[0136] generating an attention score matrix based on a cross-attention mechanism, the initial semantic features, and the initial speech features;
[0137] Determining a correlation coefficient between the initial semantic feature and the initial speech feature according to the attention score matrix;
[0138] The fusion weight of the initial semantic feature and the initial speech feature is determined according to the correlation coefficient, and the initial semantic feature and the initial speech feature are fused according to the fusion weight to generate the speech fusion feature.
[0139] In one embodiment, a fusion weight of the initial semantic feature and the initial speech feature is determined according to the correlation coefficient, and the initial semantic feature and the initial speech feature are fused according to the fusion weight to generate the speech fusion feature, which is used to achieve:
[0140] The initial semantic features and the initial speech features are aligned based on a dynamic time warping algorithm.
[0141] In one embodiment, based on a preset speech identification decision model and the speech fusion features, a speech identification result of the initial speech information is generated to achieve:
[0142] performing layer normalization processing on each of the speech fusion features to determine the correlation between the speech fusion features;
[0143] Based on the correlation, each of the speech fusion features is projected and transformed through the fully connected layer and the channel mixing layer of the gMLP model to generate a transformed speech fusion feature;
[0144] The speech identification result is generated by fusing features of the output layer of the gMLP with the transformed speech.
[0145] In one embodiment, after generating the speech identification result of the initial speech information based on the preset speech identification decision model and the speech fusion feature, it is used to achieve:
[0146] Calculating a confidence score for the speech identification result, wherein the confidence score is an uncertainty estimate or an entropy of an output probability distribution of the speech identification decision model;
[0147] When the confidence score is lower than a preset confidence threshold, a repeated evaluation mechanism is triggered, and a performance evaluation of the speech identification decision model is performed based on the repeated evaluation mechanism.
[0148] In one embodiment, initial semantic features of the initial speech information are extracted using a preset semantic feature model to achieve:
[0149] Extracting each initial voice keyword of the initial voice information by using the preset semantic feature model;
[0150] The initial speech keyword features corresponding to the initial speech keywords are fused to generate the initial semantic features.
[0151] A computer-readable storage medium is also provided in an embodiment of the present application, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and the processor executes the program instructions to implement any one of the large-model-based multi-feature speech identification methods provided in the embodiments of the present application.
[0152] The computer-readable storage medium may be an internal storage unit of the computer device described in the aforementioned embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a SmartMedia Card (SMC), a Secure Digital (SD) card, a flash memory card, etc., equipped on the computer device.
[0153] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present application, and such modifications or substitutions should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.
Claims
1. A multi-feature speech identification method based on a large model, characterized in that: include: Extracting initial semantic features of initial speech information through a preset semantic feature model; Extracting initial speech features of the initial speech information by using a preset speech feature model; generating a speech fusion feature according to the initial semantic feature and the initial speech feature; Based on a preset speech identification decision model and the speech fusion features, a speech identification result of the initial speech information is generated.
2. The multi-feature speech identification method based on a large model according to claim 1, characterized in that: The preset speech feature model includes a waveform language model WavLM, and the extracting of the initial speech features of the initial speech information by using the preset speech feature model includes: generating an initial speech waveform according to the initial speech information; Performing frame-level feature extraction processing on the initial speech waveform by the WavLM to generate frame-level time-frequency features, frame-level pitch features, and frame-level timbre features; Performing deep characterization on the frame-level time-frequency features, the frame-level pitch features, and the frame-level timbre features through a residual network (ResNet) to map the frame-level time-frequency features, the frame-level pitch features, and the frame-level timbre features to the same feature space; The frame-level time-frequency features, the frame-level pitch features, and the frame-level timbre features are fused in the same feature space to generate the initial speech features.
3. The multi-feature speech identification method based on a large model according to claim 2, characterized in that: Generating speech fusion features according to the initial semantic features and the initial speech features includes: generating an attention score matrix based on a cross-attention mechanism, the initial semantic features, and the initial speech features; Determining a correlation coefficient between the initial semantic feature and the initial speech feature according to the attention score matrix; The fusion weight of the initial semantic feature and the initial speech feature is determined according to the correlation coefficient, and the initial semantic feature and the initial speech feature are fused according to the fusion weight to generate the speech fusion feature.
4. The multi-feature speech identification method based on a large model according to claim 3 is characterized in that: The step of determining a fusion weight of the initial semantic feature and the initial speech feature according to the correlation coefficient, and fusing the initial semantic feature and the initial speech feature according to the fusion weight to generate the speech fusion feature comprises: The initial semantic features and the initial speech features are aligned based on a dynamic time warping algorithm.
5. The multi-feature speech identification method based on a large model according to claim 2, characterized in that: The preset speech identification decision model includes a multi-layer perceptron model gMLP model, and the speech identification result of the initial speech information is generated based on the preset speech identification decision model and the speech fusion feature, including: performing layer normalization processing on each of the speech fusion features to determine the correlation between the speech fusion features; Based on the correlation, each of the speech fusion features is projected and transformed through the fully connected layer and the channel mixing layer of the gMLP model to generate a transformed speech fusion feature; The speech identification result is generated by fusing features of the output layer of the gMLP with the transformed speech.
6. The multi-feature speech identification method based on a large model according to claim 5 is characterized in that: After generating the speech identification result of the initial speech information based on the preset speech identification decision model and the speech fusion feature, the method includes: Calculating a confidence score for the speech identification result, wherein the confidence score is an uncertainty estimate or an entropy of an output probability distribution of the speech identification decision model; When the confidence score is lower than a preset confidence threshold, a repeated evaluation mechanism is triggered, and a performance evaluation of the speech identification decision model is performed based on the repeated evaluation mechanism.
7. The multi-feature speech identification method based on a large model according to any one of claims 1 to 6, characterized in that: The extracting of initial semantic features of the initial speech information by using a preset semantic feature model includes: Extracting each initial voice keyword of the initial voice information through the preset semantic feature model; The initial speech keyword features corresponding to the initial speech keywords are fused to generate the initial semantic features.
8. A multi-feature speech identification device based on a large model, characterized in that: include: An initial semantic feature generation module, used to extract initial semantic features of initial speech information through a preset semantic feature model; An initial speech feature extraction module, configured to extract initial speech features of the initial speech information using a preset speech feature model; A speech fusion feature generation module, configured to generate speech fusion features based on the initial semantic features and the initial speech features; The speech identification result generating module is used to generate the speech identification result of the initial speech information based on a preset speech identification decision model and the speech fusion feature.
9. A computer device, characterized in that: The computer device includes a memory and a processor; The memory is used to store computer programs; The processor is configured to execute the computer program and implement the large model-based multi-feature speech identification method according to any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, enables the processor to implement the large model-based multi-feature speech identification method according to any one of claims 1 to 7.