A large model-based fraud speech recognition method and device, storage medium and electronic equipment

Abstract

The application discloses a fraud speech recognition method and device based on a large model, a storage medium and an electronic device. The method comprises the following steps: constructing a first data set; separating the roles of a call voice based on an open source library to generate separated voice; extracting a physiological feature vector of the separated voice according to an optimized mel-frequency cepstrum coefficient algorithm; weighting and fusing the extracted physiological feature vector, a semantic feature vector and a scene risk feature vector to generate a fixed-dimension multi-modal feature vector; inputting the multi-modal feature vector into a first large model for fine-tuning training to generate a fraud speech recognition model; obtaining a to-be-recognized voice, inputting a multi-modal feature vector generated based on the to-be-recognized voice into the fraud speech recognition model to output a fraud probability and a result label of the to-be-recognized voice. The application solves the problem that the existing fraud speech recognition technology cannot simultaneously achieve high recognition accuracy and fast recognition efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, in particular to a fraud speech recognition method and device based on a large model, a storage medium and an electronic device. BACKGROUND

[0002] Current network frauds present characteristics such as professionalization of speech tactics and concealment of means. As the core carrier of fraud implementation, speech recognition technology faces various difficulties. For example, existing technologies rely on semantic information after speech-to-text, ignoring acoustic features such as sudden changes in speech rate, emotional fluctuations and abnormal intonation of fraudsters, resulting in high misjudgment rate in scenarios where speech tactics are similar but intentions are different or intentions are fraudulent but speech tactics are ambiguous. Moreover, relying on a single text or voiceprint feature makes it difficult to resist new fraud means such as voice cloning and speech rate disguise, resulting in low recognition accuracy of fraud speech. In addition, existing technologies use a unified inference process for the speech to be recognized, resulting in slow recognition efficiency of fraud speech. That is, existing fraud speech recognition technologies have difficulty in balancing high recognition accuracy and fast recognition efficiency.

[0003] Currently, there is no effective solution to the above problems. SUMMARY

[0004] The embodiments of the present application provide a fraud speech recognition method and device based on a large model, a storage medium and an electronic device to at least solve the problem that existing fraud speech recognition technologies have difficulty in balancing high recognition accuracy and fast recognition efficiency.

[0005] According to an aspect of the embodiments of the present application, a fraud speech recognition method based on a large model is provided, which comprises: constructing a first data set containing call speech, scene data, result label and fraud probability; extracting a scene risk feature vector of the scene data by fine-tuning a trained gradient boosting decision tree model; separating the call speech based on an open source library to generate separated speech; extracting frame-level features of the separated speech according to an optimized mel-frequency cepstral coefficient algorithm, and inputting the frame-level features into a long short-term memory network for time series modeling to output a fixed-dimension physiological feature vector; converting the separated speech into separated text, extracting deep semantic features and attention weight features of the separated text based on a pre-trained language model, extracting static fraud features in the deep semantic features according to an intent recognition model, extracting dynamic interaction features in the attention weight features according to an interaction analysis model, and splicing and normalizing the static fraud features and the dynamic interaction features to generate a fixed-dimension semantic feature vector; weighting and fusing the physiological feature vector, the semantic feature vector and the scene risk feature vector to generate a fixed-dimension multi-modal feature vector; inputting the multi-modal feature vector into a first large model for fine-tuning training to generate a fraud speech recognition model; obtaining a to-be-recognized speech, inputting the multi-modal feature vector generated based on the to-be-recognized speech into the fraud speech recognition model to output the fraud probability and the result label of the to-be-recognized speech; wherein the scene data is used to represent background information of the call speech, the result label is represented as fraud or normal, the intent recognition model is a fine-tuned classification neural network, the interaction analysis model has a pre-designed calculation rule, and the gradient boosting decision tree model is fine-tuned based on the scene data and the result label.

[0006] Optionally, in the embodiments of the present application, the constructing a first data set containing call speech, scene data, result label and fraud probability comprises: collecting a second data set containing the call speech, the scene data and the result label; inputting the second data set and a preset prompt word into a second large model; the second large model outputs the fraud probability of each call speech in the second data set; in the case that the result label is fraud and the fraud probability is lower than 50%, correcting the result label or the fraud probability based on the audit result of the call speech by artificial; in the case that the result label is normal and the fraud probability is higher than 50%, correcting the result label or the fraud probability based on the audit result of the call speech by artificial; determining the call speech, the scene data, the result label and the fraud probability as the first data set, wherein the preset prompt word is used to prompt the second large model to output the fraud probability of the call speech.

[0007] Optionally, in the embodiment of the present application, the above extracting the frame-level feature of the separated speech according to the optimized mel-frequency cepstral coefficient algorithm and inputting the frame-level feature into the long short-term memory network for time sequence modeling to output the physiological feature vector of a fixed dimension comprises: in the preprocessing stage of the mel-frequency cepstral coefficient algorithm, using a hybrid denoising model combining spectral subtraction and wavelet threshold denoising to perform adaptive noise suppression on the separated speech, and dynamically adjusting the pre-emphasis coefficient according to the fundamental frequency of the separated speech, wherein the adaptive noise suppression dynamically adjusts the denoising strength of the spectral subtraction and the wavelet threshold denoising according to the real-time signal-to-noise ratio of the separated speech, and the fundamental frequency is negatively correlated with the pre-emphasis coefficient; in the filtering and cepstral calculation stage of the mel-frequency cepstral coefficient algorithm, the frequency range of the mel filter bank is divided into a physiological feature sensitive area and a semantic feature area, the number of mel filters is set densely in the physiological feature sensitive area, the number of mel filters is set sparsely in the semantic feature area, and the cepstral coefficients related to physiological features are increased to expand the cepstral feature vector, wherein the increased cepstral coefficients include the fundamental frequency and its first-order difference coefficient, the spectral flatness coefficient, and the formant frequency coefficient; in the feature post-processing stage of the mel-frequency cepstral coefficient algorithm, the cepstral feature vector is dynamically weighted according to the speech activity to obtain the frame-level feature, and the frame-level feature is input into the long short-term memory network for time sequence modeling to output the physiological feature vector of a fixed dimension.

[0008] Optionally, in the embodiment of the present application, the above inputting the multi-modal feature vector into the first large model for fine-tuning training to generate a fraud speech recognition model comprises: obtaining the first large model; performing structural pruning on the decoder of the first large model using a layer sensitivity analysis algorithm; performing INT8 quantization perception training on the pruned first large model based on the multi-modal feature vector; updating the parameters of the attention layer and the output layer of the first large model according to the mixed fine-tuning strategy of LoRA and Adapter during training; and determining the generated model after fine-tuning training as the fraud speech recognition model.

[0009] Optionally, in the embodiment of the present application, before the above inputting the multi-modal feature vector into the first large model for fine-tuning training to generate a fraud speech recognition model, it further comprises: vectorizing and storing the sample data of the first data set with the result label as fraud into a cache database, wherein the sample data comprises the hash value of the call speech, the multi-modal feature vector, the result label and the fraud probability.

[0010] Optionally, in the embodiment of the present application, the above obtaining the to-be-recognized speech, inputting the above multi-modal feature vector generated based on the above to-be-recognized speech into the fraud speech recognition model to output the above fraud probability and the above result label of the to-be-recognized speech comprises: obtaining the to-be-recognized speech; calculating the hash value of the to-be-recognized speech; in the case that the hash value of the to-be-recognized speech is queried in the cache database, outputting the fraud probability and the result label corresponding to the hash value of the to-be-recognized speech in the cache database as the recognition result of the to-be-recognized speech; in the case that the hash value of the to-be-recognized speech is not queried in the cache database, obtaining the multi-modal feature vector of the to-be-recognized speech, retrieving Top-K historical multi-modal feature vectors similar to the multi-modal feature vector of the to-be-recognized speech from the cache database based on FAISS; in the case that there is a historical multi-modal feature vector in the Top-K historical multi-modal feature vectors, the similarity of which to the multi-modal feature vector of the to-be-recognized speech exceeds a first preset threshold, outputting the fraud probability and the result label corresponding to the historical multi-modal feature vector with the highest similarity to the multi-modal feature vector of the to-be-recognized speech in the cache database as the recognition result of the to-be-recognized speech, and storing the hash value, the multi-modal feature vector, the result label and the fraud probability of the to-be-recognized speech in the cache database, wherein the historical multi-modal feature vector refers to the multi-modal feature vector stored in the cache database; in the case that there is no historical multi-modal feature vector in the Top-K historical multi-modal feature vectors, the similarity of which to the multi-modal feature vector of the to-be-recognized speech exceeds the first preset threshold, inputting the multi-modal feature vector of the to-be-recognized speech into the fraud speech recognition model to output the fraud probability and the result label of the to-be-recognized speech, and storing the hash value, the multi-modal feature vector, the result label and the fraud probability of the to-be-recognized speech in the cache database in the case that the result label is fraud.

[0011] According to another aspect of the embodiments of the present application, a fraud speech recognition device based on a large model is also provided, comprising: a construction unit configured to construct a first data set containing call speech, scene data, result label, and fraud probability; a first extraction unit configured to extract a scene risk feature vector of the scene data by fine-tuning a trained gradient boosting decision tree model; a separation unit configured to separate roles based on an open source library to generate separated speech; a second extraction unit configured to extract frame-level features of the separated speech according to an optimized mel-frequency cepstral coefficient algorithm, and input the frame-level features into a long short-term memory network for time series modeling to output a fixed-dimension physiological feature vector; a third extraction unit configured to convert the separated speech into separated text, extract deep semantic features and attention weight features of the separated text based on a pre-trained language model, extract static fraud features in the deep semantic features according to an intent recognition model, extract dynamic interaction features in the attention weight features according to an interaction analysis model, and splice and normalize the static fraud features and the dynamic interaction features to generate a fixed-dimension semantic feature vector; a fusion unit configured to weight and fuse the physiological feature vector, the semantic feature vector, and the scene risk feature vector to generate a fixed-dimension multi-modal feature vector; a fine-tuning unit configured to input the multi-modal feature vector into a first large model for fine-tuning training to generate a fraud speech recognition model; and an acquisition unit configured to acquire a to-be-recognized speech, input the multi-modal feature vector generated based on the to-be-recognized speech into the fraud speech recognition model to output the fraud probability and the result label of the to-be-recognized speech; wherein the scene data is used to represent background information of the call speech, the result label is represented as fraud or normal, the intent recognition model is a fine-tuned classification neural network, the interaction analysis model has a pre-designed calculation rule, and the gradient boosting decision tree model is fine-tuned based on the scene data and the result label.

[0012] Optionally, in the embodiment of the present application, the construction unit comprises: a collection module, configured to collect a second data set comprising the call voice, the scene data and the result label; an input module, configured to input the second data set and a preset prompt word into the second large model; a first output module, configured to output, by the second large model, the fraud probability of each call voice in the second data set; a first correction module, configured to correct the result label or the fraud probability based on an artificial review result of the call voice in a case where the result label is the fraud and the fraud probability is lower than 50%; a second correction module, configured to correct the result label or the fraud probability based on an artificial review result of the call voice in a case where the result label is the normal and the fraud probability is higher than 50%; and a first determination module, configured to determine the call voice, the scene data, the result label and the fraud probability as the first data set, wherein the preset prompt word is used to prompt the second large model to output the fraud probability of the call voice.

[0013] Optionally, in the embodiment of the present application, the second extraction unit comprises: a preprocessing module, configured to, in a preprocessing stage of the mel-frequency cepstral coefficient algorithm, perform adaptive noise suppression on the separated voice by using a hybrid noise reduction model combining spectral subtraction and wavelet threshold denoising, and dynamically adjust a pre-emphasis coefficient according to a fundamental frequency of the separated voice, wherein the adaptive noise suppression refers to dynamically adjusting noise reduction intensity of the spectral subtraction and the wavelet threshold denoising according to a real-time signal-to-noise ratio of the separated voice, and the fundamental frequency is negatively correlated with the pre-emphasis coefficient; a first calculation module, configured to, in a filtering and cepstral calculation stage of the mel-frequency cepstral coefficient algorithm, divide a frequency range of a mel filter bank into a physiological feature sensitive area and a semantic feature area, densely set a number of mel filters in the physiological feature sensitive area, sparsely set a number of mel filters in the semantic feature area, and increase cepstral coefficients related to physiological features, so as to expand a cepstral feature vector, wherein the increased cepstral coefficients include a fundamental frequency and a first-order difference coefficient thereof, a spectral flatness coefficient and a formant frequency coefficient; and a post-processing module, configured to, in a feature post-processing stage of the mel-frequency cepstral coefficient algorithm, dynamically weight the cepstral feature vector according to voice activity to obtain a frame-level feature, and input the frame-level feature into the long short-term memory network for time series modeling, so as to output the physiological feature vector of a fixed dimension.

[0014] Optionally, in the embodiments of the present application, the fine-tuning unit comprises: a first acquisition module configured to acquire the first large model; a pruning module configured to perform structural pruning on the decoder of the first large model using a layer sensitivity analysis algorithm; a training module configured to perform INT8 quantization-aware training on the pruned first large model based on the multi-modal feature vector; an updating module configured to update the parameters of the attention layer and the output layer of the first large model according to the mixed fine-tuning strategy of LoRA and Adapter during the training process; and a second determination module configured to determine the model generated after the fine-tuning training is completed as the fraud speech recognition model. Optionally, in the embodiments of the present application, the device further comprises a storage unit configured to store the sample data whose result label in the first data set is fraudulent into a cache database in a vectorized manner, wherein the sample data comprises the hash value of the call voice, the multi-modal feature vector, the result label and the fraud probability.

[0015] Optionally, in the embodiments of the present application, the obtaining unit comprises: a second obtaining module, configured to obtain the to-be-recognized speech; a second calculating module, configured to calculate the hash value of the to-be-recognized speech; a second output module, configured to, in the case where the hash value of the to-be-recognized speech is queried in the cache database, output the fraud probability and the result label corresponding to the hash value of the to-be-recognized speech in the cache database as the recognition result of the to-be-recognized speech; a third obtaining module, configured to, in the case where the hash value of the to-be-recognized speech is not queried in the cache database, obtain the multi-modal feature vector of the to-be-recognized speech, and retrieve Top-K historical multi-modal feature vectors similar to the multi-modal feature vector of the to-be-recognized speech from the cache database based on the FAISS; a third output module, configured to, in the case where the similarity between the multi-modal feature vector of the to-be-recognized speech and any of the Top-K historical multi-modal feature vectors exceeds a first preset threshold, output the fraud probability and the result label corresponding to the historical multi-modal feature vector with the highest similarity to the multi-modal feature vector of the to-be-recognized speech in the cache database as the recognition result of the to-be-recognized speech, and store the hash value, the multi-modal feature vector, the result label and the fraud probability of the to-be-recognized speech in the cache database, wherein the historical multi-modal feature vector refers to the multi-modal feature vector stored in the cache database; and a fourth output module, configured to, in the case where the similarity between the multi-modal feature vector of the to-be-recognized speech and any of the Top-K historical multi-modal feature vectors does not exceed the first preset threshold, input the multi-modal feature vector of the to-be-recognized speech into the fraud speech recognition model to output the fraud probability and the result label of the to-be-recognized speech, and in the case where the result label is fraud, store the hash value, the multi-modal feature vector, the result label and the fraud probability of the to-be-recognized speech in the cache database.

[0016] According to still another aspect of the embodiments of the present application, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program. The computer program is configured to execute the fraud speech recognition method based on a large model provided by any of the preceding embodiments when running.

[0017] According to still another aspect of the embodiments of the present application, an electronic device is provided, and the electronic device comprises a memory and a processor. The memory stores a computer program, and the processor is configured to execute the fraud speech recognition method based on a large model provided by any of the preceding embodiments through the computer program.

[0018] In the embodiment of the present application, the physiological feature vector, the semantic feature vector and the scene risk feature vector are weighted and fused to generate a multi-modal feature vector, and the first large model is fine-tuned based on the multi-modal feature vector, wherein the physiological feature vector is generated based on the optimized mel frequency cepstral coefficient algorithm, the semantic feature vector is generated based on the pre-trained language model, the intent recognition model and the interaction analysis model, and the scene risk feature vector is generated based on the gradient boosting decision tree model after fine-tuning. That is, the fraud voice recognition model is generated based on the fine-tuning of the large model in multiple dimensions, thereby improving the recognition accuracy of the fraud voice. BRIEF DESCRIPTION OF DRAWINGS

[0019] The drawings described herein are used to provide further understanding of the present application, and form a part of the present application. The illustrative embodiments of the present application and their descriptions serve to explain the present application, and do not constitute an improper limitation on the present application. In the drawings:

[0020] Figure 1 is a schematic diagram of an optional fraud voice recognition method based on a large model according to an embodiment of the present application;

[0021] Figure 2 is a schematic diagram of another optional fraud voice recognition method based on a large model according to an embodiment of the present application;

[0022] Figure 3 is a schematic diagram of yet another optional fraud voice recognition method based on a large model according to an embodiment of the present application;

[0023] Figure 4 is a schematic diagram of yet another optional fraud voice recognition method based on a large model according to an embodiment of the present application;

[0024] Figure 5 is a schematic diagram of yet another optional fraud voice recognition method based on a large model according to an embodiment of the present application;

[0025] Figure 6 is a schematic diagram of an optional fraud voice recognition device based on a large model according to an embodiment of the present application;

[0026] Figure 7 is a schematic diagram of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0027] In order for those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should fall within the scope of protection of the present application.

[0028] It should be noted that the terms "first", "second", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to only those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0029] Optionally, as an optional embodiment, as shown in Figure 1 The above fraud voice recognition method based on a large model includes:

[0030] S102, constructing a first data set containing call voice, scene data, result label and fraud probability;

[0031] S104-1, extracting a scene risk feature vector of the scene data by fine-tuning the trained gradient boosting decision tree model;

[0032] S104-2, performing role separation on the call voice based on an open source library to generate separated voice;

[0033] S106-1, extracting frame-level features of the separated voice according to an optimized mel-frequency cepstral coefficient algorithm, and inputting the frame-level features into a long short-term memory network for time series modeling to output a fixed-dimension physiological feature vector;

[0034] S106-2, converting the separated voice into separated text, extracting deep semantic features and attention weight features of the separated text based on a pre-trained language model, extracting static fraud features in the deep semantic features according to an intent recognition model, extracting dynamic interaction features in the attention weight features according to an interaction analysis model, and splicing and normalizing the static fraud features and the dynamic interaction features to generate a fixed-dimension semantic feature vector;

[0035] S108, the physiological feature vector, the semantic feature vector and the scene risk feature vector are weighted and fused to generate a fixed-dimension multi-modal feature vector;

[0036] S110, inputting the multi-modal feature vector into the first large model for fine-tuning training to generate a fraud speech recognition model;

[0037] S112, obtaining the to-be-recognized speech, inputting the multi-modal feature vector generated based on the to-be-recognized speech into the fraud speech recognition model to output a fraud probability and a result label of the to-be-recognized speech.

[0038] Optionally, in the embodiment of the present application, the fraud speech recognition method based on the large model can be applied to the field of natural language processing. The call speech can be real-time speech of a real person, pre-recorded speech or AI synthesized speech. The scene data is background information irrelevant to the call content itself but reflecting the risk attribute of the call, which is an important auxiliary basis for judging fraud. The scene data includes but is not limited to the risk level of the number home location, the call time period, the enterprise outbound call or the private call, the age and occupation label of the called person. The risk level of the number home location: based on historical data to label the risk value of the number home location.

[0039] Optionally, in the embodiment of the present application, the open source library can be Pyannote.audio to realize role separation. The physiological feature vector includes but is not limited to the speaker vocal cord vibration frequency and airflow intensity, which is not affected by the change of speech speed and intonation, effectively resisting voice disguise. The separated speech is converted into separated text through the Whisper-large-v3 model, or the separated speech is converted into separated text through the optimized Whisper-large-v3 model. The optimization method is to perform lightweight modification on the Whisper-large-v3 model through knowledge distillation and structured pruning, while retaining the recognition ability of dialect words and tone words, thereby improving the recognition efficiency of fraud speech.

[0040] Optionally, in the embodiments of the present application, the deep semantic features and attention weight features of the separated text can be extracted by the pre-trained language model BERT-WWM-Ext model. The intent recognition model is generated based on the fine-tuning training of a classification neural network, and the static fraud features include identity impersonation (whether impersonating an official, customer service, familiar person, or other authoritative role), information request (whether requesting verification code, password, bank card number, or other highly sensitive information), behavior induction (whether requiring a transfer, screen sharing, clicking on an unknown link, etc.), logic loophole (whether the call content has unreasonable or contradictory places), etc. The dynamic interaction features include right inequality (whether one party continuously uses command, intimidation, or authoritative language to suppress the other party's questions or doubts), induced questioning (whether through leading questions to make the other party unknowingly reveal information or agree to the operation), emotional manipulation (whether the conversation leads to obvious emotional changes such as anxiety, fear, or excessive trust in one party), rhythm control (whether one party interrupts the normal thinking of the other party by creating a sense of urgency such as "limited time" or "frozen immediately"), etc.

[0041] Optionally, in the embodiments of the present application, the gradient boosting decision tree can be a LightGBM model. By using the historical fraud / normal result labels, the LightGBM model learns the correlation between the scene data and the result labels, and finally outputs the risk contribution weight of each scene data dimension. For example, a first data set containing multiple data is established, each data contains a call voice, its corresponding scene data, result label and fraud probability. The gradient boosting decision tree model is fine-tuned to extract the scene data 16 as a scene risk feature vector. The voices of different speakers in the call voice are separated by the open source library Pyannote.audio to obtain separated voices. The frame-level features of the separated voices are extracted according to the optimized mel-frequency cepstral coefficient algorithm, and the frame-level features are input into the LSTM network for time series modeling, thereby outputting a 128-dimensional physiological feature vector. The separated voices are converted into separated texts by the Whisper-large-v3 model, and the deep semantic features and attention weight features of the separated texts are extracted based on the BERT-WWM-Ext model. The static fraud features in the deep semantic features are extracted by the intent recognition model, and the dynamic interaction features in the attention weight features are extracted by the interaction analysis model. The static fraud features and the dynamic interaction features are concatenated and normalized to generate a 48-dimensional semantic feature vector. The physiological feature vector, the semantic feature vector and the scene risk feature vector are weighted and fused according to the self-defined weights, thereby generating a 192-dimensional multi-modal feature vector. The 192-dimensional multi-modal feature vector is input into the first large model, and the first large model is fine-tuned according to the result label and the fraud probability corresponding to the 192-dimensional multi-modal feature vector. When the training termination condition is reached, the parameter updating is stopped, and the trained first large model is determined as the fraud voice recognition model. The to-be-recognized voice is obtained, and the multi-modal feature vector generated based on the to-be-recognized voice is input into the fraud voice recognition model to output the fraud probability and the result label of the to-be-recognized voice, thereby realizing the recognition of the fraud voice.

[0042] According to the embodiments of the present application, the physiological feature vector, the semantic feature vector and the scene risk feature vector are weighted and fused to generate a multi-modal feature vector, and the first large model is fine-tuned based on the multi-modal feature vector, wherein the physiological feature vector is generated based on the optimized mel-frequency cepstral coefficient algorithm, the semantic feature vector is generated based on the pre-trained language model, the intent recognition model and the interaction analysis model, and the scene risk feature vector is generated based on the fine-tuned gradient boosting decision tree model. That is, the fraud voice recognition model is generated based on the fine-tuning of the large model in multiple dimensions, thereby improving the recognition accuracy of the fraud voice.

[0043] Based on one or each of the above embodiments, the present application is further improved, such as Figure 2As shown, constructing a first data set containing call voice, scene data, result label, and fraud probability includes:

[0044] S202, collecting a second data set containing call voice, scene data, and result label;

[0045] S204, inputting the second data set and the preset prompt word into the second large model;

[0046] S206, the second large model outputs the fraud probability of each call voice in the second data set;

[0047] S208-1, in the case that the result label is fraud and the fraud probability is lower than 50%, correcting the result label or the fraud probability based on the manual review result of the call voice;

[0048] S208-2, in the case that the result label is normal and the fraud probability is higher than 50%, correcting the result label or the fraud probability based on the manual review result of the call voice;

[0049] S210, determining the call voice, scene data, result label, and fraud probability as the first data set, wherein the preset prompt word is used to prompt the second large model to output the fraud probability of the call voice.

[0050] Optionally, in the embodiment of the present application, the API is called to call the large language model to output the fraud probability of each call voice in the second data set.

[0051] On the basis of one or each of the above embodiments, the present application is further improved, such as Figure 3 As shown, the frame-level features of the separated voice are extracted according to the optimized mel-frequency cepstral coefficient algorithm, and the frame-level features are input into the long short-term memory network for time series modeling to output a fixed-dimension physiological feature vector including:

[0052] S302, in the preprocessing stage of the mel-frequency cepstral coefficient algorithm, a hybrid denoising model combining spectral subtraction and wavelet threshold denoising is used for adaptive noise suppression of the separated voice, and the pre-emphasis coefficient is dynamically adjusted according to the fundamental frequency of the separated voice;

[0053] S304, in the filtering and cepstral calculation stage of the mel-frequency cepstral coefficient algorithm, the frequency range of the mel filter bank is divided into a physiological feature sensitive area and a semantic feature area, the number of mel filters is set in the physiological feature sensitive area, the number of mel filters is set in the semantic feature area, and the cepstral coefficient related to the physiological feature is increased to expand the cepstral feature vector;

[0054] S306, in the feature post-processing stage of the mel-frequency cepstrum coefficient algorithm, the cepstrum feature vector is dynamically weighted according to the speech activity to obtain a frame-level feature, and the frame-level feature is input into a long short-term memory network for time sequence modeling to output a physiological feature vector with a fixed dimension.

[0055] Based on one or each of the above embodiments, the present application is further improved, for example Figure 4 As shown, the multi-modal feature vector is input into the first large model for fine-tuning training to generate a fraud speech recognition model, which includes:

[0056] S402, acquiring a first large model; S404, using a layer sensitivity analysis algorithm to perform structural pruning on the decoder of the first large model; S406-1, performing INT8 quantization perception training on the pruned first large model based on the multi-modal feature vector; S406-2, during the training process, updating the parameters of the attention layer and the output layer of the first large model according to the mixed fine-tuning strategy of LoRA and Adapter;

[0057] S408, determining the model generated after the fine-tuning training as the fraud speech recognition model.

[0058] Based on one or each of the above embodiments, the present application is further improved, for example Figure 5 As shown, the multi-modal feature vector generated based on the to-be-recognized speech is input into the fraud speech recognition model to output the fraud probability and the result label of the to-be-recognized speech, which includes:

[0059] S502, acquiring a to-be-recognized speech;

[0060] S504, calculating a hash value of the to-be-recognized speech;

[0061] S506-1, in the case that the hash value of the to-be-recognized speech is queried in the cache database, outputting the fraud probability and the result label corresponding to the hash value of the to-be-recognized speech in the cache database as the recognition result of the to-be-recognized speech;

[0062] S506-2, in the case that the hash value of the to-be-recognized speech is not queried in the cache database, acquiring a multi-modal feature vector of the to-be-recognized speech, and retrieving Top-K historical multi-modal feature vectors similar to the multi-modal feature vector of the to-be-recognized speech from the cache database based on FAISS;

[0063] S508-1, in a case where the similarity between the multi-modal feature vector of the to-be-identified speech and the multi-modal feature vectors in the Top-K historical multi-modal feature vectors exceeds a first preset threshold, outputting the fraud probability and the result label corresponding to the historical multi-modal feature vector with the highest similarity to the multi-modal feature vector of the to-be-identified speech in the cache database as the recognition result of the to-be-identified speech, and storing the hash value, the multi-modal feature vector, the result label and the fraud probability of the to-be-identified speech into the cache database;

[0064] S508-2, in a case where the similarity between the multi-modal feature vector of the to-be-identified speech and the multi-modal feature vectors in the Top-K historical multi-modal feature vectors does not exceed the first preset threshold, inputting the multi-modal feature vector of the to-be-identified speech into the fraud speech recognition model to output the fraud probability and the result label of the to-be-identified speech, and storing the hash value, the multi-modal feature vector, the result label and the fraud probability of the to-be-identified speech into the cache database in a case where the result label is fraud.

[0065] Optionally, as an optional implementation, the above method is exemplarily and wholly explained as follows:

[0066] Step 1: Data set construction

[0067] 3000 positive samples (fraudulent speech) and 3000 negative samples (normal speech) are collected as a second data set, and each sample contains a call recording (5 minutes), scene data (such as number home risk level, call period, enterprise outbound call or private call, called person age and occupation label, etc.), and a result label (fraud / normal).

[0068] According to different industries, some characteristics of the relevant industry or some general characteristics of all frauds are collected, and special prompt words are designed based on these characteristics, such as:

[0069] You are an anti-fraud specialist in the {XX industry} field. Please evaluate the following call content based on your understanding of the common fraud methods in the {XX industry} field:

[0070]

Industry fraud feature reminder

[0071] In the {XX industry}, the fraud-related language that needs to be highly vigilant may include (but is not limited to): [Here, 1-3 most typical fraud features of this industry need to be filled in manually, for example: impersonating officials to ask for verification codes, asking for screen sharing under the pretext of refund, false investment and financial promise of high return, etc.];

[0072]

Evaluation task

[0073] Please carefully analyze the call text below and judge its degree of agreement with the above fraud features;

[0074] Call content: {recording content}

[0075]

Output requirements

[0076] 1. Probability value: Give a 1-100 integer of the fraud probability;

[0077] 2. Evaluation summary: Briefly describe your core judgment basis in 1-2 sentences;

[0078] 3. Fraud feature matching item: Clearly indicate which of the above "industry fraud features" in the call content is matched (if no match, write "none").

[0079] The second data set and the designed special prompt word are input into the second large language model to determine whether the fraud probability output by the second large language model corresponds to the result label: if the result label is fraud but the fraud probability output by the second large language model is less than 50%, then the call audio is manually reviewed, and the result label or fraud probability is corrected according to the review result, such as manually reviewing the call audio as fraud, but the fraud performance is secret, then the fraud probability can be corrected to a value between 50%-70%, the fraud performance is obvious, then the fraud probability can be corrected to a value between 80%-100%, determined according to the actual manual review; If the result label is normal but the fraud probability output by the second large language model is greater than 50%, then the call audio is manually reviewed, and the result label or fraud probability is corrected according to the review result, such as manually reviewing the call audio as normal, then the fraud probability can be corrected to a value less than 50%, determined according to the actual manual review; If the call audio is reviewed as fraud, the result label can be corrected to fraud. Organize the manually corrected results, and determine the call audio and its corresponding scene data, result label and fraud probability as the first data set.

[0080] Step 2: Scene risk feature extraction

[0081] Train the LightGBM model with a large number of annotated historical data (including number attribution, call time period, called person label, etc. Scene data and its corresponding fraud / normal result label), and extract the 16-dimensional scene risk feature vector of the scene data using the trained LightGBM model.

[0082] Step 3: Physiological feature extraction

[0083] Role separation of the call voice based on an open source library such as Pyannote.audio, that is, distinguishing different speakers, so as to generate separated voices. A physiological feature label unique to each role is generated by extracting 128-dimensional physiological features such as the frequency of vocal cord vibration and airflow intensity of the speaker by an optimized mel-frequency cepstral coefficient (MFCC) algorithm, which is not affected by the speed and tone of the speech and effectively resists voice disguising.

[0084] The optimized MFCC algorithm innovatively optimizes three key links of preprocessing, filtering and feature fusion, and the specific implementation steps are as follows:

[0085] (1) Preprocessing stage: noise adaptive filtering and pre-emphasis coefficient optimization

[0086] Adaptive noise suppression: a hybrid denoising model of spectral subtraction + wavelet threshold denoising is introduced, which first eliminates the steady-state noise (such as current sound) by spectral subtraction, and then processes the non-steady-state noise (such as human voice interference) by wavelet threshold denoising. In the denoising process, the audio signal-to-noise ratio (SNR) is calculated in real time, and the denoising intensity is dynamically adjusted to avoid excessive denoising that leads to loss of physiological features.

[0087] Dynamic adjustment of pre-emphasis coefficient: the traditional MFCC uses a fixed pre-emphasis coefficient (usually 0.97), and the optimized MFCC algorithm dynamically adjusts the pre-emphasis coefficient according to the voice fundamental frequency (F0) - when the fundamental frequency is high (such as female and child speech), the pre-emphasis coefficient is reduced, and when the fundamental frequency is low (such as male speech), the pre-emphasis coefficient is increased, to ensure that high-frequency physiological features (such as vocal cord vibration harmonics) are not attenuated.

[0088] (2) Filtering and cepstral calculation stage: mel filter bank optimization + physiological feature dimension expansion

[0089] Reconstruction of the band-pass filter bank: the center frequencies of the traditional mel filter bank are uniformly distributed according to the mel scale, and the optimized MFCC algorithm divides the filter bank into a physiological feature sensitive area and a semantic feature area: in the physiological feature sensitive area (200Hz-5000Hz), the number of filters is increased (from the traditional 24 to 36), which focuses on capturing the spectral details of physiological features such as vocal cord vibration and airflow resonance; in the semantic feature area (5000Hz-8000Hz), the filters are sparsely distributed to meet the basic needs of semantic recognition.

[0090] Cepstral coefficient expansion: based on the traditional 12-dimensional MFCC coefficient, three types of physiological feature-related coefficients are added, expanding to a 28-dimensional feature vector. The three types include: fundamental frequency (F0) and its first-order difference coefficient: reflecting the variation law of vocal cord vibration frequency; spectral flatness coefficient: reflecting the periodicity of the speech signal, distinguishing between natural voice and synthetic cloned speech; formant (F1-F3) frequency coefficient: corresponding to the physiological structure features of the oral cavity, nasal cavity and other resonance cavities.

[0091] (3) Feature post-processing stage: dynamic feature weighting and inter-frame fusion

[0092] Inter-frame feature dynamic weighting: After the audio is framed (frame length 25 ms, frame shift 10 ms), the 28-dimensional features of each frame are weighted - high weight is given to speech active frames (energy higher than threshold), and low weight is given to silent frames / noise frames to avoid invalid frames interfering with physiological feature extraction.

[0093] Temporal feature fusion: Introduce long short-term memory network (LSTM) to model the feature vectors of consecutive frames in time sequence, capture the physiological feature change rule across frames (such as the stability of the speaker's voice), and finally output a 128-dimensional speech physiological feature vector.

[0094] Step 4: Semantic feature extraction

[0095] Through the Whisper-large-v3 model, the role-separated audio file (i.e. separated speech) is converted into a role-separated dialogue text (i.e. separated text), or the Whisper-large-v3 model can be optimized and then converted, and the optimization method can be to optimize the model through knowledge distillation, structured pruning, and light-weight modification, while retaining the ability to recognize dialect words and mood words. The separated text can include complete role-separated dialogue text, and complete text of each role itself. Based on the role-separated dialogue text, the BERT-WWM-Ext model is used to extract contextual semantic features, and the BERT-WWM-Ext outputs deep semantic features of the dialogue (e.g. 768-dimensional vectors for each word and each sentence) and an attention weight matrix (records which words each word in the model pays attention to).

[0096] At the same time, an intent recognition model and an interaction analysis model are constructed. The intent recognition model is used to identify identity impersonation (whether impersonating an official, customer service, familiar person, or other authoritative role), sensitive information request (whether requesting a verification code, password, bank card number, or other highly sensitive information), behavior induction (whether requiring a transfer, screen sharing, clicking on unknown links, etc.), and logic loopholes (whether the conversation content contains illogical or contradictory parts, etc. 6 types of static fraud signals. The interaction analysis model captures dynamic features such as power suppression (whether one party continuously uses commanding, intimidation, or authoritative language to suppress the other party's questions or doubts), emotional manipulation (whether the conversation leads to obvious changes in anxiety, fear, or excessive trust in one party), rhythm control (whether one party interrupts the normal thinking of the other party by creating a sense of urgency such as time limit or immediate freezing), and induced questioning (whether the other party unknowingly reveals information or agrees to an operation through guided questions). A 48-dimensional semantic dynamic vector is generated. The deep semantic features are input into the intent recognition model, which is a fine-tuned classification neural network that outputs the probability of the conversation voice belonging to identity impersonation, sensitive information request, etc. The interaction analysis model performs specialized analysis and calculation on the attention weight features. For example, whether the attention intensity of the scammer's speech to the victim's speech is overwhelming (power suppression), or whether the attention flow of emotional words is inductive (emotional manipulation). Whether the victim's speech contains confusion or compliance. These calculation rules and extracted features (such as power suppression index) are output by the interaction analysis model.

[0097] The two types of results obtained above (static fraud features and dynamic interaction features) are spliced, normalized, etc. Finally, a fixed-length, comprehensive 48-dimensional semantic dynamic vector is formed as the risk representation of the conversation voice.

[0098] Step 5: Generate a fraud voice recognition model

[0099] The physiological features, semantic features, and scene risk feature vectors obtained above are weighted and fused to generate a 192-dimensional multi-modal feature vector. The 192-dimensional multi-modal feature vector is input into the first large model for fine-tuning training, where the first large model can use deepseek-r1 as the base model. Through layer sensitivity analysis algorithm, the decoder of the base model is pruned and optimized, retaining 16 core feature extraction layers, and using INT8 quantization perception training. Using the LlamaFactory framework and based on the hybrid fine-tuning strategy of LoRA+Adapter, only the attention layer and output layer of the base model are updated for the multi-modal feature vector. When the first large model training reaches the convergence condition, stop parameter updating, and determine the trained model as the fraud voice recognition model.

[0100] When the number of new fraud samples exceeds 50, the incremental fine-tuning is automatically started. An API interface is established with the anti-fraud platform to synchronize the latest fraud script library, high-risk number library, and fraud mode characteristics every fixed time period, dynamically update the fraud feature labels and scene risk weights of the semantic analysis model, and realize rapid adaptation to new fraud methods.

[0101] Step 6: Fraud voice recognition model inference

[0102] The hash value of the call voice with the result label as fraud in the first data set is calculated, and the hash value, multi-modal feature vector, result label, and fraud probability of the call voice in the first data set are stored in the cache database (such as Redis) in a vectorized manner. The hash value can be an MD5 value, and the hash value is stored in Redis as a key (Key), and the corresponding multi-modal feature vector, result label, and fraud probability are stored as a value (Value) in Redis, to build a key-value pair for quick query.

[0103] The hash value of the call voice with the result label as fraud in the first data set is calculated, and the hash value, multi-modal feature vector, result label, and fraud probability of the call voice in the first data set are stored in the cache database (such as Redis) in a vectorized manner. The hash value can be an MD5 value, and the hash value is stored in Redis as a key (Key), and the corresponding multi-modal feature vector, result label, and fraud probability are stored as a value (Value) in Redis, to build a key-value pair for quick query.

[0104] If there is no sample with a similarity of ≥ 85% (i.e., a multi-modal feature vector), the multi-modal feature vector of the speech to be recognized is input into the fraud speech recognition model, the fraud speech recognition model outputs the fraud probability and result label of the speech to be recognized, and in the case of a fraud result label, the hash value, multi-modal feature vector, result label and fraud probability of the speech to be recognized are stored in the cache database. In the actual deployment process, a synchronous + asynchronous processing architecture is adopted to ensure the stability and response speed of the system in a high-concurrency scenario. The architecture includes: (1) Request access and routing layer. This layer serves as the unified entrance of the system, is built based on the Spring WebFlux and Project Reactor framework, adopts a non-blocking, event-driven reactive programming model to achieve efficient processing of high-concurrency HTTP requests. The core responsibility of this layer is request routing and protocol adaptation, supporting two interaction modes: ① Asynchronous direct-through mode. For asynchronous requests initiated by the client, this layer acts as a high-performance reverse proxy, directly routing the request to the backend Web service layer and bridging the response stream to achieve transparent forwarding of requests and responses. ② Synchronous-to-asynchronous mode: To be compatible with traditional synchronous calling clients, this layer implements synchronous-to-asynchronous (Sync-to-Async) interaction adaptation. Request interception and context preservation: When a synchronous request is received, the system generates a unique association ID and temporarily stores the response context (such as ServerHttpResponse) of the original request in a suspended request mapping table in memory. Asynchronous task delivery and callback registration: The system sends the processing task along with its own callback notification address to the Web service layer, and then the original HTTP connection is suspended non-blockingly. Asynchronous response and association recovery: When the backend service completes processing and calls the preset callback address, this layer retrieves and restores the corresponding suspended request context from the mapping table based on the association ID carried in the callback, and writes the final result to the response stream, thus completing the entire synchronous calling loop.

[0105] (2) Web service layer: developed based on the Spring Boot framework, providing standard Web API interfaces. The interface receives requests for speech to be recognized, and the request parameters at least include: the download address (URL) of the speech to be recognized, and the receiving address of the result callback after recognition is completed. (3) Message queue: after the service receives the request, it does not directly perform time-consuming processing, but encapsulates the task information into a message and delivers it to a RocketMQ or other message queue, and immediately returns a "request received" response to the client. This method decouples the service interface and the core processing logic, improving the throughput and fault tolerance of the system.

[0106] It should be noted that, for the foregoing method embodiments, in order to simply describe, they are all expressed as a combination of a series of actions, but those skilled in the art should know that the present application is not limited by the order of the actions described, because according to the present application, certain steps can be performed in other orders or at the same time. Secondly, those skilled in the art should know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily the application. That is, the embodiments or examples of the present disclosure are not exhaustive, but only a part of the embodiments or examples, and are not specifically limited to the protection scope of the present disclosure. In the case of no contradiction, each step in a certain embodiment or example can be implemented as an independent embodiment, and the steps can be combined arbitrarily, for example, the scheme after removing some steps in a certain embodiment or example can be implemented as an independent embodiment, and the order of the steps in a certain embodiment or example can be exchanged arbitrarily, in addition, the optional mode or optional example in a certain embodiment or example can be combined arbitrarily; in addition, the embodiments or examples can be combined arbitrarily, for example, the steps of different embodiments or examples can be combined arbitrarily, a certain embodiment or example can be combined with the optional mode or optional example of other embodiments or examples.

[0107] According to another aspect of the embodiments of the present application, a large model-based fraud voice recognition device for implementing the large model-based fraud voice recognition method described above is also provided. As shown in the figure, the device comprises: Figure 6

[0108] The construction unit 602 is configured to construct a first data set containing call voice, scene data, result label and fraud probability;

[0109] The first extraction unit 604-1 is configured to extract the scene risk feature vector of the scene data by fine-tuning the trained gradient boosting decision tree model;

[0110] The separation unit 604-2 is configured to separate the call voice into roles based on an open source library to generate separated voice;

[0111] The second extraction unit 606-1 is configured to extract the frame-level features of the separated voice according to the optimized mel-frequency cepstral coefficient algorithm, and input the frame-level features into the long short-term memory network for time series modeling to output the physiological feature vector of fixed dimension;

[0112] ​The third extraction unit 606-2 is configured to convert the separated voice into separated text, extract deep semantic features and attention weight features of the separated text based on a pre-trained language model, extract static fraud features in the deep semantic features according to an intent recognition model, extract dynamic interaction features in the attention weight features according to an interaction analysis model, splice and normalize the static fraud features and the dynamic interaction features to generate a semantic feature vector with a fixed dimension.

[0113] The fusion unit 608 is configured to weight and fuse the physiological feature vector, the semantic feature vector, and the scene risk feature vector to generate a multi-modal feature vector with a fixed dimension. The fine-tuning unit 610 is configured to input the multi-modal feature vector into the first large model for fine-tuning training to generate a fraud voice recognition model. The acquisition unit 612 is configured to acquire a to-be-recognized voice, input a multi-modal feature vector generated based on the to-be-recognized voice into the fraud voice recognition model, and output a fraud probability and a result label of the to-be-recognized voice.

[0114] The embodiments in the present scheme can refer to the above method embodiments, but are not limited thereto, and the present application is not limited thereto.

[0115] As an optional scheme, the construction unit 602 includes: a collection module configured to collect a second data set containing the call voice, the scene data, and the result label; an input module configured to input the second data set and a preset prompt word into a second large model; a first output module configured to output, by the second large model, the fraud probability of each call voice in the second data set; a first correction module configured to, in a case where the result label is the fraud and the fraud probability is lower than 50%, correct the result label or the fraud probability based on an artificial review result of the call voice; a second correction module configured to, in a case where the result label is the normal and the fraud probability is higher than 50%, correct the result label or the fraud probability based on an artificial review result of the call voice; and a first determination module configured to determine the call voice, the scene data, the result label, and the fraud probability as the first data set, wherein the preset prompt word is used to prompt the second large model to output the fraud probability of the call voice.

[0116] The embodiments in the present scheme can refer to the above method embodiments, but are not limited thereto, and the present application is not limited thereto.

[0117] As an optional solution, the second extraction unit 606-1 includes: a preprocessing module, configured to perform adaptive noise suppression on the separated voice by using a hybrid noise reduction model combining spectral subtraction and wavelet threshold denoising in a preprocessing stage of the mel-frequency cepstral coefficient algorithm, and dynamically adjust a pre-emphasis coefficient according to a fundamental frequency of the separated voice, wherein the adaptive noise suppression dynamically adjusts noise reduction strength of the spectral subtraction and the wavelet threshold denoising according to a real-time signal-to-noise ratio of the separated voice, and the fundamental frequency is negatively correlated with the pre-emphasis coefficient; a first calculation module, configured to divide a frequency range of a mel filter bank into a physiological feature sensitive area and a semantic feature area, densely set a number of mel filters in the physiological feature sensitive area, sparsely set a number of mel filters in the semantic feature area, and increase a cepstral coefficient related to a physiological feature to expand a cepstral feature vector, wherein the increased cepstral coefficient includes a fundamental frequency and a first-order difference coefficient thereof, a spectral flatness coefficient, and a formant frequency coefficient; and a post-processing module, configured to dynamically weight the cepstral feature vector according to voice activity in a feature post-processing stage of the mel-frequency cepstral coefficient algorithm to obtain the frame-level feature, and input the frame-level feature into the long short-term memory network for time series modeling to output the physiological feature vector of a fixed dimension.

[0118] The embodiments in the present scheme can refer to the above method embodiments, but are not limited thereto, and the present application is not limited thereto.

[0119] As an optional solution, the fine-tuning unit 610 includes: a first acquisition module, configured to acquire the first large model; a pruning module, configured to perform structural pruning on a decoder of the first large model by using a layer sensitivity analysis algorithm; a training module, configured to perform INT8 quantization perception training on the pruned first large model based on the multi-modal feature vector; an updating module, configured to perform parameter updating on an attention layer and an output layer of the first large model according to a hybrid fine-tuning strategy of LoRA and Adapter during the training process; and a second determination module, configured to determine a model generated after fine-tuning training as the fraud voice recognition model.

[0120] The embodiments in the present scheme can refer to the above method embodiments, but are not limited thereto, and the present application is not limited thereto. As an optional solution, the device further includes a storage unit 609, configured to store sample data in which a result label of the first data set is labeled as fraud into a cache database, wherein the sample data includes a hash value of the call voice, the multi-modal feature vector, the result label, and the fraud probability.

[0121] The embodiments in the present scheme can, but are not limited to, referring to the above method embodiments, and the present application embodiments do not make any limitation in this regard.

[0122] As an optional scheme, the acquisition unit 612 includes:

[0123] The second acquisition module is configured to acquire the to-be-recognized speech.

[0124] The second calculation module is configured to calculate the hash value of the to-be-recognized speech.

[0125] The second output module is configured to, in a case where the hash value of the to-be-recognized speech is queried in the cache database, output the fraud probability and the result label corresponding to the hash value of the to-be-recognized speech in the cache database as the recognition result of the to-be-recognized speech.

[0126] The third acquisition module is configured to, in a case where the hash value of the to-be-recognized speech is not queried in the cache database, acquire the multi-modal feature vector of the to-be-recognized speech, and retrieve, based on the FAISS, Top-K historical multi-modal feature vectors similar to the multi-modal feature vector of the to-be-recognized speech from the cache database.

[0127] The third output module is configured to, in a case where the similarity of the multi-modal feature vector of the to-be-recognized speech exceeds a first preset threshold in the Top-K historical multi-modal feature vectors, output the fraud probability and the result label corresponding to the historical multi-modal feature vector with the highest similarity to the multi-modal feature vector of the to-be-recognized speech in the cache database as the recognition result of the to-be-recognized speech, and store the hash value, the multi-modal feature vector, the result label and the fraud probability of the to-be-recognized speech in the cache database, wherein the historical multi-modal feature vector refers to the multi-modal feature vector stored in the cache database.

[0128] The fourth output module is configured to, in a case where the similarity of the multi-modal feature vector of the to-be-recognized speech does not exceed the first preset threshold in the Top-K historical multi-modal feature vectors, input the multi-modal feature vector of the to-be-recognized speech into the fraud speech recognition model to output the fraud probability and the result label of the to-be-recognized speech, and in a case where the result label is fraud, store the hash value, the multi-modal feature vector, the result label and the fraud probability of the to-be-recognized speech in the cache database. The embodiments in the present scheme can, but are not limited to, referring to the above method embodiments, and the present application embodiments do not make any limitation in this regard.

[0129] Optionally, in the embodiment, the application embodiments of each unit module can refer to the above-mentioned application embodiments of the fraud voice recognition method based on a large model, which will not be described here.

[0130] According to an aspect of the present application, a computer readable storage medium is provided, and a processor of a computer device reads computer instructions from the computer readable storage medium. The processor executes the computer instructions, so that the computer device executes the steps in any one of the method embodiments.

[0131] Optionally, in the embodiment, the computer readable storage medium can be configured to store a computer program for executing the following steps:

[0132] S1, constructing a first data set containing call voice, scene data, result label and fraud probability;

[0133] S2, extracting a scene risk feature vector of the scene data by fine-tuning the trained gradient boosting decision tree model;

[0134] S3, performing role separation on the call voice based on an open source library to generate separated voice;

[0135] S4, extracting frame-level features of the separated voice according to an optimized mel-frequency cepstral coefficient algorithm, and inputting the frame-level features into a long short-term memory network for time series modeling to output a physiological feature vector of a fixed dimension;

[0136] S5, converting the separated voice into separated text, extracting deep semantic features and attention weight features of the separated text based on a pre-trained language model, extracting static fraud features in the deep semantic features according to an intent recognition model, extracting dynamic interaction features in the attention weight features according to an interaction analysis model, and splicing and normalizing the static fraud features and the dynamic interaction features to generate a semantic feature vector of a fixed dimension;

[0137] S6, weighting and fusing the physiological feature vector, the semantic feature vector and the scene risk feature vector to generate a multi-modal feature vector of a fixed dimension;

[0138] S7, inputting the multi-modal feature vector into a first large model for fine-tuning training to generate a fraud voice recognition model;

[0139] S8, obtaining a to-be-recognized voice, inputting a multi-modal feature vector generated based on the to-be-recognized voice into the fraud voice recognition model to output a fraud probability and a result label of the to-be-recognized voice.

[0140] Optionally, in the embodiment, all or part of the steps in the various methods of the above embodiments can be understood by those of ordinary skill in the art that the steps can be completed by instructing the terminal device related hardware through a program, and the program can be stored in a computer readable storage medium, and the storage medium can include a flash disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0141] According to an aspect of the present application, a computer program product is provided, which includes computer programs / instructions containing program codes for executing the above method. In such an embodiment, the computer program can be downloaded and installed from a network through a communication part, and / or installed from a detachable medium. When the computer program is executed by a central processing unit, various functions provided by the embodiments of the present application are executed.

[0142] According to another aspect of the embodiments of the present application, an electronic device for implementing the above confusion-based text chunking method is further provided. In the embodiments, the electronic device is taken as a server for example. As shown in the figure, the electronic device includes a memory 702 and a processor 704, the memory 702 stores a computer program, and the processor 704 is configured to execute the steps in any of the method embodiments through the computer program. Figure 7

[0143] Optionally, in the embodiment, the above electronic device can be located in at least one of the network devices in the computer network.

[0144] Optionally, in the embodiment, the processor can be configured to execute the following steps through the computer program:

[0145] S1, a first data set containing call voice, scene data, result label and fraud probability is constructed;

[0146] S2, scene risk feature vectors of the scene data are extracted by fine-tuning the trained gradient boosting decision tree model;

[0147] S3, role separation is performed on the call voice based on an open source library to generate separated voice;

[0148] S4, frame-level features of the separated voice are extracted according to an optimized mel-frequency cepstral coefficient algorithm, and the frame-level features are input into a long short-term memory network for time series modeling to output physiological feature vectors of a fixed dimension;

[0149] ​S5, convert the isolated speech into isolated text, extract deep semantic features and attention weight features of the isolated text based on a pre-trained language model, extract static fraud features in the deep semantic features according to an intent recognition model, extract dynamic interaction features in the attention weight features according to an interaction analysis model, and splice and normalize the static fraud features and the dynamic interaction features to generate a fixed-dimension semantic feature vector;

[0150] S6, weight and fuse the physiological feature vector, the semantic feature vector and the scene risk feature vector to generate a fixed-dimension multi-modal feature vector;

[0151] S7, input the multi-modal feature vector into a first large model for fine-tuning training to generate a fraud speech recognition model;

[0152] S8, obtain a to-be-recognized speech, and input a multi-modal feature vector generated based on the to-be-recognized speech into the fraud speech recognition model to output a fraud probability and a result label of the to-be-recognized speech.

[0153] Optionally, those skilled in the art can understand that, Figure 7 The structure shown is only schematic, and the electronic device can also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a palm computer, a Mobile Internet Device (MID), a PAD, or the like. Figure 7 It does not limit the structure of the above-mentioned electronic device. For example, the electronic device can further include more or less components (such as a network interface, etc.) than those shown in Figure 7 , or have a different configuration from Figure 7 .

[0154] The memory 702 can be used to store software programs and modules, such as program instructions / modules corresponding to the text chunking method and device based on perplexity in the embodiments of the present application. The processor 704 performs various functional applications and data processing by running the software programs and modules stored in the memory 702, that is, implements the text chunking method based on perplexity. The memory 702 can include a high-speed random access memory, and can also include a non-volatile memory, such as one or more magnetic storage devices, a flash memory, or other non-volatile solid-state memories. In some examples, the memory 702 can further include a memory remotely arranged with respect to the processor 704, which can be connected to the terminal through a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof. The memory 702 can specifically but not limited to be used to store input text and the like. As an example, the memory 702 can be used to store the input text and the like, and the processor 704 can be used to perform the text chunking method based on perplexity. ​As shown, the memory 702 can include, but is not limited to, the construction unit 602, the first extraction unit 604-1, the separation unit 604-2, the second extraction unit 606-1, the third extraction unit 606-2, the fusion unit 608, the fine-tuning unit 610, and the acquisition unit 612 in the large model-based fraud speech recognition device.

[0155] In addition, the electronic device further includes a connection bus 706 for connecting various module components in the electronic device.

[0156] In other embodiments, the server can be a node in a distributed system, where the distributed system can be a blockchain system, which can be a distributed system formed by the plurality of nodes connected through network communication. Among them, the nodes can form a peer-to-peer (P2P, Peer To Peer) network, and any form of computing device, such as a server, a terminal, and other electronic devices, can become a node in the blockchain system by joining the peer-to-peer network.

[0157] The integrated units in the above embodiments, if implemented in the form of software function units and sold or used as independent products, can be stored in the computer-readable storage medium. Based on this understanding, the technical solutions of the present application or the whole or part of the technical solutions that essentially contribute to the prior art can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing one or more computer devices (which can be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application.

[0158] In the above embodiments of the present application, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0159] In the several embodiments provided by the present application, it should be understood that the disclosed client can be implemented in other ways. Among them, the above-mentioned device embodiment is only schematic, for example, the division of the unit is only a logical function division, and in actual implementation, there can be another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point, the coupling or direct coupling or communication connection between the displayed or discussed each other can be through some interface, indirect coupling or communication connection between units or modules, which can be electrical or other forms.

Claims

1. A fraud speech recognition method based on a large model, characterized in that, include: Construct a first dataset that includes call voice recordings, scenario data, result labels, and fraud probabilities; The scenario risk feature vector of the scenario data is extracted by fine-tuning the trained gradient boosting decision tree model. The call audio is role-separated using an open-source library to generate separate audio. The frame-level features of the separated speech are extracted using the optimized Mel frequency cepstral coefficient algorithm, and the frame-level features are input into a long short-term memory network for temporal modeling to output a fixed-dimensional physiological feature vector. The separated speech is converted into separated text. Deep semantic features and attention weight features of the separated text are extracted based on a pre-trained language model. Static fraud features are extracted from the deep semantic features based on an intent recognition model. Dynamic interaction features are extracted from the attention weight features based on an interaction analysis model. The static fraud features and the dynamic interaction features are concatenated and normalized to generate a fixed-dimensional semantic feature vector. The physiological feature vector, the semantic feature vector, and the scene risk feature vector are weighted and fused to generate a fixed-dimensional multimodal feature vector. The multimodal feature vectors are input into the first large model for fine-tuning training to generate a fraud speech recognition model; The system acquires the speech to be identified, inputs the multimodal feature vector generated based on the speech to be identified into the fraud speech recognition model, and outputs the fraud probability and the result label of the speech to be identified. The scenario data is used to represent the background information of the call voice, the result label indicates fraud or normal, the intent recognition model is a fine-tuned classification neural network, the interaction analysis model has preset calculation rules, and the gradient boosting decision tree model is fine-tuned based on the scenario data and the result label.

2. The method according to claim 1, characterized in that, The first dataset, which includes call voice recordings, scenario data, result labels, and fraud probabilities, comprises: Collect a second dataset containing the call audio, the scene data, and the result labels; Input the second dataset and the preset prompts into the second large model; The second large model outputs the fraud probability for each of the aforementioned voice calls in the second dataset; If the result label is fraud and the fraud probability is less than 50%, the result label or the fraud probability is corrected based on the results of manual review of the call audio. If the result label is "normal" and the fraud probability is higher than 50%, the result label or the fraud probability is corrected based on the results of manual review of the call audio. The call audio, the scene data, the result label, and the fraud probability are determined as the first dataset, wherein the preset prompt word is used to prompt the second model to output the fraud probability of the call audio.

3. The method according to claim 1, characterized in that, The step of extracting frame-level features of the separated speech using the optimized Mel-frequency cepstral coefficient algorithm, and inputting the frame-level features into a long short-term memory network for temporal modeling to output a fixed-dimensional physiological feature vector includes: In the preprocessing stage of the Mel frequency cepstral coefficient algorithm, a hybrid noise reduction model combining spectral subtraction and wavelet threshold denoising is used to adaptively suppress noise in the separated speech, and the pre-emphasis coefficient is dynamically adjusted according to the fundamental frequency of the separated speech. The adaptive noise suppression refers to dynamically adjusting the noise reduction intensity of the spectral subtraction and wavelet threshold denoising according to the real-time signal-to-noise ratio of the separated speech. The fundamental frequency is negatively correlated with the pre-emphasis coefficient. In the filtering and cepstral calculation stage of the Mel frequency cepstral coefficient algorithm, the frequency range of the Mel filter bank is divided into a physiological feature sensitive region and a semantic feature region. In the physiological feature sensitive region, the number of Mel filters is increased. In the semantic feature region, the number of Mel filters is decreased. Cepstral coefficients related to physiological features are added to expand the cepstral feature vector. The added cepstral coefficients include the fundamental frequency and its first-order difference coefficient, spectral flatness coefficient, and formant frequency coefficient. In the feature post-processing stage of the Mel frequency cepstral coefficient algorithm, the cepstral feature vector is dynamically weighted according to speech activity to obtain the frame-level features. The frame-level features are then input into the long short-term memory network for temporal modeling to output the physiological feature vector with fixed dimensions.

4. The method according to claim 1, characterized in that, The step of inputting the multimodal feature vector into the first large model for fine-tuning training to generate a fraud speech recognition model includes: Obtain the first large model; perform structured pruning on the decoder of the first large model using a layer sensitivity analysis algorithm; perform INT8 quantization perceptual training on the pruned first large model based on the multimodal feature vector; update the parameters of the attention layer and output layer of the first large model during training according to the hybrid fine-tuning strategy of LoRA and Adapter; determine the model generated after fine-tuning training as the fraud speech recognition model.

5. The method according to claim 1, characterized in that, Before inputting the multimodal feature vector into the first large model for fine-tuning training to generate the fraud speech recognition model, the method further includes: The sample data in the first dataset that are labeled as fraud are vectorized and stored in the cache database, wherein the sample data includes the hash value of the call voice, the multimodal feature vector, the result label, and the fraud probability.

6. The method according to claim 1, characterized in that, The process of acquiring the speech to be identified, and inputting the multimodal feature vector generated based on the speech to be identified into the fraud speech recognition model to output the fraud probability and the result label of the speech to be identified, includes: Obtain the speech to be recognized; Calculate the hash value of the speech to be recognized; If the hash value of the speech to be recognized is found in the cache database, the fraud probability corresponding to the hash value in the cache database and the result label are output as the recognition result of the speech to be recognized. If the hash value of the speech to be recognized is not found in the cache database, the multimodal feature vector of the speech to be recognized is obtained, and the Top-K historical multimodal feature vectors similar to the multimodal feature vector of the speech to be recognized are retrieved from the cache database based on FAISS. If, among the Top-K historical multimodal feature vectors, there exists a multimodal feature vector whose similarity to the multimodal feature vector of the speech to be recognized exceeds a first preset threshold, the fraud probability and the result label corresponding to the historical multimodal feature vector with the highest similarity to the multimodal feature vector of the speech to be recognized in the cache database are output as the recognition result of the speech to be recognized. The hash value, the multimodal feature vector, the result label, and the fraud probability of the speech to be recognized are stored in the cache database, wherein the historical multimodal feature vector refers to the multimodal feature vector stored in the cache database. If none of the Top-K historical multimodal feature vectors have a similarity exceeding a first preset threshold with the multimodal feature vector of the speech to be identified, the multimodal feature vector of the speech to be identified is input into the fraud speech recognition model to output the fraud probability and the result label of the speech to be identified. If the result label is fraud, the hash value, the multimodal feature vector, the result label, and the fraud probability of the speech to be identified are stored in the cache database.

7. A fraud speech recognition device based on a large model, characterized in that, include: The building unit is used to construct the first dataset, which includes call voice, scene data, result labels, and fraud probability. The first extraction unit is used to extract the scene risk feature vector of the scene data by fine-tuning the trained gradient boosting decision tree model. A separation unit is used to perform role separation on the call voice based on an open-source library to generate separated voice; The second extraction unit is used to extract the frame-level features of the separated speech according to the optimized Mel frequency cepstral coefficient algorithm, and input the frame-level features into a long short-term memory network for temporal modeling to output a fixed-dimensional physiological feature vector. The third extraction unit is used to convert the separated speech into separated text, extract deep semantic features and attention weight features of the separated text based on a pre-trained language model, extract static fraud features from the deep semantic features based on an intent recognition model, extract dynamic interaction features from the attention weight features based on an interaction analysis model, and concatenate and normalize the static fraud features and the dynamic interaction features to generate a fixed-dimensional semantic feature vector. The fusion unit is used to weightedly fuse the physiological feature vector, the semantic feature vector, and the scene risk feature vector to generate a fixed-dimensional multimodal feature vector; The fine-tuning unit is used to input the multimodal feature vector into the first large model for fine-tuning training to generate a fraud speech recognition model; The acquisition unit is used to acquire the speech to be identified, input the multimodal feature vector generated based on the speech to be identified into the fraud speech recognition model, and output the fraud probability and the result label of the speech to be identified; The scenario data is used to represent the background information of the call voice, the result label indicates fraud or normal, the intent recognition model is a fine-tuned classification neural network, the interaction analysis model has preset calculation rules, and the gradient boosting decision tree model is fine-tuned based on the scenario data and the result label.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program is executed by a processor to perform the method described in any one of claims 1 to 6.

9. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the method described in any one of claims 1 to 6 through the computer program.

Images