Fraud speech recognition method and device based on large model, storage medium and electronic equipment

Abstract

The invention discloses a fraud speech recognition method and device based on a large model, a storage medium and electronic equipment. The method comprises the following steps: constructing a first data set; performing role separation on the call voice based on an open source library to generate separated voice; according to an optimized Mel-frequency cepstrum coefficient algorithm, extracting a physiological feature vector of the separated voice; performing weighted fusion on the extracted physiological feature vector, the semantic feature vector and the scene risk feature vector to generate a fixed-dimension multi-modal feature vector; inputting the multi-modal feature vector into a first large model for fine tuning training so as to generate a fraud speech recognition model; and obtaining a to-be-recognized voice, and inputting a multi-modal feature vector generated based on the to-be-recognized voice into the fraud voice recognition model to output a fraud probability and a result label of the to-be-recognized voice. According to the invention, the problem that the existing fraud speech recognition technology is difficult to consider high recognition precision and high recognition efficiency at the same time is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, in particular to a fraud speech recognition method and device based on a large model, a storage medium and an electronic device. BACKGROUND

[0002] Current network frauds present characteristics such as professionalization of speech tactics and concealment of means. As the core carrier of fraud implementation, speech recognition technology faces many difficulties. For example, existing technologies rely on semantic information after speech-to-text, ignoring acoustic features such as sudden changes in speech rate, emotional fluctuations and abnormal intonation of fraudsters, resulting in high misjudgment rate in scenarios where speech tactics are similar but intentions are different or intentions are fraudulent but speech tactics are ambiguous. Moreover, relying on a single text or voiceprint feature makes it difficult to resist new fraud methods such as voice cloning and speech rate disguise, resulting in low recognition accuracy of fraud speech. In addition, existing technologies use a unified inference process for speech to be recognized, resulting in slow recognition efficiency of fraud speech. That is, existing fraud speech recognition technologies have difficulty in balancing high recognition accuracy and fast recognition efficiency.

[0003] Currently, there is no effective solution to the above problems. SUMMARY

[0004] The embodiments of the present application provide a fraud speech recognition method and device based on a large model, a storage medium and an electronic device to at least solve the problem that existing fraud speech recognition technologies have difficulty in balancing high recognition accuracy and fast recognition efficiency.

[0005] According to an aspect of the embodiments of the present application, a fraud speech recognition method based on a large model is provided, which comprises: constructing a first data set containing call speech, scene data, result label and fraud probability; extracting a scene risk feature vector of the scene data by fine-tuning a trained gradient boosting decision tree model; separating the call speech based on an open source library to generate separated speech; extracting frame-level features of the separated speech according to an optimized mel-frequency cepstral coefficient algorithm, and inputting the frame-level features into a long short-term memory network for time series modeling to output a fixed-dimension physiological feature vector; converting the separated speech into separated text, extracting deep semantic features and attention weight features of the separated text based on a pre-trained language model, extracting static fraud features in the deep semantic features according to an intent recognition model, extracting dynamic interaction features in the attention weight features according to an interaction analysis model, and splicing and normalizing the static fraud features and the dynamic interaction features to generate a fixed-dimension semantic feature vector; weighting and fusing the physiological feature vector, the semantic feature vector and the scene risk feature vector to generate a fixed-dimension multi-modal feature vector; inputting the multi-modal feature vector into a first large model for fine-tuning training to generate a fraud speech recognition model; obtaining a to-be-recognized speech, inputting the multi-modal feature vector generated based on the to-be-recognized speech into the fraud speech recognition model to output the fraud probability and the result label of the to-be-recognized speech; wherein the scene data is used to represent background information of the call speech, the result label is represented as fraud or normal, the intent recognition model is a fine-tuned classification neural network, the interaction analysis model has a pre-designed calculation rule, and the gradient boosting decision tree model is fine-tuned based on the scene data and the result label.

[0006] Optionally, in the embodiments of the present application, the constructing a first data set containing call speech, scene data, result label and fraud probability comprises: collecting a second data set containing the call speech, the scene data and the result label; inputting the second data set and a preset prompt word into a second large model; the second large model outputs the fraud probability of each call speech in the second data set; in the case that the result label is fraud and the fraud probability is lower than 50%, correcting the result label or the fraud probability based on the audit result of the call speech by artificial; in the case that the result label is normal and the fraud probability is higher than 50%, correcting the result label or the fraud probability based on the audit result of the call speech by artificial; determining the call speech, the scene data, the result label and the fraud probability as the first data set, wherein the preset prompt word is used to prompt the second large model to output the fraud probability of the call speech.

[0007] Optionally, in the embodiment of the present application, the above extracting the frame-level feature of the separated speech according to the optimized mel-frequency cepstral coefficient algorithm and inputting the frame-level feature into the long short-term memory network for time sequence modeling to output the physiological feature vector of a fixed dimension comprises: in the preprocessing stage of the mel-frequency cepstral coefficient algorithm, using a hybrid denoising model combining spectral subtraction and wavelet threshold denoising to perform adaptive noise suppression on the separated speech, and dynamically adjusting the pre-emphasis coefficient according to the fundamental frequency of the separated speech, wherein the adaptive noise suppression dynamically adjusts the denoising strength of the spectral subtraction and the wavelet threshold denoising according to the real-time signal-to-noise ratio of the separated speech, and the fundamental frequency is negatively correlated with the pre-emphasis coefficient; in the filtering and cepstral calculation stage of the mel-frequency cepstral coefficient algorithm, the frequency range of the mel filter bank is divided into a physiological feature sensitive area and a semantic feature area, the number of mel filters is set densely in the physiological feature sensitive area, the number of mel filters is set sparsely in the semantic feature area, and the cepstral coefficients related to physiological features are increased to expand the cepstral feature vector, wherein the increased cepstral coefficients include the fundamental frequency and its first-order difference coefficient, the spectral flatness coefficient, and the formant frequency coefficient; in the feature post-processing stage of the mel-frequency cepstral coefficient algorithm, the cepstral feature vector is dynamically weighted according to the speech activity to obtain the frame-level feature, and the frame-level feature is input into the long short-term memory network for time sequence modeling to output the physiological feature vector of a fixed dimension.

[0008] Optionally, in the embodiment of the present application, the above inputting the multi-modal feature vector into the first large model for fine-tuning training to generate a fraud speech recognition model comprises: obtaining the first large model; performing structural pruning on the decoder of the first large model using a layer sensitivity analysis algorithm; performing INT8 quantization perception training on the pruned first large model based on the multi-modal feature vector; updating the parameters of the attention layer and the output layer of the first large model according to the mixed fine-tuning strategy of LoRA and Adapter during training; and determining the generated model after fine-tuning training as the fraud speech recognition model.

[0009] Optionally, in the embodiment of the present application, before the above inputting the multi-modal feature vector into the first large model for fine-tuning training to generate a fraud speech recognition model, it further comprises: vectorizing and storing the sample data of the first data set with the result label as fraud into a cache database, wherein the sample data comprises the hash value of the call speech, the multi-modal feature vector, the result label and the fraud probability.

[0010] Optionally, in the embodiment of the present application, the above obtaining the to-be-recognized speech, inputting the above multi-modal feature vector generated based on the above to-be-recognized speech into the fraud speech recognition model to output the above fraud probability and the above result label of the to-be-recognized speech comprises: obtaining the to-be-recognized speech; calculating the hash value of the to-be-recognized speech; in the case that the hash value of the to-be-recognized speech is queried in the cache database, outputting the fraud probability and the result label corresponding to the hash value of the to-be-recognized speech in the cache database as the recognition result of the to-be-recognized speech; in the case that the hash value of the to-be-recognized speech is not queried in the cache database, obtaining the multi-modal feature vector of the to-be-recognized speech, retrieving Top-K historical multi-modal feature vectors similar to the multi-modal feature vector of the to-be-recognized speech from the cache database based on FAISS; in the case that there is a historical multi-modal feature vector in the Top-K historical multi-modal feature vectors, the similarity of which to the multi-modal feature vector of the to-be-recognized speech exceeds a first preset threshold, outputting the fraud probability and the result label corresponding to the historical multi-modal feature vector with the highest similarity to the multi-modal feature vector of the to-be-recognized speech in the cache database as the recognition result of the to-be-recognized speech, and storing the hash value, the multi-modal feature vector, the result label and the fraud probability of the to-be-recognized speech in the cache database, wherein the historical multi-modal feature vector refers to the multi-modal feature vector stored in the cache database; in the case that there is no historical multi-modal feature vector in the Top-K historical multi-modal feature vectors, the similarity of which to the multi-modal feature vector of the to-be-recognized speech exceeds the first preset threshold, inputting the multi-modal feature vector of the to-be-recognized speech into the fraud speech recognition model to output the fraud probability and the result label of the to-be-recognized speech, and storing the hash value, the multi-modal feature vector, the result label and the fraud probability of the to-be-recognized speech in the cache database in the case that the result label is fraud.

[0011] According to another aspect of the embodiments of the present application, a fraud speech recognition device based on a large model is also provided, comprising: a construction unit configured to construct a first data set containing call speech, scene data, result label, and fraud probability; a first extraction unit configured to extract a scene risk feature vector of the scene data by fine-tuning a trained gradient boosting decision tree model; a separation unit configured to separate roles based on an open source library to generate separated speech; a second extraction unit configured to extract frame-level features of the separated speech according to an optimized mel-frequency cepstral coefficient algorithm, and input the frame-level features into a long short-term memory network for time series modeling to output a fixed-dimension physiological feature vector; a third extraction unit configured to convert the separated speech into separated text, extract deep semantic features and attention weight features of the separated text based on a pre-trained language model, extract static fraud features in the deep semantic features according to an intent recognition model, extract dynamic interaction features in the attention weight features according to an interaction analysis model, and splice and normalize the static fraud features and the dynamic interaction features to generate a fixed-dimension semantic feature vector; a fusion unit configured to weight and fuse the physiological feature vector, the semantic feature vector, and the scene risk feature vector to generate a fixed-dimension multi-modal feature vector; a fine-tuning unit configured to input the multi-modal feature vector into a first large model for fine-tuning training to generate a fraud speech recognition model; and an acquisition unit configured to acquire a to-be-recognized speech, input the multi-modal feature vector generated based on the to-be-recognized speech into the fraud speech recognition model to output the fraud probability and the result label of the to-be-recognized speech; wherein the scene data is used to represent background information of the call speech, the result label is represented as fraud or normal, the intent recognition model is a fine-tuned classification neural network, the interaction analysis model has a pre-designed calculation rule, and the gradient boosting decision tree model is fine-tuned based on the scene data and the result label.

[0012] Optionally, in the embodiment of the present application, the construction unit comprises: a collection module, configured to collect a second data set comprising the call voice, the scene data and the result label; an input module, configured to input the second data set and a preset prompt word into the second large model; a first output module, configured to output, by the second large model, the fraud probability of each call voice in the second data set; a first correction module, configured to correct the result label or the fraud probability based on an artificial review result of the call voice in a case where the result label is the fraud and the fraud probability is lower than 50%; a second correction module, configured to correct the result label or the fraud probability based on an artificial review result of the call voice in a case where the result label is the normal and the fraud probability is higher than 50%; and a first determination module, configured to determine the call voice, the scene data, the result label and the fraud probability as the first data set, wherein the preset prompt word is used to prompt the second large model to output the fraud probability of the call voice.

[0013] Optionally, in the embodiment of the present application, the second extraction unit comprises: a preprocessing module, configured to, in a preprocessing stage of the mel-frequency cepstral coefficient algorithm, perform adaptive noise suppression on the separated voice by using a hybrid noise reduction model combining spectral subtraction and wavelet threshold denoising, and dynamically adjust a pre-emphasis coefficient according to a fundamental frequency of the separated voice, wherein the adaptive noise suppression refers to dynamically adjusting noise reduction intensity of the spectral subtraction and the wavelet threshold denoising according to a real-time signal-to-noise ratio of the separated voice, and the fundamental frequency is negatively correlated with the pre-emphasis coefficient; a first calculation module, configured to, in a filtering and cepstral calculation stage of the mel-frequency cepstral coefficient algorithm, divide a frequency range of a mel filter bank into a physiological feature sensitive area and a semantic feature area, densely set a number of mel filters in the physiological feature sensitive area, sparsely set a number of mel filters in the semantic feature area, and increase cepstral coefficients related to physiological features, so as to expand a cepstral feature vector, wherein the increased cepstral coefficients include a fundamental frequency and a first-order difference coefficient thereof, a spectral flatness coefficient and a formant frequency coefficient; and a post-processing module, configured to, in a feature post-processing stage of the mel-frequency cepstral coefficient algorithm, dynamically weight the cepstral feature vector according to voice activity to obtain a frame-level feature, input the frame-level feature into the long short-term memory network for time series modeling, and output the physiological feature vector of a fixed dimension.

[0014] Optionally, in the embodiments of the present application, the fine-tuning unit comprises: a first acquisition module configured to acquire the first large model; a pruning module configured to perform structural pruning on the decoder of the first large model using a layer sensitivity analysis algorithm; a training module configured to perform INT8 quantization-aware training on the pruned first large model based on the multi-modal feature vector; an updating module configured to update the parameters of the attention layer and the output layer of the first large model according to the mixed fine-tuning strategy of LoRA and Adapter during the training process; and a second determination module configured to determine the model generated after the fine-tuning training is completed as the fraud speech recognition model. Optionally, in the embodiments of the present application, the device further comprises a storage unit configured to store the sample data whose result label in the first data set is fraudulent into a cache database in a vector form, wherein the sample data comprises the hash value of the call voice, the multi-modal feature vector, the result label and the fraud probability.

[0015] Optionally, in the embodiments of the present application, the obtaining unit comprises: a second obtaining module, configured to obtain the to-be-recognized speech; a second calculating module, configured to calculate the hash value of the to-be-recognized speech; a second output module, configured to, in the case where the hash value of the to-be-recognized speech is queried in the cache database, output the fraud probability and the result label corresponding to the hash value of the to-be-recognized speech in the cache database as the recognition result of the to-be-recognized speech; a third obtaining module, configured to, in the case where the hash value of the to-be-recognized speech is not queried in the cache database, obtain the multi-modal feature vector of the to-be-recognized speech, and retrieve Top-K historical multi-modal feature vectors similar to the multi-modal feature vector of the to-be-recognized speech from the cache database based on the FAISS; a third output module, configured to, in the case where the similarity between the multi-modal feature vector of the to-be-recognized speech and any of the Top-K historical multi-modal feature vectors exceeds a first preset threshold, output the fraud probability and the result label corresponding to the historical multi-modal feature vector with the highest similarity to the multi-modal feature vector of the to-be-recognized speech in the cache database as the recognition result of the to-be-recognized speech, and store the hash value, the multi-modal feature vector, the result label and the fraud probability of the to-be-recognized speech in the cache database, wherein the historical multi-modal feature vector refers to the multi-modal feature vector stored in the cache database; and a fourth output module, configured to, in the case where the similarity between the multi-modal feature vector of the to-be-recognized speech and any of the Top-K historical multi-modal feature vectors does not exceed the first preset threshold, input the multi-modal feature vector of the to-be-recognized speech into the fraud speech recognition model to output the fraud probability and the result label of the to-be-recognized speech, and in the case where the result label is fraud, store the hash value, the multi-modal feature vector, the result label and the fraud probability of the to-be-recognized speech in the cache database.

[0016] According to still another aspect of the embodiments of the present application, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program. The computer program is configured to execute the fraud speech recognition method based on a large model provided by any of the preceding embodiments when running.

[0017] According to still another aspect of the embodiments of the present application, an electronic device is provided, and the electronic device comprises a memory and a processor. The memory stores a computer program, and the processor is configured to execute the fraud speech recognition method based on a large model provided by any of the preceding embodiments through the computer program.

[0018] In the embodiment of the present application, the physiological feature vector, the semantic feature vector and the scene risk feature vector are weighted and fused to generate a multi-modal feature vector, and the first large model is fine-tuned based on the multi-modal feature vector, wherein the physiological feature vector is generated based on the optimized mel frequency cepstral coefficient algorithm, the semantic feature vector is generated based on the pre-trained language model, the intent recognition model and the interaction analysis model, and the scene risk feature vector is generated based on the gradient boosting decision tree model after fine-tuning. That is, the fraud voice recognition model is generated based on the fine-tuning of the large model in multiple dimensions, thereby improving the recognition accuracy of the fraud voice. BRIEF DESCRIPTION OF DRAWINGS

[0019] The drawings described herein are used to provide further understanding of the present application, and form a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application, and do not constitute an improper limitation on the present application. In the drawings: Figure 1 is a schematic diagram of an optional fraud voice recognition method based on a large model according to an embodiment of the present application; Figure 2 is a schematic diagram of another optional fraud voice recognition method based on a large model according to an embodiment of the present application; Figure 3 is a schematic diagram of yet another optional fraud voice recognition method based on a large model according to an embodiment of the present application; Figure 4 is a schematic diagram of yet another optional fraud voice recognition method based on a large model according to an embodiment of the present application; Figure 5 is a schematic diagram of yet another optional fraud voice recognition method based on a large model according to an embodiment of the present application; Figure 6 is a structural schematic diagram of an optional fraud voice recognition device based on a large model according to an embodiment of the present application; Figure 7 is a structural schematic diagram of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0020] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should fall within the scope of protection of the present application.

[0021] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and in the above drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to only those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0022] Optionally, as an optional embodiment, as shown in Figure 1 The above fraud voice recognition method based on large model includes: S102, constructing a first data set containing call voice, scene data, result label and fraud probability; S104-1, extracting the scene risk feature vector of the scene data by fine-tuning the trained gradient boosting decision tree model; S104-2, role separation is performed on the call voice based on an open source library to generate separated voice; S106-1, extracting frame-level features of the separated voice according to an optimized mel-frequency cepstral coefficient algorithm, and inputting the frame-level features into a long short-term memory network for time series modeling to output a fixed-dimension physiological feature vector; S106-2, converting the separated voice into separated text, extracting deep semantic features and attention weight features of the separated text based on a pre-trained language model, extracting static fraud features in the deep semantic features according to an intent recognition model, extracting dynamic interaction features in the attention weight features according to an interaction analysis model, and splicing and normalizing the static fraud features and the dynamic interaction features to generate a fixed-dimension semantic feature vector; S108, weighting and fusing the physiological feature vector, the semantic feature vector and the scene risk feature vector to generate a fixed-dimension multi-modal feature vector; S110, inputting the multi-modal feature vector into a first large model for fine-tuning training to generate a fraud voice recognition model; S112, obtaining a to-be-recognized voice, inputting a multi-modal feature vector generated based on the to-be-recognized voice into the fraud voice recognition model to output a fraud probability and a result label of the to-be-recognized voice.

[0023] Optionally, in the embodiment of the present application, the fraud voice recognition method based on a large model can be applied to the field of natural language processing. The call voice can be real-time voice of a real person, pre-recorded voice, or AI synthesized voice. The scene data is background information that is irrelevant to the call content itself but can reflect the risk attributes of the call. It is an important auxiliary basis for judging fraud. The scene data includes, but is not limited to, the risk level of the number's home location, the call time period, enterprise outbound or private call, and the age and occupation tags of the called person. The risk level of the number's home location: based on historical data to label the risk value of the number's home location.

[0024] Optionally, in the embodiment of the present application, the open source library can be Pyannote.audio to realize role separation. The physiological feature vector includes, but is not limited to, the frequency of vocal cord vibration of the speaker, airflow intensity, etc. This feature is not affected by changes in speech speed and intonation, effectively resisting voice disguising. The separated voice is converted into separated text through the Whisper-large-v3 model, or the separated voice is converted into separated text through the optimized Whisper-large-v3 model. The optimization method is to perform lightweight modification on the Whisper-large-v3 model through knowledge distillation and structured pruning, while retaining the recognition ability of dialect words and mood words, thereby improving the recognition efficiency of fraud voice.

[0025] Optionally, in the embodiment of the present application, the deep semantic features and attention weight features of the separated text can be extracted through the pre-trained language model BERT-WWM-Ext model. The intent recognition model is generated based on the fine-tuning training of a classification neural network. The static fraud features include identity impersonation (whether impersonating an official, customer service, or familiar person as an authoritative role), information request (whether requesting verification code, password, bank card number, and other highly sensitive information), behavior induction (whether requiring a transfer, screen sharing, clicking on unknown links, etc.), logic loopholes (whether the call content has unreasonable or contradictory parts), etc. The dynamic interaction features include unequal rights (whether one party continuously uses command, intimidation, or authoritative language to suppress the other party's questions or doubts), induced questioning (whether through leading questions to make the other party unknowingly reveal information or agree to operations), emotional manipulation (whether the conversation leads to obvious emotional changes such as anxiety, fear, or excessive trust in one party), rhythm control (whether one party interrupts the normal thinking of the other party by creating a sense of urgency such as "limited time" or "frozen immediately"), etc.

[0026] Optionally, in the embodiments of the present application, the gradient boosting decision tree can be a LightGBM model. By using the historical fraud / normal result labels, the LightGBM model learns the correlation between the scene data and the result labels, and finally outputs the risk contribution weight of each scene data dimension. For example, a first data set containing multiple data is established, each data contains a call voice, its corresponding scene data, result label and fraud probability. The gradient boosting decision tree model is fine-tuned to extract the scene data 16 as a scene risk feature vector. The voices of different speakers in the call voice are separated by the open source library Pyannote.audio to obtain separated voices. The frame-level features of the separated voices are extracted according to the optimized mel-frequency cepstral coefficient algorithm, and the frame-level features are input into the LSTM network for time series modeling, thereby outputting a 128-dimensional physiological feature vector. The separated voices are converted into separated texts by the Whisper-large-v3 model, and the deep semantic features and attention weight features of the separated texts are extracted based on the BERT-WWM-Ext model. The static fraud features in the deep semantic features are extracted by the intent recognition model, and the dynamic interaction features in the attention weight features are extracted by the interaction analysis model. The static fraud features and the dynamic interaction features are concatenated and normalized to generate a 48-dimensional semantic feature vector. The physiological feature vector, the semantic feature vector and the scene risk feature vector are weighted and fused according to the self-defined weights, thereby generating a 192-dimensional multi-modal feature vector. The 192-dimensional multi-modal feature vector is input into the first large model, and the first large model is fine-tuned according to the result label and the fraud probability corresponding to the 192-dimensional multi-modal feature vector. When the training termination condition is reached, the parameter updating is stopped, and the trained first large model is determined as the fraud voice recognition model. The to-be-recognized voice is obtained, and the multi-modal feature vector generated based on the to-be-recognized voice is input into the fraud voice recognition model to output the fraud probability and the result label of the to-be-recognized voice, thereby realizing the recognition of the fraud voice.

[0027] According to the embodiments of the present application, the physiological feature vector, the semantic feature vector and the scene risk feature vector are weighted and fused to generate a multi-modal feature vector, and the first large model is fine-tuned based on the multi-modal feature vector, wherein the physiological feature vector is generated based on the optimized mel-frequency cepstral coefficient algorithm, the semantic feature vector is generated based on the pre-trained language model, the intent recognition model and the interaction analysis model, and the scene risk feature vector is generated based on the fine-tuned gradient boosting decision tree model. That is, the fraud voice recognition model is generated based on the fine-tuning of the large model in multiple dimensions, thereby improving the recognition accuracy of the fraud voice.

[0028] Based on one or each of the above embodiments, the present application is further improved, such as Figure 2As shown, constructing the first data set containing call voice, scene data, result label and fraud probability includes: S202, collecting a second data set containing call voice, scene data and result label; S204, inputting the second data set and the preset prompt word into the second large model; S206, the second large model outputs the fraud probability of each call voice in the second data set; S208-1, in the case that the result label is fraud and the fraud probability is lower than 50%, correcting the result label or the fraud probability based on the manual review result of the call voice; S208-2, in the case that the result label is normal and the fraud probability is higher than 50%, correcting the result label or the fraud probability based on the manual review result of the call voice; S210, determining the call voice, scene data, result label and fraud probability as the first data set, wherein the preset prompt word is used to prompt the second large model to output the fraud probability of the call voice.

[0029] Optionally, in the embodiment of the present application, the fraud probability of each call voice in the second data set is output by calling the large language model through the API.

[0030] On the basis of one or each of the above embodiments, the present application is further improved, such as Figure 3 As shown, the frame-level features of the isolated voice are extracted according to the optimized mel-frequency cepstral coefficient algorithm, and the frame-level features are input into the long short-term memory network for time series modeling to output the fixed-dimension physiological feature vector, including: S302, in the preprocessing stage of the mel-frequency cepstral coefficient algorithm, a hybrid denoising model combining spectral subtraction and wavelet threshold denoising is used for adaptive noise suppression of the isolated voice, and the preemphasis coefficient is dynamically adjusted according to the fundamental frequency of the isolated voice; S304, in the filtering and cepstral calculation stage of the mel-frequency cepstral coefficient algorithm, the frequency range of the mel filter bank is divided into a physiological feature sensitive area and a semantic feature area, the number of mel filters is encrypted in the physiological feature sensitive area, the number of mel filters is sparse in the semantic feature area, and the cepstral coefficient related to the physiological feature is increased to expand the cepstral feature vector; S306, in the feature post-processing stage of the mel-frequency cepstral coefficient algorithm, the cepstral feature vector is dynamically weighted according to the voice activity to obtain the frame-level feature, and the frame-level feature is input into the long short-term memory network for time series modeling to output the fixed-dimension physiological feature vector.

[0031] On the basis of one or each of the above embodiments, the present application is further improved, such as Figure 4As shown, inputting the multi-modal feature vector into the first large model for fine-tuning training to generate the fraud speech recognition model comprises: S402, obtaining the first large model; S404, using a layer sensitivity analysis algorithm to perform structured pruning on the decoder of the first large model; S406-1, performing INT8 quantization perception training on the pruned first large model based on the multi-modal feature vector; S406-2, during the training process, updating the parameters of the attention layer and the output layer of the first large model according to the mixed fine-tuning strategy of LoRA and Adapter; S408, determining the model generated after the fine-tuning training as the fraud speech recognition model.

[0032] Based on the one or each embodiment described above, the present application is further improved, such as Figure 5 As shown, obtaining the to-be-recognized speech, inputting the multi-modal feature vector generated based on the to-be-recognized speech into the fraud speech recognition model to output the fraud probability and the result label of the to-be-recognized speech comprises: S502, obtaining the to-be-recognized speech; S504, calculating the hash value of the to-be-recognized speech; S506-1, in the case where the hash value of the to-be-recognized speech is queried in the cache database, outputting the fraud probability and the result label corresponding to the hash value of the to-be-recognized speech in the cache database as the recognition result of the to-be-recognized speech; S506-2, in the case where the hash value of the to-be-recognized speech is not queried in the cache database, obtaining the multi-modal feature vector of the to-be-recognized speech, and retrieving Top-K historical multi-modal feature vectors similar to the multi-modal feature vector of the to-be-recognized speech from the cache database based on FAISS; S508-1, in the case where there is a historical multi-modal feature vector similar to the multi-modal feature vector of the to-be-recognized speech in the Top-K historical multi-modal feature vectors, the similarity of which exceeds the first preset threshold, outputting the fraud probability and the result label corresponding to the historical multi-modal feature vector with the highest similarity to the multi-modal feature vector of the to-be-recognized speech in the cache database as the recognition result of the to-be-recognized speech, and storing the hash value, the multi-modal feature vector, the result label and the fraud probability of the to-be-recognized speech into the cache database; S508-2, in the case where there is no historical multi-modal feature vector similar to the multi-modal feature vector of the to-be-recognized speech in the Top-K historical multi-modal feature vectors, the similarity of which exceeds the first preset threshold, inputting the multi-modal feature vector of the to-be-recognized speech into the fraud speech recognition model to output the fraud probability and the result label of the to-be-recognized speech, and storing the hash value, the multi-modal feature vector, the result label and the fraud probability of the to-be-recognized speech into the cache database in the case where the result label is fraud.

[0033] Optionally, as an optional embodiment, the above method is exemplarily and integrally explained by the following steps: Step 1: Data set construction 3000 positive samples (fraudulent speech) and 3000 negative samples (normal speech) are collected as a second data set, each sample containing a call recording (5 minutes), scene data (such as number home risk level, call period, enterprise outbound call or private call, called person age occupation label, etc.), and result label (fraud / normal).

[0034] According to different industries, some characteristics of the relevant industry or some general characteristics of all frauds are collected, and special prompt words are designed based on these characteristics, such as: You are an anti-fraud specialist in the {XX industry} field. Please evaluate the following call content based on your understanding of the common fraud methods in the {XX industry} field:

Industry fraud feature reminder

Evaluation task

Output requirements

[0035] The second data set and the designed special prompt word are input into the second large language model, and it is judged whether the fraud probability output by the second large language model corresponds to the result label: if the result label is fraud but the fraud probability output by the second large language model is less than 50%, then the call voice is manually audited, and the result label or the fraud probability is corrected according to the audit result. For example, if the manual audit of the call voice is fraud, but the fraud performance is hidden, the fraud probability can be corrected to a value between 50% and 70%; if the fraud performance is obvious, the fraud probability can be corrected to a value between 80% and 100%, which is determined according to the actual manual audit; if the manual audit of the call voice is normal, the result label can be corrected to normal. If the result label is normal but the fraud probability output by the second large language model is greater than 50%, then the call voice is manually audited, and the result label or the fraud probability is corrected according to the audit result. For example, if the manual audit of the call voice is normal, the fraud probability can be corrected to a value less than 50%, which is determined according to the actual manual audit; if the manual audit of the call voice is fraud, the result label can be corrected to fraud. The result after manual correction is arranged, and the call voice and its corresponding scene data, result label and fraud probability are determined as the first data set.

[0036] Step 2: Scene risk feature extraction A large number of annotated historical data (containing number of origin, call period, called person label and other scene data and its corresponding fraud / normal result label) are used to train a LightGBM model, and a 16-dimensional scene risk feature vector of the scene data is extracted by using the trained LightGBM model.

[0037] Step 3: Physiological feature extraction Based on open source libraries such as Pyannote.audio, the call voice is separated into different speakers, thereby generating separated voice. Through an optimized Mel frequency cepstrum coefficient (MFCC) algorithm, 128-dimensional physiological features such as speaker vocal cord vibration frequency and airflow intensity are extracted, and a unique physiological feature label is generated for each role. The feature is not affected by changes in speech speed and tone, and effectively resists voice disguise.

[0038] The optimized MFCC algorithm innovatively optimizes three key links of preprocessing, filtering and feature fusion, and the specific implementation steps are as follows: (1) Preprocessing stage: noise adaptive filtering and pre-emphasis coefficient optimization Adaptive noise suppression: a hybrid denoising model of spectral subtraction + wavelet threshold denoising is introduced. First, the spectral subtraction method is used to eliminate steady-state noise (such as current sound), and then the wavelet threshold denoising method is used to process non-steady-state noise (such as human voice interference). During the denoising process, the audio signal-to-noise ratio (SNR) is calculated in real time, and the denoising intensity is dynamically adjusted to avoid excessive denoising and loss of physiological features.

[0039] Pre-emphasis coefficient dynamic adjustment: The traditional MFCC adopts a fixed pre-emphasis coefficient (usually 0.97), and the optimized MFCC algorithm dynamically adjusts the pre-emphasis coefficient according to the speech fundamental frequency (F0) - when the fundamental frequency is high (such as female and child speech), the pre-emphasis coefficient is reduced, and when the fundamental frequency is low (such as male speech), the pre-emphasis coefficient is increased, to ensure that high-frequency physiological characteristics (such as vocal cord vibration harmonics) are not attenuated.

[0040] (2) Filter and cepstrum calculation stage: Mel filter bank optimization + physiological feature dimension expansion Band-pass filter bank reconstruction: The center frequency of the traditional Mel filter bank is uniformly distributed according to the Mel scale, and the optimized MFCC algorithm divides the filter bank into a physiological feature sensitive area and a semantic feature area: among them, the physiological feature sensitive area (200Hz-5000Hz): the number of filters is increased (from the traditional 24 to 36), focusing on capturing the spectral details corresponding to physiological features such as vocal cord vibration and airflow resonance; semantic feature area (5000Hz-8000Hz): maintain sparse distribution of filters to meet the basic needs of semantic recognition.

[0041] Cepstrum coefficient expansion: Based on the traditional 12-dimensional MFCC coefficient, 3 types of physiological feature-related coefficients are added, expanding to a 28-dimensional feature vector. The 3 types include: fundamental frequency (F0) and its first-order difference coefficient: reflecting the change law of vocal cord vibration frequency; spectral flatness coefficient: reflecting the periodicity of the speech signal, distinguishing natural voice from synthetic cloned speech; formant (F1-F3) frequency coefficient: corresponding to the physiological structure characteristics of the resonance cavity such as the oral cavity and the nasal cavity.

[0042] (3) Feature post-processing stage: Dynamic feature weighting and inter-frame fusion Inter-frame feature dynamic weighting: After framing the audio (frame length 25ms, frame shift 10ms), the 28-dimensional features of each frame are weighted - high weight is given to speech active frames (energy higher than threshold), and low weight is given to silent frames / noisy frames, to avoid interference of invalid frames with physiological feature extraction.

[0043] Temporal feature fusion: Long short-term memory network (LSTM) is introduced to model the temporal features of the feature vector of consecutive frames, capturing the physiological feature change law across frames (such as the stability of the speaker's voice), and finally outputting a 128-dimensional speech physiological feature vector.

[0044] Step 4: Semantic feature extraction The role-separated audio file (i.e., separated speech) is converted into a role-separated dialogue text (i.e., separated text) by the Whisper-large-v3 model, or the Whisper-large-v3 model can be optimized and then converted, and the optimization method can be to perform lightweight modification on the model through knowledge distillation and structured pruning, while retaining the recognition ability of dialect words and mood words. The separated text can include complete role-separated dialogue text, and complete text of each role itself. Based on the role-separated dialogue text, the BERT-WWM-Ext model extracts context semantic features, and the BERT-WWM-Ext outputs deep semantic features of the dialogue (e.g., 768-dimensional vectors of each word and each sentence) and an attention weight matrix (which records the relationship data of which words each word in the model pays attention to).

[0045] At the same time, an intent recognition model and an interaction analysis model are constructed. The intent recognition model is used to identify identity impersonation (whether impersonating an official, a customer service, a familiar person, or other authoritative roles), sensitive information request (whether requesting a verification code, a password, a bank card number, or other highly sensitive information), behavior induction (whether requiring a transfer, screen sharing, clicking on unknown links, etc.), and logic loopholes (whether the conversation content has unreasonable or contradictory places, etc. 6 types of static fraud signals. The interaction analysis model captures dynamic features such as power suppression (whether one party continuously uses commanding, intimidating, or authoritative language to suppress the other party's questioning or questioning), emotional manipulation (whether the conversation leads to a significant change in the other party's anxiety, fear, or excessive trust), rhythm control (whether one party interrupts the other party's normal thinking by creating a sense of urgency such as time limit, freeze immediately), and induced questioning (whether through leading questions, the other party unknowingly reveals information or agrees to the operation), and generates a 48-dimensional semantic dynamic vector. The deep semantic features are input into the intent recognition model, which is a fine-tuned classification neural network that outputs the probability of the call voice belonging to identity impersonation, sensitive information request, etc. The interaction analysis model performs special analysis and calculation on the attention weight features. For example, whether the attention intensity of the scammer's speech to the victim's speech is overwhelming (power suppression), or whether the attention flow of emotional words is inductive (emotional manipulation). Whether the victim's speech has confusion or compliance. These calculation rules and features to be extracted (such as the power suppression index) are output by the interaction analysis model.

[0046] The two types of results obtained above (static fraud features and dynamic interaction features) are spliced, normalized, etc. Finally, a fixed-length, comprehensive 48-dimensional semantic dynamic vector is formed as a risk representation of the call voice.

[0047] Step 5: Generate a fraud voice recognition model The physiological feature, semantic feature and scene risk feature vectors obtained above are weighted and fused to generate a 192-dimensional multi-modal feature vector. The 192-dimensional multi-modal feature vector is input into a first large model for fine-tuning training, wherein the first large model can select deepseek-r1 as a base model, the decoder of the base model is pruned and optimized through a layer sensitivity analysis algorithm, 16 core feature extraction layers are retained, and INT8 quantization perception training is adopted. A LlamaFactory framework is adopted, and a hybrid fine-tuning strategy based on LoRA+Adapter is adopted for the multi-modal feature vector, and only the parameter update of the attention layer and the output layer of the base model is performed. When the first large model training reaches the convergence condition, the parameter update is stopped, and the trained model is determined as the fraud voice recognition model.

[0048] When the number of new fraud samples is more than 50, incremental fine-tuning is automatically started. An API interface linkage is established with the anti-fraud platform, and the latest fraud script library, high-risk number library and fraud mode features are synchronized every fixed time period, the fraud feature labels and scene risk weights of the semantic analysis model are dynamically updated, and the rapid adaptation to new fraud methods is realized.

[0049] Step 6: Fraud voice recognition model inference The hash value of the call voice with the result label as fraud in the first data set is calculated, and the hash value of the call voice in the first data set, the multi-modal feature vector, the result label and the fraud probability are stored in the cache database (such as Redis) in a vector form, wherein the hash value can be an MD5 value, the hash value is stored in Redis as a key (Key), and the corresponding multi-modal feature vector, result label and fraud probability are stored as a value (Value) in Redis, and a key-value pair is constructed for quick query.

[0050] Get the voice to be recognized, calculate the hash value of the voice to be recognized, if the same hash value is queried in Redis, return the result label and fraud probability corresponding to the hash value; if the same hash value as the voice to be recognized is not retrieved, calculate the multi-modal feature vector of the voice to be recognized, retrieve Top-10 most similar historical multi-modal feature vectors (i.e. multi-modal feature vectors cached in Redis) from Redis based on FAISS, if there is a sample (i.e. multi-modal feature vector) with a similarity ≥ 85%, the call voice is directly judged as high risk, and the result label and fraud probability corresponding to the sample with the highest similarity are output as the recognition result of the voice to be recognized, and the hash value, multi-modal feature vector, result label and fraud probability of the voice to be recognized are stored in the cache database.

[0051] If there is no sample with a similarity of ≥ 85% (i.e., a multi-modal feature vector), the multi-modal feature vector of the speech to be recognized is input into the fraud speech recognition model, the fraud speech recognition model outputs the fraud probability and result label of the speech to be recognized, and in the case of a fraud result label, the hash value, multi-modal feature vector, result label and fraud probability of the speech to be recognized are stored in the cache database. In the actual deployment process, a synchronous + asynchronous processing architecture is adopted to ensure the stability and response speed of the system in a high-concurrency scenario. The architecture includes: (1) Request access and routing layer. This layer serves as the unified entrance of the system, is built based on the Spring WebFlux and Project Reactor framework, adopts a non-blocking, event-driven reactive programming model to achieve efficient processing of high-concurrency HTTP requests. The core responsibility of this layer is request routing and protocol adaptation, supporting two interaction modes: ① Asynchronous direct-through mode. For asynchronous requests initiated by the client, this layer acts as a high-performance reverse proxy, directly routing the request to the backend Web service layer and bridging the response stream to achieve transparent forwarding of requests and responses. ② Synchronous-to-asynchronous mode: To be compatible with traditional synchronous calling clients, this layer implements synchronous-to-asynchronous (Sync-to-Async) interaction adaptation. Request interception and context preservation: When a synchronous request is received, the system generates a unique association ID and temporarily stores the response context (such as ServerHttpResponse) of the original request in a suspended request mapping table in memory. Asynchronous task delivery and callback registration: The system sends the processing task along with its own callback notification address to the Web service layer, and then the original HTTP connection is suspended non-blockingly. Asynchronous response and association recovery: When the backend service completes processing and calls the preset callback address, this layer retrieves and restores the corresponding suspended request context from the mapping table based on the association ID carried in the callback, and writes the final result to the response stream, thus completing the entire synchronous calling loop.

[0052] (2) Web service layer: developed based on the Spring Boot framework, providing standard Web API interfaces. The interface receives requests for speech to be recognized, and the request parameters at least include: the download address (URL) of the speech to be recognized, and the receiving address of the result callback after recognition is completed. (3) Message queue: after the service receives the request, it does not directly perform time-consuming processing, but encapsulates the task information into a message and delivers it to a RocketMQ or other message queue, and immediately returns a "request received" response to the client. This method decouples the service interface and the core processing logic, improving the throughput and fault tolerance of the system.

[0053] It should be noted that, for the foregoing method embodiments, in order to simply describe, they are all expressed as a combination of a series of actions, but those skilled in the art should know that the present application is not limited by the order of the described actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily the application. That is, the embodiments or examples of the present disclosure are not exhaustive, but only a part of the embodiments or examples, and are not specifically limited to the protection scope of the present disclosure. In the case of no contradiction, each step in a certain embodiment or example can be implemented as an independent embodiment, and the steps can be combined arbitrarily, for example, the scheme after removing some steps in a certain embodiment or example can be implemented as an independent embodiment, and the order of the steps in a certain embodiment or example can be exchanged arbitrarily, in addition, the optional mode or optional example in a certain embodiment or example can be combined arbitrarily; in addition, the embodiments or examples can be combined arbitrarily, for example, the steps of different embodiments or examples can be combined arbitrarily, a certain embodiment or example can be combined with the optional mode or optional example of other embodiments or examples.

[0054] According to another aspect of the embodiments of the present application, a large model-based fraud voice recognition device for implementing the large model-based fraud voice recognition method described above is also provided. As shown in the figure, the device comprises: Figure 6 The construction unit 602 is configured to construct a first data set containing call voice, scene data, result label and fraud probability; The first extraction unit 604-1 is configured to extract the scene risk feature vector of the scene data by fine-tuning the trained gradient boosting decision tree model; The separation unit 604-2 is configured to separate the call voice into roles based on an open source library to generate separated voice; The second extraction unit 606-1 is configured to extract the frame-level features of the separated voice according to the optimized mel-frequency cepstral coefficient algorithm, and input the frame-level features into the long short-term memory network for time series modeling to output the physiological feature vector of fixed dimension; The third extraction unit 606-2 is configured to convert the separated voice into separated text, extract deep semantic features and attention weight features of the separated text based on a pre-trained language model, extract static fraud features in the deep semantic features according to an intent recognition model, extract dynamic interaction features in the attention weight features according to an interaction analysis model, and splice and normalize the static fraud features and the dynamic interaction features to generate the semantic feature vector of fixed dimension; ​The fusion unit 608 is configured to fuse and weight the physiological feature vector, the semantic feature vector, and the scene risk feature vector to generate a fixed-dimension multi-modal feature vector; the fine-tuning unit 610 is configured to input the multi-modal feature vector into the first large model for fine-tuning training to generate a fraud speech recognition model; and the acquisition unit 612 is configured to acquire a to-be-recognized speech, input a multi-modal feature vector generated based on the to-be-recognized speech into the fraud speech recognition model, and output a fraud probability and a result label of the to-be-recognized speech.

[0055] The embodiments in the present scheme can, but are not limited to, refer to the above method embodiments, and the present application is not limited in this regard.

[0056] As an optional scheme, the construction unit 602 includes: an acquisition module configured to acquire a second data set containing the call speech, the scene data, and the result label; an input module configured to input the second data set and a preset prompt word into the second large model; a first output module configured to output, by the second large model, the fraud probability of each call speech in the second data set; a first correction module configured to correct, in a case where the result label is the fraud and the fraud probability is lower than 50%, the result label or the fraud probability based on an artificial review result of the call speech; a second correction module configured to correct, in a case where the result label is the normal and the fraud probability is higher than 50%, the result label or the fraud probability based on an artificial review result of the call speech; and a first determination module configured to determine the call speech, the scene data, the result label, and the fraud probability as the first data set, where the preset prompt word is used to prompt the second large model to output the fraud probability of the call speech.

[0057] The embodiments in the present scheme can, but are not limited to, refer to the above method embodiments, and the present application is not limited in this regard.

[0058] As an optional solution, the second extraction unit 606-1 includes: a preprocessing module, configured to perform adaptive noise suppression on the separated voice by using a hybrid noise reduction model combining spectral subtraction and wavelet threshold denoising in a preprocessing stage of the mel-frequency cepstral coefficient algorithm, and dynamically adjust a pre-emphasis coefficient according to a fundamental frequency of the separated voice, wherein the adaptive noise suppression dynamically adjusts noise reduction strength of the spectral subtraction and the wavelet threshold denoising according to a real-time signal-to-noise ratio of the separated voice, and the fundamental frequency is negatively correlated with the pre-emphasis coefficient; a first calculation module, configured to divide a frequency range of a mel filter bank into a physiological feature sensitive area and a semantic feature area, densely set a number of mel filters in the physiological feature sensitive area, sparsely set a number of mel filters in the semantic feature area, and increase a cepstral coefficient related to a physiological feature to expand a cepstral feature vector, wherein the increased cepstral coefficient includes a fundamental frequency and a first-order difference coefficient, a spectral flatness coefficient, and a formant frequency coefficient; and a post-processing module, configured to dynamically weight the cepstral feature vector according to voice activity in a feature post-processing stage of the mel-frequency cepstral coefficient algorithm to obtain the frame-level feature, and input the frame-level feature into the long short-term memory network for time series modeling to output the physiological feature vector of a fixed dimension.

[0059] The embodiments in the present scheme can refer to the above method embodiments, but are not limited thereto, and the present application is not limited thereto.

[0060] As an optional solution, the fine-tuning unit 610 includes: a first acquisition module, configured to acquire the first large model; a pruning module, configured to perform structural pruning on a decoder of the first large model by using a layer sensitivity analysis algorithm; a training module, configured to perform INT8 quantization perception training on the pruned first large model based on the multi-modal feature vector; an updating module, configured to perform parameter updating on an attention layer and an output layer of the first large model according to a hybrid fine-tuning strategy of LoRA and Adapter during the training process; and a second determination module, configured to determine a model generated after fine-tuning training as the fraud voice recognition model.

[0061] The embodiments in the present scheme can refer to the above method embodiments, but are not limited thereto, and the present application is not limited thereto. As an optional solution, the device further includes a storage unit 609, configured to store sample data in which a result label of the first data set is labeled as fraud into a cache database, wherein the sample data includes a hash value of the call voice, the multi-modal feature vector, the result label, and the fraud probability.

[0062] The embodiments in the present scheme can, but are not limited to, referring to the above method embodiments, and the present application embodiments do not make any limitation in this regard.

[0063] As an optional scheme, the above acquisition unit 612 comprises: The second acquisition module is configured to acquire the above-mentioned to-be-recognized speech; The second calculation module is configured to calculate the above-mentioned hash value of the above-mentioned to-be-recognized speech; The second output module is configured to, in the case that the above-mentioned hash value of the above-mentioned to-be-recognized speech is queried in the above-mentioned cache database, output the above-mentioned fraud probability and the above-mentioned result label corresponding to the above-mentioned hash value in the above-mentioned cache database as the recognition result of the above-mentioned to-be-recognized speech; The third acquisition module is configured to, in the case that the above-mentioned hash value of the above-mentioned to-be-recognized speech is not queried in the above-mentioned cache database, acquire the above-mentioned multi-modal feature vector of the above-mentioned to-be-recognized speech, and retrieve Top-K historical multi-modal feature vectors similar to the above-mentioned multi-modal feature vector of the above-mentioned to-be-recognized speech from the above-mentioned cache database based on FAISS; The third output module is configured to, in the case that there is a similarity between the above-mentioned multi-modal feature vector of the above-mentioned to-be-recognized speech and the Top-K historical multi-modal feature vectors exceeding a first preset threshold, output the above-mentioned fraud probability and the above-mentioned result label corresponding to the historical multi-modal feature vector with the highest similarity to the above-mentioned multi-modal feature vector of the above-mentioned to-be-recognized speech in the above-mentioned cache database as the recognition result of the above-mentioned to-be-recognized speech, and store the above-mentioned hash value, the above-mentioned multi-modal feature vector, the above-mentioned result label and the above-mentioned fraud probability of the above-mentioned to-be-recognized speech into the above-mentioned cache database, wherein the historical multi-modal feature vector refers to the above-mentioned multi-modal feature vector stored in the above-mentioned cache database; The fourth output module is configured to, in the case that there is no similarity between the above-mentioned multi-modal feature vector of the above-mentioned to-be-recognized speech and the Top-K historical multi-modal feature vectors exceeding the first preset threshold, input the above-mentioned multi-modal feature vector of the above-mentioned to-be-recognized speech into the fraud speech recognition model to output the above-mentioned fraud probability and the above-mentioned result label of the above-mentioned to-be-recognized speech, and in the case that the above-mentioned result label is fraud, store the above-mentioned hash value, the above-mentioned multi-modal feature vector, the above-mentioned result label and the above-mentioned fraud probability of the above-mentioned to-be-recognized speech into the above-mentioned cache database. The embodiments in the present scheme can, but are not limited to, referring to the above method embodiments, and the present application embodiments do not make any limitation in this regard.

[0064] Optionally, in the present embodiment, the application embodiments of each unit module can refer to the embodiments of the above fraud speech recognition method based on large models, which will not be described here.

[0065] According to an aspect of the present application, a computer readable storage medium is provided, from which a processor of a computer device reads computer instructions, and the processor executes the computer instructions to enable the computer device to perform the steps in any of the method embodiments described above.

[0066] Optionally, in the present embodiment, the computer readable storage medium described above can be configured to store a computer program for performing the following steps: S1, constructing a first data set containing call voice, scene data, result label and fraud probability; S2, extracting a scene risk feature vector of the scene data by fine-tuning the trained gradient boosting decision tree model; S3, performing role separation on the call voice based on an open source library to generate separated voice; S4, extracting frame-level features of the separated voice according to an optimized mel-frequency cepstral coefficient algorithm, and inputting the frame-level features into a long short-term memory network for time series modeling to output a physiological feature vector of a fixed dimension; S5, converting the separated voice into separated text, extracting deep semantic features and attention weight features of the separated text based on a pre-trained language model, extracting static fraud features in the deep semantic features according to an intent recognition model, extracting dynamic interaction features in the attention weight features according to an interaction analysis model, and splicing and normalizing the static fraud features and the dynamic interaction features to generate a semantic feature vector of a fixed dimension; S6, weighting and fusing the physiological feature vector, the semantic feature vector and the scene risk feature vector to generate a multi-modal feature vector of a fixed dimension; S7, inputting the multi-modal feature vector into a first large model for fine-tuning training to generate a fraud voice recognition model; S8, obtaining a voice to be recognized, inputting a multi-modal feature vector generated based on the voice to be recognized into the fraud voice recognition model to output a fraud probability and a result label of the voice to be recognized.

[0067] Optionally, in the present embodiment, a person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above-described embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer readable storage medium, which can include a flash disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0068] According to an aspect of the present application, a computer program product is provided, which comprises computer programs / instructions containing program codes for executing the above method. In such embodiments, the computer program can be downloaded and installed from a network by a communication part, and / or installed from a detachable medium. When the computer program is executed by a central processing unit, various functions provided by embodiments of the present application are executed.

[0069] According to still another aspect of the embodiments of the present application, an electronic device for implementing the above confusion-based text chunking method is also provided. Embodiments of the present application take the electronic device as a server as an example. As shown in Figure 7 The electronic device comprises a memory 702 in which a computer program is stored, and a processor 704 configured to execute the steps in any of the above method embodiments by the computer program.

[0070] Optionally, in the present embodiment, the above electronic device can be located in at least one of a plurality of network devices of a computer network.

[0071] Optionally, in the present embodiment, the above processor can be configured to execute the following steps by the computer program: S1, constructing a first data set containing call voice, scene data, result label, and fraud probability; S2, extracting a scene risk feature vector of the scene data by fine-tuning the trained gradient boosting decision tree model; S3, performing role separation on the call voice based on an open source library to generate separated voice; S4, extracting frame-level features of the separated voice according to an optimized mel-frequency cepstral coefficient algorithm, and inputting the frame-level features into a long short-term memory network for time series modeling to output a physiological feature vector of a fixed dimension; S5, converting the separated voice into separated text, extracting deep semantic features and attention weight features of the separated text based on a pre-trained language model, extracting static fraud features in the deep semantic features according to an intent recognition model, extracting dynamic interaction features in the attention weight features according to an interaction analysis model, and splicing and normalizing the static fraud features and the dynamic interaction features to generate a semantic feature vector of a fixed dimension; S6, weighting and fusing the physiological feature vector, the semantic feature vector, and the scene risk feature vector to generate a multi-modal feature vector of a fixed dimension; S7, inputting the multi-modal feature vector into a first large model for fine-tuning training to generate a fraud voice recognition model; S8, obtaining the to-be-recognized speech, inputting a multi-modal feature vector generated based on the to-be-recognized speech into the fraud speech recognition model to output a fraud probability and a result label of the to-be-recognized speech.

[0072] Optionally, those skilled in the art can understand that, Figure 7 The structure shown is only schematic, and the electronic device can also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a palm computer, a Mobile Internet Device (MID), a PAD, or the like. Figure 7 It does not limit the structure of the above-mentioned electronic device. For example, the electronic device can further include more or fewer components (such as a network interface, etc.) than Figure 7 or have a different configuration from Figure 7 the structure shown.

[0073] The memory 702 can be used to store software programs and modules, such as program instructions / modules corresponding to the text chunking method and device based on perplexity in the embodiments of the present application. The processor 704 performs various functional applications and data processing by running the software programs and modules stored in the memory 702, that is, implements the text chunking method based on perplexity described above. The memory 702 can include a high-speed random access memory, and can also include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some examples, the memory 702 can further include a memory remotely arranged with respect to the processor 704, which can be connected to the terminal through a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof. The memory 702 can specifically but not limited to be used to store input text and the like. As an example, as ​ shown, the memory 702 can but not limited to include the construction unit 602, the first extraction unit 604-1, the separation unit 604-2, the second extraction unit 606-1, the third extraction unit 606-2, the fusion unit 608, the fine-tuning unit 610, and the acquisition unit 612 in the fraud speech recognition device based on a large model. In addition, other module units in the fraud speech recognition device based on a large model can also be included but not limited to, which will not be described in detail in this example.

[0074] In addition, the electronic device further includes a connection bus 706 for connecting various module components in the electronic device.

[0075] In other embodiments, the server described above can be a node in a distributed system, where the distributed system can be a blockchain system, which can be a distributed system formed by the plurality of nodes connected in the form of network communication. Among them, the nodes can form a peer-to-peer (P2P, Peer To Peer) network, and any form of computing device, such as a server, terminal, and other electronic devices, can become a node in the blockchain system by joining the peer-to-peer network.

[0076] The integrated units in the above embodiments, if realized in the form of software functional units and sold or used as independent products, can be stored in the above computer-readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in part, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a number of instructions for causing one or more computer devices (which can be personal computers, servers, or network devices, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present application.

[0077] In the above embodiments of the present application, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0078] In the several embodiments provided by the present application, it should be understood that the disclosed client can be implemented in other ways. Among them, the above-mentioned device embodiments are only schematic, for example, the division of the units is only a logical function division, and in actual implementation, another division mode can be adopted, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the shown or discussed mutual ones can be indirect coupling or communication connection through some interfaces, units or modules, and can be electrical or other forms.

Claims

1. A fraud speech recognition method based on a large model, characterized in that, include: Construct a first dataset that includes call voice recordings, scenario data, result labels, and fraud probabilities; The scenario risk feature vector of the scenario data is extracted by fine-tuning the trained gradient boosting decision tree model. The call audio is role-separated using an open-source library to generate separate audio. The frame-level features of the separated speech are extracted using the optimized Mel frequency cepstral coefficient algorithm, and the frame-level features are input into a long short-term memory network for temporal modeling to output a fixed-dimensional physiological feature vector. The separated speech is converted into separated text. Deep semantic features and attention weight features of the separated text are extracted based on a pre-trained language model. Static fraud features are extracted from the deep semantic features based on an intent recognition model. Dynamic interaction features are extracted from the attention weight features based on an interaction analysis model. The static fraud features and the dynamic interaction features are concatenated and normalized to generate a fixed-dimensional semantic feature vector. The physiological feature vector, the semantic feature vector, and the scene risk feature vector are weighted and fused to generate a fixed-dimensional multimodal feature vector. The multimodal feature vectors are input into the first large model for fine-tuning training to generate a fraud speech recognition model; The system acquires the speech to be identified, inputs the multimodal feature vector generated based on the speech to be identified into the fraud speech recognition model, and outputs the fraud probability and the result label of the speech to be identified. The scenario data is used to represent the background information of the call voice, the result label indicates fraud or normal, the intent recognition model is a fine-tuned classification neural network, the interaction analysis model has preset calculation rules, and the gradient boosting decision tree model is fine-tuned based on the scenario data and the result label.

2. The method according to claim 1, characterized in that, The first dataset, which includes call voice recordings, scenario data, result labels, and fraud probabilities, comprises: Collect a second dataset containing the call audio, the scene data, and the result labels; Input the second dataset and the preset prompts into the second large model; The second large model outputs the fraud probability for each of the aforementioned voice calls in the second dataset; If the result label is fraud and the fraud probability is less than 50%, the result label or the fraud probability is corrected based on the results of manual review of the call audio. If the result label is "normal" and the fraud probability is higher than 50%, the result label or the fraud probability is corrected based on the results of manual review of the call audio. The call audio, the scene data, the result label, and the fraud probability are determined as the first dataset, wherein the preset prompt word is used to prompt the second model to output the fraud probability of the call audio.

3. The method according to claim 1, characterized in that, The step of extracting frame-level features of the separated speech using the optimized Mel-frequency cepstral coefficient algorithm, and inputting the frame-level features into a long short-term memory network for temporal modeling to output a fixed-dimensional physiological feature vector includes: In the preprocessing stage of the Mel frequency cepstral coefficient algorithm, a hybrid noise reduction model combining spectral subtraction and wavelet threshold denoising is used to adaptively suppress noise in the separated speech, and the pre-emphasis coefficient is dynamically adjusted according to the fundamental frequency of the separated speech. The adaptive noise suppression refers to dynamically adjusting the noise reduction intensity of the spectral subtraction and wavelet threshold denoising according to the real-time signal-to-noise ratio of the separated speech. The fundamental frequency is negatively correlated with the pre-emphasis coefficient. In the filtering and cepstral calculation stage of the Mel frequency cepstral coefficient algorithm, the frequency range of the Mel filter bank is divided into a physiological feature sensitive region and a semantic feature region. In the physiological feature sensitive region, the number of Mel filters is increased. In the semantic feature region, the number of Mel filters is decreased. Cepstral coefficients related to physiological features are added to expand the cepstral feature vector. The added cepstral coefficients include the fundamental frequency and its first-order difference coefficient, spectral flatness coefficient, and formant frequency coefficient. In the feature post-processing stage of the Mel frequency cepstral coefficient algorithm, the cepstral feature vector is dynamically weighted according to speech activity to obtain the frame-level features. The frame-level features are then input into the long short-term memory network for temporal modeling to output the physiological feature vector with fixed dimensions.

4. The method according to claim 1, characterized in that, The step of inputting the multimodal feature vector into the first large model for fine-tuning training to generate a fraud speech recognition model includes: Obtain the first large model; perform structured pruning on the decoder of the first large model using a layer sensitivity analysis algorithm; perform INT8 quantization perceptual training on the pruned first large model based on the multimodal feature vector; update the parameters of the attention layer and output layer of the first large model during training according to the hybrid fine-tuning strategy of LoRA and Adapter; determine the model generated after fine-tuning training as the fraud speech recognition model.

5. The method according to claim 1, characterized in that, Before inputting the multimodal feature vector into the first large model for fine-tuning training to generate the fraud speech recognition model, the method further includes: The sample data in the first dataset that are labeled as fraud are vectorized and stored in the cache database, wherein the sample data includes the hash value of the call voice, the multimodal feature vector, the result label, and the fraud probability.

6. The method according to claim 1, characterized in that, The process of acquiring the speech to be identified, and inputting the multimodal feature vector generated based on the speech to be identified into the fraud speech recognition model to output the fraud probability and the result label of the speech to be identified, includes: Obtain the speech to be recognized; Calculate the hash value of the speech to be recognized; If the hash value of the speech to be recognized is found in the cache database, the fraud probability corresponding to the hash value in the cache database and the result label are output as the recognition result of the speech to be recognized. If the hash value of the speech to be recognized is not found in the cache database, the multimodal feature vector of the speech to be recognized is obtained, and the Top-K historical multimodal feature vectors similar to the multimodal feature vector of the speech to be recognized are retrieved from the cache database based on FAISS. If, among the Top-K historical multimodal feature vectors, there exists a multimodal feature vector whose similarity to the multimodal feature vector of the speech to be recognized exceeds a first preset threshold, the fraud probability and the result label corresponding to the historical multimodal feature vector with the highest similarity to the multimodal feature vector of the speech to be recognized in the cache database are output as the recognition result of the speech to be recognized. The hash value, the multimodal feature vector, the result label, and the fraud probability of the speech to be recognized are stored in the cache database, wherein the historical multimodal feature vector refers to the multimodal feature vector stored in the cache database. If none of the Top-K historical multimodal feature vectors have a similarity exceeding a first preset threshold with the multimodal feature vector of the speech to be identified, the multimodal feature vector of the speech to be identified is input into the fraud speech recognition model to output the fraud probability and the result label of the speech to be identified. If the result label is fraud, the hash value, the multimodal feature vector, the result label, and the fraud probability of the speech to be identified are stored in the cache database.

7. A fraud speech recognition device based on a large model, characterized in that, include: The building unit is used to construct the first dataset, which includes call voice, scene data, result labels, and fraud probability. The first extraction unit is used to extract the scene risk feature vector of the scene data by fine-tuning the trained gradient boosting decision tree model. A separation unit is used to perform role separation on the call voice based on an open-source library to generate separated voice; The second extraction unit is used to extract the frame-level features of the separated speech according to the optimized Mel frequency cepstral coefficient algorithm, and input the frame-level features into a long short-term memory network for temporal modeling to output a fixed-dimensional physiological feature vector. The third extraction unit is used to convert the separated speech into separated text, extract deep semantic features and attention weight features of the separated text based on a pre-trained language model, extract static fraud features from the deep semantic features based on an intent recognition model, extract dynamic interaction features from the attention weight features based on an interaction analysis model, and concatenate and normalize the static fraud features and the dynamic interaction features to generate a fixed-dimensional semantic feature vector. The fusion unit is used to weightedly fuse the physiological feature vector, the semantic feature vector, and the scene risk feature vector to generate a fixed-dimensional multimodal feature vector; The fine-tuning unit is used to input the multimodal feature vector into the first large model for fine-tuning training to generate a fraud speech recognition model; The acquisition unit is used to acquire the speech to be identified, input the multimodal feature vector generated based on the speech to be identified into the fraud speech recognition model, and output the fraud probability and the result label of the speech to be identified; The scenario data is used to represent the background information of the call voice, the result label indicates fraud or normal, the intent recognition model is a fine-tuned classification neural network, the interaction analysis model has preset calculation rules, and the gradient boosting decision tree model is fine-tuned based on the scenario data and the result label.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program is executed by a processor to perform the method described in any one of claims 1 to 6.

9. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the method described in any one of claims 1 to 6 through the computer program.

Images