Speaker log-based fraud early warning method and system, and server

Abstract

The invention discloses a fraud early warning method based on a speaker log, and the method comprises the following steps: 1, carrying out the preprocessing of a call audio, segmenting the call audio through a VAD model, and keeping the voice content; 2, voiceprint feature information is extracted from the voice content, a speaker is confirmed, and a speaker log is generated; 3, respectively inputting different speaker logs into the semantic recognition model, and respectively judging whether the scenes are fraud scenes or not; and 4, if the speaker of one party is identified to be in the fraud scene, sending early warning information to the speaker of the other party. According to the method, the respective call logs of the two talkers in the call audio can be generated, the identities and the fraud types of the two talkers are distinguished through a text classification method, and finally victims are reminded through short messages or frauders are tracked and alarmed, so that the telecommunication fraud can be conveniently and timely intervened and prevented, and the user experience is improved. Or anti-fraud education and the like are carried out after the conversation is finished.

Description

Technical Field

[0001] The present application belongs to the field of information security technology, and in particular relates to a fraud warning method, system and server based on a speaker log. Background Art

[0002] In recent years, the mobile information service industry based on mobile communication networks has been welcomed by the majority of users due to its convenience and fast service. However, with the rapid development of mobile communications, the complexity of the call environment has made the types of telephone fraud complex and diverse. These difficult to identify, complex and large-scale fraud calls have caused great interference to the work and life of the majority of mobile phone users, had a negative impact on the corporate image of China Unicom operators, and also brought great difficulties to government supervision.

[0003] The existing technology can only perform an overall analysis of the content of the call. Due to the diversity of fraud methods and the concealment of the fraudsters' rhetoric, it is difficult for the existing anti-fraud technology to distinguish between fraudsters and victims from the content of the call, resulting in the inability to carry out targeted treatment of fraudsters and victims, such as tracking and reporting fraudsters, reminding and educating victims, etc.

[0004] For example, if the content of the call is "Hello, I am a staff member of the Bank of China. Your bank card was stolen this morning. Do you know?", the existing technical solutions may score or mark the call content as high risk, but cannot identify whether the call content is spoken by the fraudster or the victim, so it is impossible to take targeted anti-fraud measures against the fraudster and the victim. This leads to problems such as false detection, missed detection, and poor detection effect of the anti-fraud early warning system.

[0005] Therefore, a technical solution for call content fraud warning is needed to solve the above problems. Summary of the invention

[0006] In order to address the deficiencies of the prior art, the present application provides a fraud warning method, system and server based on speaker logs, which can generate separate call logs for the two callers in the call audio, and then distinguish the identities and fraud types of the two interlocutors through text classification methods, that is, the fraudsters and victims, and the types of fraud used by the fraudsters, so as to facilitate operators to identify fraud scenarios and fraud processes as well as the risks of calls, etc. Finally, victims can be reminded through text messages, etc., or fraudsters can be tracked and reported to the police, so as to facilitate timely intervention and prevention of telecommunications fraud, or anti-fraud education can be carried out after the call.

[0007] The technical effects to be achieved by this application are achieved through the following solutions:

[0008] According to a first aspect of the present application, a fraud warning method based on a speaker log is provided, comprising the following steps:

[0009] Step 1: After preprocessing the call audio, segment the call audio using a VAD model to retain the voice content;

[0010] Step 2: extracting voiceprint feature information from the voice content, confirming the speaker and generating a speaker log;

[0011] Step 3: Input different speaker logs into the semantic recognition model to determine whether it is a fraud scenario;

[0012] Step 4: If one speaker is identified as a fraudulent scenario, a warning message is sent to the other speaker.

[0013] Preferably, in step 1, preprocessing the call audio includes: converting the call audio format, audio downsampling, and audio noise reduction.

[0014] Preferably, the VAD model includes a feature extraction part and a speech judgment part. The feature extraction part uses an energy-based method to extract features, and the speech judgment part uses a threshold-based method to judge speech content or non-speech content according to the extracted features, and retains the speech content after cutting off the non-speech content.

[0015] Preferably, the specific method of determining voice content or non-voice content in a threshold-based manner is as follows:

[0016] Using a feature extraction part to extract features representing speech features from the speech content, the features including energy, zero crossing rate, spectrum characteristics and Mel frequency cepstral coefficients;

[0017] The speech judgment part is used to compare the feature with a preset threshold. If the feature meets the threshold condition, the speech content is judged to be a speech part, otherwise it is a non-speech part.

[0018] Preferably, in step 2, a speaker recognition model of a densely connected time-delay neural network is used to extract voiceprint feature information. The speaker recognition model of the densely connected time-delay neural network includes four residual convolutional networks at the front end and three layers of D-TDNN time-delay neural network layers at the back end. After clustering the voiceprint feature vectors output by the speaker recognition model of the densely connected time-delay neural network, different speakers are distinguished.

[0019] Preferably, the D-TDNN time-delay neural network layer includes five parts, wherein:

[0020] The first part is layer 1, which is used to initialize the channel;

[0021] The second part is layers 2 to 8, which are D-TDNN blocks with frame offset values ​​set to 1, used to learn local features;

[0022] The third part is layers 9 to 21, which is another D-TDNN block with the frame offset value set to 3, used to capture the long-term dependencies between speech frames;

[0023] The fourth part is the 22nd layer, which is used to aggregate frame-level features and output speech features;

[0024] The fifth part is the 23rd layer, which is used to output the speaker embedding vector to represent the speech characteristics of the speaker.

[0025] Preferably, after the speaker is confirmed, the audio clips of different speakers are converted into text using an ASR model, and the generated text is divided into speaker logs of different speakers according to different voiceprint features.

[0026] Preferably, different speaker logs are respectively input into the semantic recognition model, and the semantic recognition model extracts the semantic information in the speaker log to determine whether it is a fraud scenario. If the current speaker log is a fraud scenario, the speaker is marked as a fraudster, and the other speaker is marked as a victim; anti-fraud information corresponding to the scenario is sent to the victim via text message or flash message.

[0027] According to a second aspect of the present application, a fraud warning system using the above-mentioned fraud warning method based on speaker log is provided, comprising:

[0028] An audio processing module, used to pre-process the call audio, segment the call audio through a VAD model, and retain the voice content;

[0029] A log generation module, used to extract voiceprint feature information from the voice content, identify the speaker and generate a speaker log;

[0030] The fraud scene recognition module is used to input different speaker logs into the semantic recognition model to determine whether it is a fraud scene;

[0031] The early warning sending module is used to send early warning information to victims in fraud scenarios.

[0032] According to a third aspect of the present application, there is provided a server, comprising: a memory and at least one processor;

[0033] The memory stores a computer program, and the at least one processor executes the computer program stored in the memory to implement the above-mentioned fraud warning method based on the speaker log.

[0034] Through an embodiment of the present application, the beneficial effect of using the fraud warning method based on the speaker log of the present application is that: by processing the voiceprint information of the call audio, the audio data of the two interlocutors in the call can be accurately extracted, thereby improving the recognition effect in the field of anti-fraud;

[0035] By performing semantic analysis on the text converted from the audio, the fraud scenario and the speaker role can be determined, and a text log containing the call role information can be generated, which effectively solves the shortcomings of the existing technical solutions that cannot distinguish between the two interlocutors in a call and cannot monitor and feedback the fraud process, improves the accuracy and timeliness of anti-fraud, and enhances the pertinence and effectiveness of anti-fraud;

[0036] The concept of conversation segments is adopted to divide the call log into several conversation segments. Each conversation segment contains the speech content of a conversation partner, which makes it easier to perform speaker recognition and semantic analysis on each conversation segment, improve the meticulousness and comprehensiveness of anti-fraud, and enhance the explainability and comprehensibility of anti-fraud. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present application or the existing technical solutions, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0038] Figure 1 This is a flow chart of a fraud warning method based on a speaker log in one embodiment of the present application;

[0039] Figure 2 This is a structural block diagram of a fraud warning system based on a speaker log in an embodiment of the present application;

[0040] Figure 3 This is a structural block diagram of a server in an embodiment of the present application. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0042] like Figure 1 As shown, a fraud warning method based on a speaker log in an embodiment of the present application includes the following steps:

[0043] Step 1: After preprocessing the call audio, segment the call audio using a VAD model to retain the voice content;

[0044] In this step, the call audio preprocessing includes: converting the call audio format, downsampling the audio, and reducing the audio noise, so as to ensure the uniform format of the input data and speed up the processing speed and accuracy of the model in the subsequent steps.

[0045] The VAD model is used to segment the preprocessed call audio, remove non-voice segments in the call audio, and retain the voice content, thereby further speeding up the processing speed and accuracy in subsequent steps.

[0046] The VAD model (Voice Activity Detection) usually includes two parts, a feature extraction part and a speech / non-speech judgment part. The feature extraction part uses an energy-based method to extract features. The speech judgment part uses a threshold-based method to judge speech content or non-speech content based on the extracted features, and removes the non-speech content and retains the speech content. Specifically:

[0047] Using a feature extraction part to extract features representing speech features from the speech content, the features including energy, zero crossing rate, spectrum characteristics and Mel frequency cepstral coefficients;

[0048] The speech judgment part is used to compare the features with the preset thresholds; in this step, one or more judgment thresholds are set, which are usually determined based on experience, experimental data or the needs of specific application scenarios. For example, an energy threshold can be set, and when the energy of the input signal exceeds this threshold, it is considered to be speech; or a zero-crossing rate threshold can be set, and when the zero-crossing rate is within a certain range, it is considered to be speech. In other embodiments, the threshold can be set for judgment according to spectral characteristics and other methods.

[0049] If the feature meets the threshold condition, the speech content is determined to be a speech part, otherwise it is a non-speech part.

[0050] The threshold-based method is used to judge the speech content, which is simple to calculate and easy to implement. Moreover, since the amount of calculation is relatively small, this method is suitable for real-time applications.

[0051] Step 2: extracting voiceprint feature information from the voice content, confirming the speaker and generating a speaker log;

[0052] In this step, the speaker recognition model of densely connected time-delay neural network is used to extract voiceprint feature information. The speaker recognition model of densely connected time-delay neural network includes four residual convolutional networks at the front end and three layers of D-TDNN time-delay neural network layers at the back end. The voiceprint feature vectors output by the speaker recognition model of densely connected time-delay neural network are clustered to distinguish different speakers. Compared with the mainstream ResNet34 model and ECAPA-TDNN model, it has better speaker recognition performance and faster reasoning speed, and can quickly generate speaker logs to achieve the purpose of rapid early warning.

[0053] By using multi-granularity pooling network operations, the model can extract contextual information of the audio at different scales. The generated mask can remove irrelevant noise in the features and retain key speaker information.

[0054] The above D-TDNN time-delay neural network layer consists of five parts, among which:

[0055] The first part is layer 1, which is used to initialize the channel;

[0056] The second part is layers 2 to 8, which are D-TDNN blocks with frame offset values ​​set to 1, used to learn local features;

[0057] The third part is layers 9 to 21, which is another D-TDNN block with the frame offset value set to 3, used to capture the long-term dependencies between speech frames;

[0058] The fourth part is the 22nd layer, which is used to aggregate frame-level features and output speech features;

[0059] The fifth part is the 23rd layer, which is used to output the speaker embedding vector to represent the speech characteristics of the speaker.

[0060] After the input audio data is processed by the speaker verification model, the voiceprint feature vector output by the model is clustered to distinguish between speaker A and speaker B.

[0061] In order to generate speaker logs, the audio clips that have been clustered by speaker confirmation are converted into text using the ASR model, and the generated text is divided into speaker logs of two people, A and B, according to different voiceprint features. For example:

[0062] Caller A:

[0063] A: Hello, I am a customer service representative from JD.com. Our system shows that you have an application credit limit agreement. If you do not cancel it, it will have a great impact on your credit report and may even cause financial losses.

[0064] A: You need to download a conference app called "zoom", and then I will remotely operate your phone through this app to help you cancel this agreement.

[0065] A: Now I need you to open the mobile banking and follow the steps I told you. We need to verify your bank account balance.

[0066] Caller B:

[0067] B: Oh, how do I cancel it?

[0068] B: OK, I'll download it now.

[0069] B: OK, I will do it now.

[0070] Step 3: Input different speaker logs into the semantic recognition model to determine whether it is a fraud scenario;

[0071] In this step, after the audio is processed using the speaker confirmation model, the logs of the two speakers AB are respectively input into the semantic recognition model, and the semantic information in the speaker logs is extracted to determine whether it is a fraud scene. If it is a fraud scene, speakers AB are marked as fraudsters and victims based on semantic features, and the type of fraud scene is determined, such as rebate for fake orders, telecommunications fraud, etc.

[0072] The semantic recognition model can distinguish fraud scenarios based on traditional recognition methods such as keywords and word vector matching, or it can be based on new methods such as large-model question answering and deep learning for semantic recognition.

[0073] Step 4: If one speaker is identified as a fraudulent scenario, a warning message is sent to the other speaker.

[0074] In this step, if the current speaker log is a fraud scenario, the speaker is marked as a fraudster, and the other speaker is marked as a victim; anti-fraud information of the corresponding scenario is sent to the victim via SMS or flash message to remind the victim to be vigilant and prevent fraud.

[0075] like Figure 2 As shown, in one embodiment of the present application, a fraud warning system using the above-mentioned fraud warning method based on speaker logs includes:

[0076] An audio processing module, used to pre-process the call audio, segment the call audio through a VAD model, and retain the voice content;

[0077] The VAD model in this module identifies speech segments and non-speech segments from speech (clean or noisy) signals, and performs segmentation processing, retaining the speech content for subsequent recognition.

[0078] A log generation module, used to extract voiceprint feature information from the voice content, identify the speaker and generate a speaker log;

[0079] This module is equipped with a speaker recognition model of a densely connected time-delay neural network, through which voiceprint feature information is extracted. After the voiceprint features are extracted, the segmented speech content is distinguished, different speakers are identified, and the speech content of different speakers is clustered.

[0080] The clustered audio segments are converted to text using the ASR model to generate speaker logs. The ASR model can achieve high-accuracy speech recognition through deep learning and large-scale data training. The ASR model can use contextual information such as historical conversations to further improve the recognition accuracy of text, making it suitable for use in calls.

[0081] The fraud scene recognition module is used to input different speaker logs into the semantic recognition model to determine whether it is a fraud scene;

[0082] For example, the semantic recognition model in this module is:

[0083] The N-gram model can better capture local vocabulary and phrase information in the text, and has a good recognition effect on some common, fixed collocation expressions;

[0084] Neural network models (such as convolutional neural network (CNN), recurrent neural network (RNN), long short-term memory (LSTM), etc.) can automatically learn and extract features from texts and have a high ability to recognize complex semantic relationships and patterns.

[0085] Transformer architecture: Compared with traditional recurrent neural networks, Transformer can better utilize the parallel computing capabilities of modern computing hardware to improve the speed of training and reasoning;

[0086] Unet model, which can achieve high-precision segmentation in many cases. Unet has fewer parameters and trainable variables, which enables it to achieve good performance even on small sample data;

[0087] The ERN IE model can accurately identify entities in text and provide a basis for subsequent semantic analysis.

[0088] The above semantic recognition models can be used in combination as needed to achieve better semantic recognition effect.

[0089] The early warning sending module is used to send early warning information to victims in fraud scenarios.

[0090] The warning information is, for example, text messages, flash messages, multimedia messages, etc., which can enable the deceived person to receive the warning information during a call (during the implementation of the fraud); a special warning center can also be set up, and when a fraud scenario is identified, the staff of the warning center or the automatic dialing method will issue a telephone warning, or directly help the victim to report the crime.

[0091] like Figure 3 As shown, the server in an embodiment of the present application includes: a memory 301 and at least one processor 302;

[0092] The memory 301 stores a computer program, and the at least one processor 302 executes the computer program stored in the memory 301 to implement the above-mentioned fraud warning method based on the speaker log.

[0093] Through the above embodiments, the beneficial effect of the fraud warning method based on speaker log of the present application is that: by processing the voiceprint information of the call audio, the audio data of the two interlocutors in the call can be accurately extracted, thereby improving the recognition effect in the field of anti-fraud;

[0094] By performing semantic analysis on the text converted from the audio, the fraud scenario and the speaker role can be determined, and a text log containing the call role information can be generated, which effectively solves the shortcomings of the existing technical solutions that cannot distinguish between the two interlocutors in a call and cannot monitor and feedback the fraud process, improves the accuracy and timeliness of anti-fraud, and enhances the pertinence and effectiveness of anti-fraud;

[0095] The concept of conversation segments is adopted to divide the call log into several conversation segments. Each conversation segment contains the speech content of a conversation partner, which makes it easier to perform speaker recognition and semantic analysis on each conversation segment, improve the meticulousness and comprehensiveness of anti-fraud, and enhance the explainability and comprehensibility of anti-fraud.

[0096] It should be noted that the above detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the art to which the present application belongs.

[0097] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should also be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0098] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein.

[0099] In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or inherent to these processes, methods, products, or apparatuses.

[0100] For ease of description, spatially relative terms, such as "above", "above", "on the upper surface of", "above", etc., may be used herein to describe the spatial positional relationship between a device or feature and other devices or features as shown in the figure. It should be understood that spatially relative terms are intended to include different orientations of the device in use or operation in addition to the orientation described in the figure. For example, if the device in the accompanying drawings is inverted, the device described as "above other devices or structures" or "above other devices or structures" will be positioned as "below other devices or structures" or "below other devices or structures". Thus, the exemplary term "above" may include both "above" and "below". The device may also be positioned in other different ways, such as rotated 90 degrees or in other orientations, and the spatially relative descriptions used herein are interpreted accordingly.

[0101] In the above detailed description, reference is made to the accompanying drawings, which form a part of this document. In the accompanying drawings, similar symbols typically identify similar components unless the context indicates otherwise. The illustrated embodiments described in the detailed description, drawings, and claims are not meant to be limiting. Other embodiments may be used, and other changes may be made, without departing from the spirit or scope of the subject matter presented herein.

[0102] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A fraud warning method based on a speaker log, characterized in that: The steps include: Step 1: After preprocessing the call audio, segment the call audio using a VAD model to retain the voice content; Step 2: extracting voiceprint feature information from the voice content, confirming the speaker and generating a speaker log; Step 3: Input different speaker logs into the semantic recognition model to determine whether it is a fraud scenario; Step 4: If one speaker is identified as a fraudulent scenario, a warning message is sent to the other speaker.

2. The fraud warning method based on speaker log according to claim 1 is characterized in that: In step 1, preprocessing the call audio includes: converting the call audio format, downsampling the audio, and reducing audio noise.

3. The fraud warning method based on speaker log according to claim 1 is characterized in that: The VAD model includes a feature extraction part and a speech judgment part. The feature extraction part uses an energy-based method to extract features. The speech judgment part uses a threshold-based method to judge the speech part or the non-speech part according to the extracted features, and cuts off the non-speech part and retains the speech part.

4. The fraud warning method based on speaker log according to claim 3 is characterized in that: The specific method of using a threshold-based method to determine speech content or non-speech content is as follows: Using a feature extraction part to extract features representing speech features from the speech content, the features including energy, zero crossing rate, spectrum characteristics and Mel frequency cepstral coefficients; The speech judgment part is used to compare the feature with a preset threshold. If the feature meets the threshold condition, the speech content is judged to be a speech part, otherwise it is a non-speech part.

5. The fraud early warning method based on speaker log according to claim 1 is characterized in that: In step 2, a speaker recognition model of a densely connected time-delay neural network is used to extract voiceprint feature information. The speaker recognition model of the densely connected time-delay neural network includes four residual convolutional networks at the front end and three layers of D-TDNN time-delay neural network layers at the back end. After clustering the voiceprint feature vectors output by the speaker recognition model of the densely connected time-delay neural network, different speakers are distinguished.

6. The fraud warning method based on speaker log according to claim 5 is characterized in that: The D-TDNN time-delay neural network layer includes five parts, among which: The first part is layer 1, which is used to initialize the channel; The second part is layers 2 to 8, which are D-TDNN blocks with frame offset values ​​set to 1, used to learn local features; The third part is layers 9 to 21, which is another D-TDNN block with the frame offset value set to 3, used to capture the long-term dependencies between speech frames; The fourth part is the 22nd layer, which is used to aggregate frame-level features and output speech features; The fifth part is the 23rd layer, which is used to output the speaker embedding vector to represent the speech characteristics of the speaker.

7. The fraud warning method based on speaker log according to claim 1 is characterized in that: After the speaker is confirmed, the audio clips of different speakers are converted into text using the ASR model, and the generated text is divided into speaker logs of different speakers according to different voiceprint features.

8. The fraud warning method based on speaker log according to claim 7 is characterized in that: Different speaker logs are input into the semantic recognition model respectively, and the semantic recognition model extracts the semantic information in the speaker log to determine whether it is a fraud scene. If the current speaker log is a fraud scene, the speaker is marked as a fraudster, and the other speaker is marked as a victim. Send anti-fraud information for the corresponding scenario to the victim via SMS or flash message.

9. A fraud warning system using the fraud warning method based on speaker log according to any one of claims 1 to 8, characterized in that: include: An audio processing module, used to pre-process the call audio, segment the call audio through a VAD model, and retain the voice content; A log generation module, used to extract voiceprint feature information from the voice content, identify the speaker and generate a speaker log; The fraud scene recognition module is used to input different speaker logs into the semantic recognition model to determine whether it is a fraud scene; The early warning sending module is used to send early warning information to victims in fraud scenarios.

10. A server, characterized in that: include: memory and at least one processor; The memory stores a computer program, and the at least one processor executes the computer program stored in the memory to implement the fraud warning method based on the speaker log as described in any one of claims 1 to 8.

Images