A call detection method and system based on multi-modal feature fusion

Abstract

The application relates to the technical field of communication security, in particular to a call detection method and system based on multi-modal feature fusion, which comprises the following steps: collecting voice signals, video signals and / or communication metadata in real time during a call process of a first communication party and a second communication party; in a first target scene, obtaining a voice signal stream corresponding to the voice signals, a video signal stream corresponding to the video signals and recently collected communication metadata; the first target scene refers to an abnormal call scene with risks determined based on voice keywords; performing feature extraction and feature fusion on the voice signal stream, the video signal stream and the recently collected communication metadata to obtain multi-modal features; and inputting the multi-modal features into a deep learning model to obtain suspicious behavior detection results. It can be seen that the method can realize real-time and accurate detection of suspicious behaviors in calls through multi-modal data fusion, effectively make up for the limitations of single-modal detection, and has high identification timeliness and accuracy.

Description

Technical Field

[0001] This application relates to the field of communication security technology, and in particular to a call detection method and system based on multimodal feature fusion. Background Technology

[0002] With the popularization of communication technology and the diversification of communication scenarios, fraudsters are no longer limited to simple persuasive language, but widely use various technical tools to enhance their deceptiveness.

[0003] Traditional single-modal anti-fraud solutions typically rely on only two core methods: one is to match keywords in the content of voice calls to identify fraudulent calls; the other is to use number tagging mechanisms to assess the risk of call numbers.

[0004] It is evident that traditional single-modal anti-fraud solutions are limited in scope and cannot identify new and complex fraud schemes such as those using virtual numbers for spoofing. This results in high rates of missed and false detections, making it difficult to form effective prevention and control measures and posing a serious threat to users' property security. Summary of the Invention

[0005] This application provides a call detection method and system based on multimodal feature fusion to solve the technical problem of the limited range of traditional single-modal call detection methods.

[0006] In a first aspect, embodiments of this application provide a call detection method based on multimodal feature fusion. The method includes: real-time acquisition of voice signals, video signals, and / or communication metadata during a call between a first communicating party and a second communicating party; wherein the call includes video calls, VoLTE calls, and 5G new calls; in a first target scenario, acquiring the voice signal stream corresponding to the voice signal, the video signal stream corresponding to the video signal, and the recently acquired communication metadata; the first target scenario refers to an abnormal call scenario where risks are judged based on voice keywords; performing feature extraction and feature fusion on the voice signal stream, video signal stream, and recently acquired communication metadata to obtain multimodal features; and inputting the multimodal features into a deep learning model to obtain suspicious behavior detection results.

[0007] In one possible implementation, the suspicious behavior detection result includes a risk score; after the step of inputting multimodal features into a deep learning model to obtain the suspicious behavior detection result, the method further includes: determining whether the risk score is greater than a preset threshold; if the risk score is greater than the preset threshold, outputting a blocking signal, which is used to block the call between the first communication party and the second communication party; if the risk score is less than or equal to the preset threshold, outputting a warning signal, which is used to alert the first communication party that there is a risk in the call.

[0008] In one possible implementation, feature extraction and feature fusion are performed on the speech signal stream, video signal stream, and recently acquired communication metadata to obtain multimodal features. This includes: time-aligning the speech signal stream, video signal stream, and communication metadata with the time axis of the first communication party as a reference; extracting the speech feature vector corresponding to the speech signal stream; extracting the video feature vector corresponding to the video signal stream; and extracting the communication feature vector corresponding to the recently acquired communication metadata; and using a feature fusion formula to perform weighted fusion of the speech feature vector, video feature vector, and communication feature vector to obtain multimodal features.

[0009] In one possible implementation, extracting the speech feature vector corresponding to the speech signal stream includes: denoising the speech signal stream to obtain a denoised speech signal stream; performing frame segmentation and windowing on the denoised speech signal stream to obtain a windowed speech signal stream; performing short-time Fourier transform on the windowed speech signal stream frame by frame to obtain a spectral feature sequence; calculating the power spectrum on the spectral feature sequence frame by frame to obtain a power spectrum sequence; and extracting Mel-frequency cepstral coefficients from the power spectrum sequence to obtain the speech feature vector.

[0010] In one possible implementation, extracting the video feature vector corresponding to the video signal stream includes: decoding the video signal stream frame by frame to obtain the original image frame sequence; using a face detection model to perform face detection and tracking on the original image frame sequence to obtain a face region coordinate sequence; extracting key points from the face region coordinate sequence to obtain a face key point coordinate sequence; using a depth feature extraction model to perform facial expression recognition on the face key point coordinate sequence, extracting facial expression features to obtain a facial expression feature vector; using optical flow to perform motion detection on the original image frame sequence, extracting motion features to obtain a motion feature vector; and concatenating or weighting the facial expression feature vector with the motion feature vector to form a video feature vector.

[0011] In one possible implementation, the communication metadata includes the call number, call duration, collection timestamp, and / or calling and called party identifiers; extracting the communication feature vector corresponding to the most recently collected communication metadata includes: cleaning the most recently collected communication metadata to obtain cleaned communication metadata; extracting features from the cleaned communication metadata to obtain call duration features, number location features, and timestamp features; and encoding and concatenating the call duration features, number location features, and timestamp features to form a communication feature vector.

[0012] In one possible implementation, the video signal is real-time image data captured by the camera of the second communicating party; the method further includes: in a second target scenario, the first communicating party initiates a video acquisition request to the second communicating party through a call terminal; the second target scenario includes an abnormal call scenario where risks are judged based on communication metadata, and the second target scenario also includes the first communicating party performing a target action through the call terminal; in response to the second communicating party's authorization confirmation of the video acquisition request, the acquisition of the video signal is triggered.

[0013] In one possible implementation, the method further includes: acquiring a labeled dataset containing call samples with risk and normal labels; performing data augmentation on the call samples in the labeled dataset; tuning the hyperparameters of the deep learning model to be trained using grid search or random search to determine the optimal hyperparameter combination; training the deep learning model to be trained using the cross-entropy loss function and the Adam optimizer based on the augmented labeled dataset and the optimal hyperparameter combination to obtain the deep learning model; wherein the deep learning model includes one or more networks such as convolutional neural network (CNN), recurrent neural network (RNN), and transformer; and employing a dynamic learning rate strategy during training, with the learning rate gradually decreasing with each training epoch.

[0014] In one possible implementation, the method further includes: lightweighting the trained deep learning model, which includes pruning and / or quantization.

[0015] Secondly, embodiments of this application provide a call detection system based on multimodal feature fusion. The system includes: a data acquisition module configured to: acquire in real time voice signals, video signals, and / or communication metadata during a call between a first communication party and a second communication party; wherein the call includes video calls, VoLTE calls, and 5G new calls; a data preprocessing module configured to: acquire, in a first target scenario, the voice signal stream corresponding to the voice signal, the video signal stream corresponding to the video signal, and the recently acquired communication metadata; the first target scenario refers to an abnormal call scenario where risk is judged based on voice keywords; a feature extraction and fusion module configured to: extract and fuse features from the voice signal stream, the video signal stream, and the recently acquired communication metadata to obtain multimodal features; and a call detection module configured to: input the multimodal features into a deep learning model to obtain suspicious behavior detection results.

[0016] As described above, this application provides a call detection method and system based on multimodal feature fusion. The method includes: real-time acquisition of voice signals, video signals, and / or communication metadata during a call between a first and a second communicating party; wherein the call includes video calls, VoLTE calls, and 5G calls; in a first target scenario, acquiring the voice signal stream corresponding to the voice signal, the video signal stream corresponding to the video signal, and the most recently acquired communication metadata; the first target scenario refers to an abnormal call scenario where risk is identified based on voice keywords; feature extraction and feature fusion are performed on the voice signal stream, video signal stream, and the most recently acquired communication metadata to obtain multimodal features; the multimodal features are input into a deep learning model to obtain suspicious behavior detection results. It is evident that this method, through multimodal data fusion, can achieve real-time and accurate detection of suspicious call behavior, effectively compensating for the limitations of single-modal detection, and exhibiting high timeliness and accuracy in identification. Attached Figure Description

[0017] Figure 1 A flowchart illustrating the call detection method based on multimodal feature fusion provided in this application embodiment; Figure 2 A schematic diagram illustrating the process of extracting speech feature vectors provided in an embodiment of this application; Figure 3 A schematic diagram illustrating the process of extracting video feature vectors provided in an embodiment of this application; Figure 4 A schematic diagram illustrating the process of extracting communication feature vectors provided in an embodiment of this application; Figure 5 A schematic diagram illustrating the real-time early warning and blocking process provided in the embodiments of this application; Figure 6 This is a schematic diagram of the structure of a call detection system based on multimodal feature fusion provided in an embodiment of this application. Detailed Implementation

[0018] To enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of this application.

[0019] To address the limitations of traditional single-modal call detection methods, this application provides a call detection method and system based on multimodal feature fusion. This method can be deployed on user terminals to perform real-time voice monitoring and blocking based on multimodal fusion data, thereby improving the accuracy and timeliness of abnormal call detection.

[0020] Figure 1 This is a flowchart illustrating the call detection method based on multimodal feature fusion provided in an embodiment of this application.

[0021] like Figure 1 As shown in the figure, this application provides a call detection method based on multimodal feature fusion, which includes the following steps S100-S400.

[0022] S100: Real-time collection of voice signals, video signals and / or communication metadata during a call between the first and second communication parties; wherein, the call includes video call, Voice over Long-Term Evolution (VoLTE) call and 5th Generation New Call (5G NC).

[0023] Specifically, video calls can be calls using instant messaging applications based on real-time audio and video communication technology. This application does not limit the specific type of instant messaging application. VoLTE calls can also be called "4G HD voice calls," which are voice call technologies based on 4G / LTE networks. 5G new calls are new call technologies based on the 5G network architecture, integrating core capabilities such as HD voice, ultra-HD video, and real-time interaction.

[0024] Furthermore, the first and second communicating parties are two parties conducting a VoLTE or 5G call via a calling terminal. Alternatively, the first and second communicating parties can be two parties conducting a call via an instant messaging application. Alternatively, the first and second communicating parties can be two participants in an online conference, specifically a telephone conference or a video conference. The first communicating party can be the caller and the second communicating party can be the called party, or vice versa. The calling terminal can be a computer, smartphone, 5G smart terminal, tablet device supporting VoLTE or 5G calling functionality, or smart wearable device, etc.

[0025] Furthermore, after the call is connected, this embodiment of the application can acquire the voice signal according to a preset sampling rate and quantization bit depth. In practical applications, the voice signal can be acquired through the microphone of the call terminal. The microphone can convert sound wave vibrations into continuous-time analog electrical signals, which can then be digitally sampled to obtain a digital voice signal. For example, the preset sampling rate can be 16kHz, and the quantization bit depth can be 16 bits. Thus, each frame of the voice signal contains 320 sampling points. The specific sampling formula is as follows: x[n]=x a (nTs )×quantization_factor; Where: x[n] represents the digital speech signal; x a (nT s () is the analog speech signal in time nT s The value at T; s It is the sampling period; quantization_factor represents the quantization factor, which is used to convert analog signals into digital signals.

[0026] Understandably, with a preset sampling rate of 16kHz, the sampling period is 1 / 16000 seconds.

[0027] Furthermore, after the call is connected, this embodiment of the application can collect communication metadata at a first preset frequency. , where C t This can include the call number, call duration, collection timestamp, and caller / called identifiers.

[0028] Specifically, the communication metadata can be obtained through the communication module of the communication terminal. The first preset frequency can be 25Hz, that is, it is collected once every 40ms; this embodiment of the application does not specifically limit this. The collection timestamp refers to the specific timestamp of the collected communication metadata. It is understood that the communication metadata can be continuously updated as the call progresses.

[0029] In some implementations, embodiments of this application can align communication metadata with voice signals to ensure a unique association between communication metadata and voice acquisition data (voice signals).

[0030] Furthermore, after the call is connected, this embodiment can acquire video signals at a preset resolution and frame rate. The preset resolution can be 720p or 1080p, and the preset frame rate can be 30fps, ensuring that the facial details, micro-expressions, and body movements of the second communicating party can be clearly captured. It is worth noting that, with the authorization of the second communicating party, this embodiment can directly acquire the raw video stream captured in real time by the camera of the second communicating party's calling terminal. Also, in scenarios where the call includes video (such as video calls, 5G new calls, video conferencing), the screen recording data of the first or second communicating party can be directly acquired. As acquisition continues, a continuous video signal stream can be obtained.

[0031] In some implementations, the embodiments of this application may also include the following steps S1001-S1002.

[0032] S1001: In the second target scenario, the first communication party initiates a video capture request to the second communication party through a call terminal; the second target scenario includes abnormal call scenarios where risks are judged based on communication metadata, and the second target scenario also includes the first communication party performing a target action through a call terminal.

[0033] Specifically, in this application embodiment, after obtaining communication metadata through the communication module, it can be determined through analysis and judgment that the first communication party is currently or is about to communicate with an unknown number (without historical call records or not stored in the contact list), or identify risk characteristics such as abnormal number location (e.g., overseas virtual number, number from a high-risk area) or call time period being a high-risk period, and thus directly determine that the second target scenario is established.

[0034] Furthermore, the target action could be, for example, the first party making a transfer or payment through the call terminal. If the first party performs the target action through the communication terminal during the call, the second target scenario can also be determined to be valid.

[0035] Furthermore, the video capture request can be presented in a two-way pop-up window format. For the first communicating party, the terminal displays a prominent pop-up reminder (e.g., "The current call has a potential risk; do you wish to request video capture of the other party for security verification?"), allowing the first communicating party to choose whether to initiate the video capture request. For the second communicating party, upon receiving the video capture request, their terminal can display the reason for the capture and authorization options in a pop-up window (e.g., "The other party needs to capture your video feed to verify the security of the call; do you agree?"), clearly informing them of the purpose of the capture and ensuring the second communicating party's right to know.

[0036] S1002: In response to the authorization confirmation of the video acquisition request by the second communicating party, trigger the acquisition of video signals.

[0037] Understandably, if the second party clicks the "Agree" button in the pop-up window to complete the authorization confirmation, the video signal acquisition process will be triggered immediately.

[0038] In some implementations, the interaction between video capture requests and authorization confirmations can be achieved through in-band signaling channels embedded in 5G new calls or VoLTE calls.

[0039] It is understood that in the embodiments of this application, voice signals, video signals and communication metadata may be acquired asynchronously.

[0040] S200: In the first target scenario, acquire the voice signal stream corresponding to the voice signal, the video signal stream corresponding to the video signal, and the recently collected communication metadata; the first target scenario refers to an abnormal call scenario where risks are judged based on voice keywords.

[0041] This application embodiment can perform LMS adaptive filtering algorithm noise reduction and real-time text conversion on a certain length of speech signal stream at a second preset frequency to obtain a text sequence. Then, the text sequence can be matched against a preset risk keyword database. If at least one risk keyword is matched, a first target scenario can be triggered. For example, the second preset frequency is 3 seconds / time, 5 seconds / time, or 10 seconds / time. Risk keywords include, for example, transfer, verification code, suspected money laundering, etc., and this application embodiment does not specifically limit these.

[0042] In some implementations, when a risky keyword is detected, this application embodiment can extract the signal by taking the timestamp of the keyword as the center, going back 30 seconds and continuing forward 30 seconds, that is, the continuous voice signal stream and the synchronous video signal stream within the 60-second time window, as the input data for subsequent multimodal feature extraction and fusion analysis.

[0043] In some implementations, embodiments of this application may also acquire a certain length of voice signal stream and video signal at a third preset frequency for periodic call recognition. The third preset frequency may be, for example, 1 minute / time or 2 minutes / time. Embodiments of this application do not specifically limit this frequency.

[0044] In some implementations, embodiments of this application may also acquire the voice signal stream corresponding to the voice signal, the video signal stream corresponding to the video signal, and the most recently collected communication metadata after the call ends, and then perform detection. In this implementation, embodiments of this application may determine the length of the voice signal stream and the video signal stream based on the actual call duration.

[0045] It is understood that since communication metadata is constantly updated as the call progresses, embodiments of this application can obtain the most recently collected communication metadata as the basis for data processing. In some implementations, embodiments of this application may also perform the communication metadata acquisition step after the first target scenario or after the call ends.

[0046] In some implementations, the step of determining whether the second target scenario is valid can be performed before or simultaneously with the step of determining whether the first target scenario is valid, in order to ensure that the video signal can be acquired in a timely manner.

[0047] S300: Performs feature extraction and feature fusion on voice signal stream, video signal stream, and recently acquired communication metadata to obtain multimodal features.

[0048] It is understood that embodiments of this application can extract speech feature vectors corresponding to speech signal streams, specifically Mel-Frequency Cepstral Coefficients (MFCCs). Furthermore, embodiments of this application can extract video feature vectors corresponding to video signal streams, specifically facial expression feature vectors, motion feature vectors, etc. Additionally, embodiments of this application can extract features from recently collected communication metadata, specifically extracting call duration features, number location features, and timestamp features, ultimately obtaining communication feature vectors. Specific feature extraction steps will be detailed below and will not be repeated here.

[0049] Furthermore, in this embodiment of the application, the feature vectors of the speech modality, video modality and communication modality can be weighted and fused using the feature fusion formula to obtain multimodal features. The specific fusion steps will be detailed below and will not be repeated here.

[0050] S400: Input multimodal features into a deep learning model to obtain suspicious behavior detection results.

[0051] It is understood that the embodiments of this application can utilize pre-trained deep learning models for suspicious behavior detection. The deep learning model includes one or more networks such as Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), and Transformers. Specifically, CNNs are used to extract local features from the speech spectrum and video frames, RNNs are used to capture the temporal dependence of speech signals and the regularity of video action sequences, and Transformers are used to mine global correlations between multimodal data. Thus, the embodiments of this application can achieve real-time blocking by "acquiring, analyzing, and responding simultaneously," resulting in accurate and timely identification.

[0052] As can be seen from the above, this application provides a call detection method based on multimodal feature fusion. The method includes: real-time acquisition of voice signals, video signals, and / or communication metadata during a call between a first and a second communicating party; wherein the call includes video calls, VoLTE calls, and 5G new calls; in a first target scenario, acquiring the voice signal stream corresponding to the voice signal, the video signal stream corresponding to the video signal, and the most recently acquired communication metadata; the first target scenario refers to an abnormal call scenario where risk is judged based on voice keywords; feature extraction and feature fusion are performed on the voice signal stream, video signal stream, and the most recently acquired communication metadata to obtain multimodal features; the multimodal features are input into a deep learning model to obtain suspicious behavior detection results. It is evident that this method, through multimodal data fusion (MDF), can achieve real-time and accurate detection of suspicious call behavior, effectively compensating for the limitations of single-modal detection, and exhibiting high timeliness and accuracy in identification.

[0053] Furthermore, step S300 may specifically include the following steps S301-S305.

[0054] S301: Time alignment of voice signal stream, video signal stream and communication metadata is performed based on the time axis of the first communicating party.

[0055] Specifically, in this embodiment, the system timestamp of the first communicating party's terminal can be used as a unified benchmark to limit the voice signal stream, video signal stream, and communication metadata to the same time window. For example, the voice signal stream and video signal stream can be truncated to ensure that they correspond to the same time window. At the same time, the communication metadata is bound to the corresponding voice data stream and video data stream, and finally a time-synchronized multimodal data queue is output to ensure that the voice frames, video frames, and communication metadata correspond one-to-one in the same time dimension.

[0056] In some implementations, embodiments of this application can also perform time-series normalization processing on the voice data stream and video data stream, that is, map the feature sequences of both to the same set of equally spaced analysis time points using an interpolation algorithm. Afterwards, the communication metadata can be bound to the corresponding voice data stream and video data stream.

[0057] S302: Extract the speech feature vector corresponding to the speech signal stream.

[0058] Figure 2 This is a schematic diagram of the process for extracting speech feature vectors provided in an embodiment of this application.

[0059] Specifically, such as Figure 2 As shown, step S302 may include the following steps S3021-S3025.

[0060] S3021: Perform noise reduction processing on the speech signal stream to obtain the noise-reduced speech signal stream.

[0061] In this application embodiment, an adaptive filter (AF) can be used to remove environmental noise and circuit noise, and this step is performed frame by frame. For example, the adaptive filter can employ the Least Mean Square (LMS) algorithm.

[0062] The adaptive filtering formula is as follows: x0[n] = x[n] - w[n] T ·d[n]; Where: x0[n] represents the denoised speech signal; x[n] is the original speech signal; d[n] represents the noise reference signal; w[n] are the adaptive filter coefficients; T represents the vector transpose operation; n is the sampling point number, with a value range of [0, N-1]; and N is the total number of sampling points corresponding to the signal frame of the currently processed speech signal stream (e.g., if each frame has 32 sampling points, then n = 0……319).

[0063] S3022: Perform frame segmentation and windowing processing on the noise-reduced speech signal stream to obtain the windowed speech signal stream.

[0064] The framing step specifically refers to dividing the denoised speech signal stream into frames of fixed length, such as one frame every 20ms. Further, the windowing process can involve adding a Hamming window frame-by-frame to reduce spectral leakage.

[0065] Specifically, the formulas for framing and windowing can be: w[n] = 0.54 - 0.46cos( ); Where: n is the intra-frame sampling point index, with a value range of 0, 1, ..., N-1, N is the total number of sampling points per frame (e.g., N = 320 for a 20ms frame length), and w[n] is the Hamming window coefficient corresponding to the nth sampling point.

[0066] Therefore, the windowed speech signal can be X1[mN+n]=x0[mN+n]·w[n].

[0067] Where x0[mN+n] is the denoised speech signal, w[n] is the Hamming window coefficient, n is the sampling point index in the current frame (consistent with n in the Hamming window coefficient formula, with a value range of 0, 1, ..., N-1), and m is the frame index (with a value range of 0, 1, ..., M-1, where M is the total number of frames of the windowed speech signal).

[0068] S3023: Perform a short-time fourier transform (STFT) on each frame of the windowed speech signal stream to obtain the spectral feature sequence.

[0069] Specifically, the formula for the short-time Fourier transform is: ; in, Let m be the spectral characteristics of the m-th frame of the speech signal, where m is the frame index (ranging from 0, 1, ..., M-1, where M is the total number of frames of the windowed speech signal). m is the frequency index (value range 0, ..., N / 2), and m is the frame index. The input signal is the windowed audio signal, n is the index of the sampling point within the frame, and the value range is 0, 1, ..., N-1. N is the total number of sampling points per frame.

[0070] S3024: Calculate the power spectrum frame by frame for the spectral feature sequence to obtain the power spectrum sequence.

[0071] The formula for calculating the power spectrum can be: P[f,m]=| | 2 ; Where P[k,m] is the power spectrum of the m-th frame of the speech signal. Let m be the spectral characteristics of the m-th frame of the speech signal, where m is the frame index (ranging from 0, 1, ..., M-1, where M is the total number of frames of the windowed speech signal).

[0072] S3025: Extract Mel frequency cepstral coefficients from the power spectrum sequence to obtain the speech feature vector.

[0073] Specifically, in this embodiment, MFCC can be extracted using a Mel filter bank and a discrete cosine transform (DCT). The specific formula is as follows: ; in, Let be the j-th MFCC coefficient, where j ranges from 1 to ... to 13. Let be the DCT transform coefficients, lnP[k] be the logarithmic power spectrum of the output after filtering by the kth Mel filter, and K be the total number of Mel filters.

[0074] Understandably, the Mel filter bank can filter and weight the power spectrum, transforming it from a linear frequency scale to a Mel frequency scale, obtaining the Mel frequency domain energy corresponding to each Mel filter, and then taking the logarithm of it to obtain the logarithmic power spectrum.

[0075] Subsequently, embodiments of this application can convert the 13th-order MFCC coefficients of each frame of speech signal. Arranged frame by frame, a speech feature matrix of dimension M×13 is formed, where M is the total number of frames in the windowed speech signal stream. After normalization, the final global speech feature vector F is obtained. 语音 This is used for subsequent multimodal feature fusion.

[0076] In some implementations, embodiments of this application may further perform weighted fusion of the power spectrum and Mel frequency cepstral coefficients to obtain the speech feature vector F. 语音 .

[0077] S303: Extract the video feature vector corresponding to the video signal stream.

[0078] Figure 3This is a schematic diagram of the process for extracting video feature vectors provided in an embodiment of this application.

[0079] Specifically, such as Figure 3 As shown, step S303 may include the following steps S3031-S3036.

[0080] S3031: Decode the video signal stream frame by frame to obtain the original image frame sequence.

[0081] For example, the video signal stream is a continuous data stream acquired at a frame rate of 30fps (corresponding to 720p or 1080p resolution), and after decoding, it outputs a temporally continuous sequence of original image frames. , among which, I t It is the image of the t-th frame, where T is the total number of video frames.

[0082] S3032: Using a face detection model, face detection and tracking are performed on the original image frame sequence to obtain a face region coordinate sequence.

[0083] The face detection model can be a YOLO series deep learning model (preferably YOLOv5), or it can be an open-source cross-platform computer vision library (OpenCV) model; this embodiment does not specifically limit the choice. This allows the output of the facial region coordinate sequence of the second communicating party. ,in, Let T be the coordinates of the facial region of the second communicator in the t-th frame image, and T be the total number of video frames.

[0084] S3033: Extract key points from the facial region coordinate sequence to obtain the facial key point coordinate sequence.

[0085] The key points can be the core facial features such as the eyes, mouth, and nose, as well as contour features (preferably 68 key feature points, covering the main facial muscle movement areas). This allows the output of the facial key point coordinate sequence for the second communicating party. ,in, Specifically, it can be the sequence of facial keypoint coordinates corresponding to the t-th frame image, where T is the total number of video frames. In the case of 68 feature points... This includes the x and y coordinates of the r-th keypoint in the t-th frame image (r=1,2,...,68).

[0086] S3034: Using a deep feature extraction model, facial expression recognition is performed on the sequence of facial key point coordinates to extract facial expression features and obtain a facial expression feature vector.

[0087] Among them, deep feature extraction models, such as the FaceNet model, can extract micro-expressions such as raised eyebrows, tight corners of the mouth, fake smiles, pupil constriction, stiff facial expressions, and eye avoidance of the camera, ultimately obtaining a multi-dimensional facial expression feature vector F. 面部表情 .

[0088] S3035: Use optical flow to perform motion detection on the original image frame sequence, extract motion features, and obtain motion feature vectors.

[0089] Among them, the Optical Flow Method (OFM) can capture the pixel motion state of two adjacent frames in the original image frame sequence globally, outputting an optical flow field matrix containing motion direction and velocity, accurately depicting the limb movement trajectory of the second communicator, and outputting a motion feature vector. Specifically, the embodiments of this application can use the Farneback algorithm. In this way, the motion features of the second communicator can be accurately identified, and a multi-dimensional motion feature vector F can be obtained. 动作 For example, it can detect frequent taps on the screen by the other party's hands (suggesting operation of a suspicious app), with the frequency of these actions being three times higher than during a normal phone call.

[0090] S3036: Concatenate or weightedly fuse facial expression feature vectors with motion feature vectors to form video feature vectors.

[0091] In this embodiment of the application, the splicing method can be to directly connect the facial expression feature vector (such as the 512-dimensional vector extracted based on FaceNet) and the action feature vector (such as the 256-dimensional vector extracted by optical flow) end to end to form a high-dimensional joint feature.

[0092] Weighted fusion can employ a dynamic weight optimization strategy, and the fusion formula can be F. 视频 =ω1·F 面部表情 +ω2·F 动作 Where ω1 and ω2 are weighting coefficients, and ω1 + ω2 = 1.

[0093] For example, ω1 is preferably 0.6 and ω2 is preferably 0.4 to highlight the core role of facial micro-expressions in risk identification, while retaining the auxiliary discriminative value of body movements, thus forming the final video feature vector.

[0094] This allows for the comprehensive integration of visual cues, providing strong support for subsequent multimodal fusion analysis.

[0095] S304: Extract the communication feature vector corresponding to the most recently collected communication metadata.

[0096] Figure 4 This is a schematic diagram of the process for extracting communication feature vectors provided in an embodiment of this application.

[0097] Specifically, such as Figure 4 As shown, step S304 may include the following steps S3041-S3043.

[0098] S3041: Perform data cleaning on the recently collected communication metadata to obtain the cleaned communication metadata.

[0099] One step in data cleaning is, for example, removing base station codes.

[0100] S3042: Extract features from the cleaned communication metadata to obtain call duration features, number location features, and timestamp features.

[0101] In some implementations, if the location of a phone number cannot be determined based on the call number, the location feature can be set to a preset default value.

[0102] S3043: Encode and concatenate the call duration feature, number location feature, and timestamp feature to form a communication feature vector.

[0103] Specifically, the feature encoding and splicing can be implemented according to the following steps in the embodiments of this application.

[0104] ① Call duration feature encoding: The collected cumulative call duration is normalized, and the duration value is mapped to the [0,1] interval using a formula. For example, 120 seconds outputs 0.75 after normalization, resulting in a one-dimensional duration feature value (denoted as T). 通话 =0.75);

[0105] ②Number Location Feature Encoding: The location of a number is mapped using a pre-defined location classification method (such as commonly used provinces / regions within China, virtual numbers from overseas, unknown locations, etc.). For example, if the number's location is Beijing, the encoding is [1,0,0] (denoted as L). 号码 =[1,0,0]), if it is an overseas virtual number, it is encoded as [0,1,0] (denoted as L). 号码 =[0,1,0]), ensuring that the risk attributes of different locations can be identified by the model.

[0106] ③ Timestamp feature encoding: Based on the timestamp, the time period attribute (such as morning rush hour, noon, night, early morning) and date attribute (such as weekday, weekend, public holiday) of the call are parsed. For example, if the call is "morning rush hour + weekday", it is encoded as [1,0,1] (denoted as T). 时间 =[1,0,1]), accurately capturing the time distribution characteristics of abnormal calls.

[0107] After completing the above encoding, this embodiment of the application can concatenate the call duration feature, number location feature, and timestamp feature in sequence to finally output a numerical communication feature vector F of a unified dimension. 通信 For example, combining the encoding results of the above examples, F 通信 =[0.75, 1, 0, 0, 1, 0, 1], this vector fully contains the quantified features of normalized duration, geographic classification and time attributes, and can be directly used for subsequent multimodal feature fusion.

[0108] S305: Using the feature fusion formula, the speech feature vector, video feature vector and communication feature vector are weighted and fused to obtain multimodal features.

[0109] It is understood that embodiments of this application can fuse voice features (such as MFCC, power spectrum), video features (such as facial expressions, actions), and communication features (such as call duration, number location). The specific formula is as follows: F=αF 语音 +βF 视频 +γF 通信 ; Where: F represents multimodal features; F 语音 F represents the speech feature vector; 视频 F represents the video feature vector; 通信 This represents the communication feature vector; α, β, and γ are the fusion weights. For example, α can be 0.4, β can be 0.4, and γ can be 0.2. In practical applications, the values ​​of the fusion weights can be optimized through training.

[0110] Figure 5 This is a schematic diagram illustrating the real-time early warning and blocking process provided in the embodiments of this application.

[0111] In this embodiment of the application, the suspicious behavior detection result may include a risk score. Further, such as... Figure 5 As shown, step S400 may be followed by steps S501-S503.

[0112] S501: Determine whether the risk score is greater than the preset threshold.

[0113] The preset threshold is, for example, 0.8, but this application embodiment does not specifically limit it.

[0114] S502: If the risk score is greater than the preset threshold, output a blocking signal. The blocking signal is used to block the call between the first communication party and the second communication party.

[0115] S503: If the risk score is less than or equal to a preset threshold, output a warning signal. The warning signal is used to indicate that the first communication party's call is at risk.

[0116] It is understandable that the judgment logic can be quantified and expressed by a formula as follows: Block= ; Where S represents the risk score, T represents the preset threshold, 0 represents the warning signal, 1 represents the blocking signal, and Block is the identifier variable.

[0117] In practical applications, when Block=1 (risk score S>T), this embodiment can output a blocking signal. Specifically, it can trigger a call termination mechanism through the 4G / 5G core network signaling interface to immediately terminate the call between the first and second communicating parties. Alternatively, it can immediately cut off the application's network access process or data stream locally on the terminal via VPN or firewall rules, interrupting its data stream and signaling. Simultaneously, this embodiment can add suspicious numbers to a blacklist to prevent users from suffering financial losses. When Block=0 (risk score S≤T), this embodiment can output a warning signal. Specifically, it can prompt the first communicating party that "the current call has potential risks" through terminal pop-ups, voice broadcasts, etc., without interrupting normal calls, only guiding the user to be more vigilant.

[0118] The call detection process of this application embodiment is described below with reference to examples.

[0119] User A receives a video call from someone claiming to be "official customer service," who asks for their cooperation in handling "account anomalies" and guides them to download a designated app for "fund verification." During this process, the embodiments described in this application can initiate a multimodal risk detection process.

[0120] After the call is connected, User A's mobile phone microphone can collect the call voice at a sampling rate of 16kHz and a quantization precision of 16 bits. The original signal contains ambient noise and the other party's deliberately lowered serious tone. After noise reduction by the LMS adaptive filtering algorithm, the voice clarity is improved, and keywords such as "suspected violation" and "limited-time transfer" can be identified, and the first target scenario is established.

[0121] Furthermore, this application embodiment can collect communication metadata to determine if the second target scenario is established and trigger video signal acquisition. At this time, the "customer service personnel's" mobile phone camera can capture video at 720p resolution and 30fps frame rate. In the picture, the "customer service personnel" are wearing fake official customer service clothing, with a stiff expression and avoiding eye contact.

[0122] Furthermore, in this embodiment of the application, the communication metadata can carry a lot of communication-related information. For example, high-risk metadata such as the caller ID being an overseas virtual number (with the location displayed as "unknown"), the call duration lasting 15 minutes (far exceeding the normal duration of an official business call), and the timestamp being 2 AM (non-official office hours).

[0123] Furthermore, when the first target scenario is established, the speech signal stream is segmented into frames (20ms / frame) and processed with a Hamming window to obtain the windowed speech signal stream. Additionally, embodiments of this application can detect and track facial regions based on the YOLOv5 model, identifying instances where the person frequently looks down at their phone (suspected of reading lines from a script), and determining that their facial expression is unnatural by extracting the coordinate sequence of key facial points, showing a significant difference from normal law enforcement scenarios.

[0124] Furthermore, this embodiment can extract 13th-order MFCC features from the windowed speech signal stream. This reveals concentrated high-frequency energy, a speech rate 20% faster than normal conversation, and abnormally prominent power spectrum energy in the 2000-4000Hz frequency band, corresponding to threatening word pronunciation. Additionally, this embodiment can identify micro-expressions such as "fake smile" and "pupil constriction" from facial expression features extracted from the video signal stream using the FaceNet model, achieving an 85% match with the "deceptive behavior" feature library. It also uses optical flow to detect that the frequency of frequent hand taps on the screen is three times higher than normal conversation; communication features directly mark anomalies in number, duration, and time slot. Subsequently, a weighted fusion formula is used to fuse the three types of features into a unified multimodal feature.

[0125] Furthermore, in this embodiment, the fused features can be input into a pre-trained deep learning model for inference, outputting a risk score S=0.92, exceeding a preset threshold T=0.8. This embodiment can immediately determine it as a high-risk call, triggering a dual response: on one hand, a pop-up message is pushed to user A stating "The current call may be risky, please operate with caution!", and a safety reminder voice is played; on the other hand, the call is automatically disconnected, and the number is added to the local blacklist to prevent secondary calls. Thus, this embodiment can achieve early warning and proactive blocking at critical stages of abnormal behavior through real-time collection, processing, fusion, and intelligent analysis of multimodal data.

[0126] The model training process is illustrated below.

[0127] Specifically, the call detection method based on multimodal feature fusion provided in this application embodiment may further include the following steps S601-S604.

[0128] S601: Obtain the labeled dataset, which contains call samples with risk labels and normal labels.

[0129] In practical applications, the embodiments of this application can construct a labeled dataset with more than 100,000 samples, comprehensively covering various suspicious scenarios (such as impersonating official staff, fake investments, and inducement by acquaintances) and normal communication scenarios, ensuring the diversity and representativeness of the data.

[0130] Furthermore, the embodiments of this application can clean the call samples. For example, data with severely unclear voice signals (such as excessive background noise or low volume) can be removed, as can samples with severely blurry or obstructed video images that prevent effective face detection and feature extraction. Additionally, invalid data in the communication metadata can be removed, such as data lacking caller and callee information, data with abnormal duration or incorrect format, base station codes can be removed, and redundant data in the communication metadata, such as duplicate records and customer service call records, can also be removed.

[0131] S602: Perform data augmentation on call samples in the labeled dataset.

[0132] Augmentation processing is a data processing technique that expands the scale and enriches the distribution of data to enhance the model's adaptability to complex scenarios and adversarial behaviors (such as voice changing and speech synthesis). For example, for voice data, new samples are generated by adding environmental noise of different intensities (such as office noise and outdoor wind noise), speeding (0.8-1.2 times speed), pitch shifting (±0.5 pitch), and audio segment cropping and splicing. For video data, image enhancement techniques such as random cropping, flipping, brightness or contrast adjustment, and adding slight blur are used to simulate video images under different shooting angles and lighting conditions. For communication metadata, sample diversity is expanded by perturbing the duration within a reasonable range and replacing similar location categories, effectively alleviating the data imbalance problem and improving the model's generalization ability.

[0133] Understandably, the augmented training dataset can be used to... ,in, These are input features. It is a true label (0 or 1, where 0 represents a risk label and 1 represents a normal label, or 0 represents a normal label and 1 represents a risk label).

[0134] S603: Optimize the hyperparameters of the deep learning model to be trained by grid search or random search to determine the optimal combination of hyperparameters.

[0135] It is understandable that hyperparameter tuning is a model optimization technique that systematically traverses or randomly samples a pre-defined hyperparameter space and combines it with validation set performance metrics (such as detection accuracy, loss value, and inference latency) to select the optimal parameter combination. In the embodiments of this application, the hyperparameters to be tuned include, but are not limited to: the kernel size, number of convolutional layers, and pooling window size of CNN networks; the number of hidden layer neurons, number of layers, and dropout ratio of RNN networks; the number of attention heads, number of encoder and decoder layers, and word embedding dimension of Transformer networks; and training-related hyperparameters such as batch size and initial learning rate.

[0136] S604: Based on the enhanced labeled dataset and the optimal hyperparameter combination, the deep learning model to be trained is trained using the cross-entropy loss function and the Adam optimizer to obtain the deep learning model; wherein, the deep learning model includes one or more networks such as convolutional neural network (CNN), recurrent neural network (RNN), and transformer; and, a dynamic learning rate strategy is adopted during the training process, with the learning rate gradually decreasing with each training round.

[0137] For example, the cross-entropy loss function is as follows: L=− ; Where: L represents the loss function; It is a real label (0 or 1); I is the model's predicted probability; I is the sample size.

[0138] Understandably, based on the aforementioned cross-entropy loss function, the input is the model's predicted probability. and real labels The output is the loss value L.

[0139] It is understood that various optimization strategies can be employed in the embodiments of this application. For example, the Adam optimizer can be used, a gradually decreasing dynamic learning rate can be employed, and model parameters can be optimized through grid search or random search. For instance, the initial learning rate is set to a preset baseline value (e.g., 0.001), and then decays to 0.9 times the original learning rate every 10 iterations. This ensures rapid convergence in the early stages of training while avoiding parameter oscillations caused by excessively high learning rates in the later stages, thus ensuring that the model converges stably to the optimal solution.

[0140] In this way, we can obtain the optimized model parameters and finally obtain the trained deep learning model.

[0141] Furthermore, the method provided in the embodiments of this application may also include the following step S605.

[0142] S605: Perform lightweight processing on the trained deep learning model. Lightweight processing includes pruning and / or quantization.

[0143] In practical applications, this embodiment can reduce the total number of parameters in the deep learning model by 40% through pruning and quantization, thus compressing the model size. As a result, after deploying the deep learning model on a mobile edge computing node, the entire process from data acquisition to blocking takes approximately 2.3 seconds, meeting real-time requirements.

[0144] In some implementations, embodiments of this application may also provide a model performance monitoring mechanism, specifically including: monitoring model inference time to ensure real-time performance (e.g., processing time per frame < 50ms), and monitoring indicators such as detection accuracy, recall, and F1 score to continuously optimize model performance, forming a closed-loop feedback mechanism.

[0145] In some implementations, embodiments of this application can also optimize model performance through data annotation. Specifically, this application can annotate data from real-world cases and supplement it into the annotation dataset. For example, for the case data of "User A" mentioned above, embodiments of this application can annotate it as "abnormal call impersonating an official identity" and supplement it into the annotation dataset to optimize the model's ability to identify "authoritative identity forgery" scenarios.

[0146] As described above, this application provides a multimodal fusion-based call detection method. This method constructs a detection process of "data acquisition - preprocessing - feature extraction - multimodal fusion - intelligent detection - real-time response." It collects voice, video, and communication metadata through a mobile terminal, then extracts acoustic features from the voice to identify abnormal speech, analyzes micro-expressions and actions in the video to supplement non-verbal cues, and mines behavioral anomalies from the communication metadata. Weighted fusion significantly improves the detection accuracy compared to single-modal methods. Furthermore, this application uses CNN, RNN, and Transformer to build the detection engine, combining lightweight models and edge computing, achieving inference latency within 50ms, making it suitable for new 5G call scenarios. Moreover, the method provided in this application can offer operators a deployment solution, can be migrated to multiple scenarios, and has a wide range of applications.

[0147] Figure 6 This is a schematic diagram of the structure of a call detection system based on multimodal feature fusion provided in an embodiment of this application.

[0148] like Figure 6 As shown in the figure, this application provides a call detection system based on multimodal feature fusion, the system comprising: The data acquisition module 1001 is configured to: collect voice signals, video signals and / or communication metadata during a call between the first communication party and the second communication party in real time; wherein, the call includes video call, VoLTE call and 5G new call, and the video signal is the image data captured in real time by the camera of the second communication party; The data preprocessing module 1002 is configured to: in the first target scenario, acquire the voice signal stream corresponding to the voice signal, the video signal stream corresponding to the video signal, and the recently collected communication metadata; the first target scenario refers to an abnormal call scenario where risks are judged based on voice keywords; The feature extraction and fusion module 1003 is configured to: extract and fuse features from the speech signal stream, video signal stream, and recently acquired communication metadata to obtain multimodal features; The call detection module 1004 is configured to input multimodal features into a deep learning model to obtain suspicious behavior detection results.

[0149] In one possible implementation, the suspicious behavior detection result includes a risk score; the system provided in this application embodiment also includes a real-time warning and blocking module 1005, specifically configured to: determine whether the risk score is greater than a preset threshold; if the risk score is greater than the preset threshold, output a blocking signal, the blocking signal being used to block the call between the first communication party and the second communication party; if the risk score is less than or equal to the preset threshold, output a warning signal, the warning signal being used to alert the first communication party that there is a risk in the call.

[0150] In one possible implementation, the data preprocessing module 1002 is specifically configured to: time-align the speech signal stream, video signal stream, and communication metadata based on the time axis of the first communicating party; extract the speech feature vector corresponding to the speech signal stream; extract the video feature vector corresponding to the video signal stream; and extract the communication feature vector corresponding to the most recently acquired communication metadata; and use a feature fusion formula to perform weighted fusion of the speech feature vector, video feature vector, and communication feature vector to obtain multimodal features.

[0151] In one possible implementation, the data preprocessing module 1002 is further configured to: perform noise reduction processing on the speech signal stream to obtain a noise-reduced speech signal stream; perform frame segmentation and windowing processing on the noise-reduced speech signal stream to obtain a windowed speech signal stream; perform short-time Fourier transform on the windowed speech signal stream frame by frame to obtain a spectral feature sequence; calculate the power spectrum on the spectral feature sequence frame by frame to obtain a power spectrum sequence; and extract the Mel-frequency cepstral coefficients from the power spectrum sequence to obtain a speech feature vector.

[0152] In one possible implementation, the data preprocessing module 1002 is further configured to: decode the video signal stream frame by frame to obtain the original image frame sequence; use a face detection model to perform face detection and tracking on the original image frame sequence to obtain a face region coordinate sequence; extract key points from the face region coordinate sequence to obtain a face key point coordinate sequence; use a depth feature extraction model to perform facial expression recognition on the face key point coordinate sequence, extract facial expression features, and obtain a facial expression feature vector; use optical flow to perform motion detection on the original image frame sequence, extract motion features, and obtain a motion feature vector; and concatenate or weightedly fuse the facial expression feature vector and the motion feature vector to form a video feature vector.

[0153] In one possible implementation, the communication metadata includes the call number, call duration, collection timestamp, and / or calling and called party identifiers; the data preprocessing module 1002 is further configured to: perform data cleaning on the recently collected communication metadata to obtain cleaned communication metadata; extract features from the cleaned communication metadata to obtain call duration features, number location features, and timestamp features; and encode and concatenate the call duration features, number location features, and timestamp features to form a communication feature vector.

[0154] In one possible implementation, the video signal is real-time image data captured by the camera of the second communicating party; the data acquisition module 1001 is further configured to: in the second target scenario, the first communicating party initiates a video acquisition request to the second communicating party through a call terminal; the second target scenario includes abnormal call scenarios where risks are judged based on communication metadata, and the second target scenario also includes the first communicating party performing a target action through the call terminal; in response to the second communicating party's authorization confirmation of the video acquisition request, the acquisition of the video signal is triggered.

[0155] In one possible implementation, the system provided in this application embodiment further includes a model training and optimization module 1006, configured to: acquire a labeled dataset containing call samples with risk and normal labels; perform data augmentation on the call samples in the labeled dataset; fine-tune the hyperparameters of the deep learning model to be trained using grid search or random search to determine the optimal hyperparameter combination; train the deep learning model to be trained using the cross-entropy loss function and the Adam optimizer based on the augmented labeled dataset and the optimal hyperparameter combination to obtain the deep learning model; wherein the deep learning model includes one or more networks such as convolutional neural network (CNN), recurrent neural network (RNN), and transformer; and adopt a dynamic learning rate strategy during training, with the learning rate gradually decreasing with each training round.

[0156] In one possible implementation, the model training and optimization module 1006 is further configured to perform lightweight processing on the trained deep learning model, including pruning and / or quantization.

[0157] In a specific implementation, the present invention also provides a computer storage medium, wherein the computer storage medium may store a program, which, when executed, may include some or all of the steps of the call detection method based on multimodal feature fusion provided by the present invention. The storage medium may be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0158] It is readily understood that, based on the several embodiments provided in this application, those skilled in the art can combine, split, or reorganize the embodiments of this application to obtain other embodiments, none of which exceed the protection scope of this application.

[0159] The above detailed embodiments further illustrate the purpose, technical solution, and beneficial effects of the embodiments of this application. It should be understood that the above are merely specific embodiments of the embodiments of this application and are not intended to limit the protection scope of the embodiments of this application. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solutions of the embodiments of this application should be included within the protection scope of the embodiments of this application.

Claims

1. A call detection method based on multimodal feature fusion, characterized in that, The method includes: Real-time acquisition of voice signals, video signals, and / or communication metadata during a call between the first and second communicating parties; wherein the call includes video calls, VoLTE calls, and 5G new calls; In the first target scenario, the voice signal stream corresponding to the voice signal, the video signal stream corresponding to the video signal, and the recently collected communication metadata are acquired; the first target scenario refers to an abnormal call scenario where risks are identified based on voice keywords. Feature extraction and feature fusion are performed on the voice signal stream, the video signal stream, and the recently acquired communication metadata to obtain multimodal features; The multimodal features are input into a deep learning model to obtain suspicious behavior detection results.

2. The call detection method based on multimodal feature fusion according to claim 1, characterized in that, The suspicious behavior detection result includes a risk score; after the step of inputting the multimodal features into the deep learning model to obtain the suspicious behavior detection result, it further includes: Determine whether the risk score is greater than a preset threshold; If the risk score is greater than the preset threshold, a blocking signal is output, which is used to block the call between the first communication party and the second communication party. If the risk score is less than or equal to the preset threshold, an early warning signal is output, which is used to alert the first communicating party that there is a risk in the call.

3. The call detection method based on multimodal feature fusion according to claim 1, characterized in that, The step of extracting and fusing features from the speech signal stream, the video signal stream, and the most recently acquired communication metadata to obtain multimodal features includes: Based on the timeline of the first communicating party, the voice signal stream, the video signal stream, and the communication metadata are time-aligned. Extract the speech feature vector corresponding to the speech signal stream; extract the video feature vector corresponding to the video signal stream; and extract the communication feature vector corresponding to the most recently acquired communication metadata. The speech feature vector, the video feature vector, and the communication feature vector are weighted and fused using a feature fusion formula to obtain the multimodal features.

4. The call detection method based on multimodal feature fusion according to claim 3, characterized in that, The extraction of the speech feature vector corresponding to the speech signal stream includes: The speech signal stream is subjected to noise reduction processing to obtain a noise-reduced speech signal stream; The denoised speech signal stream is framed and windowed to obtain the windowed speech signal stream. The windowed speech signal stream is subjected to short-time Fourier transform frame by frame to obtain the spectral feature sequence; The power spectrum is calculated frame by frame from the spectral feature sequence to obtain the power spectrum sequence; Mel frequency cepstral coefficients are extracted from the power spectrum sequence to obtain the speech feature vector.

5. The call detection method based on multimodal feature fusion according to claim 3, characterized in that, The extraction of the video feature vector corresponding to the video signal stream includes: The video signal stream is decoded frame by frame to obtain the original image frame sequence; Using a face detection model, face detection and tracking are performed on the original image frame sequence to obtain a face region coordinate sequence; Key points are extracted from the facial region coordinate sequence to obtain a facial key point coordinate sequence; Using a deep feature extraction model, facial expression recognition is performed on the facial key point coordinate sequence to extract facial expression features and obtain a facial expression feature vector. The original image frame sequence is subjected to motion detection using optical flow, and motion features are extracted to obtain motion feature vectors; The facial expression feature vector and the action feature vector are concatenated or weighted and fused to form a video feature vector.

6. The call detection method based on multimodal feature fusion according to claim 3, characterized in that, The communication metadata includes call number, call duration, collection timestamp and / or caller and called party identifiers; The extraction of the communication feature vector corresponding to the most recently collected communication metadata includes: The recently collected communication metadata is cleaned to obtain cleaned communication metadata; Feature extraction is performed on the cleaned communication metadata to obtain call duration features, number location features, and timestamp features. The call duration feature, the number location feature, and the timestamp feature are encoded and concatenated to form a communication feature vector.

7. The call detection method based on multimodal feature fusion according to claim 1, characterized in that, The video signal is real-time image data captured by the camera of the second communicating party; the method further includes: In the second target scenario, the first communication party initiates a video capture request to the second communication party through a call terminal; the second target scenario includes abnormal call scenarios where risks are determined based on the communication metadata, and the second target scenario also includes the first communication party performing a target action through the call terminal; In response to the second communicating party's authorization confirmation of the video acquisition request, the acquisition of the video signal is triggered.

8. The call detection method based on multimodal feature fusion according to claim 1, characterized in that, The method further includes: Obtain a labeled dataset, which contains call samples with risk labels and normal labels; Data augmentation processing is performed on the call samples in the labeled dataset; The hyperparameters of the deep learning model to be trained are tuned by grid search or random search to determine the optimal combination of hyperparameters. Based on the enhanced labeled dataset and the optimal hyperparameter combination, the deep learning model to be trained is trained using the cross-entropy loss function and the Adam optimizer to obtain the deep learning model; wherein, the deep learning model includes one or more networks such as convolutional neural network (CNN), recurrent neural network (RNN), and transformer; and, a dynamic learning rate strategy is adopted during training, with the learning rate gradually decreasing with each training round.

9. The call detection method based on multimodal feature fusion according to claim 8, characterized in that, The method further includes: The trained deep learning model is then subjected to lightweight processing, which includes pruning and / or quantization.

10. A call detection system based on multimodal feature fusion, characterized in that, The system includes: The data acquisition module is configured to: collect voice signals, video signals and / or communication metadata during a call between the first and second communication parties in real time; wherein the call includes video calls, VoLTE calls and 5G new calls; The data preprocessing module is configured to: in a first target scenario, acquire the voice signal stream corresponding to the voice signal, the video signal stream corresponding to the video signal, and the most recently collected communication metadata; the first target scenario refers to an abnormal call scenario where risks are identified based on voice keywords; The feature extraction and fusion module is configured to: extract and fuse features from the speech signal stream, the video signal stream, and the recently acquired communication metadata to obtain multimodal features; The call detection module is configured to input the multimodal features into a deep learning model to obtain suspicious behavior detection results.

Images