Method for analyzing risk information in public data based on multi-modal artificial intelligence

Abstract

The invention relates to the technical field of information security, and discloses a method for analyzing risk information in public data based on multi-modal artificial intelligence, and the method comprises the steps: collecting multi-modal data, and carrying out the data preprocessing of the multi-modal data; extracting high-dimensional features of the preprocessed modal data; obtaining a confidence coefficient score of the high-dimensional feature of each modal data; inputting the modal availability vector and the confidence score into a multi-modal fusion model, and calculating an attention fusion weight; performing weighted fusion on the high-dimensional features of the modal data according to the attention fusion weight to generate multi-modal joint feature representation; based on the multi-modal joint feature representation, outputting a risk probability and a type label; and when the output risk probability exceeds a preset threshold value, generating a structured early warning event, and automatically pushing the structured early warning event to a subsequent processing system to trigger a linkage response. According to the method, the risk information in the public data is accurately, efficiently and adaptively identified and disposed, and the automation level and response efficiency of network security protection are effectively improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of information security, and particularly relates to a method for analyzing risk information in public data based on multi-modal artificial intelligence. BACKGROUND

[0002] With the rapid development of the Internet and digital communication technology, a large amount of data is generated on various public network platforms (such as social media, forums, video sharing websites, etc.), which contains a large amount of risk information with security threats. Such information usually has the characteristics of concealment, inducement and diffusion, and poses a potential threat to personal rights and interests and the network ecological environment.

[0003] At present, the identification of such risk information mainly relies on traditional information filtering technology. In the prior art, a method based on rule and keyword matching is used, which relies on a keyword list or a regular expression rule library defined by artificial pre-matching. Since the rule library needs to be maintained by artificial pre-matching, it is difficult to exhaust all variants; and this method is extremely inflexible and cannot effectively deal with new, transformed or metaphorical expressions of risk content, with high false positive and false negative rates. SUMMARY

[0004] The purpose of the present application is to accurately, efficiently and adaptively identify and dispose of risk information in public data, and effectively improve the automation level and response efficiency of network security protection.

[0005] In a first aspect, the embodiments of the present application provide a method for analyzing risk information in public data based on multi-modal artificial intelligence, comprising: asynchronously collecting multi-modal data from a plurality of heterogeneous public data sources, and performing data preprocessing on the multi-modal data using mode-specific cleaning strategies, wherein the multi-modal data includes text, speech, images and videos; extracting high-dimensional features of each mode of the preprocessed data using a pre-trained deep model, wherein the high-dimensional features include semantic features of text, acoustic and semantic features of speech, visual and spatial features of images, and time sequence change features of videos; evaluating the availability state of each mode of data to generate a mode availability vector; the mode availability vector is a continuous value vector, and the mode availability vector is generated by evaluating the quality of each mode of data using a lightweight neural network; the value at each position in the mode availability vector represents the availability probability of the corresponding mode of data; obtaining a confidence score of the high-dimensional features of each mode of data output by the pre-trained deep model; ​inputting the modality availability vector and the confidence score into a multi-modal fusion model to calculate a set of normalized attention fusion weights representing importance of features of each modality; the multi-modal fusion model comprises a gating network based on an attention mechanism; performing weighted fusion on high-dimensional features of the each modality data according to the attention fusion weights to generate a multi-modal joint feature representation; and outputting a risk probability and a type label based on the multi-modal joint feature representation; when the output risk probability exceeds a preset threshold, generating a structured early warning event and automatically pushing the structured early warning event to a subsequent handling system to trigger a linkage response; the structured early warning event at least includes a risk type, a confidence score, a key evidence segment and context-related information.

[0006] Optionally, the method further comprises an updating process of the multi-modal fusion model, and the updating process specifically comprises: continuously monitoring classification performance indicators of the multi-modal fusion model on new labeled data through an online learning framework; and when the classification performance indicators are lower than a preset threshold, automatically triggering an incremental learning process; the incremental learning process comprises: obtaining new labeled data, fine-tuning the multi-modal fusion model using the new labeled data, and introducing a historical data replay mechanism to maintain performance of the multi-modal fusion model on historical tasks; the new labeled data is obtained by monitoring new data in a public data stream and periodically labeling the new data through a continuous learning mechanism.

[0007] Optionally, the multi-modal data is preprocessed by using a modality-specific cleaning strategy, comprising: special characters and invalid links in the text are removed through a regular expression, and word segmentation and part-of-speech tagging are performed using a natural language processing tool; the speech is converted into text after noise reduction processing and endpoint detection; the endpoint detection process comprises: calculating short-time energy and short-time zero-crossing rate of the speech; setting a first threshold for preliminary detection of possible speech segments; setting a second threshold for confirming reliable speech segment start and end points from the preliminary detection segments; for a region between the first threshold and the second threshold, speech / silence is determined in combination with the zero-crossing rate to accurately determine the start and end positions of the speech segment; the first threshold is lower than the second threshold; for images and videos, format unification, size adjustment and key frame extraction are performed through an OpenCV library, and color histogram, texture feature and target detection information are further extracted.

[0008] Optionally, the semantic features of the text are represented by context embedding vectors generated by a pre-trained language model. The acoustic and semantic features of the voice include Mel-frequency cepstral coefficient features, linear prediction cepstral coefficient features extracted from the audio signal, and semantic keyword features extracted from the voice transcription text. The visual and spatial features of the image include deep visual features extracted by a convolutional neural network and target bounding box information obtained by a target detection algorithm. The temporal change features of the video are extracted by calculating the feature difference or optical flow information between the key frame sequences.

[0009] Optionally, for text, the key evidence segment is a sentence containing a high-risk keyword, for voice, the key evidence segment is a timestamp segment where a high-risk word is located, and for image or video, the key evidence segment is an image region or video frame sequence where a target bounding box is located.

[0010] Optionally, the method further includes a risk pattern mining process, which includes: Periodically clustering the high-risk events identified by the multi-modal fusion model, extracting emerging risk pattern features, and feeding the risk pattern features back to the training stage of the multi-modal fusion model as prior knowledge to construct targeted difficult example samples to enhance the discriminative ability of the model.

[0011] Optionally, the attention mechanism-based gating network calculates the attention fusion weight by the following formula: α = softmax(W * [v avail o v conf] + b) Wherein, α represents the fusion weight vector, W and b are learnable parameters, v avail is the modality availability vector, v conf is the confidence score vector, o represents element-wise multiplication, and [·] represents the vector concatenation operation.

[0012] Optionally, the method further includes a multi-modal fusion model construction and training process, which includes: Constructing a multi-modal fusion model including an attention mechanism-based gating network; Collecting a large amount of labeled multi-modal data from multiple public data sources to form a training set and a test set; Initializing the parameters of the multi-modal fusion model; Using a stochastic gradient descent or Adam optimization algorithm, taking cross-entropy loss function as the optimization objective, and using the training set to iteratively train the multi-modal fusion model; After each training cycle, evaluate the performance of the multi-modal fusion model using the test set, and use the early stopping strategy according to the test set loss to prevent overfitting; Save the model parameters with the best performance, and obtain a final deployable multi-modal fusion model.

[0013] Optionally, the visual and spatial features of the image further include low-level visual features extracted by a traditional image processing algorithm, including a color histogram and a texture feature; the color histogram is calculated in an HSV color space, and the texture feature is extracted by a gray level co-occurrence matrix or a local binary pattern algorithm.

[0014] In a second aspect, an embodiment of the present application provides a system for analyzing risk information in public data based on multi-modal artificial intelligence, comprising: A data preprocessing module is configured to asynchronously collect multi-modal data from a plurality of heterogeneous public data sources, and perform data preprocessing on the multi-modal data by using a mode-specific cleaning strategy, wherein the multi-modal data includes text, voice, images, and videos. A high-dimensional feature extraction module is configured to extract high-dimensional features of each mode of the preprocessed data by using a pre-trained deep model, wherein the high-dimensional features include semantic features of text, acoustic and semantic features of voice, visual and spatial features of images, and time sequence change features of videos. A mode availability vector generation module is configured to evaluate the availability state of each mode of data and generate a mode availability vector; the mode availability vector is a continuous value vector, and the mode availability vector is generated by evaluating the quality of each mode of data by using a lightweight neural network; the value of each position in the mode availability vector represents the availability probability of the corresponding mode of data. A confidence score acquisition module is configured to acquire a confidence score of the high-dimensional features of each mode of data output by the pre-trained deep model. An attention fusion weight calculation module is configured to input the mode availability vector and the confidence score into a multi-modal fusion model to calculate a set of normalized attention fusion weights representing the importance of each mode of feature; the multi-modal fusion model includes a gating network based on an attention mechanism. A multi-modal feature fusion module is configured to weight and fuse the high-dimensional features of each mode of data according to the attention fusion weights to generate a multi-modal joint feature representation; based on the multi-modal joint feature representation, a risk probability and a type label are output. A structured early warning event generation module is configured to generate a structured early warning event when the output risk probability exceeds a preset threshold, and automatically push the structured early warning event to a subsequent handling system to trigger a linkage response; the structured early warning event at least includes a risk type, a confidence, a key evidence segment, and context association information.

[0015] The technical scheme provided by the embodiment of the present application can more comprehensively capture the complex mode of risk information by fusing the multi-modal features of text, voice, image and video instead of relying on single modal information. The introduction of the attention mechanism-based gating network can dynamically adjust the fusion weight according to the availability and quality of the data and the confidence of the model itself, thereby automatically reducing the contribution of the missing or poor quality modal data, and significantly improving the accuracy and robustness of the system in a real complex environment. Moreover, the present application not only outputs the risk judgment result, but also provides "key evidence fragments" and "context-related information". For example, it can accurately locate which area in the image or which time period in the voice leads to a high-risk judgment, providing clear evidence support for the research and judgment of regulatory personnel and greatly improving the disposal efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 A flowchart of a method for analyzing risk information in public data based on multi-modal artificial intelligence provided by the embodiment of the present application is shown in Figure 2 A flowchart of the overall technical scheme provided by the embodiment of the present application is shown in DETAILED DESCRIPTION

[0017] The present application will be described in detail below through embodiments.

[0018] With the rapid development of the Internet and digital communication technology, a large amount of data has been generated on various public network platforms (such as social media, forums, video sharing websites, etc.). These data, while containing great value, also contain a large amount of risk information with security threats. Such information usually has the characteristics of concealment, inducement and diffusion, and poses a potential threat to personal rights and interests and the network ecological environment.

[0019] Currently, the identification of such risk information mainly relies on traditional information filtering technology. In the prior art, a method based on rule and keyword matching is used, which relies on a keyword list or regular expression rule library predefined by artificial matching. Since the rule library needs to be continuously maintained by artificial, it is difficult to exhaust all variants; and this method is extremely inflexible and cannot effectively deal with new, transformed or metaphorically expressed risk content, with high false positive rate and false negative rate.

[0020] In order to solve the above technical problems existing in the prior art, the embodiment of the present application provides a method for analyzing risk information in public data based on multi-modal artificial intelligence, as shown in Figure 1 The method can include: S110, asynchronously collecting multi-modal data from a plurality of heterogeneous public data sources, and using modal-specific cleaning strategies to pre-process the multi-modal data.

[0021] Multimodal data includes text, speech, images, and video.

[0022] In practice, this step involves deploying a distributed crawler cluster to asynchronously collect text, voice, image, and video data from multiple heterogeneous public data sources, such as social media, news websites, forums, and video platforms. The collected raw data typically contains a large amount of noise and requires preprocessing using modality-specific cleaning strategies.

[0023] For text data, the Scrapy framework and BeautifulSoup library in Python were used to scrape and parse web page text. The NLTK library and the re (regular expression) module were used for cleaning: invalid URLs and special characters were removed using regular expressions; nltk.word_tokenize() was used for tokenization, and nltk.pos_tag() was used for part-of-speech tagging, laying the foundation for subsequent feature extraction.

[0024] For example, for a text message “Dear, click this link https: / / fake.link to receive a high-value red envelope!!!”, after noise reduction and removal of invalid links, the word segmentation result is ["Dear", "click", "receive", "high-value", "red envelope"], and the part-of-speech tagging result is [("Dear", "NN"), ("click", "VB"), ("receive", "VB"), ("high-value", "JJ"), ("red envelope", "NN")].

[0025] Part-of-speech tagging can be viewed as a classification problem, where the model needs to predict the part-of-speech label for each word. The following is a simplified mathematical representation: .

[0026] in: It is the i-th word. T It is the set of all possible part-of-speech tags. θ These are model parameters.

[0027] In the given word and model parameters θ In the case of part-of-speech tags t The posterior probability.

[0028] For audio data, the Librosa library is used to load audio files, and noise reduction is performed using the noise.reduce_noise() function based on spectral subtraction. Endpoint detection is performed using the pyaudio library. The speech recognition API is then used to convert the speech to text. For example, suppose a speech message is "Hello, this is customer service. Your account has an anomaly; please transfer funds to the designated account immediately to unfreeze it." After speech conversion, the above text message is obtained.

[0029] For image and video data, preprocessing is performed using the OpenCV library. For images, the `cv2.calcHist` function is used to calculate the histograms for different color channels when extracting color histograms. Texture features are extracted using the Gray-Level Co-occurrence Matrix (GLCM) method, where a custom function calculates the GLCM matrix and extracts relevant texture features. For videos, OpenCV's `VideoCapture` function is used to read the video, extract keyframes, and perform feature extraction operations on the keyframes similar to those used for images.

[0030] The Gray-Level Co-occurrence Matrix (GLCM) can construct a matrix reflecting the texture characteristics of an image, thereby extracting useful features for image analysis and classification. The mathematical formula is as follows: GLCM(i,j)=∑_{x,y}[I(x, y)=i and I(x+d·cosθ,y+d·sinθ)=j], where I(x,y) represents the gray value of the image at position (x,y). i and j are two distinct gray values. d is the distance between pixel pairs. θ is the angle between pixel pairs. GLCM(i,j) records the number of pixel pairs with gray values ​​i and j at distance d and angle θ.

[0031] S120 uses a pre-trained deep model to extract high-dimensional features from each preprocessed modality of data.

[0032] Among them, high-dimensional features include semantic features of text, acoustic and semantic features of speech, visual and spatial features of images, and temporal variation features of video.

[0033] This step utilizes a pre-trained deep model to extract high-dimensional, abstract features from the cleaned data.

[0034] For text semantic features, the preprocessed text is input into the pre-trained BERT-base model, and the output vector (768 dimensions) corresponding to the [CLS] label is taken as the context embedding semantic representation of the entire sentence.

[0035] For speech acoustic and semantic features, the Mel-frequency cepstral coefficients (MFCC, e.g. 13-dimension) and linear prediction cepstral coefficients (LPCC) are extracted using the Librosa library. The recognized text from the speech is also input into the BERT model to extract its semantic features.

[0036] For image visual and spatial features, the image is input into a pre-trained ResNet-50 model with the last fully connected layer removed, and the 2048-dimension vector after global average pooling is taken as the deep visual feature. Meanwhile, the bounding box coordinates ([x_center, y_center, width, height]) and its class confidence from the target detection algorithm (e.g. YOLO) are taken as the spatial feature.

[0037] For video temporal change features, the cosine distance between the deep visual features (e.g. ResNet features) of adjacent frames in the extracted key frame sequence is calculated to form a temporal difference sequence, which represents the trend of video content change over time.

[0038] S130, evaluate the availability state of each modality data to generate a modality availability vector.

[0039] The modality availability vector is a continuous value vector, and the modality availability vector is generated by evaluating the quality of each modality data using a lightweight neural network. The value at each position in the modality availability vector represents the availability probability of the corresponding modality data.

[0040] The modality availability vector can be a four-dimensional continuous vector. A lightweight neural network (e.g. two-layer fully connected layer) is used to evaluate the quality of each modality data. The input of the network is the original statistical features of each modality (e.g. text length, speech signal-to-noise ratio, image clarity, video frame number), and the output is the corresponding availability probability value.

[0041] S140, obtain the confidence score of the high-dimensional feature of each modality data output by the pre-trained deep model.

[0042] Specifically, the confidence score is directly derived from the output of each feature extraction model. For BERT text features, the entropy of the output probability distribution is calculated, and the lower the entropy, the higher the confidence; for YOLO target detection, the object class confidence output by it is directly used; for MFCC features, the reconstruction error of the extraction process can be taken as a reference for confidence.

[0043] S150, input the modality availability vector and the confidence score into the multi-modal fusion model to calculate a set of normalized attention fusion weights representing the importance of each modality feature.

[0044] The multi-modal fusion model includes a gating network based on an attention mechanism.

[0045] In S160, the high-dimensional features of each modality data are weighted and fused according to the attention fusion weight to generate a multi-modal joint feature representation. Based on the multi-modal joint feature representation, a risk probability and a type label are output.

[0046] All the features, the modality availability vector v avail and the confidence score vector v conf are input into the multi-modal fusion model. The core of the model is a gating network based on an attention mechanism, which calculates a set of normalized attention fusion weight vectors a by the formula: a = softmax(W * [v avail o v conf] + b). The weight dynamically reflects the importance of each modality feature under the current input. W and b are learnable parameters, o represents element-wise multiplication, and [·] represents the vector concatenation operation.

[0047] Subsequently, the model performs weighted summation on each modality feature according to the weight a to generate a unified multi-modal joint feature representation, and finally outputs a risk probability and a type label through a fully connected classification layer.

[0048] In S170, when the output risk probability exceeds a preset threshold, a structured early warning event is generated and automatically pushed to a subsequent handling system to trigger a linkage response. The structured early warning event at least includes a risk type, a confidence, a key evidence segment and context-related information.

[0049] When the risk probability exceeds a preset threshold (such as 0.85), the system generates a structured early warning event. The event is encapsulated in JSON format, including the risk type, the confidence, the key evidence segment (the timestamp of the risk information, the text of the risk information link, the bounding box coordinates of the risk picture), and the context-related information (such as the publisher ID, the timestamp, and the propagation path). Subsequently, it is automatically pushed to the risk control handling system through the RESTful API to trigger linkage response mechanisms such as content interception, account restriction, manual review, etc.

[0050] The technical solution provided by this invention, by fusing multimodal features from text, speech, images, and video, rather than relying on single-modal information, can more comprehensively capture the complex patterns of risk information. By introducing an attention-based gating network, the fusion weights can be dynamically adjusted according to data availability and quality, as well as the model's own confidence level. This automatically reduces the contribution of missing or poor-quality data from certain modalities, significantly improving the system's accuracy and robustness in real-world complex environments. Furthermore, this invention not only outputs risk assessment results but also provides "key evidence fragments" and "contextual information." For example, it can precisely pinpoint which region in an image or which time period in a speech led to the high-risk assessment, providing clear evidence support for regulatory personnel's judgment and greatly improving processing efficiency.

[0051] Furthermore, the entire process, from data collection, preprocessing, feature extraction, fusion analysis to final risk warning and handling, is highly automated. The system can perform millisecond-level inference on massive amounts of publicly available data and instantly complete warning reporting and handling triggering after risk assessment, forming an efficient closed loop. This greatly shortens response time, effectively curbs the spread of risk information, and protects user rights.

[0052] exist Figure 1 Based on the illustrated embodiment, in order to ensure that the multimodal fusion model can continuously adapt to the dynamic evolution of risk information patterns, as an implementation of this embodiment, the method further includes an update process for the multimodal fusion model, which specifically includes the following steps: The online learning framework continuously monitors the classification performance metrics of the multimodal fusion model on newly labeled data; when the classification performance metrics fall below a preset threshold, the incremental learning process is automatically triggered.

[0053] The incremental learning process includes: acquiring new labeled data, fine-tuning the multimodal fusion model using the new labeled data, and introducing a historical data replay mechanism to maintain the performance of the multimodal fusion model on historical tasks. The new labeled data is obtained by establishing a continuous learning mechanism to monitor new data in the public data stream and label it regularly.

[0054] Specifically, the system deploys an online learning framework (which can be implemented using machine learning pipeline platforms such as Kubeflow or TFX). This framework has a built-in performance monitor that continuously evaluates the model's performance on newly labeled data. Monitored performance metrics include, but are not limited to: accuracy, precision, and recall. F1-Score, which is the harmonic mean of precision and recall, is a commonly used core metric; AUC, which stands for Area Under the ROC Curve, is used to measure the overall classification performance of the model.

[0055] Furthermore, preset thresholds are crucial for triggering model updates. For example, if the F1-Score is below 0.80 for three consecutive hours (this threshold can be adjusted based on business sensitivity), the monitoring system will automatically send a signal to the model management platform, triggering the incremental learning process. This judgment based on continuous performance degradation, rather than a fixed time period, ensures the necessity and efficiency of the update behavior.

[0056] Once triggered, the incremental learning process starts immediately. The incremental learning process mainly includes the following steps: (1) Acquiring New Annotated Data. The system acquires recently collected and annotated data from a dedicated new data buffer. This data comes from continuous monitoring of public data streams and is quickly annotated by a professional data annotation platform (or high-value samples selected in conjunction with active learning strategies) to form a high-quality incremental training set.

[0057] (2) The system will load a copy of the model from the current online service for fine-tuning. During fine-tuning, a small learning rate (e.g., 1e-5), an optimizer (e.g., Adam), and a loss function (e.g., cross-entropy loss) are used to train the model so that it can quickly learn new knowledge from the incremental training set.

[0058] Furthermore, to overcome the catastrophic forgetting problem (i.e., forgetting old knowledge after learning new knowledge) that is most challenging in incremental learning of neural networks, this system introduces an experience replay buffer. This buffer stores representative samples sampled from historical training data (e.g., retaining a certain number of samples from each historical risk category using a stratified sampling strategy). During fine-tuning, the training data for each iteration consists of a mixture of new and historical data in a certain ratio (e.g., 7:3). In this way, the model is constantly reminded of old patterns while adapting to new ones, thus achieving a balance between exploration and memorization and significantly maintaining its performance on historical tasks.

[0059] After incremental fine-tuning, the new model is comprehensively evaluated on a test set isolated from the online environment. This test set also includes both new and old data to ensure that the performance improvement is comprehensive. The new model is only approved for deployment when a comprehensive performance metric (such as the average F1-Score on both new and old data) significantly outperforms the old model. Subsequently, it is automatically deployed, seamlessly replacing the old model, completing a full iterative update.

[0060] The implementation of this update process brings the following significant beneficial effects to the present invention: (1) Realizes lifelong learning and dynamic adaptability of the model: This invention breaks the static paradigm of "train once, use forever" of traditional AI models and gives the model the ability to continuously evolve. It captures and adapts to the latest dynamics of risk patterns in real time, effectively solving the problem of rising false negative rates caused by outdated models, and ensuring that the risk defense system always maintains foresight and high sensitivity.

[0061] (2. Fundamentally solves the technical challenge of catastrophic forgetting: By introducing a historical data playback mechanism, this invention provides a key technical guarantee for incremental learning. It does not simply fine-tune with new data, but cleverly mixes new and old data, allowing the model to continuously review old knowledge while learning new knowledge. This ensures that the model does not lose its memory of existing risk patterns while rapidly learning emerging risk patterns, thus maintaining a comprehensive and stable high-precision recognition capability.)

[0062] (3) A fully automated model operation and maintenance closed loop has been constructed, greatly reducing labor costs. From performance monitoring, trigger judgment, data preparation, incremental training to model verification and deployment, the entire process is fully automated and requires no manual intervention. This greatly reduces the manpower burden of machine learning operation and maintenance, improves system maintenance efficiency, and avoids delays and errors that may be caused by manual operation, ensuring the timeliness and reliability of the risk response system.

[0063] (4) Improved system resource utilization efficiency. Compared with the method of periodically retraining the model with the full dataset, this incremental learning process only uses new data and a small number of historical core samples, significantly reducing computational resource consumption and training time. This allows the model to be iteratively optimized more frequently and at a lower cost, achieving the best balance between resource consumption and model performance.

[0064] exist Figure 1 Based on the illustrated embodiment, in order to ensure the quality of the input data and lay a solid foundation for subsequent high-precision analysis, as one implementation of this invention, a mode-specific cleaning strategy is used for data preprocessing of multimodal data, which may include the following steps: Step a1 involves removing special characters and invalid links from the text using regular expressions, and then performing word segmentation and part-of-speech tagging using natural language processing tools.

[0065] Specifically, the core objectives of text data preprocessing are to remove noise, achieve standardization, and provide structured input for deep models.

[0066] The method for removing special characters and invalid links is to use Python's `re` library (regular expressions) to perform pattern matching and replacement. By writing specific regular expression patterns (such as `r'[^\w\s]'` to match all characters except letters, numbers, underscores, and whitespace; and `r'https?: / / \S+'` to match all HTTP / HTTPS links), meaningless symbols, garbled characters, and potentially malicious or disruptive links in text can be efficiently and in batches.

[0067] The word segmentation and part-of-speech tagging process is as follows: The cleaned text is fed into a natural language processing tool (such as the NLTK library or Jieba (Chinese)) for further processing. The `word_tokenize()` function is used to segment the continuous string into independent lexical units (tokens). For example, the sentence "Click the link to receive a red envelope" will be segmented into ['click', 'link', 'receive', 'red envelope']. Then, the `pos_tag()` function is used to tag each segmented word with its part-of-speech tag (such as noun, verb, adjective, etc.). For example, the tagging results for the above segmentation are [('click', 'VB'), ('link', 'NN'), ('receive', 'VB'), ('red envelope', 'NN')]. This step provides rich linguistic structural information for subsequent semantic feature extraction, helping the model to understand the text content more accurately.

[0068] Step a2: After noise reduction and endpoint detection, the speech is converted into text.

[0069] The endpoint detection process includes: calculating the short-time energy and short-time zero-crossing rate of the speech; setting a first threshold for preliminary detection of possible speech segments; setting a second threshold to confirm reliable start and end points of speech segments from the preliminary detection segments; for the region between the first and second thresholds, combining the zero-crossing rate for speech / voiceless speech determination to accurately determine the start and end positions of speech segments; and ensuring that the first threshold is lower than the second threshold.

[0070] Specifically, speech preprocessing aims to enhance signal quality, locate effective speech segments, and convert them into information that can be processed by text models.

[0071] The noise reduction process involves using an audio processing library based on spectral subtraction. This algorithm estimates and analyzes the noise spectrum in the audio signal and subtracts it from the original speech spectrum, thereby effectively suppressing interference such as ambient background noise and current noise, and significantly improving the clarity and signal-to-noise ratio (SNR) of the speech signal.

[0072] Endpoint detection (dual-threshold method) is a key technique for accurately locating speech segments. Specifically, it may include the following steps: Step 1: Calculate short-time energy and zero-crossing rate. First, divide the speech signal into frames (usually 20-40ms per frame) and calculate the short-time energy (STE, representing volume) and short-time zero-crossing rate (ZCR, representing frequency) for each frame.

[0073] Step 2: Set dual thresholds and judgment.

[0074] First threshold (low threshold). Set a low STE threshold (e.g., 1 / 10 of the global maximum energy). Frames with STE exceeding this threshold are marked as "possible speech segments", but their start and end points are not yet precise.

[0075] Second threshold (high threshold). Set a high STE threshold (e.g., 1 / 3 of the global maximum energy). Search both ends of the "possible speech segments", and regions where the STE consistently exceeds this high threshold are identified as reliable speech segments.

[0076] For the transition region between high and low thresholds, a fine judgment is made by combining high zero-crossing rate characteristics (voiceless sounds, such as the fricative / s / with a high ZCR), so as to accurately locate the true start and end of the speech segment while eliminating voiceless noise, thus avoiding the truncation of effective speech.

[0077] Next, the detected valid speech segments are input into the automatic speech recognition system, which converts them into corresponding text information to provide input for subsequent semantic analysis.

[0078] Step a3: For images and videos, use the OpenCV library to unify the format, adjust the size, and extract keyframes, and further extract color histograms, texture features, and object detection information.

[0079] Specifically, image and video preprocessing focuses on format standardization, information condensation, and feature extraction to reduce computational complexity and extract key information.

[0080] First, the `cv2.imread()` and `cv2.VideoCapture()` functions from the OpenCV library are used to read image and video files. Then, `cv2.cvtColor()` is used to convert the image color space to RGB. Finally, the `cv2.resize()` function is called to scale all image and video frames to the fixed size required by the model (e.g., 224x224 pixels) to eliminate resolution differences and ensure input consistency.

[0081] Secondly, for video, to avoid processing a large number of redundant frames, a shot boundary detection-based method is adopted. The difference in the RGB histogram between adjacent frames is calculated. When the difference value exceeds a certain threshold, it is determined to be a shot change, and the frame is extracted as a keyframe, thus using a small number of representative frames to summarize the main content of the video.

[0082] Furthermore, the cv2.calcHist() function is used to calculate the histogram of each channel of the image in the HSV color space, quantifying the color distribution of the image. It is insensitive to object rotation and slight deformation, and has good robustness.

[0083] Furthermore, the Gray Co-occurrence Matrix (GLCM) can also be used. The GLCM is calculated using skimage.feature.greycomatrix(), and then greycoprops() is used to extract statistics such as contrast, correlation, energy, and homogeneity from it, which numerically describe the texture roughness and regularity of the image.

[0084] Finally, object detection information is extracted. Deep learning-based object detection algorithms (such as YOLOv5 or Faster R-CNN) are employed. The model outputs the category labels (such as "face," "car," and "logo") of objects of interest in the image, along with their bounding box information (i.e., the precise coordinates of the object's location in the image). This information is crucial for understanding the semantics of the image.

[0085] The present invention implements the above-described modality-specific preprocessing procedure, which brings the following significant beneficial effects: (1) It greatly improves the quality and consistency of input data. Through targeted cleaning algorithms, it effectively removes the inherent and diverse noises (text garbled characters, background noise of speech, differences in image resolution, etc.) in various modal data, providing pre-processed data for downstream AI models and laying the foundation for high-precision analysis from the source.

[0086] (2) Enhanced richness and robustness of feature representation. This invention does not perform simple data cleaning, but rather deep feature preprocessing. It simultaneously extracts low-level statistical features (such as color and texture) and high-level semantic features (such as object detection results and text part-of-speech tags). This multi-granularity feature representation complements each other, enabling the subsequent fusion model to capture both detailed textures and understand macro-level semantics, greatly enhancing the system's ability to identify diverse and mutated risky content.

[0087] (3) It achieves the best balance between computational efficiency and model performance. By extracting keyframes and filtering high-quality data (such as accurate speech endpoint detection), the amount of data to be processed is significantly reduced without losing core information. This significantly reduces the computational overhead and time cost of subsequent feature extraction and model inference, enabling the entire system to efficiently process massive public data streams and meet real-time requirements.

[0088] exist Figure 1 Based on the illustrated embodiment, as one implementation of the present invention, the semantic features of the text are represented by context embedding vectors generated by a pre-trained language model. The acoustic and semantic features of speech include Mel frequency cepstral coefficient features extracted from audio signals, linear prediction cepstral coefficient features, and semantic keyword features extracted from speech-to-text. The visual and spatial features of an image include depth visual features extracted through a convolutional neural network and object detection bounding box information obtained through an object detection algorithm; The temporal variation features of the video are extracted by calculating the feature differences or optical flow information between keyframe sequences.

[0089] The semantic features of text, the acoustic and semantic features of speech, the visual and spatial features of images, and the temporal variation features of video will be described in detail below.

[0090] I. Extraction of semantic features from text.

[0091] The semantic features of the text are represented by context embedding vectors generated by a pre-trained language model. Specifically, BERT (Bidirectional Encoder Representations from Transformers) or its variants (such as RoBERTa and ERNIE) are used as feature extractors.

[0092] The implementation process is as follows: First, the preprocessed (segmented and cleaned) text sequence (e.g., "Your account is at risk and needs verification") is input into the BERT model. The model generates a high-dimensional context-aware vector for each token in the sequence. Typically, the output vector corresponding to the [CLS] token (e.g., 768-dimensional or 1024-dimensional) is taken as the semantic representation summary of the entire sentence. This vector deeply integrates the contextual information of all words in the sentence, accurately capturing subtle semantic differences and potential risk intentions.

[0093] Compared with traditional bag-of-words models or static word vectors (such as Word2Vec), Transformer-based contextual embedding can effectively solve the problem of polysemy and is more robust to spoofing techniques such as word transformation and word order adjustment, which greatly improves the accuracy of text risk identification.

[0094] II. Speech Acoustics and Semantic Feature Extraction: Speech feature extraction integrates low-level acoustic features and transformed high-level semantic features to form a comprehensive feature representation. Acoustic feature extraction includes: Mel-frequency cepstral coefficients (MFCCs) are extracted using the `librosa.feature.mfcc` function from the Librosa library. MFCCs simulate the characteristics of human hearing. First, the short-time power spectrum of the speech signal is filtered using a Mel-filter bank, and then a discrete cosine transform (DCT) is performed to obtain the cepstral coefficients. They effectively characterize the timbre and tone quality features of speech.

[0095] Linear predictive cepstral coefficients (LPCC). Based on linear predictive analysis (LPC), they are calculated using relevant functions such as lpc from the python_speech_features library. LPCC focuses on characterizing the spectral envelope of the speech signal and is highly sensitive to the formant structure of the speech. MFCC and LPCC features complement each other, together constituting the acoustic feature representation.

[0096] Semantic feature extraction. The valid speech segments after endpoint detection are converted into text using an Automatic Speech Recognition (ASR) engine. This text is then input into the pre-trained BERT model described above, and the context embedding vectors corresponding to the [CLS] tags are extracted as semantic keyword features of the speech content. This enables the model not only to capture sound characteristics but also to understand the content of the speech.

[0097] III. Image visual and spatial feature extraction.

[0098] Image feature extraction combines deep semantic information with explicit spatial structural information.

[0099] Regarding deep visual features, use deep convolutional neural networks (CNNs) pre-trained on large datasets such as ImageNet, such as ResNet-50, VGG16, or EfficientNet. Remove the last classification layer of the network and take the output of the final global average pooling layer as a high-dimensional feature vector (e.g., ResNet-50 outputs 2048 dimensions). This feature vector contains high-level, abstract semantic information about the image.

[0100] Regarding object detection bounding box information: Advanced object detection algorithms (such as YOLOv5 and Faster R-CNN) are used to analyze the image. This algorithm outputs the category labels (such as "face," "ID card," and "screen") for all detected object instances in the image, along with their precise bounding box coordinates (typically represented as normalized center point x, y and width and height w, h). This spatial information is crucial for understanding the image's composition and identifying specific high-risk objects.

[0101] The visual and spatial features of the image also include low-level visual features extracted by traditional image processing algorithms, including color histograms and texture features; the color histogram is calculated in the HSV color space, and the texture features are extracted using gray-level co-occurrence matrix or local binary mode algorithms.

[0102] Compared to the commonly used RGB (red, green, blue) color space, the HSV (Hue, Saturation, Value) color space is more in line with how humans perceive color. In risk information identification, risk images may be taken under different lighting conditions or have undergone simple color adjustments. Using the HSV color space can more stably capture their core color distribution and reduce environmental interference.

[0103] Texture features describe visual patterns such as texture, roughness, and regularity of an object's surface. Integrating traditional features such as color histograms and texture features with deep visual features brings the following significant advantages to this invention: (1) Feature complementarity, improving model robustness. Deep learning features are good at capturing high-level semantic information (such as "what it is"), but are relatively sensitive to changes in lighting, rotation, etc. Traditional features (color, texture) are invariant to geometric and optical ornamentation and are good at describing low-to-mid-level visual attributes (such as "how it looks"). The fusion of the two makes the model both "understand the connotation" and "see the surface", making it more robust in dealing with fraudulent images processed by filters, color adjustment, scaling, etc. (2) Enhanced interpretability, providing dual evidence: When the model determines that an image is high-risk, we can backtrack and analyze which color distributions are abnormal (such as color histograms that are very different from normal images) or which areas have suspicious textures (such as abnormally high GLCM contrast, suggesting forgery), providing AI's decision-making with human-understandable, quantifiable physical evidence, which is a powerful complement to the black-box decision-making of deep neural networks. (3) Computational efficiency, supplementing model input. These traditional features have minimal computational overhead and can be extracted almost in real time. Inputting them as additional feature channels along with deep features into the fusion model provides the model with more dimensional, low-cost information, which helps improve the accuracy and confidence of the final classification.

[0104] IV. Extraction of temporal variation features from video.

[0105] Video feature extraction aims to capture its dynamic characteristics that evolve over time.

[0106] Regarding the feature-based difference approach: First, deep visual features (such as ResNet features for each frame) are extracted from the video keyframe sequence using the methods described above. Then, the cosine or Euclidean distance between the feature vectors of adjacent keyframes is calculated, forming a temporal difference signal sequence. This sequence clearly reflects the drasticness and rhythm of changes in the video content.

[0107] Regarding methods based on optical flow information, dense optical flow fields are calculated using algorithms such as cv2.calcOpticalFlowFarneback or DualTVL1 in OpenCV. The optical flow field represents the motion vector of each pixel in a video across consecutive frames. These motion vectors can be statistically analyzed (e.g., calculating the average optical flow amplitude and orientation histogram) or input into a convolutional network to extract temporal features representing global motion patterns. This method effectively captures motion information in a scene, such as object movement and camera zoom.

[0108] The present invention achieves the following significant beneficial effects through the above-described multimodal feature extraction scheme: (1) A hierarchical and complementary comprehensive feature representation system was constructed. This invention does not rely solely on a single type of feature, but extracts features from multiple dimensions and levels, including acoustics (MFCC, LPCC), shallow semantics (part-of-speech tags), deep semantics (BERT embedding), visual semantics (CNN features), spatial structure (bounding boxes), and temporal dynamics (feature differences, optical flow). This three-dimensional feature engineering strategy ensures that risk information can be effectively captured and represented regardless of which dimension it is hidden in, laying a solid data foundation for subsequent high-precision fusion recognition.

[0109] (2) It greatly enhances the model's ability to generalize to complex and ever-changing risk patterns. By combining the powerful semantic understanding capabilities of the pre-trained model with the robustness of traditional signal processing features, this invention overcomes the limitations of single-type features. For example, BERT features are effective when the wording changes (textual semantic changes); MFCC / LPCC features are effective when it uses speech synthesis technology (acoustic anomalies); and object detection features are effective when it displays screenshots of phishing websites (visual spatial information). This multi-faceted capability enables the system to cope with various unknown and mutated risk patterns.

[0110] (3) It provides interpretable evidence to support risk assessment. The extracted features themselves contain interpretable evidence. For example, the target detection box directly indicates the risk area ("Where is the image problematic?"), and the text keywords transcribed by ASR indicate the risk content ("What was said?"). These features are not only used for model classification, but also serve as part of the output of structured evidence, which greatly assists regulators in verifying and assessing the machine's judgment results, and realizes intelligent risk control through human-machine collaboration.

[0111] exist Figure 1Based on the illustrated embodiment, as one implementation of the present invention, for text, the key evidence fragment is a sentence containing high-risk keywords; for speech, the key evidence fragment is the timestamp fragment where the high-risk words are located; for images or videos, the key evidence fragment is the image region or video frame sequence where the target detection box is located.

[0112] To provide clear and traceable evidence for risk identification, this invention enables precise location and extraction of key evidence fragments for different modalities. This function transforms the abstract decision-making process of the model into concrete and interpretable evidence presentation, as detailed below: 1. Extraction of key evidence fragments from text.

[0113] For text data, key evidence fragments are sentences containing high-risk keywords. The implementation process is as follows: The system maintains a dynamically updated high-risk keyword database (e.g., containing "transfer", "safe account", "brushing orders", "unsubscribe reply T", etc.). When the model determines that a piece of text is high-risk, the system segments the text into sentences. Subsequently, each sentence is scanned using exact matching or semantic similarity calculation (by comparing sentence embeddings with word embeddings in the keyword database). The first sentence detected to contain a high-risk keyword or a highly similar word is extracted as a key evidence fragment.

[0114] For example, given the text "Hello, I am e-commerce customer service. Your package has been lost. Please click the link http: / / fake.com to process a refund," the model will extract the sentence "Please click the link http: / / fake.com to process a refund" which contains the high-risk keyword "link" and a suspicious URL as key evidence.

[0115] 2. Extraction of key audio evidence segments.

[0116] For audio data, the key evidence segment is the timestamp segment containing high-risk words. The process is as follows: First, using an Automatic Speech Recognition (ASR) system and its timestamp alignment function, the start and end times corresponding to each word in the transcribed text are obtained. Then, a similar scan is performed against a high-risk keyword database. Once a high-risk word is identified, the system can accurately locate the time interval of that word in the original audio stream (e.g., from 12.3 seconds to 12.8 seconds) based on the timestamps provided by the ASR.

[0117] For example, a voice message is transcribed as "Please transfer funds to safe account xxx immediately," and the high-risk term "safe account" is identified, with timestamps at [05.2 seconds, 05.9 seconds]. The system will then extract and save the audio segment from 05.0 seconds to 06.5 seconds (with a small buffer time allowed before and after) as key evidence.

[0118] III. Extraction of key evidence fragments from images and videos.

[0119] For image or video data, key evidence is the image region or video frame sequence containing the object detection bounding box. For images, the process is as follows: when an object detection algorithm (such as YOLO) identifies a pre-defined high-risk object category in an image, the bounding box coordinates of that object are the most direct key evidence. The system will either draw this bounding box directly on the original image or crop the image region within the bounding box and save it as evidence.

[0120] For video, the implementation process is similar to that for images, but with the addition of a temporal dimension. The system continuously performs target detection on video frames. Once a high-risk target is detected in any one or more frames, evidence extraction is triggered. The extracted content includes: Keyframe sequence: Frames containing high-risk targets and their preceding and following frames (e.g., 1 second before and 2 seconds after the target is detected) form a short video clip of evidence.

[0121] Spatial location information: The bounding box coordinates of high-risk targets in these keyframes.

[0122] The present invention achieves the following significant beneficial effects through the above-mentioned key evidence fragment extraction scheme: (1) It greatly enhances the explainability and credibility of risk management. This invention goes beyond the black-box model that only provides "yes / no" judgments, and can accurately locate the specific data fragments that lead to the risk judgment (which sentence, which time point of sound, which area of ​​the image). This provides regulators with clear, intuitive, and irrefutable evidence for judgment, enabling them to quickly understand the decision-making basis of AI, greatly enhancing their trust in the results of automated management, and promoting the efficient operation of human-machine collaboration.

[0123] (2) A significant leap in processing efficiency has been achieved. By providing precisely located and structured evidence fragments, rather than raw massive amounts of data (such as a complete video of up to one hour or lengthy chat logs), this invention greatly reduces the time required for manual review. Reviewers do not need to browse through all the content from the beginning; they can directly view the "evidence" fragments highlighted by the system, thereby achieving second-level analysis and processing, significantly accelerating the response speed to risk events, and effectively curbing the further spread of risks.

[0124] (3) It provides a high-quality data foundation for risk tracing and pattern analysis. The extracted key evidence fragments are condensed and labeled. These fragments can be systematically archived into the risk case library for subsequent in-depth analysis, correlation tracing, and attack pattern mining. For example, analyzing the frequently occurring sentence structures in different risk scripts, or tracing the propagation path of a specific logo on multiple platforms, can provide data support for developing more forward-looking defense strategies.

[0125] exist Figure 1 Based on the illustrated embodiment, as one implementation of the present invention, the method further includes a risk pattern mining process, which includes: Regularly perform cluster analysis on high-risk events identified by the multimodal fusion model to extract emerging risk pattern features. Use these risk pattern features as prior knowledge to feed back into the training phase of the multimodal fusion model to construct targeted difficult example samples, thereby enhancing the model's discriminative ability.

[0126] Specifically, to enable the risk identification system to self-evolve and proactively discover unknown threats, this invention designs a closed-loop risk pattern mining process. This process, as an advanced intelligent module, runs periodically offline, aiming to extract knowledge from historical data and feed it back into model training, thereby forming a continuously self-enhancing intelligent system. Its specific implementation is as follows: I. Cluster analysis of high-risk events.

[0127] The system periodically (e.g., weekly) exports all high-risk events identified by the multimodal fusion model (including all their multimodal data, feature vectors, and final labels) to form a high-risk event set. Subsequently, unsupervised clustering algorithms (such as density-based DBSCAN clustering, hierarchical clustering, or deep learning-based autoencoder clustering) are used to mine these events.

[0128] The implementation process is as follows: First, all features of each high-risk event (text BERT vectors, speech MFCC features, image CNN features, etc.) are concatenated or mapped to a unified risk feature space through a dedicated projection network. Then, cluster analysis is performed in this space. The algorithm automatically groups events with similar features into several clusters.

[0129] Each cluster represents a potential risk pattern. For example, the clustering results might show that: cluster A mostly contains text and QR code images related to "risky part-time jobs"; cluster B mostly contains voice calls and fake images impersonating relevant departments; while cluster C is a new pattern that is unprecedented and composed of specific image textures and novel dialogue.

[0130] II. Extraction of characteristics of emerging risk patterns.

[0131] For each cluster obtained after clustering, the system will conduct in-depth analysis to extract its pattern features.

[0132] Regarding text patterns: For all text within a cluster, use topic modeling (such as LDA) or keyword frequent itemset mining to extract the core topic words, high-frequency word combinations, and sentence structures (such as "Your flight has been cancelled, please process a refund and rebooking") under that pattern.

[0133] Regarding visual / acoustic patterns: Calculate the centroid or average vector of image / speech features within a cluster as representative visual / acoustic features of that pattern. For example, a risk pattern might correspond to a specific, recurring counterfeit logo design style or audio background noise.

[0134] Regarding multimodal association patterns, analyze the association rules between features of different modalities. For example, "When keyword X appears in text and object Y appears in image, the probability of risk is extremely high."

[0135] III. Construction of a Difficult Example Library and Model Retraining: Using the extracted pattern features as prior knowledge and feeding them back into the model's training phase is key to achieving a leap in capabilities.

[0136] (1) Constructing targeted difficult case samples. The system will proactively construct or screen difficult case samples based on the characteristics of the new patterns discovered.

[0137] (2) For variants of known patterns, samples with similar features but not successfully identified by the model can be retrieved from unlabeled data (missed samples).

[0138] (3) For new models, a batch of highly realistic difficult sample samples can be synthesized based on the extracted keywords and visual features using data augmentation techniques (such as text paraphrasing and image style transfer).

[0139] These samples are added to a dedicated hard example library. During the model's next periodic training or incremental learning cycle, samples are taken from this hard example library and mixed with the regular training data to train the model. This is equivalent to having the model specifically practice problems it is not good at or has never seen before, thereby forcing the model to learn more robust and refined decision boundaries, significantly improving its ability to discriminate emerging risks and complex evasion techniques.

[0140] This invention achieves the following significant benefits by introducing a risk pattern mining process: (1) It endows the system with the ability to proactively discover unknown threats: This invention transcends the paradigm of passively identifying known risks. Through unsupervised clustering, it can proactively mine emerging and potential unknown risk patterns from historical data. This gives the system an early warning function, enabling it to capture early characteristics of a new method before it breaks out on a large scale, providing regulators with valuable early warning time and realizing a strategic shift from "post-event response" to "pre-event warning".

[0141] (2) A continuously self-optimizing closed-loop learning ecosystem has been formed: This invention constructs a complete "identification-mining-feedback-enhancement" closed loop. The risk events identified by the model become the raw materials for mining, and the newly mined knowledge, in turn, enhances the model's identification ability. This positive feedback loop makes the entire system no longer static, but an intelligent life form that can continuously learn from data and evolve, fundamentally solving the problem of AI models becoming outdated quickly due to the iteration of threat methods.

[0142] (3) Significantly improved the robustness of the model to complex adversarial attacks. By constructing a hard sample library in a targeted manner and strengthening the training of the model, the model’s ability to distinguish risky content after slight variations and combined disguises has been greatly enhanced, effectively responding to adversarial attacks taken by risky personnel to evade detection and significantly reducing the false negative rate of the model.

[0143] exist Figure 1 Based on the illustrated embodiment, as one implementation of the present invention, the method further includes a process for constructing and training a multimodal fusion model, which includes the following steps: Step b1: Construct a multimodal fusion model that includes a gating network based on an attention mechanism.

[0144] Specifically, a multimodal fusion model is constructed, including a gating network based on an attention mechanism. This model is an end-to-end deep learning architecture, and its main components are as follows: First, modality-specific encoders. Each modality is equipped with a dedicated feature encoding sub-network. The text modality uses a pre-trained BERT model as the encoder; speech acoustic features are encoded using a one-dimensional convolutional neural network (1D-CNN); and the image / video modality uses a pre-trained ResNet-50CNN as the encoder. These encoders are responsible for further abstracting and refining the original high-dimensional features.

[0145] Second, the attention-based gating network is the core innovative module of the model. Its input consists of the modality availability vector generated in steps S130 and S140 and the confidence scores of each modality feature. This network typically comprises several fully connected layers and non-linear activation functions (such as ReLU), and a set of normalized, dynamic attention fusion weights is calculated using a formula. These attention fusion weights precisely quantify the importance of each modality's information within the current input context.

[0146] Third, the feature fusion and classification layer. The feature vectors output by each modality encoder are weighted and summed with the calculated attention weights to generate a multimodal joint feature representation. Subsequently, this joint representation is input into a classifier consisting of a fully connected layer and a softmax function, which finally outputs the risk probability and type label.

[0147] Step b2 involves collecting a large amount of labeled multimodal data from multiple public data sources to form training and testing sets.

[0148] Collect a large amount of labeled multimodal data from multiple public data sources (such as social media, forums, and video platforms). Data labeling should be carried out by domain experts, who should label each data point as "normal" or "specific risk type".

[0149] The collected labeled data was randomly divided into training and test sets in an 8:2 ratio. The training set was used to directly train the model parameters; the test set was used to evaluate the model's generalization ability and was completely invisible throughout the training process.

[0150] Implement data augmentation strategies (such as randomly cropping or flipping images, and replacing text with synonyms) to increase data diversity and improve model robustness.

[0151] Step b3: Initialize the parameters of the multimodal fusion model.

[0152] For pre-trained encoders (such as BERT and ResNet), parameters pre-trained on large public datasets (such as ImageNet and BookCorpus) are loaded as initial values. This is a transfer learning strategy that can significantly accelerate convergence and improve performance.

[0153] For newly added modules (such as gating networks and classification layers), strategies such as Xavier initialization or Kaiming initialization are used to randomly initialize parameters according to their activation functions to ensure stability in the early stages of training.

[0154] Step b4: Using stochastic gradient descent or Adam optimization algorithm, with cross-entropy loss function as the optimization objective, the multimodal fusion model is iteratively trained using the training set.

[0155] The multimodal fusion model is iteratively trained using the training set with the cross-entropy loss function as the optimization objective and stochastic gradient descent (SGD) or its improved algorithm Adam optimizer.

[0156] Set training hyperparameters, such as batch size (e.g., 32), initial learning rate (e.g., 1e-4), and learning rate decay strategy.

[0157] In each iteration (Epoch), the optimization algorithm calculates the gradient of the loss function with respect to the model parameters and updates the parameters in the opposite direction of the gradient, aiming to continuously minimize the difference between the model prediction and the true label.

[0158] Step b5: After each training cycle, evaluate the performance of the multimodal fusion model using the test set, and adopt an early stopping strategy based on the test set loss to prevent overfitting.

[0159] After each training cycle, the performance of the multimodal fusion model is evaluated using a test set. The model's loss, accuracy, F1 score, and other metrics are calculated on the test set.

[0160] An early stopping strategy is employed to prevent overfitting. The test set loss is continuously monitored, and training is terminated and the model is rolled back to the state of the epoch with the best test set performance when the test set loss no longer decreases over several consecutive epochs (e.g., 10). This effectively avoids the model over-memorizing noise in the training data instead of learning general patterns.

[0161] Step b6: Save the optimal model parameters to obtain the final deployable multimodal fusion model.

[0162] Specifically, the optimal model parameters are saved to obtain the final deployable multimodal fusion model. After training, the final model is evaluated on an independent test set to report its true generalization performance. The model's structural definition and trained parameter weights are saved together to form a complete model asset for subsequent deployment to a production environment for real-time inference.

[0163] Through the above-described systematic model construction and training process, this invention achieves the following significant beneficial effects: (1) End-to-end automated model production is achieved. This invention provides a complete, standardized, and reproducible model training pipeline. From data preparation, model building, parameter initialization, training optimization to verification and storage, the entire process is clear and explicit, greatly reducing the reliance on expert experience, realizing the standardized production of high-performance models, and laying the foundation for large-scale applications.

[0164] (2) It ensures the best performance and generalization ability of the model. By combining transfer learning (pre-trained encoder), strict validation monitoring and early stopping strategy, this process effectively balances the model's fitting ability and generalization ability. It can make full use of the knowledge of large-scale pre-trained models, and prevent overfitting through a carefully designed training mechanism. The final delivered model shows excellent and stable recognition accuracy and robustness on unknown data.

[0165] (3) Provides a reliable guarantee for core innovations. The core innovations such as the "attention-based gating network" in the above embodiments require a stable and efficient training environment to fully realize their effectiveness. This training process ensures that these innovative modules can be trained fully and correctly through reasonable hyperparameter settings, optimization algorithm selection, and loss function design, thereby ultimately achieving dynamic and intelligent multimodal information fusion. This is the technical guarantee for the success of this invention.

[0166] To ensure clarity, the overall technical solution of this invention will be described in detail below with specific examples. For instance... Figure 2 As shown, the process may include the following: 1. Data Preprocessing. Data preprocessing is the foundation for subsequent analysis and aims to transform raw, heterogeneous data into a clean, well-organized format. Specifically, it includes: Noise reduction: For text data, regular expressions are used to remove invalid links, special characters, and garbled text; for speech data, audio processing techniques such as spectral subtraction are used to filter ambient background noise.

[0167] Speech conversion: The effective speech segments after noise reduction are converted into text information using automatic speech recognition (ASR) technology (such as the Kaldi engine) for semantic analysis.

[0168] Format conversion and video frame extraction: Image and video data are processed using the OpenCV library. Images are uniformly converted to RGB format and resized to a fixed size (e.g., 224x224 pixels). For video streams, keyframes are extracted by calculating inter-frame differences or based on lens boundary detection algorithms, significantly reducing the amount of data required for subsequent processing.

[0169] Word segmentation and tagging: Natural language processing tools (such as NLTK) are used to segment and tag the text (including text transcribed from speech) to prepare for the extraction of deep semantic features.

[0170] 2. Multimodal feature extraction. High-dimensional features that can characterize risk information are extracted from the preprocessed data.

[0171] Text feature extraction: Input the cleaned text into a pre-trained language model (such as BERT) to obtain its context embedding vectors, so as to deeply represent the semantic information of the sentences.

[0172] Speech feature extraction: Extract acoustic features (such as Mel-frequency cepstral coefficients (MFCC) and linear predictive cepstral coefficients (LPCC)) from the original audio signal; at the same time, extract semantic keyword features from the speech-to-text.

[0173] Image feature extraction: For keyframes of images and videos, two types of features are extracted: (1) Deep visual features: High-level semantic features are extracted through pre-trained convolutional neural networks (such as ResNet); (2) Low-level visual features: Including color histograms calculated in the HSV color space, and texture features extracted through gray-level co-occurrence matrix (GLCM) or local binary mode (LBP) algorithms.

[0174] Video keyframe feature extraction: In addition to the image features mentioned above, feature differences or optical flow information between keyframe sequences are also calculated to capture the temporal change patterns of video content.

[0175] III. Algorithm Model Construction (Multimodal Fusion Model Construction) A deep learning-based multimodal fusion algorithm model is constructed. This model includes: Modality-Specific Processing Algorithms: Dedicated processing branches are designed for text, speech, and image modalities (e.g., using BERT for text and CNN for images) to further learn and refine the features of each modality. Fusion Processing Algorithm: As the core of the model, an attention-based gating network is employed. This network takes the availability state and feature confidence of each modality as input, dynamically calculates a set of normalized attention weights, and then performs weighted fusion of the features of each modality to generate a unified multimodal joint feature representation. Finally, a fully connected classification layer outputs the final risk probability and type label based on the fused features.

[0176] 4. Model Training and Optimization. This stage aims to optimize model parameters through a data-driven approach.

[0177] Data sample collection: Collect a large amount of labeled data from multiple open source data platforms, including normal data and risk data, and divide them into training set, test set and test set according to proportion.

[0178] Training parameter settings: Set the hyperparameters for training, such as optimizer (Adam), learning rate (e.g., 0.001), batch size (e.g., 24), number of training epochs, etc.

[0179] Model training: The model is iteratively trained using the training set data, and the model parameters are optimized by minimizing the cross-entropy loss function.

[0180] Model evaluation: After a certain number of training rounds, the model's performance metrics, such as accuracy, recall, and F1 score, are evaluated using a test set.

[0181] Model tuning: Based on the evaluation results, adjust the model structure or hyperparameters (such as introducing L2 regularization to prevent overfitting) to improve the model's accuracy and generalization ability.

[0182] 5. Dynamic Model Iteration. To enable the model to adapt to dynamic changes in risk patterns, a continuous learning mechanism is established.

[0183] Data monitoring and updates: Establish a real-time data acquisition pipeline to continuously monitor and collect newly generated data from public data sources, and perform manual or semi-automatic annotations regularly.

[0184] Model evaluation and analysis: Continuously monitor the model's classification performance metrics (such as F1 score) on new data in an online environment.

[0185] Model optimization and tuning: When model performance drops to a preset threshold, the optimization process is automatically triggered. Incremental learning techniques are used to fine-tune the model using newly labeled data.

[0186] Model retraining: During incremental learning, a historical data playback mechanism is introduced to mitigate the problem of catastrophic forgetting. If performance degradation is severe, a complete retraining of the model is triggered.

[0187] Verification and Testing: The updated model must undergo rigorous verification and testing to ensure its performance is improved before it can be redeployed.

[0188] 6. Information identification and processing. Deploy the trained and optimized model in real-world application scenarios.

[0189] When new communication content is generated on the open-source data platform, the system automatically executes the above preprocessing and feature extraction process, and inputs the results into the deployed model for analysis and prediction.

[0190] Information Classification: The model outputs risk probability and type label. The system classifies the risk according to its confidence level (e.g., high, medium, low).

[0191] Information handling: For information identified as high-risk, the system immediately takes preset handling measures, such as real-time pop-up alerts, automatic interception, or pushing to the manual review queue.

[0192] Results Feedback: The results of the actions and related data can be recorded and fed back into the model training library for subsequent model optimization iterations, forming a self-reinforcing closed-loop system.

[0193] Through the above complete process, this invention achieves accurate, efficient, and adaptive identification and handling of risk information in publicly available data, effectively improving the automation level and response efficiency of network security protection.

[0194] This invention also provides a system for analyzing risk information in publicly available data based on multimodal artificial intelligence, comprising: The data preprocessing module is used to asynchronously collect multimodal data from multiple heterogeneous public data sources, and to preprocess the multimodal data using a modality-specific cleaning strategy. The multimodal data includes text, voice, images, and video. The high-dimensional feature extraction module is used to extract high-dimensional features of each modality data after preprocessing using a pre-trained deep model. The high-dimensional features include semantic features of text, acoustic and semantic features of speech, visual and spatial features of images, and temporal variation features of video. A modal availability vector generation module is used to evaluate the availability status of each modal data and generate a modal availability vector. The modal availability vector is a continuous value vector, and it is generated after evaluating the quality of each modal data through a lightweight neural network. The value at each position in the modal availability vector represents the availability probability of the corresponding modal data. The confidence score acquisition module is used to obtain the confidence scores of high-dimensional features of each modality output by the pre-trained deep model. The attention fusion weight calculation module is used to input the modality availability vector and the confidence score into the multimodal fusion model to calculate a set of normalized attention fusion weights that characterize the importance of each modality feature; the multimodal fusion model includes a gating network based on an attention mechanism; The multimodal feature fusion module is used to perform weighted fusion of the high-dimensional features of each modality data according to the attention fusion weight to generate a multimodal joint feature representation; based on the multimodal joint feature representation, it outputs the risk probability and type label; The structured early warning event generation module is used to generate a structured early warning event when the output risk probability exceeds a preset threshold, and automatically push it to the subsequent handling system to trigger a linkage response; the structured early warning event includes at least the risk type, confidence level, key evidence fragments, and contextual information.

[0195] Although embodiments of the present invention have been shown and described above, these embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention without departing from the principles and spirit of the present invention.

Claims

1. A method for analyzing risk information in publicly available data based on multimodal artificial intelligence, characterized in that, include: Multimodal data is asynchronously collected from multiple heterogeneous public data sources, and modality-specific cleaning strategies are used to preprocess the multimodal data, which includes text, voice, images, and video. High-dimensional features of each modality data after preprocessing are extracted using a pre-trained deep model. These high-dimensional features include semantic features of text, acoustic and semantic features of speech, visual and spatial features of images, and temporal variation features of video. The availability status of each modality of data is evaluated to generate a modality availability vector. The modality availability vector is a continuous value vector, and it is generated after evaluating the quality of each modality of data through a lightweight neural network. The value at each position in the modality availability vector represents the availability probability of the corresponding modality of data. Obtain the confidence scores of high-dimensional features of each modality output by the pre-trained deep model; The modality availability vector and the confidence score are input into the multimodal fusion model to calculate a set of normalized attention fusion weights that characterize the importance of each modality feature; the multimodal fusion model includes a gating network based on an attention mechanism; The high-dimensional features of each modality data are weighted and fused according to the attention fusion weights to generate a multimodal joint feature representation; based on the multimodal joint feature representation, the risk probability and type label are output. When the output risk probability exceeds a preset threshold, a structured early warning event is generated and automatically pushed to the subsequent handling system to trigger a linkage response; the structured early warning event includes at least the risk type, confidence level, key evidence fragments, and contextual information.

2. The method according to claim 1, characterized in that, The method also includes an update process for the multimodal fusion model, the update process specifically including: The classification performance of the multimodal fusion model on newly labeled data is continuously monitored through an online learning framework; when the classification performance is lower than a preset threshold, an incremental learning process is automatically triggered. The incremental learning process includes: acquiring new labeled data, fine-tuning the multimodal fusion model using the new labeled data, and introducing a historical data replay mechanism to maintain the performance of the multimodal fusion model on historical tasks; the new labeled data is obtained by establishing a continuous learning mechanism to monitor new data in the public data stream and label it periodically.

3. The method according to claim 1, characterized in that, The preprocessing of the multimodal data using a modality-specific cleaning strategy includes: Special characters and invalid links in the text are removed using regular expressions, and word segmentation and part-of-speech tagging are performed using natural language processing tools. The speech is converted into text after noise reduction and endpoint detection. The endpoint detection process includes: calculating the short-time energy and short-time zero-crossing rate of the speech; setting a first threshold for preliminary detection of possible speech segments; setting a second threshold to confirm reliable start and end points of speech segments from the preliminary detection segments; for the region between the first threshold and the second threshold, combining the zero-crossing rate to determine speech / unvoiced speech, so as to accurately determine the start and end positions of the speech segment; the first threshold is lower than the second threshold. For images and videos, the OpenCV library is used for format unification, size adjustment, and keyframe extraction, and further extraction of color histograms, texture features, and object detection information.

4. The method according to claim 1, characterized in that, The semantic features of the text are represented by context embedding vectors generated by a pre-trained language model; The acoustic and semantic features of the speech include Mel frequency cepstral coefficient features extracted from the audio signal, linear prediction cepstral coefficient features, and semantic keyword features extracted from the speech-to-text. The visual and spatial features of the image include depth visual features extracted by a convolutional neural network and target detection box information obtained by a target detection algorithm. The temporal variation features of the video are extracted by calculating the feature differences or optical flow information between keyframe sequences.

5. The method according to claim 1, characterized in that, For text, the key evidence fragment is a sentence containing high-risk keywords; for speech, the key evidence fragment is a timestamp fragment containing high-risk words; for images or videos, the key evidence fragment is an image region or video frame sequence containing a target detection box.

6. The method according to claim 1, characterized in that, The method further includes a risk pattern mining process, which includes: Cluster analysis is performed periodically on the high-risk events identified by the multimodal fusion model to extract emerging risk pattern features. These risk pattern features are then fed back as prior knowledge to the training phase of the multimodal fusion model to construct targeted difficult example samples, thereby enhancing the model's discriminative ability.

7. The method according to claim 1, characterized in that, The attention-based gating network calculates the attention fusion weights using the following formula: α=softmax(W*[v_avail⊙v_conf]+b) Where α represents the fusion weight vector, W and b are learnable parameters, v_avail is the modality availability vector, v_conf is the confidence score vector, ⊙ represents element-wise multiplication, and [·] represents vector concatenation operation.

8. The method according to any one of claims 1 to 7, characterized in that, The method also includes a process for constructing and training a multimodal fusion model, the construction and training process including: Construct a multimodal fusion model that includes an attention-based gating network; We collected a large amount of labeled multimodal data from multiple public data sources to form the training set and the test set; Initialize the parameters of the multimodal fusion model; The multimodal fusion model is iteratively trained using the training set with stochastic gradient descent or Adam optimization algorithm and cross-entropy loss function as the optimization objective. After each training cycle, the performance of the multimodal fusion model is evaluated using the test set, and an early stopping strategy is adopted based on the test set loss to prevent overfitting. Save the optimal model parameters to obtain the final deployable multimodal fusion model.

9. The method according to claim 4, characterized in that, The visual and spatial features of the image also include low-level visual features extracted by traditional image processing algorithms, including color histograms and texture features; the color histogram is calculated in the HSV color space, and the texture features are extracted using gray-level co-occurrence matrix or local binary mode algorithms.

10. A system for analyzing risk information in publicly available data based on multimodal artificial intelligence, characterized in that, include: The data preprocessing module is used to asynchronously collect multimodal data from multiple heterogeneous public data sources, and to preprocess the multimodal data using a modality-specific cleaning strategy. The multimodal data includes text, voice, images, and video. The high-dimensional feature extraction module is used to extract high-dimensional features of each modality data after preprocessing using a pre-trained deep model. The high-dimensional features include semantic features of text, acoustic and semantic features of speech, visual and spatial features of images, and temporal variation features of video. A modal availability vector generation module is used to evaluate the availability status of each modal data and generate a modal availability vector. The modal availability vector is a continuous value vector, and it is generated after evaluating the quality of each modal data through a lightweight neural network. The value at each position in the modal availability vector represents the availability probability of the corresponding modal data. The confidence score acquisition module is used to obtain the confidence scores of high-dimensional features of each modality output by the pre-trained deep model. The attention fusion weight calculation module is used to input the modality availability vector and the confidence score into the multimodal fusion model to calculate a set of normalized attention fusion weights that characterize the importance of each modality feature; the multimodal fusion model includes a gating network based on an attention mechanism; The multimodal feature fusion module is used to perform weighted fusion of the high-dimensional features of each modality data according to the attention fusion weight to generate a multimodal joint feature representation; based on the multimodal joint feature representation, it outputs the risk probability and type label; The structured early warning event generation module is used to generate a structured early warning event when the output risk probability exceeds a preset threshold, and automatically push it to the subsequent handling system to trigger a linkage response; the structured early warning event includes at least the risk type, confidence level, key evidence fragments, and contextual information.

Images