CLIP model-based batch multi-modal data alignment method and system

Abstract

The invention discloses a batch multi-modal data alignment method and system based on a CLIP model, and belongs to the technical field of information processing, and the method comprises the steps: S1, receiving batch multi-modal data; s2, data preprocessing; s3, performing batch feature extraction based on a CLIP model to realize feature alignment; s4, carrying out batch classification based on a Prompt template; s5, generating and outputting a result, and performing visualization processing; the method is suitable for user behavior analysis, abnormal traffic detection and network content security management in a communication big data environment, fully utilizes strong cross-modal semantic alignment capability and zero sample migration capability of the CLIP, and is combined with a batch optimization strategy, dynamic task scheduling and a batch classification method based on a Prompt template. Therefore, efficient, flexible and extensible multi-modal data processing is realized, bad texts, malicious images and cross-modal risk links are accurately identified, and real-time monitoring and intelligent prevention and control are performed on bad network behaviors.

Description

Technical Field

[0001] The present invention belongs to the field of information processing technology, and in particular relates to a batch multimodal data alignment method and system based on a CLIP model. Background Art

[0002] Understanding and aligning multimodal data (such as images and text) has long been a key research area in artificial intelligence. With the rapid development of the internet and communications technologies, the volume of multimodal data has exploded. Examples include user behavior logs on communication networks, graphic content on social media, massive amounts of product images and descriptions on e-commerce platforms, and audio and video data transmitted in network traffic. Efficiently processing and analyzing this data is crucial for telecommunications operators' network optimization, content security governance, and intelligent customer service response, as well as for cybersecurity agencies' real-time monitoring and intelligent prevention of unhealthy industry practices.

[0003] Traditional multimodal data processing methods typically rely on models designed for specific tasks, such as image classification, text classification, and image description generation. These models often require large amounts of manually annotated data for training, which is costly and difficult to scale to new data modalities or task types. Furthermore, traditional methods face bottlenecks in computational efficiency and resource utilization for batch processing of large-scale data. This is particularly true in the context of big data communications, where the need for concurrent processing of massive amounts of real-time data and the limitations of traditional architectures are increasingly at odds.

[0004] In recent years, contrastive learning has become an important research direction in the field of multimodal representation learning. By learning the correlations between data from different modalities, this method can effectively map them into a shared semantic space, thereby achieving cross-modal understanding and alignment. The contrastive language-image pre-training model (CLIP) is a representative achievement in this field. CLIP achieves powerful cross-modal semantic representation capabilities through contrastive learning on large-scale image-text pair datasets and demonstrates significant zero-shot transfer potential. This enables CLIP to be directly applied to a variety of downstream tasks without the need for task-specific fine-tuning, greatly improving the model's flexibility and generalization capabilities. In the context of communications big data, CLIP's zero-shot capabilities are particularly suitable for quickly identifying new types of bad industry variants, such as bad language and bad link templates.

[0005] Currently, there are some solutions that use CLIP for multimodal data processing, but there are still deficiencies in feature alignment and efficient processing of large-scale batch data. For example:

[0006] CLIP-based feature extraction, either line by line or in small batches: In communication network log analysis, processing user behavior data line by line results in excessive computational latency, making it difficult to meet real-time requirements.

[0007] Batch processing of specific tasks based on fine-tuning CLIP: The classification task of bad industries requires frequent data labeling and model adjustment, which makes it difficult to adapt to the rapid iterative changes of bad industries.

[0008] Simple parallel CLIP feature extraction: When processing mixed image-text data in communication traffic, the lack of a dynamic resource scheduling mechanism can easily lead to GPU / CPU load imbalance, affecting system throughput.

[0009] Currently, traditional methods face the following core challenges when processing communications big data and adverse industry analysis scenarios:

[0010] Efficiency bottleneck: The processing efficiency of massive multimodal data (such as billions of user messages and millions of inappropriate images every day) is low, making it difficult to meet real-time monitoring needs.

[0011] Scalability limitations: The bad feature library that relies on manual annotation is updated laggingly and cannot quickly adapt to new bad methods.

[0012] Insufficient cross-modal association: Existing methods have difficulty establishing a cross-modal association network between bad text, bad app icons, and bad links, which affects the ability to trace risks.

[0013] Waste of resources: Traditional batch processing does not fully utilize the distributed computing resources of communication operators (such as edge node computing power), resulting in high computing costs. Summary of the Invention

[0014] In response to the problems mentioned in the background technology, the present invention proposes a batch multimodal data alignment method and system based on the CLIP model, which is aimed at communication big data and bad industry analysis. Through the image encoder and text encoder of the CLIP model, mixed modal data such as bad icons, bad words, and bad recorded voices in the communication network are jointly represented and learned to construct a cross-modal semantic space covering the implicit features of the communication protocol and the ecological behavior of bad industries; providing full-link perception capabilities for communication network attack and defense drills and bad industry crackdowns.

[0015] Technical solution: In order to solve the above technical problems, the technical solution adopted by the present invention is as follows:

[0016] A batch multimodal data alignment method based on the CLIP model, which batch processes multimodal data in communication networks and combines the cross-modal alignment capabilities of the CLIP model, can identify bad text, bad images, and cross-modal risk links, supporting communication operators in real-time monitoring and intelligent blocking of bad industry transmission links. The method specifically includes the following steps:

[0017] S1: Receive batch multimodal data;

[0018] S2: data preprocessing;

[0019] S3: Batch feature extraction based on CLIP model to achieve feature alignment;

[0020] S4: Batch classification based on Prompt template;

[0021] S5: Generate and output results and perform visualization.

[0022] Preferably, in S1, images and text information on the Internet are collected and analyzed in batches, and harmful information is identified and blocked through a dynamic keyword library and image feature matching.

[0023] As a preference, in S2, the collected multimodal data is standardized to effectively filter out normal communication noise, retain key features of adverse industries, and provide high-quality input for subsequent feature alignment.

[0024] Preferably, in S3, the specific process of performing batch feature extraction and feature alignment based on the CLIP model is as follows:

[0025] S31: Use the CLIP model’s image encoder and text encoder to batch extract pre-processed image and text features;

[0026] S32: Dynamically schedule tasks based on the extracted image features and text features to allocate computing resources;

[0027] S33: Batch contrastive learning optimizes and stores feature vectors.

[0028] Preferably, in S31, the image encoder of the pre-trained CLIP model is used to extract features from the batch image data to obtain a batch image feature matrix ;

[0029] in, Represents the batch of image feature matrix, N represents the batch size, Represents the dimension of image features, Represents the dimension of the image feature matrix;

[0030] Use the pre-trained CLIP model's text encoder to extract features from batch text data and obtain a batch text feature matrix ;

[0031] in, Represents the batch of text feature matrix, N represents the batch size, Represents the dimension of text features, Represents the dimension of the text feature matrix.

[0032] Preferably, in S33, a contrast loss function for batch data is constructed, and the contrast loss function L is expressed as:

[0033] ,

[0034] in, and Represent the feature vectors of the jth image and text respectively, represents the cosine similarity function; τ represents a temperature hyperparameter; N represents the number of image-text pairs in a batch; represents the exponential function, Represents the index of the image-text pair in the batch; Indicates the The feature vector of the image and the feature vectors of all texts in the batch The sum of the exponential functions of the similarities between them; Indicates the feature vector of the text and the feature vectors of all images in the batch The sum of the exponential functions of the similarities between them; Indicates the The feature vector of the image and the corresponding text feature vector The exponential function of the similarity between them.

[0035] Preferably, in S4, the specific process of batch classification based on the Prompt template is as follows:

[0036] S41: Predefined batch prompt template library;

[0037] S42: Dynamically schedule tasks based on the extracted image features and text features to allocate computing resources;

[0038] S43: Calculate the similarity between the image features and the Prompt features, and generate a classification result.

[0039] Preferably, in S41, the specific content of the predefined batch prompt template library is:

[0040] For batch image data that needs to be classified, CLIP's image encoder is first used to extract image features; then, based on the predefined batch Prompt template library, CLIP's text encoder is used to extract Prompt features, and corresponding text features are generated in batches according to predefined category labels.

[0041] Preferably, in S43, the similarity between the image feature and the Prompt feature is calculated, and the specific content of generating the classification result is:

[0042] Suppose an image is divided into k possible categories, and a corresponding prompt text feature vector is generated for each category ;

[0043] Traversal calculation: For each category of Prompt feature vector , based on the cosine similarity between each image’s feature vector and the text feature vector, calculate the image feature vector With the current Prompt feature vector The similarity score between ;

[0044] Comparing fractions: get Similarity scores ;

[0045] Determine category: Find this The index corresponding to the score with the largest median similarity score ;

[0046] Output result: The image is predicted to be categories;

[0047] This is performed once for each image in the batch, thus achieving batch classification.

[0048] A CLIP model-based batch multimodal data alignment system, which implements any of the CLIP model-based batch multimodal data alignment methods described above, comprising a batch data input and preprocessing module, a batch feature extraction module, a batch contrastive learning optimization module, a dynamic task scheduling and concurrent processing module, a prompt template-based batch classification module, a strategy matching and analysis module, and a result generation and output module;

[0049] Batch data input and preprocessing module: used to receive batch input image data and text data, and filter normal communication traffic noise through dynamic denoising and standardization processing;

[0050] Batch feature extraction module: Combines the cross-modal alignment capability of the CLIP model to perform joint feature representation of image-text pairs in communication big data;

[0051] Batch contrastive learning optimization module: used to calculate and optimize the similarity between image features and text features;

[0052] Dynamic task scheduling and concurrent processing module: used to decompose batch alignment tasks into multiple subtasks that can be executed in parallel; based on the real-time status of system resources and predefined scheduling policies, it dynamically allocates subtasks to available computing resources, enabling real-time monitoring and management of tasks;

[0053] Prompt template-based batch classification module: used to convert the multi-category classification problem into the matching problem of image and text templates to achieve batch classification;

[0054] Strategy matching and analysis module: Built-in knowledge base of bad industry characteristics, combined with Prompt template matching technology, automatically identifies bad text, bad images and abnormal behavior patterns, and generates risk scores and associated network topology;

[0055] Result generation and output module: used to output the results of batch data alignment.

[0056] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0057] (1) The batch multimodal data alignment method based on the CLIP model proposed in this paper can effectively solve the problems of low efficiency, poor scalability, weak generalization ability and insufficient resource utilization in the existing technology when processing large-scale bad information.

[0058] (2) The present invention significantly improves the efficiency of identifying harmful information: by performing batch processing and utilizing the GPU-accelerated CLIP model for feature extraction and similarity calculation, it is able to quickly locate potential harmful information in large-scale multimodal data. Compared with traditional one-by-one or small-batch processing methods, the processing speed is greatly improved, effectively coping with the characteristics of rapid information dissemination and large amounts in this field.

[0059] (3) This invention enhances the ability to identify new and variant malicious tactics: Leveraging CLIP's powerful cross-modal semantic understanding and zero-shot transfer capabilities, as well as its flexible Prompt Engineering technology, it can quickly adapt to evolving malicious network manipulation techniques. Without the need for time-consuming data annotation and model retraining for each new variant, the accuracy and recall of identifying new malicious information can be improved by simply updating the Prompt template or keyword library, effectively compensating for the lack of generalization capabilities of traditional fixed models.

[0060] (4) The present invention improves the ability to analyze cross-modal associations of negative information. It can align features of data from multiple modalities, such as images and text, to better understand and correlate the semantic relationships between different modalities. For example, it can identify images containing negative links and images associated with specific negative information. This improves the ability to identify and analyze complex and highly disguised negative information, breaking the limitations of traditional single-modal analysis methods.

[0061] (5) The present invention can optimize computing resource utilization and reduce operating costs: through dynamic task scheduling and concurrent processing mechanisms, it can intelligently allocate computing resources according to system load and task priority, fully utilize hardware acceleration capabilities such as GPUs, avoid idle and wasted resources, and reduce the operating costs of large-scale data processing.

[0062] (6) This invention enhances the scalability and flexibility of the system: The CLIP-based pre-trained model and PromptEngineering technology enable the system to be more easily expanded to new types of negative information and new data modalities without the need for large-scale model reconstruction. Dynamic task scheduling also enables the system to flexibly respond to the growth of data volume and changes in business needs.

[0063] (7) The present invention can improve the accuracy and timeliness of early warning: through efficient batch processing and intelligent risk assessment mechanism, it can more quickly and accurately identify and warn of potential bad information, provide more timely security protection for users and platforms, and reduce losses.

[0064] (8) The present invention uses the CLIP model's image encoder and text encoder to jointly characterize mixed-modal data such as bad icons, bad words, and bad recorded voices in the communication network, and constructs a cross-modal semantic space covering the implicit features of the communication protocol and bad industry ecological behaviors; combined with the spatiotemporal distribution law of communication traffic (such as SMS bombing frequently occurs during business peak hours and bad emails surge at night), a dynamic task scheduling strategy based on resource perception is constructed to achieve millisecond-level elastic computing resource allocation of GPU clusters; a professional bad industry classification prompt library containing "bad APP startup page features" and "fake bank transfer word templates" is pre-set, and the image-text-voice feature similarity matrix is ​​calculated in combination with the comparative learning optimization module, which not only realizes zero-sample classification and recognition of new variant attack modes, but also automatically mines cross-modal risk propagation links such as "bad SMS-bad website-malicious APP download", providing full-link threat perception capabilities for communication network attack and defense drills and bad industry crackdowns. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 This is a flow chart of the batch multimodal data alignment method based on the CLIP model;

[0066] Figure 2 It is a flow chart of the back-end implementation of the system of the present invention;

[0067] Figure 3 It is the front-end implementation logic diagram of the system of the present invention;

[0068] Figure 4 It is the network data flow logic diagram of the present invention. DETAILED DESCRIPTION

[0069] The present invention will be further illustrated below with reference to specific examples. The examples are implemented based on the technical solutions of the present invention. It should be understood that these examples are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0070] Example 1

[0071] The batch multimodal data alignment method based on the CLIP model provided in this embodiment is mainly used for monitoring and combating malicious website behaviors. Its core technical concepts include:

[0072] Batch processing: Organize large-scale image and text data into batches for processing, fully utilize the parallel computing capabilities of modern computing devices, reduce repeated calculations, and improve overall processing efficiency.

[0073] Batch Optimization for Contrastive Learning: Design and implement a contrastive loss function optimization strategy for batch data. During the training or feature extraction phase, this strategy efficiently calculates and optimizes the similarity between image and text features, improving feature discrimination and alignment.

[0074] Dynamic task scheduling and resource management: Combined with CLIP's zero-shot inference capabilities, it dynamically allocates and schedules computing tasks based on task priority, data complexity, and the real-time status of system resources (such as GPU utilization and memory usage), achieving more efficient resource utilization and lower latency.

[0075] Prompt Engineering for Batch Classification: Leveraging CLIP's Prompt Engineering technology, we generate standardized text templates in batches, transforming the multi-category classification problem into a matching problem between images and text templates. This enables efficient batch classification with good scalability.

[0076] In this embodiment, the custom nouns and their explanations are as follows:

[0077] CLIP: Contrastive Language-Image Pre-training. A neural network model trained on a large-scale image-text pair dataset through contrastive learning, capable of learning semantic associations between images and text.

[0078] ViT: Vision Transformer, an image encoder based on the Transformer architecture, is used as an image feature extractor by the CLIP model.

[0079] BERT: Bidirectional Encoder Representations from Transformers, a text encoder based on the Transformer architecture, is used as a text feature extractor by the CLIP model.

[0080] InfoNCE: Information Noise-Contrastive Estimation, a commonly used contrastive loss function for learning to distinguish positive and negative samples.

[0081] GPU: Graphics Processing Unit, a graphics processor, is an electronic circuit specialized for parallel computing, often used to accelerate deep learning tasks.

[0082] Prompt Engineering: Prompt Engineering is a technique that guides pre-trained language models (such as CLIP's text encoder) to perform specific tasks by designing specific text prompts (Prompt).

[0083] Black and gray industries: refers to industries that use the Internet and other platforms to carry out illegal or irregular activities.

[0084] Anti-fraud: Short for opposing and combating bad activities.

[0085] API: Application Programming Interface, a set of definitions and protocols that allow communication and data exchange between different software systems.

[0086] UI: User Interface, the interface through which users interact with the software system.

[0087] NLP: Natural Language Processing, a branch of computer science and artificial intelligence that studies how to enable computers to understand and process human language.

[0088] CNN: Convolutional Neural Network, a deep learning model commonly used in image processing and computer vision tasks.

[0089] Transformer: A neural network architecture based on the self-attention mechanism, which has achieved great success in natural language processing and computer vision. The text encoder and some image encoders in the CLIP model both use the Transformer architecture.

[0090] RoBERTa: A Robustly Optimized BERT Pretraining Approach, a pre-trained language model that optimizes the BERT model.

[0091] ELECTRA: Efficiently Learning an Encoder that Classifies TokenReplacements Accurately, a pre-trained language model for efficient learning encoders.

[0092] ALIGN: A Little Is Enough: Learning Highly Accurate Visual Embeddingsfor Product Search, a multimodal pre-training model focused on learning accurate visual embeddings for product search.

[0093] FILIP: Fine-grained Interactive Language-Image Pre-training, a fine-grained interactive language-image pre-training model.

[0094] CUDA: Compute Unified Device Architecture, a parallel computing platform and programming model launched by NVIDIA for general computing on its GPUs.

[0095] cuDNN: NVIDIA CUDA Deep Neural Network library, a GPU-accelerated library for deep neural networks provided by NVIDIA.

[0096] Kubernetes: An open-source container orchestration system for automating the deployment, scaling, and management of containerized applications.

[0097] Apache Mesos: An open source cluster management platform for resource sharing and task scheduling.

[0098] Milvus: An open-source vector database designed for storing, indexing, and searching large-scale feature vectors.

[0099] Faiss: Facebook AI Similarity Search, Facebook AI's open source library for efficient similarity search.

[0100] Flask: A lightweight Python web framework.

[0101] FastAPI: A modern, high-performance Python web framework for building APIs.

[0102] React: A JavaScript library for building user interfaces.

[0103] Vue.js: A progressive JavaScript framework for building user interfaces.

[0104] This invention is suitable for user behavior analysis, abnormal traffic detection and network content security management in the communication big data environment. By batch processing multimodal data such as images, text, and voice in the communication network, combined with the cross-modal alignment capability of the CLIP model, it can efficiently identify bad text, malicious images and cross-modal risk links, and support communication operators in real-time monitoring and intelligent blocking of bad industry transmission links.

[0105] like Figures 1-4 As shown, the specific implementation steps are:

[0106] S1: Receive batch multimodal data;

[0107] Batch analyze text and image information such as pictures, texts, videos, text messages, chat records, social media posts, web page content, etc. on the Internet to quickly identify and issue warnings involving harmful information.

[0108] Multimodal feature alignment can identify images disguised as official organizations, text containing inappropriate links, and harmful information combined with images and text. It can quickly extract key information and uncover harmful clues. For example, by analyzing images and text on social media, it can correlate identities and activity trajectories.

[0109] In the communication big data scenario, this method can be extended to multi-source data collection such as mobile application stores and social media platforms. Through dynamic keyword libraries and image feature matching, it can accurately intercept bad industry communication content such as bad text messages, bad links, and bad pictures.

[0110] S2: data preprocessing;

[0111] Organize large-scale image and text data into batches for processing. Specifically, preprocess the collected data, including cleaning, formatting, denoising, image resizing, and text tokenization, to ensure it meets the input requirements of the CLIP model. Leveraging the parallel computing capabilities of modern computing devices, this reduces repetitive computations and improves overall processing efficiency.

[0112] Extract features from various modal data and perform cross-modal alignment and correlation analysis. For example, align images of a specific location with related text information to extract key time, location, and person information.

[0113] For multimodal data in communication networks (such as images, text, and voice), this method effectively filters out normal communication noise through denoising and standardization, retaining key features of bad industries (such as bad speech templates and bad APP icon features), and providing high-quality input for subsequent feature alignment.

[0114] S3: Batch feature extraction based on CLIP model to achieve feature alignment;

[0115] S31: Use the CLIP model’s image encoder and text encoder to batch extract preprocessed image and text features to obtain high-dimensional feature vectors, specifically:

[0116] Use the image encoder of the pre-trained CLIP model (such as ViT) to extract features from batch image data and obtain the batch image feature matrix ;in, represents the batch of image feature matrices, N is the batch size, is the dimension of image features, Indicates the dimensions of the image feature matrix.

[0117] Use the pre-trained CLIP model's text encoder (such as Transformer) to extract features from batch text data to obtain a batch text feature matrix .

[0118] in, Represents the batch of text feature matrix, N is the batch size, is the dimension of text features, Represents the dimension of the text feature matrix.

[0119] S32: Dynamically schedule tasks based on the extracted image features and text features to allocate computing resources;

[0120] Decompose the received batch alignment task into multiple subtasks that can be executed in parallel; the division of subtasks can be based on data volume, data complexity, or other custom strategies.

[0121] Dynamically assign subtasks in the task queue to available computing resources (such as different GPU cores or CPU threads) based on the real-time status of system resources (for example, by monitoring GPU utilization, memory usage, CPU load, etc.) and predefined scheduling policies.

[0122] Maintain a task queue to manage pending subtasks.

[0123] Scheduling strategies can include scheduling based on task priority, resource allocation based on data complexity (e.g., more complex images or longer texts are allocated more computing resources), and dynamic adjustment of the number of concurrently processed subtasks based on resource utilization.

[0124] Implement real-time monitoring and management of tasks, such as monitoring the execution status of subtasks, handling errors, and performing necessary retries to ensure the stability and reliability of the overall task.

[0125] S33: Batch contrastive learning optimizes and stores feature vectors;

[0126] Construct a contrast loss function for batch data, such as a batch optimized version of the InfoNCE loss. For N image-text pairs in a batch ( , ), the goal is to maximize the similarity between the features of positive sample pairs (i.e., matching images and texts), while minimizing the similarity between the features of negative sample pairs (i.e., mismatched images and texts).

[0127] The contrast loss function L can be expressed as:

[0128] ,

[0129] in, and are the feature vectors of the jth image and text respectively, is the cosine similarity function. In the formula, it means looping over different samples in the batch; Indicates the The feature vector of an image is extracted by the image encoder of the CLIP model. Indicates the The feature vector of the text. The text is extracted by the text encoder of the CLIP model. τ represents a temperature hyperparameter. It is used to adjust the steepness of the similarity distribution. A smaller τ will make the model more focused on distinguishing positive and negative samples, which may lead to overfitting. A larger τ will make the distribution smoother, which may lead to insufficient model learning. L represents the value of the contrast loss function. The smaller this value, the better the alignment between the image and text features learned by the model. N represents the number of image-text pairs in a batch. Batch processing is the key to improving computational efficiency, where N represents the batch size. represents the exponential function, Equivalent to ,in represents the Euler number, which is approximately 2.71828. Represents the index of the image-text pair in the batch. The sum symbol indicates that the calculation is performed on all image-text pairs in a batch. This is the first term in the denominator. This part calculates the The feature vector of the image and the feature vectors of all texts in the batch The sum of the exponential functions of the similarities between them. This is the second term in the denominator. This part calculates the feature vector of the text and the feature vectors of all images in the batch The sum of the exponential functions of the similarities between them. It is the molecular part, and the calculation is The feature vector of the image and the corresponding text feature vector The exponential function of the similarity between them.

[0130] Through efficient matrix operations and parallel computing (for example, GPU acceleration), the calculation process of the contrast loss function is optimized during large-scale data training, and the parameters of the CLIP model are updated through optimization algorithms such as gradient descent (if fine-tuning is selected).

[0131] The extracted feature vectors are then stored in a high-performance vector database and indexed for fast similarity search and matching.

[0132] In the bad industry analysis scenario, dynamic task scheduling is used to optimize GPU / CPU resource allocation, improve the efficiency of large-scale feature extraction in the communication big data environment, and provide high-precision feature representation for cross-modal risk link analysis (such as the linkage between bad phone calls and bad websites).

[0133] S4: Batch classification based on Prompt template;

[0134] S41: Predefine an extensible batch prompt template library; the templates can adopt standardized forms, such as "This is a picture of {category}", "It describes a scene about {topic}", etc., where "{category}" and "{topic}" are placeholders and can be dynamically replaced with actual category labels or keywords based on the specific classification task.

[0135] For batch image data that needs to be classified, the image features are first extracted using the CLIP image encoder.

[0136] Then use CLIP's text encoder to extract prompt features and generate corresponding text features in batches according to predefined category labels. For example, if you need to classify an image into In the categories, Substitute the category labels into the Prompt template and get text prompts, and use CLIP's text encoder to extract the features of these prompts to obtain a The text feature matrix.

[0137] S42: Dynamically schedule tasks based on the extracted image features and text features to allocate computing resources;

[0138] Decompose the received batch alignment task into multiple subtasks that can be executed in parallel; the division of subtasks can be based on data volume, data complexity, or other custom strategies.

[0139] Dynamically assign subtasks in the task queue to available computing resources (such as different GPU cores or CPU threads) based on the real-time status of system resources (for example, by monitoring GPU utilization, memory usage, CPU load, etc.) and predefined scheduling policies.

[0140] Maintain a task queue to manage pending subtasks.

[0141] Scheduling strategies can include scheduling based on task priority, resource allocation based on data complexity (e.g., more complex images or longer texts are allocated more computing resources), and dynamic adjustment of the number of concurrently processed subtasks based on resource utilization.

[0142] Implement real-time monitoring and management of tasks, such as monitoring the execution status of subtasks, handling errors, and performing necessary retries to ensure the stability and reliability of the overall task.

[0143] S43: Calculate the similarity between the image features and the prompt features and generate a classification result;

[0144] Calculate the feature vector of each image and The cosine similarity between text feature vectors.

[0145] The specific formula and process for calculating the cosine similarity between the image feature vector and the text (Prompt) feature vector are as follows:

[0146] The calculation formula of cosine similarity.

[0147] Assume there are two feature vectors, vector A (for example, image feature vector ) and vector B (for example, the text feature vector of a Prompt p), their cosine similarity The calculation formula is:

[0148]

[0149] in, Represents a vector and The dot product (DotProduct). If and ,but . is the dimension of the vector. and Represents vectors and The L2 norm (L2Norm) of , also known as the Euclidean norm or the modulus of the vector. The specific calculation method is:

[0150]

[0151]

[0152] The formula calculates the degree of similarity between two vectors in terms of direction, and the result range is between [-1, 1]. The closer the value is to 1, the more similar the directions are; the closer it is to -1, the more opposite the directions are; and the closer it is to 0, the directions are orthogonal (independent). In the feature space of CLIP, the feature vectors are usually normalized. , the formula can be simplified to directly calculate the dot product: .

[0153] Specific calculation process (in S43 batch classification step);

[0154] Suppose you need to transform an image (whose eigenvector is ) into k possible categories, and a corresponding Prompt text feature vector has been generated for each category .

[0155] Traversal calculation: For each category of Prompt feature vector (in From 1 to ):

[0156] Using the cosine similarity formula above, calculate the image feature vector With the current Prompt feature vector The similarity score between .

[0157] Comparing fractions: get Similarity scores: .

[0158] Determine category: Find this The index corresponding to the score with the largest median score .

[0159]

[0160] Output result: The image is predicted to be categories, that is, The category associated with this Prompt feature vector.

[0161] This process is performed once for each image in the batch, thus achieving batch classification.

[0162] The class label corresponding to the text feature with the highest similarity to the image feature is selected as the predicted class for the image. The entire process is performed in batches, significantly improving the efficiency of multi-class classification.

[0163] S5: result generation and output, and visualization processing;

[0164] The feature vectors based on S3 and the classification results of S4 are used to generate and output the results, and then perform post-processing (visualization, storage, etc.).

[0165] Based on the specific application scenario, the batch data alignment results are output. For example, for image retrieval tasks, the most relevant text description of the query image is output; for classification tasks, the predicted category label of the image is output; and for anomaly detection tasks, the data identified as anomalies and their anomaly scores are output.

[0166] The output results can be post-processed, such as visualization and storage of the results.

[0167] This application also proposes a batch multimodal data alignment system based on the CLIP model to implement a batch multimodal data alignment method based on the CLIP model, including a batch data input and preprocessing module, a batch feature extraction module, a batch contrast learning optimization module, a dynamic task scheduling and concurrent processing module, a Prompt template-based batch classification module, and a result generation and output module.

[0168] Batch data input and preprocessing module: used to receive batch input image data and related text data; perform necessary preprocessing on the input image and text data, such as image size normalization and text tokenization, to meet the CLIP model input requirements.

[0169] It supports real-time access to massive multimodal data such as images, text, and voice in communication networks. Through dynamic denoising and standardization processing, it filters out normal communication traffic noise and accurately extracts negative industry characteristics such as bad language and bad APP icons.

[0170] Image data may include picture files or image data streams in various formats; text data may include descriptions, tags, titles, etc. related to the image.

[0171] Batch feature extraction module: used to extract image features and text features; combined with the cross-modal alignment capability of the CLIP model, it performs joint feature representation on image-text pairs in communication big data, providing a high-precision feature foundation for cross-modal risk link analysis.

[0172] The image encoder of the pre-trained CLIP model is used to extract features from batch image data to obtain a batch image feature matrix.

[0173] The text encoder of the pre-trained CLIP model is used to extract features from batch text data to obtain a batch text feature matrix.

[0174] Batch Contrastive Learning Optimization Module: This module is used to calculate and optimize the similarity between image and text features, improving feature discrimination and alignment. It also builds and adopts a contrastive loss function optimization strategy for batch data to calculate similarity during training or feature extraction.

[0175] Dynamic task scheduling and concurrent processing module: This module is used to decompose received batch alignment tasks into multiple subtasks that can be executed in parallel. The division of subtasks can be based on data volume, data complexity, or other custom strategies.

[0176] To meet the massive data processing needs during communication peak periods, intelligent scheduling of GPU / CPU resources is used to achieve batch feature extraction and comparative learning optimization with a response time of seconds, supporting real-time blocking of bad industry transmission links.

[0177] Maintain a task queue to manage pending subtasks.

[0178] Dynamically assign subtasks in the task queue to available computing resources (such as different GPU cores or CPU threads) based on the real-time status of system resources (for example, by monitoring GPU utilization, memory usage, CPU load, etc.) and predefined scheduling policies.

[0179] Scheduling strategies can include scheduling based on task priority, resource allocation based on data complexity (e.g., more complex images or longer texts are allocated more computing resources), and dynamic adjustment of the number of concurrently processed subtasks based on resource utilization.

[0180] Implement real-time monitoring and management of tasks, such as monitoring the execution status of subtasks, handling errors, and performing necessary retries to ensure the stability and reliability of the overall task.

[0181] Prompt template-based batch classification module: This module transforms the identification of multi-category negative information into an efficient batch image-text similarity calculation (converting the classification problem into a matching problem between an image and a text template). This eliminates the need for fine-tuning for specific categories, improving the efficiency and scalability of classification / recognition. This module enables efficient batch classification as an application of feature alignment.

[0182] Strategy matching and analysis module: The module has a built-in knowledge base of bad industry characteristics and combines it with Prompt template matching technology to automatically identify bad text, bad images, and abnormal behavior patterns, generate risk scores and associated network topologies, and provide a decision-making basis for security protection.

[0183] The strategy matching and analysis module includes:

[0184] Bad feature library: Maintain a knowledge base containing text keywords related to bad information, visual pattern feature vectors, behavioral pattern rules, etc.

[0185] Similarity calculation: Perform efficient batch similarity calculation (for example, using cosine similarity) on the feature vectors of the data to be analyzed and the feature vectors in the feature library.

[0186] Rule engine: It combines predefined rules (such as keyword combinations and specific behavior sequences) to perform risk assessment on matching results.

[0187] Prompt template application: To classify or identify specific types of negative information, Prompt Engineering is used to generate relevant text template features and match them with the image features of the data to be analyzed.

[0188] Risk assessment and early warning: Based on the matching results and the evaluation of the rule engine, the data is risk graded and corresponding early warning information is generated.

[0189] Early warning and analysis: Based on the matching results and set thresholds, the system automatically generates high-risk content warnings and provides visual analysis reports that display the distribution, dissemination trends, and associated networks of harmful information, assisting security personnel in conducting in-depth analysis and tracing.

[0190] Rule management: Security personnel can flexibly add, edit, and update prompt templates, keyword libraries, and visual feature libraries related to bad information through the background interface, and adjust risk assessment rules and thresholds.

[0191] Interception and warning: After the system identifies high-risk harmful information, it can take interception measures (such as blocking SMS delivery, marking risky links), and send security warnings to users.

[0192] Case Analysis and Model Updates: Experts can use the platform to analyze past incidents, extract new negative characteristics, and update the system's feature library and recognition models. Using Prompt Engineering, recognition rules can be quickly created for new negative language and tactics.

[0193] Result generation and output module: used to output the results of batch data alignment.

[0194] Based on the specific application scenario, the batch data alignment results are output. For example, for image retrieval tasks, the most relevant text description of the query image is output; for classification tasks, the predicted category label of the image is output; and for anomaly detection tasks, the data identified as anomalies and their anomaly scores are output.

[0195] Display analysis results, risk levels, and warning information through the user interface or API interface. Receive user feedback to optimize feature libraries, rules, and models.

[0196] The output results can be post-processed, that is, the analysis results can be visualized in the form of charts, relationship networks, etc.

[0197] Output risk scores, early warning signals and visual reports.

[0198] UI interface interaction:

[0199] Warning prompt: When the system identifies potential harmful information (such as text messages or chat content containing high-risk images), a prominent warning prompt will pop up on the user interface, such as "Suspected harmful information detected, please pay attention to identify it!"

[0200] Risk level display: For suspicious information, the system can display different risk level labels (such as "high risk" and "suspected risk") according to its risk level.

[0201] Information details display: Users can view the detailed content of suspicious information, including text, images, etc. The system can highlight high-risk parts.

[0202] Reporting and feedback: Users can report confirmed bad information with one click and feed the reported information back to the backend system for model iteration and optimization.

[0203] Popularization of safety knowledge: The system can regularly push relevant safety knowledge and cases to users to enhance their awareness of prevention.

[0204] The hardware and software environment for this application:

[0205] Table 1 Hardware environment

[0206]

[0207] Table 2 Software environment

[0208]

[0209] Example 2

[0210] Based on Example 1, this embodiment provides other methods for feature extraction:

[0211] Use other multimodal pre-trained models: In addition to CLIP, consider using other pre-trained models with cross-modal alignment capabilities, such as ALIGN and FILIP. Although these models may differ from CLIP in training methods and model structure, they can also extract joint feature representations of images and text.

[0212] Combining single-modal models: Consider using independent image encoders (such as ResNet and EfficientNet) and text encoders (such as RoBERTa and ELECTRA), and then learning cross-modal alignment relationships through additional fusion layers (such as Transformer Encoder and attention mechanisms). Although this may require more training data and fine-tuning, it may achieve better performance in certain specific areas.

[0213] Other ways of dynamic task scheduling:

[0214] Rule-based static resource allocation: Instead of relying entirely on dynamic scheduling, different types of tasks can be assigned to fixed computing resource pools based on pre-defined rules and task types. For example, feature extraction tasks can be assigned to GPU cluster A, while prompt matching tasks can be assigned to GPU cluster B. While less flexible, this is a viable alternative in scenarios where resources are relatively fixed and task types are well-defined.

[0215] Simple priority-based queues: A simple task queue can be maintained and scheduled based on task priority, with higher-priority tasks receiving priority. While not as intelligent as dynamic scheduling, it can simplify implementation in scenarios where task priorities are clearly distinguished.

[0216] Other ways to match batch prompt templates:

[0217] Similarity search in a vector database: You can pre-store the features of all category text descriptions (not prompt templates) in a vector database. Then, use the features of the image to be classified to perform a top-K similarity search, selecting the category corresponding to the most similar text description as the prediction result. This approach doesn't require explicitly building prompt templates, but may result in slightly different semantic matching accuracy.

[0218] Multi-label classification model: If the malicious information has multiple labels or attributes, a multi-label classification model can be trained (with image and text features as input) to directly predict multiple related labels. This requires a multi-label dataset, but can more directly output classification results.

[0219] Other ways to store feature vectors:

[0220] Traditional relational databases: Feature vectors can be stored in traditional relational databases and queried using indexes. However, for similarity searches on high-dimensional vectors, the performance may not be as good as that of specialized vector databases.

[0221] In this embodiment, the application can also perform batch analysis on the following data contents:

[0222] Content copyright protection: Batch identify unauthorized images and text content on the Internet, such as comparing product images on e-commerce platforms to see if they infringe the brand's copyright.

[0223] Disinformation Detection: This service analyzes text and image information, such as news reports and social media posts, in batches to identify and flag possible false information or rumors. For example, it compares image content with text descriptions or matches them against a database of known false information.

[0224] Intelligent marketing content generation and evaluation: Batch generate product-related advertising copy and image materials, and evaluate their attractiveness or match with the target audience.

[0225] Educational resource management and retrieval: Batch management and retrieval of graphic and text materials on the education platform, such as quickly finding relevant learning resources based on students' questions (text or pictures).

[0226] Industrial quality inspection: Batch analyze images of product defects and align them with standard product description text to determine the type and severity of the defects.

[0227] Artwork appreciation and retrieval: Batch analysis of artwork images and description information to identify styles, verify authenticity, and retrieve similar works.

[0228] Multilingual content alignment and translation assistance: Batch align text and image content in different languages ​​to assist machine translation and localization.

[0229] The present invention uses batch processing and comparative learning optimization strategies for batch data, combined with the spatiotemporal distribution characteristics of communication network traffic (such as 5G base station signaling storms and abnormal connection patterns of IoT devices) and bad industry behavior patterns (such as bad email timing patterns and bad software propagation links), to design a resource-aware dynamic task scheduling mechanism, achieve millisecond-level elastic computing resource allocation for GPU clusters, significantly shorten the processing cycle of large-scale cross-modal data (including base station logs, network traffic packets, bad speech texts, etc.), improve system throughput, reduce response delay, and effectively solve the large-scale data processing bottleneck in communication network attack and defense drills and bad industry crackdown scenarios.

[0230] Based on CLIP's zero-sample migration capability and Prompt Engineering technology, by pre-setting a communication security-specific Prompt template library (such as "5G core network topology anomaly") and a bad industry analysis template library, there is no need to perform data annotation or model fine-tuning for new attack variants. Simply by adjusting the template description, cross-modal risk characteristics can be quickly identified, shortening the response time for new attack detection and solving the problem of insufficient scalability of traditional methods for unknown threats.

[0231] The CLIP model combines its powerful cross-modal semantic representation capabilities with a batch feature optimization strategy. By constructing a joint semantic space of implicit features of communication protocols and behavioral patterns of undesirable industrial ecosystems, the model can maintain high recognition accuracy when dealing with zero-day vulnerability attacks (such as unknown signaling vulnerabilities in 5G slicing networks) or new types of forged documents (such as dynamic facial documents generated based on GAN), effectively solving the problem of insufficient generalization capabilities in complex network environments.

[0232] Through the intelligent resource scheduling engine and concurrent processing module, the system can monitor the communication link load (such as traffic surges during DDoS attacks) and bad data torrents in real time, dynamically adjust the priority of computing tasks, and maintain GPU utilization at a high level for a long time, significantly reducing the computing cost of large-scale cross-modal data analysis.

[0233] The batch classification strategy based on the Prompt template transforms the cross-modal classification of communication network attack patterns (such as the "SMS bombing-bad website-malicious APP download" attack chain) and bad industry behavior characteristics into efficient batch similarity calculation, avoiding independent feature matching for each attack sample. This improves the classification efficiency of million-level data sets, especially when processing complex attack data containing images, text, and voice.

[0234] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A batch multimodal data alignment method based on the CLIP model, characterized by: By batch processing multimodal data in communication networks and combining the cross-modal alignment capabilities of the CLIP model, we can identify bad text, bad images, and cross-modal risk links, supporting communication operators in real-time monitoring and intelligent blocking of bad industry transmission links. Specifically, we include the following steps: S1: Receive batch multimodal data; S2: data preprocessing; S3: Batch feature extraction based on CLIP model to achieve feature alignment; S4: Batch classification based on Prompt template; S5: Generate and output results and perform visualization.

2. The CLIP model-based batch multimodal data alignment method according to claim 1, characterized in that: In S1, images and text information on the Internet are collected and analyzed in batches, and harmful information is identified and blocked through dynamic keyword libraries and image feature matching.

3. The batch multimodal data alignment method based on the CLIP model according to claim 1, characterized in that: In S2, the collected multimodal data are standardized to effectively filter normal communication noise, retain key features of adverse industries, and provide high-quality input for subsequent feature alignment.

4. The CLIP model-based batch multimodal data alignment method according to claim 1, characterized in that: In S3, the specific process of batch feature extraction and feature alignment based on the CLIP model is as follows: S31: Use the CLIP model’s image encoder and text encoder to batch extract pre-processed image and text features; S32: Dynamically schedule tasks based on the extracted image features and text features to allocate computing resources; S33: Batch contrastive learning optimizes and stores feature vectors.

5. The CLIP model-based batch multimodal data alignment method according to claim 4, characterized in that: In S31, the image encoder of the pre-trained CLIP model is used to extract features from the batch image data to obtain the batch image feature matrix ; in, Represents the batch of image feature matrix, N represents the batch size, Represents the dimension of image features, Represents the dimension of the image feature matrix; Use the pre-trained CLIP model's text encoder to extract features from batch text data and obtain a batch text feature matrix ; in, Represents the batch of text feature matrix, N represents the batch size, Represents the dimension of text features, Represents the dimension of the text feature matrix.

6. The CLIP model-based batch multimodal data alignment method according to claim 4, characterized in that: In S33, a contrast loss function for batch data is constructed. The contrast loss function L is expressed as: ， in, and Represent the feature vectors of the jth image and text respectively, represents the cosine similarity function; τ represents a temperature hyperparameter; N represents the number of image-text pairs in a batch; represents the exponential function, Represents the index of the image-text pair in the batch; Indicates the The feature vector of the image and the feature vectors of all texts in the batch The sum of the exponential functions of the similarities between them; Indicates the feature vector of the text and the feature vectors of all images in the batch The sum of the exponential functions of the similarities between them; Indicates the The feature vector of the image and the corresponding text feature vector The exponential function of the similarity between them.

7. The CLIP model-based batch multimodal data alignment method according to claim 1, characterized in that: In S4, the specific process of batch classification based on the Prompt template is as follows: S41: Predefined batch prompt template library; S42: Dynamically schedule tasks based on the extracted image features and text features to allocate computing resources; S43: Calculate the similarity between the image features and the Prompt features, and generate a classification result.

8. The CLIP model-based batch multimodal data alignment method according to claim 7, characterized in that: In S41, the specific contents of the predefined batch prompt template library are: For batch image data that needs to be classified, CLIP's image encoder is first used to extract image features; then, based on the predefined batch Prompt template library, CLIP's text encoder is used to extract Prompt features, and corresponding text features are generated in batches according to predefined category labels.

9. The CLIP model-based batch multimodal data alignment method according to claim 7, characterized in that: In S43, the similarity between the image feature and the prompt feature is calculated, and the specific content of the classification result is generated as follows: Suppose an image is divided into k possible categories, and a corresponding prompt text feature vector is generated for each category ; Traversal calculation: For each category of Prompt feature vector , based on the cosine similarity between each image’s feature vector and the text feature vector, calculate the image feature vector With the current Prompt feature vector The similarity score between ; Comparing fractions: get Similarity scores ; Determine category: Find this The index corresponding to the score with the largest median similarity score ; Output result: The image is predicted to be categories; This is performed once for each image in the batch, thus achieving batch classification.

10. A CLIP model-based batch multimodal data alignment system, implementing the CLIP model-based batch multimodal data alignment method according to any one of claims 1 to 9, characterized in that: It includes batch data input and preprocessing module, batch feature extraction module, batch comparative learning optimization module, dynamic task scheduling and concurrent processing module, batch classification module based on prompt template, strategy matching and analysis module, and result generation and output module; Batch data input and preprocessing module: used to receive batch input image data and text data, and filter normal communication traffic noise through dynamic denoising and standardization processing; Batch feature extraction module: Combines the cross-modal alignment capability of the CLIP model to perform joint feature representation of image-text pairs in communication big data; Batch contrastive learning optimization module: used to calculate and optimize the similarity between image features and text features; Dynamic task scheduling and concurrent processing module: used to decompose batch alignment tasks into multiple subtasks that can be executed in parallel; Dynamically allocate subtasks to available computing resources based on the real-time status of system resources and predefined scheduling policies, enabling real-time monitoring and management of tasks. Prompt template-based batch classification module: used to convert the multi-category classification problem into the matching problem of image and text templates to achieve batch classification; Strategy matching and analysis module: Built-in knowledge base of bad industry characteristics, combined with Prompt template matching technology, automatically identifies bad text, bad images and abnormal behavior patterns, and generates risk scores and associated network topology; Result generation and output module: used to output the results of batch data alignment.

Images