Multimodal data anomaly identification method and device, electronic equipment and storage medium

Abstract

The application discloses a multi-modal data anomaly recognition method and device, electronic equipment and storage medium, and is used for solving the technical problems that the existing anomaly detection related methods are not strong in universality, mainly aim at single type data, single rule and single scene. The method comprises the following steps: firstly, a multi-modal data set is acquired; then, according to the multi-modal data set, an image-text pre-training set and an image-text fine-tuning data set are constructed; then, based on multi-modal embedding alignment, a model is pre-trained according to the image-text pre-training set, and the model obtained after pre-training is fine-tuned according to the image-text fine-tuning data set, so as to obtain a multi-modal anomaly detection model; finally, to-be-recognized data is input into the multi-modal anomaly detection model for anomaly recognition, and an anomaly recognition result is obtained.

Description

Technical Field

[0001] This invention relates to the field of anomaly detection technology, and in particular to a method, apparatus, electronic device, and storage medium for multimodal data anomaly identification. Background Technology

[0002] With the booming development of digital services, diverse service forms such as ride-hailing and internet hospitals are constantly emerging. Supported by massive amounts of data, automated anomaly detection in digital services has become possible. Anomaly data refers to unusual noise or unexpected patterns in a dataset that significantly deviate from the majority of the data. These anomalies may also indicate errors, fraud, or other types of anomalous events. The goal of anomaly detection is to identify and flag these anomalies for further investigation by technical personnel. However, traditional methods for detecting anomaly content in digital services face numerous challenges. On the one hand, the data collected and integrated in different fields of digital services vary, and most detection methods are often limited to single data types, such as images, form text, or database data, severely restricting their implementation. On the other hand, there is a lack of exploration in interactively extracting common information from multiple fields and data types, enabling complementary information from different data types, and achieving similar anomaly detection results in new application scenarios.

[0003] Current multimodal anomaly detection methods are mainly based on graph attention networks and temporal convolutional networks. This approach is not very versatile and is mainly applicable to single types of data, single rules, and single scenarios. Summary of the Invention

[0004] This invention provides a multimodal data anomaly identification method, device, electronic device, and storage medium to solve or partially solve the technical problems of existing anomaly detection methods, such as lack of versatility and targeting only a single type of data, a single rule, and a single scenario.

[0005] This invention provides a method for multimodal data anomaly identification, the method comprising:

[0006] Obtain a multimodal dataset;

[0007] Based on the multimodal dataset, construct an image-text pre-training set and an image-text fine-tuning dataset;

[0008] Based on multimodal embedding alignment, the model is pre-trained using the image-text pre-training set, and then fine-tuned using the image-text fine-tuning dataset to obtain a multimodal anomaly detection model.

[0009] The data to be identified is input into the multimodal anomaly detection model for anomaly identification, and the anomaly identification result is obtained.

[0010] Optionally, the multimodal dataset includes anomalous text data and anomalous image data, and the step of constructing an image-text pre-training set and an image-text fine-tuning dataset based on the multimodal dataset includes:

[0011] Image-text association is performed on the abnormal text data and the abnormal image data to generate multiple image-text data pairs;

[0012] Generate the corresponding descriptive text for each of the image text data pairs;

[0013] For each image-text data pair, the similarity between the descriptive text and the image in the image-text data pair is calculated, and an image-text pre-training data pair is generated based on the image-text data pair, the descriptive text, and the similarity.

[0014] Construct an image-text pre-training set based on each of the image-text pre-training data pairs;

[0015] Obtain a pre-created scene instruction set for anomaly detection, and based on the scene instruction set, expand the multiple image-text data pairs using a self-instruction method to obtain the image-text fine-tuning data pairs corresponding to each of the image-text data pairs.

[0016] Based on each of the image-text fine-tuning data pairs, construct an image-text fine-tuning dataset.

[0017] Optionally, the method further includes:

[0018] Create multiple different task instruction templates, each of which is used to perform anomaly detection for a vertical domain scenario, and each of which includes at least an anomaly description instruction, an anomaly question and answer instruction, and an anomaly analysis instruction.

[0019] Based on the multiple task instruction templates, scene instruction sets for different vertical domain scenarios are generated.

[0020] Optionally, the step of pre-training the model based on the image-text pre-training set and fine-tuning the pre-trained model based on the image-text fine-tuning dataset to obtain a multimodal anomaly detection model includes:

[0021] The LLaMA2 large language model was selected as the base model for multimodal anomaly recognition and detection. Combined with the LoRA optimization strategy, the base model was pre-trained based on multimodal embedding alignment according to the image-text pre-training set to learn the semantic relationships and alignment information between modalities and obtain a general multimodal anomaly detection model.

[0022] By combining the efficient fine-tuning strategy of LLaMA2 parameters, the general multimodal anomaly detection model is fine-tuned based on multimodal embedding alignment according to the image-text fine-tuning dataset to obtain the multimodal anomaly detection model.

[0023] Optionally, the multimodal embedding alignment process includes:

[0024] Extract the abnormal text data and the descriptive text from the dataset used in the current processing stage, and map the abnormal text data and the descriptive text to the embedding space in vector form to generate a text embedding representation;

[0025] The abnormal image data is extracted from the dataset used in the current processing stage, and the abnormal image data is input into the visual encoder for vector transformation to generate an image embedding representation;

[0026] Construct a semantic graph corresponding to the text embedding representation and a visual graph corresponding to the image embedding representation, and perform data association between the semantic graph and the visual graph to construct a multimodal heterogeneous graph, which includes multiple nodes;

[0027] A graph neural network with a cross-modal attention mechanism is introduced. Based on the message passing mechanism of the graph neural network, each target node in the multimodal heterogeneous graph receives messages from neighboring nodes during the iterative update process.

[0028] Different weights are set for different modal messages. Based on the weight ratio setting, the embeddings of different modal messages received from the neighboring nodes are differentially aggregated with the embeddings of the target node itself to update the node's own embeddings, and generate the text-aligned representation corresponding to the text embedding representation and the image-aligned representation corresponding to the image embedding representation.

[0029] Optionally, the step of inputting the data to be identified into the multimodal anomaly detection model for anomaly identification and obtaining anomaly identification results includes:

[0030] The data to be identified is input into the multimodal anomaly detection model. First, the accuracy of the text description of the data to be identified is judged by combining the information content evaluation index. Then, the anomaly judgment is performed based on the text description by combining the factual evaluation index. The anomaly recognition result is output, which includes the anomaly judgment result, anomaly description, anomaly cause inference, and anomaly location.

[0031] Optionally, before acquiring the multimodal dataset, the method includes:

[0032] Obtain a publicly available anomaly detection dataset and a raw multimodal anomaly dataset, both of which include text data, form data, image data, and video data containing anomalies.

[0033] The text data and the form data are preprocessed to obtain abnormal text data;

[0034] The image data and video data are preprocessed to obtain processed image data;

[0035] The processed image data is input into a pre-trained image anomaly detection model to supplement auxiliary information and obtain supplementary image data;

[0036] The supplementary image data is divided into image regions, and then based on a pre-trained image description model, image descriptions and title information corresponding to each image region are generated to obtain abnormal image data.

[0037] The abnormal text data and the abnormal image data are integrated together into a multimodal dataset.

[0038] The present invention also provides a multimodal data anomaly identification device, comprising:

[0039] The multimodal dataset acquisition module is used to acquire multimodal datasets.

[0040] The image-text dataset construction module is used to construct an image-text pre-training set and an image-text fine-tuning dataset based on the multimodal dataset.

[0041] The multimodal embedding alignment processing module is used to pre-train the model based on the image-text pre-training set based on multimodal embedding alignment, and fine-tune the model obtained after pre-training based on the image-text fine-tuning dataset to obtain a multimodal anomaly detection model.

[0042] The multimodal anomaly recognition module is used to input the data to be recognized into the multimodal anomaly detection model for anomaly recognition and to obtain anomaly recognition results.

[0043] The present invention also provides an electronic device, the device comprising a processor and a memory:

[0044] The memory is used to store program code and transmit the program code to the processor;

[0045] The processor is configured to execute the multimodal data anomaly identification method as described above, according to instructions in the program code.

[0046] The present invention also provides a computer-readable storage medium for storing program code for executing the multimodal data anomaly identification method as described in any of the preceding claims.

[0047] As can be seen from the above technical solutions, the present invention has the following advantages:

[0048] A multimodal data anomaly detection method is proposed. First, a multimodal dataset is acquired. Then, based on the multimodal dataset, an image-text pre-training set and an image-text fine-tuning set are constructed. Next, based on multimodal embedding alignment, a model is pre-trained using the image-text pre-training set, and then fine-tuned using the image-text fine-tuning set to obtain a multimodal anomaly detection model. Finally, the data to be identified is input into the multimodal anomaly detection model for anomaly detection, yielding the anomaly detection result. Thus, based on the multimodal embedding alignment and model fine-tuning strategy, the method's ability to capture multimodal information is improved. This not only achieves anomaly detection for multimodal data services but also enhances the model's generalization ability across multimodal data in various scenarios, resulting in greater versatility. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 A flowchart illustrating the steps of a multimodal data anomaly identification method;

[0051] Figure 2 This is a schematic diagram of a multimodal data preprocessing workflow;

[0052] Figure 3 A schematic diagram illustrating the process of constructing an image-text pre-training set and an image-text fine-tuning dataset;

[0053] Figure 4 This is a schematic diagram of a multimodal embedding alignment process;

[0054] Figure 5 This is a schematic diagram of the training and inference process of a multimodal anomaly detection model;

[0055] Figure 6 This is a schematic diagram of the overall process of a multimodal data anomaly identification method;

[0056] Figure 7 This is a structural block diagram of a multimodal data anomaly identification device. Detailed Implementation

[0057] This invention provides a multimodal data anomaly identification method, apparatus, electronic device, and storage medium to solve or partially solve the technical problems of existing anomaly detection methods, such as lack of versatility and targeting only a single type of data, a single rule, and a single scenario.

[0058] To make the objectives, features, and advantages of this invention more apparent and understandable, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0059] As an example, current multimodal anomaly detection methods are mainly based on graph attention networks and temporal convolutional networks. This approach is not very versatile and is mainly applicable to single types of data, single rules, and single scenarios.

[0060] Therefore, one of the core inventive points of this invention is: addressing the shortcomings of existing technologies, a multimodal data service anomaly identification and detection method is proposed based on a large language model to handle general anomaly detection in multimodal data and arbitrary rule scenarios. Given multimodal data including images, videos, and text, as well as rule text in natural language form, a graph structure network is constructed using data temporal, spatial, and relational attributes for specific application scenarios. A graph-based embedding and fusion strategy and instruction fine-tuning strategy are designed, and a cross-modal information aggregation algorithm is designed in conjunction with the graph model to improve the method's ability to capture multimodal information. This enhances the model's generalization ability across multimodal data in multiple scenarios while achieving anomaly identification in multimodal data services. For image and video data, a target detection algorithm targeting key violation elements is introduced to discover anomalous elements in image / frame data. By combining deep learning and natural language processing technologies, comprehensive supervision of digital services is achieved. Compared with traditional methods, the method proposed in this invention can more comprehensively identify abnormal behaviors in digital services, improving supervision efficiency and accuracy.

[0061] Reference Figure 1 The diagram illustrates a flowchart of a multimodal data anomaly identification method provided by an embodiment of the present invention, which may specifically include the following steps:

[0062] Step 101: Obtain the multimodal dataset;

[0063] In the specific implementation, we can first combine Figure 2The multimodal dataset is constructed through the following steps S201 to S206:

[0064] Step S201: Obtain the abnormal dataset.

[0065] Specifically, publicly available anomaly detection datasets and raw multimodal anomaly datasets can be obtained. Both publicly available anomaly detection datasets and raw multimodal anomaly datasets include text data, form data, image data, and video data containing anomalies.

[0066] The original multimodal anomaly dataset can also be understood as multimodal data from different vertical domains (corresponding to a single specific scenario). Public anomaly detection datasets and vertical domain multimodal data are consistent in data type. Both contain sampled frame data of text, images, and videos; the main difference lies in the content.

[0067] The publicly available anomaly detection dataset integrates multiple datasets, including industrial anomaly detection datasets and traffic accident datasets. The image, video frame, and text data cover multiple topics. In contrast, the text, images, and videos in the vertical domain multimodal data are all targeted at a single specific scenario (such as distracted driving).

[0068] Current multimodal data services include written text, forms, images, and videos. Because this data is integrated from multiple datasets and represented in various formats, inconsistencies exist in data format, units, and dimensions in the original data. Furthermore, some data contains missing values, outliers, and errors. These issues can affect the training and performance of subsequent models. Therefore, data preprocessing can be performed on both the publicly available anomaly detection dataset and the original multimodal anomaly dataset. Preprocessing unifies the data to the same format and standards, ensuring data consistency and facilitating subsequent processing and analysis.

[0069] Step S202: Text and form data preprocessing.

[0070] For text and form data in publicly available anomaly detection datasets and raw multimodal anomaly datasets, text preprocessing can be performed to obtain anomalous text data. Then, by performing preprocessing techniques such as word segmentation and cleaning on the input text data, standardized text data can be generated.

[0071] Specifically, the input form data is categorized and parsed. This primarily involves using OCR (Optical Character Recognition) to parse the text data within the form images. Missing values ​​in the form are handled using methods such as filling, deleting, or interpolating to clean the form text, replacing missing text with default values, and then the form text is concatenated.

[0072] For the input long text data, split the text into basic units such as words, sub - words or characters, and remove common stop words such as "的", "了", "是" in the text. Finally, normalize the text to make it in a consistent format, delete text redundancy, streamline the text content, and ensure that the text length meets the requirements of the recognition model.

[0073] Based on the text content obtained after processing (including the text content obtained after pre - processing form data and the text content obtained after pre - processing long text data), generate keywords corresponding to the text. Then, according to the input form, match the text with image information, and generate data pairs containing information such as image number, image size, keywords, figure title, etc. Finally, integrate all these generated data pairs into abnormal text data.

[0074] Step S203: Pre - process image and video data.

[0075] For the image data and video data in the publicly available abnormal detection data set and the original multi - modal abnormal data set, image pre - processing can be performed on them to obtain processed image data. By means of pre - processing such as chunking, extracting narration and description text from images, videos and other data, multi - modal data sample pairs can be generated.

[0076] Specifically, for video data, extract key - frame data (frame images) from multiple videos, delete noise frame data such as blurred main subjects in the picture and occluded main subjects in the picture, and extract the narration text information of the key - frame data. Adjust the image / frame data to a fixed size by methods such as cropping and scaling. Finally, convert image / frame data in different formats to the same color space, such as the RGB (Red Green Blue, optical primary colors) color space or grayscale image, to obtain standardized processed image data.

[0077] It can be understood that the processed image data obtained through the above image pre - processing method includes the data obtained by pre - processing image data and video frame data respectively.

[0078] Step S: Use the image anomaly detection model to generate supplementary image data.

[0079] Then, the processed image data can be input into a pre - trained image anomaly detection model to supplement auxiliary information, and supplementary image data can be obtained.

[0080] In actual situations, in scenarios such as minor anomalies or unclear target subjects, it is very difficult to generate accurate abnormal description text only by inputting image / frame data of RGB or grayscale images.

[0081] Therefore, this invention introduces a traditional image anomaly detection (IAD) model into existing anomaly identification methods. A pre-trained IAD model is used as an auxiliary expert model. Based on the original image embedding, image / frame data from the processed image data is input into an image segmentation model or a pre-trained IAD model to generate supplementary auxiliary image information such as anomaly masks and anomaly scores (i.e., the anomaly identification results in image form output by the IAD model). By concatenating the anomaly identification results generated by the IAD model with the original image embedding as supplementary data, additional information about the image is provided, generating supplementary image data.

[0082] In this way, anomaly segmentation maps or anomaly score maps can be generated as supplementary auxiliary image information based on the model's prior knowledge. By splicing supplementary auxiliary image information into the original image (i.e., processed image data) embedding, large models can be enabled to have the ability to perceive knowledge in the field of image anomaly detection.

[0083] It is understood that the supplementary image data referred to in the embodiments of the present invention includes both the original image data in the processed image data and the supplementary auxiliary image information data generated after information supplementation processing.

[0084] Step S205: Divide the image region.

[0085] Then, the supplementary image data can be divided into image regions, and based on the pre-trained image description model, image descriptions and title information corresponding to each image region can be generated to obtain abnormal image data.

[0086] The supplementary image data (i.e., the original image data and the anomaly identification result image) is divided into regions to facilitate the subsequent identification of anomaly locations in the image by the language model.

[0087] Specifically, based on the position of the anomaly detection result image in the whole image, a simple image division can be performed on the original image data and the anomaly detection result image, and the divided image blocks can be numbered to facilitate the model in generating more detailed image region descriptions and help the model identify anomaly regions.

[0088] Simultaneously, based on video narration, image categories, and other auxiliary information, a trained third-party image description model can be used to generate static image descriptions, image titles, and image descriptions for each frame of the video. Through the image processing steps S202 to S205, abnormal image data can finally be obtained.

[0089] Step S206: Construct a multimodal dataset

[0090] Abnormal text data and abnormal image data are integrated into a multimodal dataset.

[0091] Step 102: Based on the multimodal dataset, construct an image-text pre-training set and an image-text fine-tuning dataset;

[0092] Based on the current requirements of multimodal anomaly detection tasks, pre-training sets and fine-tuning datasets containing "image-short text" data (such as image titles, image categories, etc.) and "image-long text" data (such as anomaly descriptions, causal reasoning, etc.) are generated from the aforementioned preprocessed multimodal datasets. It can be understood that the "image-short text" data corresponds to the image-text pre-training set, and the "image-long text" data corresponds to the image-text fine-tuning dataset.

[0093] The core of this step lies in generating "image-short text" data pairs—i.e., image-text pre-training data pairs—for image / frame data by parsing or manually annotating the input image form data or text data in the multimodal dataset, including various descriptions such as image titles and image categories. Then, using a third-party large model, "image-long text" data pairs—i.e., image-text fine-tuning data pairs—are generated for the image / frame data, containing various descriptions such as anomaly questions and answers, anomaly descriptions, and anomaly inferences.

[0094] As can be seen from the foregoing, multimodal datasets include anomalous text data and anomalous image data. Specifically, combining... Figure 3 The process of constructing an image-text pre-training set and an image-text fine-tuning set based on a multimodal dataset can be achieved by performing the following steps S301 to S306:

[0095] Step S301: Associate the image and text to generate image and text data pairs.

[0096] Image-text association is performed on abnormal text and image data to generate multiple image-text data pairs. The text / narration data and image segmentation block numbers extracted in the preceding steps can be used as supplementary text information to match with the images and generate image-text data pairs.

[0097] Step S302: Generate short text such as image title and image description.

[0098] Next, generate the corresponding descriptive text for each image text data pair, such as the image title, image description, and other short texts.

[0099] In step S302, this embodiment of the invention employs a trained multimodal model based on Transformer (an artificial intelligence model) for data processing. This model can simultaneously process image and text input and generate short text that meets the task requirements. For example, the short text generated by this model can be categorized as follows: describing the main content, scene, or object contained in the image, such as "A white car was severely damaged after a collision on the highway" or "A car was spun 90 degrees after being hit"; describing the actions or behaviors of people or objects in the image, such as "A man in a blue shirt is answering a phone call while driving" or "A black car is driving against traffic in the lane"; describing the position and spatial relationships of objects or people in the image, such as "A white car is stopped on a pedestrian crossing" or "A woman in a yellow shirt is by the highway"; and describing the relationships or interactions between objects or people in the image, such as "Front-seat passengers are talking to a child in the back seat of a car."

[0100] Step S303: Calculate the similarity between text and image using a trained multimodal embedding generative model.

[0101] Next, for each image-text data pair, the similarity between the descriptive text and the image in the image-text data pair is calculated, and image-text pre-training data pairs are generated based on the image-text data pair, the descriptive text, and the similarity.

[0102] In step S303, this embodiment of the invention uses CLIP (Contrastive Language-Image Pre-Training, a text-image pre-trained model employing contrastive learning) scores to measure the similarity between the short text and its corresponding image. Specifically, by introducing the open-source CLIP model, which is first trained on a large image with a title to learn the representation of the image and text in a joint embedding space. Based on the assumption that "in this space, image embeddings are closer to their corresponding title embeddings, while unrelated image embeddings and title embeddings are farther apart," the CLIP model can extract text from the image and compare the obtained text with a given text to generate a semantic relevance score between the image and the text.

[0103] In this embodiment of the invention, information such as image number, image size, text description, and image block number are integrated together to generate "image-short text" data pairs for pre-training in a fixed format. An example of an "image-short text" data pair is shown below:

[0104] { "Image ID":,"Image Description":,"Narration / Supplementary Text":,"Image-Text Semantic Similarity":,"Height":,"Width":,"Block Number:}

[0105] Step S304: Construct “image-short text” data for pre-training of the multimodal anomaly detection model.

[0106] Based on each image-text pre-training data pair, an image-text pre-training set can be constructed.

[0107] Step S305: Write task instruction templates for exception detection, exception description, etc., and generate instruction sets for specific domain scenarios.

[0108] Generating richer data types and instruction types for pre-trained multimodal anomaly detection models can effectively improve the prediction accuracy of large models and alleviate hallucinations. Therefore, this invention provides various types of instruction scripts (anomaly identification instruction data) for image / frame data from different vertical fields such as fatigued driving, traffic accidents, and medical conditions, serving as task instruction templates for anomaly judgment and description. These instruction scripts include image anomaly descriptions, anomaly Q&A, and anomaly analysis. These task instruction templates (manual instructions) are then used as a template library and input into a third-party open-source large model to generate a specified anomaly identification instruction dataset (i.e., scene instruction sets for different vertical fields) based on image content. Simultaneously, to improve model accuracy and response quality, irrelevant or stereotypical anomaly identification instructions are removed. By decomposing complex instruction text, the model's ability to understand instructions can be improved.

[0109] Therefore, before constructing the image-text fine-tuning dataset, multiple different task instruction templates can be created. Each task instruction template is used to perform anomaly detection for a vertical domain scenario. Each task instruction template includes at least anomaly description instructions, anomaly question and answer instructions, and anomaly analysis instructions. Then, based on the multiple task instruction templates, scene instruction sets for different vertical domain scenarios are generated.

[0110] The anomaly description instruction consists of two parts. The first part describes the input image, providing information about the objects present in the image and their expected attributes. The second part queries whether there are any anomalies in the image. The model's response format is designed as follows: first, it determines whether there are any anomalies in the current image; if an anomaly is detected, the model continues to output a description of the image anomaly; otherwise, it replies "No anomalies in the image". An example of an anomaly description instruction is shown below, using a traffic accident scenario as an example:

[0111] {"Description":[{"Requirement Description":"A picture containing a car for anomaly detection. Each tire of the vehicle should be round and free from any damage, blemishes, defects, scratches, holes, or punctures.","Question":" Please determine if there is any abnormality in the vehicle and briefly describe the abnormal condition. [Abnormality Description: "A photo shows a white car severely damaged after a collision on a highway. It appears to have been hit by a truck or bus, resulting in damage to its front and sides. Other vehicles and pedestrians were passing through the area."]

[0112] For exception-based question-and-answer instructions, these instructions typically contain an image and multiple questions. The model needs to answer the questions based on the information provided in the image. An example of an exception-based question-and-answer instruction is shown below:

[0113] {"Dialogue":[{"Question":" What is shown in the picture? {"Reply":"This picture shows a city transportation hub, including a roundabout and several roads. There are trees around it, and an overpass with a train passing underneath. Vehicles, including cars and trucks, are traveling on the roads around the roundabout. The weather looks somewhat gloomy, possibly cloudy or overcast."},{"Question":" Please describe which parts were damaged? For example, the front, rear, and doors. {"answer":"In the picture, you can see two cars have collided. One car's front left side is damaged, while the other car's rear right side is damaged. Specifically, the damage to the front left side may include the bumper, headlights, and body, while the damage to the rear right side may include the bumper, taillights, and body."}

[0114] "}]}

[0115] The anomaly analysis command guides the model to reason and analyze, enabling it to understand the reasons for abnormal events or phenomena in an image. An example of an anomaly analysis command is shown below:

[0116] {"Reasoning":[{"Problem":" From level 1 to 10, determine the similarity between the image and the requirement. Explain your reasoning. {"Answer": "Based on the current information, I would tend to rate it as a level 7 or 8 damage."}, {"Reasoning": "The front of the vehicle is severely damaged; the front end is almost completely broken off, the front bumper and hood are detached, and internal parts are exposed. This damage is very obvious and involves the main structural parts of the vehicle. Generally, damage to the main structural parts of a vehicle is considered a more serious level of damage."}

[0117] Step S306: Construct “image-long text” data for fine-tuning instructions of the multimodal anomaly detection model.

[0118] Next, based on the scene instruction set, multiple image-text data pairs can be expanded using a self-instruction approach to obtain the corresponding image-text fine-tuning data pairs for each image-text data pair; then, an image-text fine-tuning dataset can be constructed based on each image-text fine-tuning data pair.

[0119] Based on the above instruction template, by using the self-instruction method on a third-party language model, "image-long text" data for fine-tuning the instructions of the anomaly detection model is obtained.

[0120] Step 103: Based on multimodal embedding alignment, perform model pre-training according to the image-text pre-training set, and fine-tune the model obtained after pre-training according to the image-text fine-tuning dataset to obtain a multimodal anomaly detection model.

[0121] In this step, a large multimodal anomaly detection model is obtained through pre-training and instruction fine-tuning. Using the two datasets obtained in the preceding steps, a general multimodal language model base is trained using an image-text pre-training set during the pre-training phase. During the instruction fine-tuning phase, the pre-trained large language model is fine-tuned using an image-text fine-tuning dataset based on the specific scene anomaly identification target and task requirements, thereby improving the model's generalization ability and adapting it to multi-scene anomaly detection and recognition tasks.

[0122] In the specific implementation, based on multimodal embedding alignment, the model is pre-trained using an image-text pre-training set, and then fine-tuned using an image-text fine-tuning dataset to obtain a multimodal anomaly detection model, which can be:

[0123] In the pre-training stage, the LLaMA2 (Large Language Model Version 2, an open-source large language model) large language model is selected as the base model for multimodal anomaly recognition and detection. Combined with the LoRA (Low-Rank Adaptation of Large Language Models) optimization strategy, the base model is pre-trained based on multimodal embedding alignment according to the image-text pre-training set to learn the semantic relationships and alignment information between modalities and obtain a general multimodal anomaly detection model.

[0124] The LoRA optimization strategy aims to reduce the number of model parameters by introducing low-rank structures, thereby improving computational efficiency while maintaining model performance as much as possible.

[0125] During the instruction fine-tuning stage, an efficient fine-tuning strategy for LLaMA2 parameters is employed. Based on the image-text fine-tuning dataset, the general multimodal anomaly detection model is fine-tuned using multimodal embedding alignment to obtain a multimodal anomaly detection model. This leverages the capabilities of a large language model, allowing it to adapt to anomaly recognition in different application scenarios through fine-tuning.

[0126] The efficient parameter fine-tuning strategies for LLaMA2 can involve various methods, such as Adapter Tuning, Prefix Tuning, and BitFit, aiming to reduce the computational resources and time required to fine-tune large language models while maintaining model performance. In practical applications, those skilled in the art can choose appropriate efficient LLaMA2 parameter fine-tuning strategies according to actual needs. It is understood that this invention does not impose any limitations on these strategies.

[0127] Furthermore, the main principle behind the multimodal embedding alignment process in this step is as follows:

[0128] First, based on the text keywords, descriptive text, and abnormal images of the abnormal text in the dataset, the data of different modalities are represented as graph structures. For example, for text data, a word co-occurrence graph or semantic graph can be constructed; for image data, a visual feature graph can be constructed. Next, based on node and edge information, a cross-modal information transfer model is designed to generate node representations containing multimodal information, i.e., a multimodal heterogeneous graph. Then, a graph neural network with a cross-modal attention mechanism is introduced, utilizing the attention-based cross-modal message passing mechanism to fuse information between different modalities.

[0129] Based on the above description, specifically, the multimodal embedding alignment processing flow proposed in the embodiments of the present invention can be implemented by executing the following steps S401 to S404:

[0130] Step S401: Text embedding generation.

[0131] Extract anomalous text data and descriptive text from the dataset used in the current processing stage (e.g., in the model pre-training stage, the dataset used is an image-text pre-training set; in the model fine-tuning stage, the dataset used is an image-text fine-tuning dataset), and map the anomalous text data and descriptive text to the embedding space in vector form to generate text embedding representations.

[0132] In text processing, each word can typically be represented as a vector. A Transformer-based encoder can map these vectors into the embedding space to generate the corresponding text embedding representation.

[0133] Step S402: Original image and supplementary image embedding generation.

[0134] Extract anomalous image data from the dataset used in the current processing stage, and input the anomalous image data into the visual encoder for vector transformation to generate image embedding representation.

[0135] Similarly, in this embodiment of the invention, the pre-trained CLIP model as described in step S303 above is used as the visual encoder. For image data, the visual encoder combines the original image with supplementary auxiliary image data to construct a fused input representation. This can be achieved by encoding the images into vectors and then performing average pooling or concatenation on the vectors.

[0136] In the specific implementation, a visual encoder is used. The main process of generating embedding representations based on anomalous image data can be as follows: For the image data portion of the anomalous image data, the original image and the supplementary auxiliary image (i.e., the anomaly detection image) are divided into blocks. The original image and the supplementary auxiliary image are input into the visual encoder for vector transformation to convert the blocks into representation vectors, generating embeddings (represented in the form of representation vectors) for the original image and the supplementary auxiliary image respectively. Then, by concatenating the above embedding results (representation vectors), the final image embedding is obtained. For the video frame data portion of the anomalous image data, each video keyframe is input into the visual encoder to generate embeddings for each frame (this process is similar to the image embedding generation process). Subsequently, the frame embeddings are aggregated using methods such as average pooling or max pooling to obtain the embedding representation corresponding to the video frame data. Thus, based on the temporal and spatial information of the existing image / frame data, an embedding representation with image association is constructed.

[0137] For ease of explanation, in this embodiment of the invention, the image embedding corresponding to the processed image data and the video embedding corresponding to the video frame data are used together as the image embedding representation in subsequent processing.

[0138] Step S403: Construction of heterogeneous graph based on multimodal data.

[0139] Based on the aforementioned image / frame data and text data, visual and semantic graphs can be constructed respectively. The visual graph is a fully connected graph, where each node represents the visual information of entities such as objects and scene text within an image block. The semantic graph is generated based on image scene text, supplementary text, and text semantic information. Each node in the semantic graph represents a text entity, and the initial representation of these nodes comes from the embedding generation results. By integrating information from different modalities, a multimodal heterogeneous graph is constructed, enabling the model to perform joint reasoning on both the visual and semantic graphs simultaneously, thereby improving the model's performance in visual question answering tasks.

[0140] Specifically, the heterogeneous graph construction process based on multimodal data can be as follows: constructing a semantic graph corresponding to text embedding representation and a visual graph corresponding to image embedding representation, and performing data association between the semantic graph and the visual graph to construct a multimodal heterogeneous graph, wherein the multimodal heterogeneous graph includes multiple nodes.

[0141] Step S404: Cross-modal data messaging and embedding alignment.

[0142] Building upon this foundation, graph neural networks or recurrent neural networks can be invoked to achieve multimodal embedding alignment. In this embodiment of the invention, by introducing a graph neural network with a cross-modal attention mechanism (Message Passing), and utilizing its message passing capability, each node in the heterogeneous graph receives messages from its neighboring nodes. By assigning different weights to messages of different modalities, the messages are aggregated with the node's own embedding in a differentiated manner, thereby updating its own embedding and generating aligned image representations and text representations respectively.

[0143] Specifically, the cross-modal data message passing and embedding alignment process can be as follows: a graph neural network with a cross-modal attention mechanism is introduced. Based on the message passing mechanism of the graph neural network, each target node in the multimodal heterogeneous graph receives messages from neighboring nodes during the iterative update process. Different weights are set for different modal messages. Based on the weight ratio setting, the embeddings of different modal messages received from neighboring nodes and the target node itself are differentially aggregated to update the node's own embedding, generating the text alignment representation corresponding to the text embedding representation and the image alignment representation corresponding to the image embedding representation.

[0144] In this step, for specific application scenarios, time and spatial information of the data is utilized to introduce sequence models and graph models to generate multimodal data-aligned embeddings, thereby enabling model training and adjustment. Furthermore, based on the aforementioned construction of multiple instructions including question answering, description, and reasoning, fine-tuning based on these model instructions allows the model to output standardized anomaly judgment results, anomaly descriptions, anomaly cause inferences, and anomaly locations. Therefore, this embodiment of the invention, by combining a large language model and a multimodal embedding fusion scheme, designs a novel model pre-training and fine-tuning scheme, achieving a strong generalization capability anomaly detection algorithm framework.

[0145] Step 104: Input the data to be identified into the multimodal anomaly detection model for anomaly identification and obtain the anomaly identification result.

[0146] This step primarily focuses on anomaly identification based on a multimodal anomaly detection model within real-world application scenarios. Specifically, the data to be identified is input into the multimodal anomaly detection model for anomaly identification, yielding the anomaly identification results. This can be achieved by first assessing the accuracy of the text description of the data using an information content evaluation index, and then using a factual evaluation index to perform anomaly judgment based on the text description. The anomaly identification results are then output, including the anomaly judgment result, anomaly description, anomaly cause inference, and anomaly location.

[0147] To address the anomaly detection problem, this invention constructs several data evaluation metrics to assess whether the data to be identified contains anomalies. These metrics are plug-and-play. Specifically, based on the previously described scheme, the data to be identified in multiple modalities is converted into text modalities. This text modal data can be regarded as a text description of the data to be identified. Based on these text descriptions of the identified data, the following metrics are constructed to comprehensively evaluate whether the text descriptions contain anomalies:

[0148] (1) Information content index:

[0149] The information content index assesses whether a text description contains useful information. It evaluates whether the text description generated by the large language model accurately, reasonably, and completely describes the data to be regulated. The calculation formula is as follows:

[0150]

[0151] In the formula, S info (k,q) represents the information content index; q represents the prompt (notification word) for detecting whether the data to be identified (regulated) is compliant / abnormal, which can refer to the query / question; k is the text description token generated by the large language model, which can refer to the learned knowledge; P θ (k_t∣k_(1:t-1),q) represents the probability of generating learned knowledge k based on q; M is the number of knowledge generated.

[0152] Therefore, the information content metric is the sum of the information entropy of all text description tokens generated by the large language model, given a regulatory problem q.

[0153] (2) Factual indicators:

[0154] Factual indicators directly assess whether there are anomalies in the text description. They evaluate the data to be identified based on the information content indicator. The calculation formula is as follows:

[0155]

[0156] In the formula, S fact(k,E) represents a factual indicator; k={s1,…,s m It consists of m sentences (knowledge); s i This represents the background knowledge retrieved based on the model for q; to determine whether the retrieved background knowledge is correct, l is provided for each piece of background knowledge. i One factual basis Therefore, e i Indicates the factual basis corresponding to background knowledge, l i Indicate the quantity of factual evidence, that is It can be understood as a factual knowledge base related to the problem, and in the context of compliance testing, it can be understood as a rule document; f(.) represents the equation for calculating sentence-level facts; NLI(.) represents the natural language reasoning model, which is used to process premise-assumption pairs to output a three-dimensional vector.

[0157] The output of the factual indicator includes scores on three dimensions. These three scores represent the scores for factual consistency, neutrality, and factual inconsistency, respectively.

[0158] To provide a more intuitive understanding of the two evaluation metrics mentioned above for those skilled in the art, it is assumed that the process of generating results from a large model can be viewed as finding a series of m knowledge points k = {s1,…,s...} most relevant to the query / question q from a learned large knowledge base. m This process generates a response 'a'. For example, suppose a technician wants to query an anomaly in the "transportation domain". The trained large model should be able to retrieve the most relevant background knowledge in the "transportation domain", such as vehicle information, from its learned knowledge, and then give an anomaly judgment in the "transportation domain".

[0159] Combining the two indicators mentioned above, we can first assess the accuracy of the text description, and then, based on the text description, assess whether there are any anomalies, thus achieving a two-stage assessment.

[0160] In this step, formatted identification results can be generated based on the input data. Furthermore, post-processing techniques can be applied to the model output to output anomaly images in parallel, obtaining anomaly identification numerical results for the images. For example: selecting appropriate portions from the model's identified text as the discrimination result; formatting the discrimination result and writing it to a file, database, or displaying it on a user interface; scoring the anomalies and classifying the severity of the input data anomalies based on the scores. In this way, the model can generate formatted output results adapted to various application scenarios when facing specific downstream tasks such as anomaly detection, anomaly region segmentation, and anomaly cause analysis.

[0161] For example, in order to facilitate a better understanding of the technical solution of the present invention by those skilled in the art, Figure 5 A schematic diagram of the training and inference process of a multimodal anomaly detection model provided in an embodiment of the present invention is shown.

[0162] Step S501: Pre-train the model based on the "image-short text" data.

[0163] In the pre-training stage, the LLaMA2 large language model was selected as the base model for multimodal anomaly recognition and detection. Combined with the LoRA optimization strategy, the base model was pre-trained based on multimodal embedding alignment according to "image-short text" data to learn the semantic relationships and alignment information between modalities and obtain a general multimodal anomaly detection model.

[0164] Step S502: Fine-tune the model instructions based on the "image-long text" data.

[0165] During the instruction fine-tuning stage, the general multimodal anomaly detection model is fine-tuned based on multimodal embedding alignment according to the "image-long text" data, combined with the efficient fine-tuning strategy of LLaMA2 parameters, to obtain the multimodal anomaly detection model.

[0166] Step S503: Anomaly Detection

[0167] In real-world application scenarios, anomaly identification is performed based on a multimodal anomaly detection model.

[0168] In this invention embodiment, a method for anomaly identification and detection in multimodal data services is proposed based on a large language model to handle general anomaly detection in multimodal data and arbitrary rule scenarios. Given multimodal data including images, videos, and text, as well as rule text in natural language form, a graph structure network is constructed using attributes such as data temporal and spatial information and correlation, and a graph-based embedding and fusion strategy and instruction fine-tuning strategy are designed. A cross-modal information aggregation algorithm is designed in conjunction with the graph model, improving the method's ability to capture multimodal information and enhancing the model's generalization ability on multimodal data in multiple scenarios, while simultaneously achieving anomaly identification in multimodal data services. For image and video data, a target detection algorithm targeting key violation elements is introduced to discover anomalous elements in image / frame data. By combining deep learning and natural language processing technologies, comprehensive supervision of digital services is achieved. Compared with traditional methods, the method proposed in this invention embodiment can more comprehensively identify abnormal behaviors in digital services, improving supervision efficiency and accuracy.

[0169] For better explanation, refer to Figure 6This diagram illustrates the overall flow of a multimodal data anomaly identification method provided by an embodiment of the present invention. It should be noted that this embodiment only provides a brief description of the general flow of multimodal data anomaly identification; the specific implementation process of each step can be understood by referring to the relevant content in the foregoing embodiments, and will not be elaborated upon here. It is understood that the present invention does not impose any limitations on this.

[0170] First, obtain the public anomaly detection dataset and the specific scenario dataset (i.e., the original multimodal anomaly dataset). Then, perform data preprocessing on the public anomaly detection dataset and the specific scenario dataset to obtain the multimodal dataset.

[0171] Next, based on the multimodal dataset, pre-trained data of "image-short text" and fine-tuned data of "image-long text" are generated;

[0172] Then, based on the "image-short text" pre-training data and the "image-long text" fine-tuning data, the model training and fine-tuning of multimodal embedding alignment are carried out. Specifically, the LLaMA2 large language model is selected as the base model for multimodal anomaly recognition and detection. First, the LoRA optimization strategy is combined with the "image-short text" pre-training data to perform model pre-training based on multimodal embedding. Then, combined with the LLaMA2 parameter efficient fine-tuning strategy, the model obtained after pre-training is fine-tuned based on multimodal embedding based on the "image-long text" fine-tuning data to obtain the multimodal anomaly detection model.

[0173] Finally, based on the multimodal anomaly detection model, anomaly identification is performed on the data to be identified, and the corresponding anomaly identification results are output.

[0174] Reference Figure 7 The diagram shows a structural block diagram of a multimodal data anomaly identification device provided by an embodiment of the present invention, which may specifically include:

[0175] The multimodal dataset acquisition module 701 is used to acquire multimodal datasets;

[0176] The image-text dataset construction module 702 is used to construct an image-text pre-training set and an image-text fine-tuning dataset based on the multimodal dataset.

[0177] The multimodal embedding alignment processing module 703 is used to pre-train the model based on the image-text pre-training set based on multimodal embedding alignment, and fine-tune the model obtained after pre-training based on the image-text fine-tuning dataset to obtain a multimodal anomaly detection model.

[0178] The multimodal anomaly recognition module 704 is used to input the data to be recognized into the multimodal anomaly detection model for anomaly recognition and to obtain anomaly recognition results.

[0179] In one optional embodiment, the multimodal dataset includes anomalous text data and anomalous image data, and the image-text dataset construction module 702 includes:

[0180] The image-text association module is used to perform image-text association on the abnormal text data and the abnormal image data to generate multiple image-text data pairs.

[0181] The descriptive text generation module is used to generate descriptive text corresponding to each of the image text data pairs.

[0182] The image-text pre-training data pair generation module is used to calculate the similarity between the description text and the image in the image-text data pair for each image-text data pair, and generate image-text pre-training data pairs based on the image-text data pair, the description text, and the similarity.

[0183] The image-text pre-training set construction module is used to construct an image-text pre-training set based on each of the image-text pre-training data pairs;

[0184] The image-text fine-tuning data pair generation module is used to obtain a pre-created scene instruction set for anomaly detection, and based on the scene instruction set, to expand the multiple image-text data pairs using a self-instruction method to obtain the image-text fine-tuning data pairs corresponding to each of the image-text data pairs.

[0185] The image-text fine-tuning dataset construction module is used to construct an image-text fine-tuning dataset based on each of the image-text fine-tuning data pairs.

[0186] In one alternative embodiment, the device further includes:

[0187] The task instruction template creation module is used to create multiple different task instruction templates. Each task instruction template is used to perform anomaly judgment for a vertical domain scenario. Each task instruction template includes at least an anomaly description instruction, an anomaly question and answer instruction, and an anomaly analysis instruction.

[0188] The scene instruction set generation module is used to generate scene instruction sets for different vertical domain scenarios based on the multiple task instruction templates.

[0189] In one alternative embodiment, the multimodal embedding alignment processing module 703 includes:

[0190] The model pre-training module is used to select the LLaMA2 large language model as the base model for multimodal anomaly recognition and detection, and combine it with the LoRA optimization strategy to perform model pre-training based on multimodal embedding alignment on the base model according to the image-text pre-training set, so as to learn the semantic relationship and alignment information between modalities and obtain a general multimodal anomaly detection model.

[0191] The model fine-tuning module is used to combine the LLaMA2 parameter efficient fine-tuning strategy to perform model fine-tuning based on multimodal embedding alignment on the general multimodal anomaly detection model according to the image-text fine-tuning dataset, so as to obtain the multimodal anomaly detection model.

[0192] In an optional embodiment, the multimodal embedding alignment processing module 703 further includes:

[0193] The text embedding representation generation module is used to extract the abnormal text data and the descriptive text from the dataset used in the current processing stage, and to map the abnormal text data and the descriptive text to the embedding space in vector form to generate a text embedding representation.

[0194] The image embedding representation generation module is used to extract the abnormal image data from the dataset used in the current processing stage, and input the abnormal image data into the visual encoder for vector transformation to generate the image embedding representation;

[0195] A multimodal heterogeneous graph construction module is used to construct a semantic graph corresponding to the text embedding representation and a visual graph corresponding to the image embedding representation, and to perform data association between the semantic graph and the visual graph to construct a multimodal heterogeneous graph, wherein the multimodal heterogeneous graph includes multiple nodes;

[0196] The message passing module is used to introduce a graph neural network with a cross-modal attention mechanism. Based on the message passing mechanism of the graph neural network, during the iterative update process, each target node in the multimodal heterogeneous graph receives messages from neighboring nodes.

[0197] The node self-embedding update module is used to set different weights for different modal messages. Based on the weight ratio setting, it performs differential aggregation of different modal messages received from the neighboring nodes and the embedding of the target node itself to update the node self-embedding, and generates the text alignment representation corresponding to the text embedding representation and the image alignment representation corresponding to the image embedding representation.

[0198] In one optional embodiment, the multimodal anomaly identification module 704 is specifically used for:

[0199] The data to be identified is input into the multimodal anomaly detection model. First, the accuracy of the text description of the data to be identified is judged by combining the information content evaluation index. Then, the anomaly judgment is performed based on the text description by combining the factual evaluation index. The anomaly recognition result is output, which includes the anomaly judgment result, anomaly description, anomaly cause inference, and anomaly location.

[0200] In one alternative embodiment, the device includes:

[0201] The multimodal anomaly data acquisition module is used to acquire a publicly available anomaly detection dataset and a raw multimodal anomaly dataset. Both the publicly available anomaly detection dataset and the raw multimodal anomaly dataset include text data, form data, image data, and video data that contain anomalies.

[0202] The text preprocessing module is used to preprocess the text data and the form data to obtain abnormal text data;

[0203] The image preprocessing module is used to perform image preprocessing on the image data and the video data to obtain processed image data;

[0204] A supplementary image data generation module is used to input the processed image data into a pre-trained image anomaly detection model to supplement auxiliary information and obtain supplementary image data.

[0205] An abnormal image data generation module is used to divide the supplementary image data into image regions, and then generate image descriptions and title information corresponding to each image region based on a pre-trained image description model to obtain abnormal image data.

[0206] The multimodal dataset integration module is used to integrate the abnormal text data and the abnormal image data into a multimodal dataset.

[0207] As the device embodiment is basically similar to the method embodiment, it is described in a relatively simple way. For relevant details, please refer to the description of the method embodiment above.

[0208] This invention also provides an electronic device, which includes a processor and a memory:

[0209] The memory is used to store program code and transfer the program code to the processor;

[0210] The processor is used to execute the multimodal data anomaly identification method of any embodiment of the present invention according to the instructions in the program code.

[0211] This invention also provides a computer-readable storage medium for storing program code for executing the multimodal data anomaly identification method of any embodiment of this invention.

[0212] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0213] In the embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between devices or units through some interfaces, and may be electrical, mechanical, or other forms.

[0214] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0215] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0216] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0217] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for multi-modal data anomaly identification, the method comprising: include: Obtain a multimodal dataset; Based on the multimodal dataset, construct an image-text pre-training set and an image-text fine-tuning dataset; Based on multimodal embedding alignment, the model is pre-trained using the image-text pre-training set, and then fine-tuned using the image-text fine-tuning dataset to obtain a multimodal anomaly detection model. The data to be identified is input into the multimodal anomaly detection model for anomaly identification, and anomaly identification results are obtained. The multimodal dataset includes anomalous text data and anomalous image data; by performing image-text association on the anomalous text data and the anomalous image data, multiple image-text data pairs are generated, and then descriptive text corresponding to each image-text data pair is generated; The multimodal embedding alignment process includes: Extract the abnormal text data and the descriptive text from the dataset used in the current processing stage, and map the abnormal text data and the descriptive text to the embedding space in vector form to generate a text embedding representation; The abnormal image data is extracted from the dataset used in the current processing stage, and the abnormal image data is input into the visual encoder for vector transformation to generate an image embedding representation; Construct a semantic graph corresponding to the text embedding representation and a visual graph corresponding to the image embedding representation, and perform data association between the semantic graph and the visual graph to construct a multimodal heterogeneous graph, which includes multiple nodes; A graph neural network with a cross-modal attention mechanism is introduced. Based on the message passing mechanism of the graph neural network, each target node in the multimodal heterogeneous graph receives messages from neighboring nodes during the iterative update process. Different weights are set for different modal messages. Based on the weight ratio setting, the embeddings of different modal messages received from the neighboring nodes are differentially aggregated with the embeddings of the target node itself to update the node's own embeddings, and generate the text-aligned representation corresponding to the text embedding representation and the image-aligned representation corresponding to the image embedding representation. 2.The multi-modal data anomaly identification method of claim 1, wherein, The construction of an image-text pre-training set and an image-text fine-tuning set based on the multimodal dataset includes: For each image-text data pair, the similarity between the descriptive text and the image in the image-text data pair is calculated, and an image-text pre-training data pair is generated based on the image-text data pair, the descriptive text, and the similarity. Construct an image-text pre-training set based on each of the image-text pre-training data pairs; Obtain a pre-created scene instruction set for anomaly detection, and based on the scene instruction set, expand the multiple image-text data pairs using a self-instruction method to obtain the image-text fine-tuning data pairs corresponding to each of the image-text data pairs. Based on each of the image-text fine-tuning data pairs, construct an image-text fine-tuning dataset. 3.The multi-modal data anomaly identification method of claim 2, wherein, Also includes: Create multiple different task instruction templates, each of which is used to perform anomaly detection for a vertical domain scenario, and each of which includes at least an anomaly description instruction, an anomaly question and answer instruction, and an anomaly analysis instruction. Based on the multiple task instruction templates, scene instruction sets for different vertical domain scenarios are generated. 4.The multi-modal data anomaly identification method of claim 2, wherein, The process of obtaining a multimodal anomaly detection model based on multimodal embedding alignment, pre-training the model using the image-text pre-training set, and fine-tuning the pre-trained model using the image-text fine-tuning dataset includes: The LLaMA2 large language model was selected as the base model for multimodal anomaly recognition and detection. Combined with the LoRA optimization strategy, the base model was pre-trained based on multimodal embedding alignment according to the image-text pre-training set to learn the semantic relationships and alignment information between modalities and obtain a general multimodal anomaly detection model. By combining the efficient fine-tuning strategy of LLaMA2 parameters, the general multimodal anomaly detection model is fine-tuned based on multimodal embedding alignment according to the image-text fine-tuning dataset to obtain the multimodal anomaly detection model. 5.The multi-modal data anomaly identification method of claim 1, wherein, The step of inputting the data to be identified into the multimodal anomaly detection model for anomaly identification and obtaining anomaly identification results includes: The data to be identified is input into the multimodal anomaly detection model. First, the accuracy of the text description of the data to be identified is judged by combining the information content evaluation index. Then, the anomaly judgment is performed based on the text description by combining the factual evaluation index. The anomaly recognition result is output, which includes the anomaly judgment result, anomaly description, anomaly cause inference, and anomaly location. 6.The multi-modal data anomaly identification method of any one of claims 1 to 5, wherein, Prior to acquiring the multimodal dataset, the method includes: Obtain a publicly available anomaly detection dataset and a raw multimodal anomaly dataset, both of which include text data, form data, image data, and video data containing anomalies. The text data and the form data are preprocessed to obtain abnormal text data; The image data and video data are preprocessed to obtain processed image data; The processed image data is input into a pre-trained image anomaly detection model to supplement auxiliary information and obtain supplementary image data; The supplementary image data is divided into image regions, and then based on a pre-trained image description model, image descriptions and title information corresponding to each image region are generated to obtain abnormal image data. The abnormal text data and the abnormal image data are integrated together into a multimodal dataset.

7. A multimodal data anomaly identification device, characterized in that, include: The multimodal dataset acquisition module is used to acquire multimodal datasets. The image-text dataset construction module is used to construct an image-text pre-training set and an image-text fine-tuning dataset based on the multimodal dataset. The multimodal embedding alignment processing module is used to pre-train the model based on the image-text pre-training set based on multimodal embedding alignment, and fine-tune the model obtained after pre-training based on the image-text fine-tuning dataset to obtain a multimodal anomaly detection model. The multimodal anomaly recognition module is used to input the data to be recognized into the multimodal anomaly detection model for anomaly recognition and to obtain anomaly recognition results; The multimodal dataset includes abnormal text data and abnormal image data; the image text dataset construction module is used to generate multiple image text data pairs by performing image-text association on the abnormal text data and the abnormal image data, and then generate the corresponding descriptive text for each image text data pair. The multimodal embedding alignment processing module includes: The text embedding representation generation module is used to extract the abnormal text data and the descriptive text from the dataset used in the current processing stage, and to map the abnormal text data and the descriptive text to the embedding space in vector form to generate a text embedding representation. The image embedding representation generation module is used to extract the abnormal image data from the dataset used in the current processing stage, and input the abnormal image data into the visual encoder for vector transformation to generate the image embedding representation; A multimodal heterogeneous graph construction module is used to construct a semantic graph corresponding to the text embedding representation and a visual graph corresponding to the image embedding representation, and to perform data association between the semantic graph and the visual graph to construct a multimodal heterogeneous graph, wherein the multimodal heterogeneous graph includes multiple nodes; The message passing module is used to introduce a graph neural network with a cross-modal attention mechanism. Based on the message passing mechanism of the graph neural network, during the iterative update process, each target node in the multimodal heterogeneous graph receives messages from neighboring nodes. The node self-embedding update module is used to set different weights for different modal messages. Based on the weight ratio setting, it performs differential aggregation of different modal messages received from the neighboring nodes and the embedding of the target node itself to update the node self-embedding, and generates the text alignment representation corresponding to the text embedding representation and the image alignment representation corresponding to the image embedding representation.

8. An electronic device, comprising: The device includes a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the multimodal data anomaly identification method according to any one of claims 1-6 according to the instructions in the program code.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store program code for executing the multimodal data anomaly identification method according to any one of claims 1-6.

Images