Risk content identification and abstract generation method of multi-modal large model

Abstract

A method for risk content identification and abstract generation by using a multi-modal large model comprises the following steps: collecting risk content data and carrying out data preprocessing, and dividing based on the preprocessed risk content data to obtain a training set and a test set; performing model fine tuning on Chameleon-7B by using the training set to obtain a risk identification large model RiskControl-Valuer; inputting the test set data into a RiskContent-Valuer for abstract feature extraction to obtain a risk content identification and abstract generation result; wherein before fine tuning of the model, a BitsAndBytesConfig interface is utilized to quantify a basic model Chameleon-7B, the weight of the model is converted into int4 through a quantization layer, the int4 is put into a GPU, and a core calculation process is completed on a CUDA (Compute Unified Device Architecture); and performing model fine tuning by adopting supervised learning and LoRA methods. According to the method, the consumption of computing resources is greatly reduced by utilizing the LoRA, the powerful language understanding ability crossing different modalities and styles is inherited, accurate risk content assessment and total results are realized, and meanwhile, risk point abstracts are provided.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence security technology, and specifically relates to a multimodal large model for risk content identification and summary generation method. Background Art

[0002] With the rapid development and popularization of communication technology and digital technology, risk content such as fraud and rumors has increased, becoming a new challenge for social management and cyberspace governance. Multimodal risk content, such as fraudulent text messages, phone calls, and fake faces, sounds, and videos, poses a huge hidden danger to national security, economic development, and social stability. However, when dealing with increasingly complex, dynamic, and diverse multimodal content, traditional deep learning-based methods have problems such as poor generalization performance of the model in different data environments and insufficient ability to capture and learn high-level abstract expressions. These problems limit the effectiveness and efficiency of existing technologies in practical applications. Therefore, it is very important for national security and social management to design an efficient, accurate, and flexible method to complete the identification of multimodal risk content. This method needs to automatically analyze risk content, output identification results, and provide a summary of risk points.

[0003] Existing methods have made great progress in risk content identification and summarization. However, there are still some limitations. First, due to the development of modern information digitization, the complexity of data is increasing, which leads to an increase in the time of model training and prediction; second, in terms of information risk content identification and summarization, the migration characteristics of current models to different types of risk content encounter the problem of insufficient model generalization ability. Model generalization requires a trained model to transfer its generation capabilities to the target domain and produce accurate outputs for previously unseen instances. In the task of information risk content identification and summarization, this requires the generation of realistic and accurate reports for various risk contents. Although image description has shown significant effectiveness in general text generation tasks, the diversity of risk content and the complexity of language have created obstacles to model generalization. Summary of the invention

[0004] In order to overcome the above problems, the purpose of this application is to propose a multimodal large model for risk content identification and summary generation method. By proposing RiskContent-Valuer on large-scale, high-quality instruction compliance data in the field of risk content identification, using LoRA (Low-Rank Adaptation) technology to significantly reduce the consumption of computing resources, it inherits powerful language understanding capabilities across different modalities and genres, coupled with complex reasoning and diverse generation capabilities. In addition, high-level domain-specific knowledge is learned in the training of risk content identification, which solves the heterogeneity gap between different risk contents, and achieves accurate risk content assessment and overall results, while providing risk point summaries.

[0005] In order to achieve the above purpose, the technical solution adopted in this application is as follows:

[0006] A multimodal large model is used for risk content identification and summary generation method, including:

[0007] S1. Collect risk content data and perform data preprocessing, and divide the risk content data after preprocessing into training set and test set;

[0008] S2. Build the basic model Chameleon-7B, use the training set to fine-tune the basic model Chameleon-7B, and obtain the risk identification model RiskContent-Valuer;

[0009] S3. Input the data in the test set into the risk identification model RiskContent-Valuer to extract abstract features and obtain the risk content identification and summary generation results;

[0010] Before fine-tuning the model, the BitsAndBytesConfig interface is used to quantize the basic model Chameleon-7B. The model weights are converted into int4 through the quantization layer and put into the GPU, and the core calculation process is completed on CUDA. Supervised learning and LoRA methods are used to fine-tune the model.

[0011] The risky content data is collected from heterogeneous public data from different sources, including but not limited to: online rumors, violent videos, non-violent videos, crowd violence videos, street fights, abuse videos, arrest videos, arson videos, assault videos, traffic accident videos, burglary videos, explosion videos, fight videos, robbery videos, shooting videos, theft videos, shoplifting videos, and vandalism videos.

[0012] Data preprocessing includes data integration and data cleaning. Data integration involves unifying the text data set, extracting key frames from the video data set, and classifying them by risk category. Data cleaning involves eliminating data that is not identified as suspected bad information, is meaningless or non-directional, and lacks sensitive information, to form risk content data containing text and images.

[0013] The data in the training set is constructed into a unified json format file, which contains the following fields: id, image, conversations, from, instruction, prompt, and value. id records the label of the training data, image records the path of the training image, and conversations is a large field including from, instruction, prompt, and value. From indicates whether the source of the conversation is human or AI. Instruct indicates system instructions, which are used to regulate the output behavior of the basic model. Prompt indicates specific human prompts. The basic model will output according to the specific human prompts. Value is the preset answer of the basic model. Supervised learning is used to fine-tune the basic model.

[0014] In the test data set, the path, name, serial number, label, and summary information of the test images are uniformly saved in a csv file to facilitate traversal and access by the test script. The test script calculates the accuracy and text similarity of the basic model according to the labels and summaries provided in the csv file, which is used to provide prerequisites for the evaluation and optimization of the basic model.

[0015] Using the small batch fine-tuning method, the optimal prompt is found through a round of small-scale fine-tuning. The optimal prompt is used to improve the model's ability to identify risky content and the quality of summary generation. The optimal prompt is suitable for the full stage of model fine-tuning.

[0016] The prompt consists of a system description, instructions, and input. The system description is called at the beginning to define the tasks and constrain the behavior of the instructions, ensuring that the model's identification of risk content and summary generation meet the preset goals; the instructions are used to provide commands for risk content identification and summary generation, guiding the model to identify and process risk content and generate relevant summaries; the input contains the text and images to be detected, which are used to support the model's processing process.

[0017] The evaluation indicators of the risk identification model RiskContent-Valuer are ROUGE and Accuracy.

[0018] A multimodal large model is used for risk content identification and summary generation system, including:

[0019] Risk content collection module: collects risk content data and performs data preprocessing, and divides the preprocessed risk content data into training sets and test sets;

[0020] Model fine-tuning module: build the basic model Chameleon-7B, use the training set to fine-tune the basic model Chameleon-7B, and obtain the risk identification model RiskContent-Valuer; before fine-tuning the model, use the BitsAndBytesConfig interface to quantize the basic model Chameleon-7B, convert the model weights into int4 through the quantization layer and put them into the GPU, and complete the core calculation process on CUDA; use supervised learning and LoRA methods to fine-tune the model;

[0021] Result generation module: The data in the test set is input into the risk identification model RiskContent-Valuer to extract abstract features and obtain risk content identification and summary generation results.

[0022] A computer storable medium stores a computer program, which, when executed by a processor, implements a multimodal large model for risk content identification and summary generation method.

[0023] Compared with the prior art, the beneficial effects of this application are:

[0024] 1. By introducing preprocessing for diverse data sets, this application can filter and clean training data more effectively compared to the existing technology, thereby reducing the negative impact of noise data on the large model training process and improving the generation quality and stability of the model.

[0025] 2. Compared with the existing technology, this application can effectively guide the model to generate more accurate and relevant content by using high-quality prompt words during the training process, thereby improving the performance of the model in complex tasks and reducing the probability of misidentification of risky content.

[0026] 3. Compared with the existing technology, this application can effectively improve the parameter efficiency of the model without significantly increasing the computing cost by using quantization and LoRA technology in the training process, enhance the adaptability of the model to specific tasks, and improve the overall performance of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 A flow chart of the method of this application;

[0028] Figure 2 Schematic diagram of the framework constructed for this application;

[0029] Figure 3A schematic diagram of the data processing flow for this application;

[0030] Figure 4 Examples of data collected for this application risk sample;

[0031] Figure 5 This is a schematic diagram of the application during use. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solution and advantages of the present application more clear, the present application is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described here are only used to explain the present application and are not used to limit the present application.

[0033] This application uses Chameleon-7B as the basic model, fine-tunes the parameters of Chameleon-7B on a public data set, and mainly uses the technical means of LoRA (Low-Rank Adaptation) to significantly reduce the consumption of computing resources, and finally obtains the RiskContent-Valuer model suitable for risk content data.

[0034] The fine-tuning technology based on the open source large model Chameleon-7B mainly includes three parts. The fine-tuning process is performed on the training set:

[0035] (1) Model quantization process: With the help of the Transformers quantization model library, the BitsAndBytesConfig interface is used to quantize the basic model. The essence is to convert the model weights into int4 through the quantization layer and put it into the GPU. The core calculation process is completed on CUDA, which can reduce memory usage and improve efficiency.

[0036] (2) Prompt (specific human prompt) screening process: In the fine-tuning stage, prompt selection plays an important role in order to effectively activate the generalization and semantic capabilities of mllm. However, thoroughly fine-tuning any mllm is a time-consuming project that requires sufficient computing resources and time capacity. Therefore, we adopt a small batch fine-tuning method to use five carefully written prompts for efficient fine-tuning. The best prompt can be found through a round of small-scale fine-tuning, which will be suitable for the full stage of model fine-tuning to ensure the stability and reliability of the model in practical applications.

[0037] (3) Model fine-tuning process: Model fine-tuning is divided into two steps: In the first step, the parameters of the transformer component of the model are refined, focusing only on training embeddings. This is done to adapt to the newly added small sample vectors without significantly interfering with the original model. In the second step, training continues for most of the remaining samples while updating the LoRA parameters. At this point, we will get a LoRA weight based on the pre-trained model, which is independent of the pre-trained model and will not affect the inherent performance of the pre-trained model, but will have stronger adaptability and capabilities for our task. After this, we merge the LoRA weight W back to the initialized pre-trained model finite value to prepare for further risk content summary and performance evaluation.

[0038] like Figure 1 , 2 As shown, a multimodal large model is used for risk content identification and summary generation method, including:

[0039] S1. Risk content data collection and data preprocessing, and division into training set and test set;

[0040] The risk content data collection is used to obtain multi-modal and multi-data types from different online public data sets. It is composed of content text and risk images, and proposes a unified and generalizable framework that can simulate the real network environment: perform network content security review across multiple modes and multiple systems to improve review efficiency and reduce the workload of relevant personnel in processing massive data. The collected risk content is taken from heterogeneous public data from seven different sources, and generally contains 420,260 samples in two modes, text and image, including: basic event content (content summary), event type, source channel, sensitive words, release date, region code & name, etc. Figure 4 As shown, this information cannot be used directly for training. We screen, clean and construct the collected data set and divide it into a test set and a training set.

[0041] Specifically, in the test data set, the path of the test image, the name of the test image, the serial number, the label (true value), the summary (true value) and other information are uniformly saved in a csv file to facilitate the traversal and access of the test script. The test script can calculate the accuracy and text similarity of the model according to the labels and summaries provided in the csv file, which provides the prerequisite for model evaluation and optimization.

[0042] The training set is constructed into a unified json format file, which contains several fields: 'id', 'image', 'conversations', 'from', 'instruct', 'prompt', and 'value'. Among them, id records the label of the training data, image records the path of the training image, and conversations is a large field, which includes from, instruct, prompt, and value. From indicates whether the source of the conversation is human or AI, and instruct is a system instruction used to regulate the output behavior of the large model. Prompt is a specific human prompt, and the large model will make outputs based on the prompt. Value is the preset answer of the large model, which is used to train the large model.

[0043] Specific risk categories include: online rumors, violent videos, non-violent videos, crowd violence videos, street fights, abuse videos, arrest videos, arson videos, assault videos, traffic accident videos, burglary videos, explosion videos, fight videos, robbery videos, shooting videos, theft videos, shoplifting videos and vandalism videos.

[0044] like Figure 3 As shown, the data preprocessing includes two parts, data integration and data cleaning, which plays a key role in ensuring the effectiveness and reliability of the model when processing a large amount of specific field risk content from different data sets.

[0045] Data integration: Aggregate the data from the above seven datasets, group them by risk category, and process them according to the category to obtain a comprehensive dataset covering all the same risk categories. Then extract the training set and test set according to the category index in the order of these risk categories. Specifically, the network rumor text dataset is unified and organized, the risk content video dataset is keyframe extracted, and classified according to the risk category, thereby forming a comprehensive network rumor text dataset and a network risk content image dataset. Then randomly select data of different risk content categories such as network rumors, violent behavior, traffic accidents, etc. to fine-tune Chameleon-7B to obtain RiskContent-Valuer, and combine it with other remaining data to test the generalization performance of RiskContent-Valuer. The selection of the training data is to randomly divide the entire comprehensive dataset into 80% training set and 20% validation set.

[0046] Data cleaning: The data cleaning phase aims to place special emphasis on mitigating undue noise caused by duplicate values ​​and report titles, inconsistent tables, and irrelevant text and images to ensure that the data used (training + testing) is high-quality and useful for making inferences. For each dataset, a rigorous data cleaning process is carried out from three different perspectives: resolving duplications, correcting inconsistencies, and filtering out meaningless content (text and images).

[0047] To address duplication issues, manual searches were performed to identify and eliminate elements that were not relevant to values ​​and report titles, resulting in a collated dataset with unique values ​​in a standardized format. Careful manual review and manual merging were used to resolve inconsistencies arising from multiple tables from different datasets. Regarding content importance, the applicant team created a corpus containing non-essential terms that are too confusing for a large multimodal language model to understand and are used to activate the reasoning performance of a large multimodal language model, but these terms are critical to practical applications. Using this corpus, a large portion of invalid data that is not suitable for the task of a large multimodal language model was deleted, resulting in a refined dataset dedicated to prompt generation.

[0048] S2. Construct the basic model Chameleon-7B, and use the training set to fine-tune the basic model Chameleon-7B to obtain the risk identification model RiskContent-Valuer; the model fine-tuning is specifically as follows: quantize the basic model using the BitsAndBytesConfig interface, and convert the model weights into int4 through the quantization layer. Use supervised learning and LoRA methods to fine-tune the model. Use the small batch fine-tuning method to find the best prompt through a round of small-scale fine-tuning. The best prompt is used to improve the model's ability to identify risk content and the quality of summary generation. The best prompt is suitable for the full stage of model fine-tuning.

[0049] The prompt consists of a system description, instructions, and input. The system description is called at the beginning to define the tasks and constrain the behavior of the instructions, ensuring that the model's identification of risk content and summary generation meet the preset goals; the instructions are used to provide commands for risk content identification and summary generation, guiding the model to identify and process risk content and generate relevant summaries; the input contains the text and images to be detected, which are used to support the model's processing process.

[0050] S3. Input the test set data into the risk identification model RiskContent-Valuer to extract abstract features and obtain risk content identification and summary generation results; the evaluation indicators of the risk identification model RiskContent-Valuer are ROUGE and Accuracy.

[0051] RiskContent-Valuer is a generalized risk content identification and summarization multimodal large language model system based on multimodal and multi-data types. It includes five parts: risk content data collection, data preprocessing, abstract feature extraction, risk content summary generation, and performance evaluation.

[0052] This application selected the Chameleon-7B, MiniGPT-4, and LLAVA-v1.5 multimodal models, involving four considerations in total:

[0053] Large-scale datasets: The training datasets of the above three models cover a wide range of content. Chameleon-7B's plain text data includes all the pre-training data used by Llama 2 and CodeLlama, totaling 2.9 trillion tokens. The text-image pairs contain some public data, totaling 1.4 billion pairs and 1.5 trillion tokens. For data that interleaves text and images, public data sources are used entirely, totaling 400 billion tokens. At the same time, MiniGPT-4 and LLAVA-v1.5 models also have large training datasets. This huge dataset gives the model the ability to learn from a variety of data sources, enabling it to effectively handle heterogeneous and diverse data.

[0054] Extension of context length: Chameleon-7B's image segmenter, based on a codebook of size 8192, encodes an image of size 512×512 into 1024 discrete tokens. The text segmenter is based on the sentencepiece open source library developed by Google, and a BPE segmenter with 65536 text tokens and 8192 image tokens is trained. This extension means that Chameleon-7B is more suitable for understanding and processing longer / larger text / image paragraphs. When processing heterogeneous data, there may be more text / image information to consider, and the extended context length helps to capture this information more comprehensively.

[0055] Multi-task adaptability: Chameleon-7B exhibits excellent adaptability across multiple tasks, indicating that it is able to perform multiple tasks simultaneously when processing heterogeneous data. This enables it to more fully exploit the diversity and complexity of the data.

[0056] Improved model architecture: Chameleon-7B, like GPT-4o, uses a unified Transformer architecture and completes training using a mixed modality of text, images, and code. Images are discretely tokenized in a manner similar to text generation, and ultimately interleaved text and image sequences are generated and reasoned. With this "early fusion" approach, all pipelines are mapped to a common representation space from the beginning, so the model can process text and images seamlessly.

[0057] Based on the Chameleon-7B, MiniGPT-4, LLAVA-v1.5 multimodal large language models and the risk content data after data preprocessing, this application trains the model and uses the following three technologies:

[0058] Model quantization: We use the Transformers model quantization library and the BitsAndBytesConfig interface to perform model quantization. Basically, this process involves converting the model weights to int4 format through a quantization layer and then putting them on the GPU. We use the int4 quantization configuration to initialize the pre-trained model for further model preparation. The core calculations are performed on CUDA, which reduces memory consumption and improves efficiency.

[0059] Model fine-tuning: Incorporates LoRA (Low Rank Adaptation) weights and training parameters. In the first step, we fix the parameters of the transformer component of the model and focus only on training embeddings. This is to adapt to the newly added small sample vectors without significantly perturbing the original model. In addition, we use supervised prompt data for fine-tuning, which helps to select the most appropriate prompts. In the second step, training continues on the remaining majority of samples while updating the LoRA parameters. The LoRA weights are then merged back into the initialized model, resulting in a fine-tuned model with updated model parameters.

[0060] Efficient prompt selection: During the fine-tuning phase, prompt selection plays an important role in order to effectively activate the generality and semantic capabilities of the multimodal large language model (MLLM). However, thoroughly fine-tuning any MLLM is a time-consuming project that requires sufficient computing resources and time. Therefore, to save time, we adopted small-epoch fine-tuning, using five training prompts designed under the guidance of prompt engineering experts. Through this small-epoch fine-tuning, we found the best prompt and its index, and then determined the optimal "model-parameter" pair based on its index to continue the complete model fine-tuning mentioned above.

[0061] For the selection of the prompt, the goal is to generate complete prompts, which are then used by the MLLM to produce effective causal inferences, directly in the training or evaluation phase.

[0062] The structure of the prompt is divided into three components: system description, instructions, and input. The system description is usually called at the beginning to define the task and constrain the behavior of the instructions, while the instructions are used to provide commands for risk content identification and summarization. The input contains the content and images to be detected to support the model's processing. The output of the model serves as the basis for our dynamic prompt and iterative optimization framework to improve the accuracy and practicality of risk identification and summary generation.

[0063] Previous studies have used fixed-form prompts for simple tasks that can be easily generalized. However, these prompts lack the prior knowledge required for complex tasks and domain-specific datasets, resulting in low performance. Therefore, we propose a hypothesis to build dynamic prompts from relevant domain-specific corpora to enhance the model's understanding and cognitive capabilities. Specifically, in addition to a corpus containing pre-processed synthetic risk content data, prompt engineering experts also developed five prompt templates for the MLLM to provide instructions specifically for the MLLM in this study. The professional advice helps to provide refined guidance to the MLLM to stimulate its ability to adapt to the risk content identification and summarization tasks and generate meaningful results. This application combines the instructions and system description designed by synthetic experts with the input data. Using this approach, five different prompt data sets were generated, fully prepared for the described abstract feature extraction stage and the subsequent evaluation stage.

[0064] like Figure 5 As shown, during the use of this application, the sample is input into the risk identification model RiskContent-Valuer, and the model outputs "risk category", "content description" and "risk summary"

[0065] In this application, the engineering indicators (1-2) used to evaluate MLLM are as follows:

[0066] (1) ROUGE metric: The performance of the model is evaluated by the language engineering metric ROUGE, and the F1 scores of ROUGE-N and ROUGE-L (denoted as R-1, R-2, and RL, respectively) are reported, which measure the word-level N-gram overlap and the longest common sequence between the reference summary and the candidate summary, respectively.

[0067] (2) Accuracy index: It represents the ratio of the number of correct judgments to the total number of judgments. It measures the training effect and performance of the model by measuring the ratio of the number of correct judgments of risk content to the total number of judgments of the trained multimodal large language model.

[0068] In the risk content identification and summarization stage, the MLLM pool for the final evaluation is first constructed, which contains a total of 4 MLLMs, including the RiskContent-Valuer of this application. For each MLLM, the weights are initialized accordingly for specific categories during risk content identification and summarization (i.e., RiskContent-Valuer is initialized with the weights after fine-tuning and updating, while other MLLMs are initialized with official weights) and are ready for causal inference. In all reasoning configurations, MLLM reasoning parameters such as "top-k", "top-p", and "temperature" are clearly defined. Use these MLLMs to generate causal inference and obtain results.

[0069] During the risk summary generation process, great attention is paid to the quality of the generated results. However, due to the high dependence of MLLM on hardware characteristics and the randomness in its output, the results sometimes encounter problems, including empty outputs and repeated words, leading to an inaccurate representation of the performance of MLLM in generating causal inference (especially the quality of the summary, without meaningless values). To address these issues, two modules are added to check whether empty outputs are generated and repeated words are obtained during the generation process. These two modules ensure that the quality of the final result is reliable to avoid interference caused by meaningless values.

[0070] In addition, in order to make the risk summary generated results more human-like in natural language, this application uses Chinese word segmentation to segment the results to improve the accuracy of ROUGE evaluation in the Chinese context. This word segmentation provides valuable predictive experimental results for future bilingual or multilingual tasks.

[0071] In this application, RiskContent-Valuer was evaluated using ROUGE and Accuracy indicators on a public data set. The results were better than most of the models used as controls and achieved the expected indicators of network security governance. Compared with existing large models, this application can greatly improve the efficiency of network risk information screening, save network security resource expenditures, and is of great significance to social security.

[0072] In summary, this application, by introducing context-aware filtering technology based on large models and applying dynamic content summarization and writing technology driven by large models, has stronger context understanding capabilities compared to the existing technology, can more effectively identify and summarize risk information, and can automatically generate high-quality summaries based on the characteristics of the input content, adapt to the needs of different fields, and improve the performance stability of the model in complex scenarios.

[0073] Those skilled in the art will appreciate that the above embodiments are specific embodiments for implementing the present application, and in actual applications, various changes may be made thereto in form and detail without departing from the spirit and scope of the present application.

Claims

1. A multimodal large model for risk content identification and summary generation method, characterized in that: include: S1. Collect risk content data and perform data preprocessing, and divide the risk content data after preprocessing into training set and test set; S2. Build the basic model Chameleon-7B, use the training set to fine-tune the basic model Chameleon-7B, and obtain the risk identification model RiskContent-Valuer; S3. Input the data in the test set into the risk identification model RiskContent-Valuer to extract abstract features and obtain the risk content identification and summary generation results; Before fine-tuning the model, the BitsAndBytesConfig interface is used to quantize the basic model Chameleon-7B. The model weights are converted into int4 through the quantization layer and put into the GPU, and the core calculation process is completed on CUDA. Supervised learning and LoRA methods are used to fine-tune the model.

2. According to claim 1, a multimodal large model for risk content identification and summary generation method is characterized in that: The risky content data is collected from heterogeneous public data from different sources, including but not limited to: online rumors, violent videos, non-violent videos, crowd violence videos, street fights, abuse videos, arrest videos, arson videos, assault videos, traffic accident videos, burglary videos, explosion videos, fight videos, robbery videos, shooting videos, theft videos, shoplifting videos, and vandalism videos.

3. The method for risk content identification and summary generation using a multimodal large model according to claim 2 is characterized in that: Data preprocessing includes data integration and data cleaning. Data integration involves unifying the text data set, extracting key frames from the video data set, and classifying them by risk category. Data cleaning involves eliminating data that is not identified as suspected bad information, is meaningless or non-directional, and lacks sensitive information, to form risk content data containing text and images.

4. The method for risk content identification and summary generation using a multimodal large model according to claim 3 is characterized in that: The data in the training set is constructed into a unified json format file, which contains the following fields: id, image, conversations, from, instruction, prompt, and value. id records the label of the training data, image records the path of the training image, and conversations is a large field including from, instruction, prompt, and value. From indicates whether the source of the conversation is human or AI. Instruct indicates system instructions, which are used to regulate the output behavior of the basic model. Prompt indicates specific human prompts. The basic model will output according to the specific human prompts. Value is the preset answer of the basic model. Supervised learning is used to fine-tune the basic model.

5. The method for risk content identification and summary generation using a multimodal large model according to claim 3 is characterized in that: In the test data set, the path, name, serial number, label, and summary information of the test images are uniformly saved in a csv file to facilitate traversal and access by the test script. The test script calculates the accuracy and text similarity of the basic model according to the labels and summaries provided in the csv file, which is used to provide prerequisites for the evaluation and optimization of the basic model.

6. The method for risk content identification and summary generation using a multimodal large model according to claim 1, characterized in that: Using the small batch fine-tuning method, the optimal prompt is found through a round of small-scale fine-tuning. The optimal prompt is used to improve the model's ability to identify risky content and the quality of summary generation. The optimal prompt is suitable for the full stage of model fine-tuning.

7. The method for risk content identification and summary generation using a multimodal large model according to claim 6 is characterized in that: The prompt consists of a system description, instructions, and input. The system description is called at the beginning to define the tasks and constrain the behavior of the instructions, ensuring that the model's identification of risk content and summary generation meet the preset goals; the instructions are used to provide commands for risk content identification and summary generation, guiding the model to identify and process risk content and generate relevant summaries; the input contains the text and images to be detected, which are used to support the model's processing process.

8. The method for risk content identification and summary generation using a multimodal large model according to claim 1, characterized in that: The evaluation indicators of the risk identification model RiskContent-Valuer are ROUGE and Accuracy.

9. A multimodal large model is used for risk content identification and summary generation system, characterized in that: include: Risk content collection module: collects risk content data and performs data preprocessing, and divides the preprocessed risk content data into training sets and test sets; Model fine-tuning module: build the basic model Chameleon-7B, use the training set to fine-tune the basic model Chameleon-7B, and obtain the risk identification model RiskContent-Valuer; before fine-tuning the model, use the BitsAndBytesConfig interface to quantize the basic model Chameleon-7B, convert the model weights into int4 through the quantization layer and put them into the GPU, and complete the core calculation process on CUDA; use supervised learning and LoRA methods to fine-tune the model; Result generation module: The data in the test set is input into the risk identification model RiskContent-Valuer to extract abstract features and obtain risk content identification and summary generation results.

10. A computer storable medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for using a multimodal large model for risk content identification and summary generation as claimed in any one of claims 1 to 8 is implemented.

Images