Hidden disaster hazard identification method, system and device, and storage medium

Abstract

The invention provides a disaster hidden danger identification method, system and device and a storage medium. The method comprises the following steps: acquiring a hidden danger data set of a real disaster; the hidden danger data set is converted into a fine-tuning instruction prompt set, the instruction prompt set comprises various different types of disaster samples and labels, and the disaster samples comprise pictures and texts; performing fine adjustment on the multi-modal large model by utilizing the instruction prompt set to obtain the multi-modal large model after fine adjustment; and inputting a disaster picture and a disaster text to be identified into the multi-modal large model after fine adjustment, and identifying whether a disaster exists or not. According to the method, multiple instruction data such as knowledge enhancement, hidden danger content and metaphor recognition are utilized to finely adjust the multi-mode large model, and the generation capacity of the large model for hidden danger judgment is improved; and finally, a prompt learning mechanism is utilized to carry out self-predictive iterative expansion training so that the model further adapts to a specific task, and thus the accuracy and efficiency of disaster hidden danger recognition are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a disaster hazard identification method, system, device and storage medium. Background Art

[0002] Natural disasters seriously affect human production and life, and cause huge losses to human society. Under the condition that it is impossible to control the occurrence of natural disasters, or even accurately predict and warn of natural disasters, the assessment of natural disaster risks is of great significance.

[0003] The assessment of key hidden dangers of natural disasters can identify high-risk areas and provide important basis for disaster prevention and mitigation, thereby reducing disaster losses. Currently, there is no method to accurately assess the risks of key hidden dangers of natural disasters. Summary of the invention

[0004] The present invention provides a disaster hazard identification method, system, device and storage medium, the main purpose of which is to provide an automated disaster hazard identification method to effectively improve the accuracy and efficiency of disaster identification.

[0005] In a first aspect, an embodiment of the present invention provides a method for identifying disaster hazards, including:

[0006] Acquire a hidden danger data set of a real disaster, wherein the hidden danger data set includes each modal information of the disaster;

[0007] Converting the hidden danger data set into a fine-tuned instruction prompt set, wherein the instruction prompt set includes various types of disaster samples and labels, and the disaster samples include pictures and texts;

[0008] Using the instruction prompt set, fine-tuning the multimodal large model to obtain a fine-tuned multimodal large model;

[0009] The disaster images and texts to be identified are input into the fine-tuned multimodal large model to identify whether a disaster exists.

[0010] Furthermore, the multimodal large model is pre-trained using the following steps:

[0011] Preprocessing sample data of different modalities, wherein the sample data includes image data and text data;

[0012] For the image data, extracting image features through a convolutional neural network model;

[0013] For the text data, extracting text features through a recurrent neural network model;

[0014] The image features and the text features are fused, and the fused features are used to train the multimodal large model.

[0015] Furthermore, before the preprocessing of the sample data of different modalities, the method further includes:

[0016] Data enhancement operations are performed on existing public data sets to obtain sample data of different modalities, wherein the sample data includes hidden danger meme attributes and metaphor-related attributes. The hidden danger meme attributes include classification labels, hidden danger objects and hidden danger contents, and the metaphor-related attributes include metaphor classifications and metaphor labels.

[0017] Furthermore, the training of the multimodal large model using the fused features includes:

[0018] The multimodal large model is updated and optimized by using the fused features through the back propagation algorithm and the gradient descent algorithm.

[0019] Furthermore, the loss function of the multimodal large model during fine-tuning is calculated as follows:

[0020] L = (1-λ) × L CE +λ×L MLM ;

[0021] Wherein, L represents the loss function, λ represents a constant, and L CE represents the cross entropy loss of the disaster potential identification task, L MM Represents the modeling loss of the multimodal large model.

[0022] Furthermore, the multimodal large model is a multimodal large language model.

[0023] Furthermore, the disasters include waterlogging monitoring, explosion-related monitoring, and icy and snowy road monitoring.

[0024] In a second aspect, an embodiment of the present invention provides a disaster hazard identification system, including:

[0025] A data acquisition module, used to obtain a hidden danger data set of a real disaster, wherein the hidden danger data set includes each modal information of the disaster;

[0026] A conversion module, used for converting the hidden danger data set into a fine-tuned instruction prompt set, wherein the instruction prompt set includes various types of disaster samples and labels, and the disaster samples include pictures and texts;

[0027] A fine-tuning module, used to fine-tune the multimodal large model using the instruction prompt set to obtain a fine-tuned multimodal large model;

[0028] The recognition module is used to input the disaster images and disaster texts to be identified into the fine-tuned multimodal large model to identify whether a disaster exists.

[0029] In a third aspect, an embodiment of the present invention provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned disaster hazard identification method when executing the computer program.

[0030] In a fourth aspect, an embodiment of the present invention provides a computer storage medium, wherein the computer storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned disaster hazard identification method are implemented.

[0031] The present invention proposes a disaster hazard identification method, system, device and storage medium. First, the general knowledge of the multimodal large model is used to expand the training samples. Then, unlabeled data is used for comparative embedding training to optimize the semantic representation ability of the text encoder, and a hazard judgment generation algorithm based on the multimodal large model is proposed. Through knowledge enhancement, hazard content and metaphor recognition and other instruction data, the multimodal large model is fine-tuned to improve the large model's ability to generate hazard judgments. Finally, a prompt learning mechanism is used to perform self-predictive iterative expansion training to further adapt the model to specific tasks, thereby improving the accuracy and efficiency of disaster hazard identification. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A flowchart of a method for identifying disaster hazards provided by an embodiment of the present invention;

[0033] Figure 2 A schematic diagram of a hidden danger identification and detection task ablation experiment results provided by an embodiment of the present invention;

[0034] Figure 3 A schematic diagram of the average accuracy of each generation of multimodal large models provided by an embodiment of the present invention;

[0035] Figure 4 A schematic diagram of experimental results of different iterative sample selection amounts provided by an embodiment of the present invention;

[0036] Figure 5 The performance comparison experimental results of different training rounds provided by the embodiment of the present invention;

[0037] Figure 6 A schematic diagram of the structure of a disaster hazard identification system provided in an embodiment of the present invention.

[0038] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0039] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as limiting the present application.

[0040] In order to enable those skilled in the art to better understand the solutions of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application.

[0041] In the embodiments of the present application, at least one refers to one or more; multiple refers to two or more. In the description of the present application, words such as "first", "second", "third", etc. are only used to distinguish the purpose of description, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying order. In addition, the terms "first" and "second" are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0042] References to "one embodiment" or "some embodiments" etc. described in this specification mean that one or more embodiments of the present application include a particular feature, structure or characteristic described in conjunction with the embodiment. Thus, in this specification, the terms "include", "comprises", "has" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.

[0043] It should be noted that in the embodiments of the present application, "connection" can be understood as electrical connection, and the connection between two electrical components can be a direct or indirect connection between the two electrical components. For example, the connection between A and B can be either a direct connection between A and B or an indirect connection between A and B through one or more other electrical components.

[0044] In order to generate comprehensive and detailed hidden danger meme explanations, the embodiment of the present invention constructs an instruction fine-tuning dataset for meme explanation generation, and proposes a hidden danger meme explanation generation algorithm based on a multimodal large language model. First, based on the existing 6 public datasets, using the rich common sense knowledge and powerful semantic understanding ability of Llama 3.2, the meme data generates corresponding explanations. The 6 public datasets include 3 metaphor meme datasets and 3 hidden danger meme datasets.

[0045] Secondly, based on the enhanced data, a multimodal instruction fine-tuning dataset is constructed. The instruction includes both the attributes of the hidden danger meme, such as classification labels, hidden danger objects, hidden danger content, etc., and metaphor-related attributes, such as metaphor classification, metaphor labels and related social background knowledge, etc. A hidden danger judgment generation algorithm based on a multimodal large model is proposed.

[0046] Finally, the constructed instruction fine-tuning dataset is used to fine-tune the multimodal large language model. Through these three steps, the method in this paper can deeply learn the hidden danger identification skills in memes, while accurately matching the hidden danger features in vision and text, and effectively generate a comprehensive and accurate explanation of the hidden danger memes.

[0047] Figure 1 A flowchart of a method for identifying disaster hazards provided by an embodiment of the present invention is shown in FIG. Figure 1 As shown, the method includes:

[0048] S110, obtaining a hidden danger data set of a real disaster, wherein the hidden danger data set includes each modal information of the disaster;

[0049] Supervised fine-tuning means using labeled data to update a pre-trained multimodal large model to complete a specific task. In order to achieve the goal of fine-tuning, it is necessary to first obtain a hidden danger dataset of real disasters. The hidden danger dataset can be obtained from many open source datasets disclosed on the Internet. The real disaster can be ice surface recognition, dust detection, etc., which can be determined according to actual conditions. The embodiments of the present invention do not make specific limitations on this.

[0050] S120, converting the hidden danger data set into a fine-tuned instruction prompt set, wherein the instruction prompt set includes various types of disaster samples and labels, and the disaster samples include pictures and texts;

[0051] There are many open source datasets that can provide insights into user behavior and preferences, even if they are not directly formatted as prescriptive data. For example, we can take a real-world hidden danger dataset and transform it into a fine-tuned prescriptive prompt dataset. The prompt template library contains many templates for different tasks and different datasets.

[0052] S130, fine-tuning the multimodal large model using the instruction prompt set to obtain a fine-tuned multimodal large model;

[0053] Fine-tuning is performed to split the hidden data set into training, validation, and testing parts. During fine-tuning, prompts are selected from the training data set and passed to the multimodal large model, which then generates the finished text.

[0054] Specifically, when the multimodal large model is exposed to a new labeled dataset for the target task, which is the instruction prompt set, it calculates the error, or difference, between its predictions and the actual labels. The multimodal large model then uses this error to adjust its weights, usually through an optimization algorithm such as gradient descent. The magnitude and direction of the weight adjustment depends on the gradient, which indicates how much each weight contributes to the error. Weights that contribute more to the error are adjusted more, while weights that contribute less are adjusted less.

[0055] Over multiple iterations (or cycles) of the dataset, the multimodal model continues to adjust its weights, gradually finding a configuration that minimizes the error on a specific task. The goal is to adapt previously learned general knowledge to the nuances and specific patterns in the set of instructional cues, making the multimodal model more specialized and effective on the target task.

[0056] Model updating, in which the large multimodal model is updated based on the labeled data. It changes based on the difference between its guess and the actual answer. This helps the large multimodal model learn the details in the labeled data. By doing so, the performance of the large multimodal model on the fine-tuned task improves.

[0057] S140, inputting the disaster image and disaster text to be identified into the fine-tuned multimodal large model to identify whether a disaster exists.

[0058] Then, the disaster image and disaster text to be identified are input into the fine-tuned multimodal large model to identify whether there are hidden dangers in the image, and thus whether a natural disaster will occur at the scene represented by the image.

[0059] A disaster hazard identification method proposed in the present invention first uses the general knowledge of a multimodal large model to expand the training samples, then uses unlabeled data for comparative embedding training to optimize the semantic representation ability of the text encoder, and proposes a hazard judgment generation algorithm based on the multimodal large model; through knowledge enhancement, hazard content and metaphor recognition and other instruction data, the multimodal large model is fine-tuned to improve the large model's ability to generate hazard judgments; finally, a prompt learning mechanism is used to perform self-predictive iterative expansion training to further adapt the model to specific tasks, thereby improving the accuracy and efficiency of disaster hazard identification.

[0060] In some embodiments, the multimodal large model is pre-trained using the following steps:

[0061] Preprocessing sample data of different modalities, wherein the sample data includes image data and text data;

[0062] For the image data, extracting image features through a convolutional neural network model;

[0063] For the text data, extracting text features through a recurrent neural network model;

[0064] The image features and the text features are fused, and the fused features are used to train the multimodal large model.

[0065] This large multimodal model is generally pre-trained before fine-tuning, so that it can quickly reach convergence during fine-tuning, thus saving training time.

[0066] For pre-training, the multimodal large model framework can be divided into three main modules: a multimodal encoder that receives and efficiently encodes, a projector that aligns data between multimodalities, and a large language model that receives the alignment signal and performs inference.

[0067] (1) Data preprocessing and alignment

[0068] Preprocess data of different modalities, including format conversion, standardization, etc. Align data using timestamps or other synchronization information to ensure consistency of data of different modalities in time and space.

[0069] (2) Feature extraction and fusion

[0070] For image data, the Convolutional Neural Networks (CNN) model can be used to extract image features; for text data, the Recurrent Neural Network (RNN) model can be used to extract text features. The extracted features are fused through specific fusion strategies, such as weighted averaging and concatenation, to form a unified feature representation.

[0071] (3) Relationship Modeling and Representation Learning

[0072] Based on the fusion features, deep learning models (such as multi-layer perceptron, graph neural network, etc.) are used for relationship modeling and representation learning. Through the learned relationship model, the intrinsic connections and rules between different modal data can be mined.

[0073] (4) Joint training and optimization

[0074] Design appropriate loss functions and objective functions to achieve joint training of multimodal data. Update and optimize model parameters through back-propagation algorithms and optimizers (such as gradient descent) to obtain a more robust and generalized model.

[0075] The lightweight visual encoder Vision Transformer architecture faces challenges in practical applications due to hardware and environmental limitations. ViTamin represents a lightweight visual model that is tailored for vision and language models. As shown in the figure below, visual encoding is completed through two layers of MBC plus a layer of attention blocks, and then compared with text for learning.

[0076] In order to avoid the forgetting problem of the pre-trained model during cued fine-tuning, an auxiliary loss is introduced into cued fine-tuning. This loss function considers both the cross entropy loss of classification and the pre-trained language modeling loss, so that the model can adapt to specific tasks while retaining its original knowledge.

[0077] The auxiliary loss is calculated as follows:

[0078] L = (1-λ) × L CE +λ×L MLM ;

[0079] Wherein, L represents the loss function, λ represents a constant, and L CE represents the cross entropy loss of the disaster potential identification task, L MM Represents the modeling loss of the multimodal large model.

[0080] Wherein, the multimodal large model is a multimodal large language model.

[0081] It should be noted that multimodal large language models (MLLMs) have demonstrated excellent performance on tasks such as visual question answering, visual understanding, and reasoning.

[0082] The following embodiment of the present invention conducts experimental verification on the disaster hidden danger identification method.

[0083] 4.1 Dataset

[0084] 200 real hidden danger images were used for experimental verification. These images have been processed into annotation files in YOLO and VOC formats and can be directly used to train deep learning models. The types of hidden dangers mainly include waterlogging monitoring, dust explosion (dust accumulation, personnel, fireworks) identification, ice and snow road monitoring, and law enforcement key inspection item identification. The screened English picture and text pairs were pre-trained, with the same weights for Chinese and English. This training method better aligns visual information to the semantic space of the large language model; in the subsequent fine-tuning stage, the model is trained on long visual question-answering data to generate answers that meet human preferences.

[0085] 4.2 Evaluation Metrics

[0086] Accuracy is mainly used to evaluate model performance. At the same time, F1-Score is also introduced to evaluate the labeling ability of multimodal large models for data with imbalanced positive and negative examples. For hidden danger analysis, the combination of model prediction and actual sentiment is as follows: True Positive (TP), which means that the model predicts positive and the actual is positive; False Negative (FN), which means that the model predicts negative but the actual is positive; True Negative (TN), which means that the model predicts negative and the actual is also negative; False Positive (FP), which means that the model predicts positive but the actual is negative.

[0087] Based on this, we have:

[0088] Accuracy calculation formula:

[0089]

[0090] F1 score calculation formula:

[0091]

[0092] 4.3 Baseline Model

[0093] The following baseline methods are selected to compare with the algorithm in this paper under the same resource conditions:

[0094] (a) YOLO: It is mainly used in the field of target detection (a subfield of machine vision). It can identify and locate objects in an image with a single view. It has the advantages of fast speed, high accuracy, strong interpretability and wide applicability. It is one of the most important representatives in the current field of target detection.

[0095] (b) Vit: is a deep learning model based on the Transformer architecture for processing image data. ViT divides the image into multiple small patches (patches), each patch is converted into a feature vector through a linear layer, and then input into a standard Transformer encoder. Unlike traditional convolutional neural networks (CNNs), ViT does not rely on local features of the image, but learns patterns in the image through a global attention mechanism.

[0096] (c) SigLIP: is an advanced multimodal model. In simple terms, SiGILP can understand and generate multimodal data, such as images and text, at the same time. It can not only understand a picture or a piece of text, but also generate new pictures or text based on this information.

[0097] (d) CLIP: It is a large multimodal model that is trained using contrastive learning. The core idea is to learn the similarities or differences between different samples in the same set of data by comparing them. The CLIP model maps images and texts to the same representation space and trains by comparing the similarities and differences between different image and text pairs, thereby learning feature representations with good generalization capabilities.

[0098] For the above baseline methods, the corresponding available training resources are as follows:

[0099] (a), 1-shot: A manually labeled training sample is provided for each category.

[0100] (b), 60-shot: A total of 60 manually labeled training samples are provided.

[0101] (c), 200-shot: A total of 200 manually labeled training samples are provided.

[0102] 4.4 Experimental Setup

[0103] For the algorithm model in this paper, the temperature hyperparameter and dropout rate of contrastive embedding training are set to 0.05 and 0.1 respectively. The auxiliary loss weight a is set to 1e-4. The maximum participation ratio of the model is set to 0.6, the generational sample increment multiplier d is set to 8, and the iterative selected sample size is set to 30. For Llama 3.2, the temperature hyperparameter is set to 0.4.

[0104] The experimental platform is based on Windows 11 system and is implemented using Python 3.9.12's deep learning library and transformers 4.23.0; the computer hardware configuration is AMD Ryzen 75800H CPU, NVIDIA GeForce RTX 3070 Laptop GPU, 1T solid-state drive, and 16GB of memory.

[0105] 4.5 Experimental Results Analysis

[0106] Based on the real hidden danger data set, hidden danger detection is carried out. The experimental evaluation index uses accuracy. Table 1 shows the hidden danger detection results. The experimental results are shown in Table 1, where the best results are marked in bold.

[0107] Table 1

[0108]

[0109] According to the observation of the experimental results, in the hidden danger identification and detection experiment, the accuracy of the algorithm model in this paper is improved by about 21 percentage points under the 1-shot training scale compared with the optimal baseline method SigLIP; under the 60-shot and 200-shot training scales, the accuracy is improved by about 15 and 12 percentage points respectively.

[0110] The algorithm in this paper ensures the performance of the model under low resource conditions by making full use of endogenous and exogenous information. In order to verify the effectiveness of each module of the model, the following ablation experiment is set for comparative analysis:

[0111] (1) The algorithm in this paper has no large model knowledge, that is, the method uses a very small amount of annotated data for a specific task for training.

[0112] (2) The algorithm in this paper is not iteratively trained with augmented samples, that is, this method does not use redundant unlabeled data for training.

[0113] (3) The algorithm in this paper does not have contrast embedding training, that is, this method does not perform targeted encoding optimization of the pre-trained model for specific tasks.

[0114] Conduct ablation experiments on hidden danger identification and detection tasks. Figure 2 A schematic diagram of a hidden danger identification and detection task ablation experiment results provided by an embodiment of the present invention. The experimental results are shown in Figure 2 The model without ablation achieved the best results in various low-resource situations, verifying the necessity of the three modules of large model knowledge, iterative training and comparative embedding training for hidden danger analysis under low-resource conditions.

[0115] It should be noted that Figure 2In the figure, under various data sets such as 1-shot, 60-shot and 200-shot, the columns represent YOLO, VIT, SigLIP, CLIP and the method proposed in the embodiment of the present invention from left to right, the horizontal axis Data Set represents the data set, and Accuracy represents the accuracy.

[0116] The analysis of the ablation experiment results is as follows:

[0117] (1) The performance of our algorithm is the lowest in the 1-shot training scale, which indicates that in hidden danger analysis tasks, relying solely on small-scale labeled samples is not enough to obtain sufficient model performance.

[0118] (2) The proposed algorithm performs slightly worse than the baseline model in all training scales, which shows that iterative training with unlabeled data is feasible for improving model performance. Iterative training can make full use of the knowledge in the pre-trained model and provide the model with richer training data.

[0119] (3) The performance of the proposed algorithm is slightly worse than that of the original model in the 1-shot training scale, but it shows similar performance to the original model in the 60-shot and 200-shot training scales. This shows that in small-scale training samples, contrastive embedding training can effectively enhance data and improve the generalization ability of the model. However, as the number of training samples increases, the diversity of the data itself increases, so the effect of contrastive embedding training may be relatively weakened.

[0120] In the training phase, the algorithm in this paper generates the training samples required for the next generation based on the model integration method. Therefore, this unsupervised process may have error accumulation and transmission. In order to verify the significance and effectiveness of iterative training, the performance of each generation model of the algorithm in this paper is analyzed below. The experimental results are shown in Figure 3 , Figure 3 A schematic diagram of the average accuracy of each generation of multimodal large models provided in an embodiment of the present invention.

[0121] It should be noted that Figure 3 The horizontal axis Number of iterations represents the number of iterations, and Accuracy represents the accuracy.

[0122] Depend on Figure 3 It can be seen that with the increase of training generations, the overall performance of the model shows an increasing trend, which shows the effectiveness of the iterative training strategy, while the specific situation of the model between generations is related to the characteristics of the task and the initial labeled data. In the hate speech detection task, the G3 generation model showed good performance at 1-shot, 60-shot and 200-shot training scales, and the performance was significantly improved compared with the G1 generation model.

[0123] The iterative sample selection amount w determines the number of samples selected for sampling when the model generates a new training set for the next generation of model training. The sample selection amount also represents the minimum confidence ranking required for the sample selection to be included in the next round of training. The following study is about the relationship between the sample selection amount and model performance. The experimental results are shown in Figure 4 , Figure 4 A schematic diagram of experimental results of different iterative sample selection amounts provided by an embodiment of the present invention, wherein: Figure 4 The middle line 1 represents 1-shot, line 2 represents 60-shot, and line 3 represents 200-shot. The horizontal axis Sample size represents the sampling scale, and Accuracy represents the accuracy.

[0124] The model performance can be guaranteed by adjusting the training rounds of each generation of models. The following is an analysis of the training rounds of each generation of models. The experimental results are shown in Figure 5 , Figure 5 The performance comparison experimental results of different training rounds provided by the embodiment of the present invention are as follows: Figure 5 The middle line 1 represents 1-shot, line 2 represents 60-shot, and line 3 represents 200-shot. The horizontal axis Training round represents the number of training rounds, and Accuracy represents accuracy.

[0125] From the experimental results, the performance of the algorithm model in this paper varies under different training rounds. When the training samples are small, 1-shot, excessive training rounds can easily lead to overfitting of the model. The training rounds should be adapted to the sample size. As the number of training samples increases, the model can achieve a better fit under a certain number of training rounds, but as the number of training rounds continues to increase, slight overfitting may occur. Therefore, in practical applications, it is necessary to select the appropriate training rounds according to the number of training samples to obtain better model performance.

[0126] During the training phase, the algorithm in this paper uses two different prompt templates: direct prompt and context-introducing prompt. Table 2 is a performance comparison table of different prompt methods. Table 2 shows the performance of the two prompt methods in the algorithm in this paper, and the best results are marked in bold:

[0127] Table 2

[0128]

[0129] Through observation, it can be found that the effects of direct prompts and contextual prompts in the hidden danger identification and detection task are: direct prompts perform better than prompts with context, which shows that for tasks such as hidden danger identification and detection, direct prompts can more effectively guide the model to make judgments, because the characteristics of hidden dangers may make them easier to identify through direct prompts.

[0130] Figure 6 A schematic diagram of a disaster hazard identification system provided by an embodiment of the present invention is shown in FIG. Figure 6 As shown, the system includes:

[0131] The data acquisition module 610 is used to obtain a hidden danger data set of a real disaster, wherein the hidden danger data set includes each modal information of the disaster;

[0132] A conversion module 620, configured to convert the hidden danger data set into a fine-tuned instruction prompt set, wherein the instruction prompt set includes various types of disaster samples and labels, and the disaster samples include pictures and texts;

[0133] A fine-tuning module 630, configured to use the instruction prompt set to fine-tune the multimodal large model to obtain a fine-tuned multimodal large model;

[0134] The identification module 640 is used to input the disaster picture and disaster text to be identified into the fine-tuned multimodal large model to identify whether a disaster exists.

[0135] This embodiment is a system embodiment corresponding to the above method, and its specific implementation process is the same as that of the above method embodiment. For details, please refer to the above method embodiment, and this system embodiment does not make specific limitations on this.

[0136] Each module in the above-mentioned disaster hazard identification system can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each of the above modules.

[0137] In one embodiment, a computer device is provided, which may be a server, and includes a processor, a memory, a network interface, and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a computer storage medium and an internal memory. The computer storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the computer storage medium. The database of the computer device is used to store data generated or obtained during the execution of the disaster hazard identification method, such as a hazard data set and various modal information. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a disaster hazard identification method is implemented.

[0138] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of a method for identifying a potential disaster hazard in the above embodiment are implemented. Alternatively, when the processor executes the computer program, the functions of each module / unit in the embodiment of a system for identifying a potential disaster hazard are implemented.

[0139] In one embodiment, a computer storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of a method for identifying a potential disaster hazard in the above embodiment are implemented. Alternatively, when the computer program is executed by a processor, the functions of each module / unit in the embodiment of the system for identifying a potential disaster hazard are implemented.

[0140] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0141] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0142] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A method for identifying disaster hazards, characterized in that: include: Acquire a hidden danger data set of a real disaster, wherein the hidden danger data set includes each modal information of the disaster; Converting the hidden danger data set into a fine-tuned instruction prompt set, wherein the instruction prompt set includes various types of disaster samples and labels, and the disaster samples include pictures and texts; Using the instruction prompt set, fine-tuning the multimodal large model to obtain a fine-tuned multimodal large model; The disaster images and texts to be identified are input into the fine-tuned multimodal large model to identify whether a disaster exists.

2. The method for identifying disaster hazards according to claim 1, characterized in that: The multimodal large model is pre-trained using the following steps: Preprocessing sample data of different modalities, wherein the sample data includes image data and text data; For the image data, extracting image features through a convolutional neural network model; For the text data, extracting text features through a recurrent neural network model; The image features and the text features are fused, and the fused features are used to train the multimodal large model.

3. The method for identifying disaster hazards according to claim 2, characterized in that: Before the preprocessing of the sample data of different modalities, the method further includes: Data enhancement operations are performed on existing public data sets to obtain sample data of different modalities, wherein the sample data includes hidden danger meme attributes and metaphor-related attributes. The hidden danger meme attributes include classification labels, hidden danger objects and hidden danger contents, and the metaphor-related attributes include metaphor classifications and metaphor labels.

4. The method for identifying disaster hazards according to claim 2, characterized in that: The step of training the multimodal large model by using the fused features includes: The multimodal large model is updated and optimized by using the fused features through the back propagation algorithm and the gradient descent algorithm.

5. The method for identifying disaster hazards according to claim 1, characterized in that: The calculation method of the loss function of the multimodal large model during fine-tuning is as follows: L=(1-λ)×L CE +λ×L MLM ; Wherein, L represents the loss function, λ represents a constant, and L CE represents the cross entropy loss of the disaster potential identification task, L MM Represents the modeling loss of the multimodal large model.

6. The method for identifying disaster hazards according to any one of claims 1 to 5, characterized in that: The multimodal large model is a multimodal large language model.

7. The method for identifying disaster hazards according to any one of claims 1 to 5, characterized in that: The disasters mentioned include waterlogging monitoring, explosion-related monitoring, and icy and snowy road monitoring.

8. A disaster hazard identification system, characterized in that: include: A data acquisition module, used to obtain a hidden danger data set of a real disaster, wherein the hidden danger data set includes each modal information of the disaster; A conversion module, used for converting the hidden danger data set into a fine-tuned instruction prompt set, wherein the instruction prompt set includes various types of disaster samples and labels, and the disaster samples include pictures and texts; A fine-tuning module, used to fine-tune the multimodal large model using the instruction prompt set to obtain a fine-tuned multimodal large model; The recognition module is used to input the disaster images and disaster texts to be identified into the fine-tuned multimodal large model to identify whether a disaster exists.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the disaster hazard identification method according to any one of claims 1 to 7 are implemented.

10. A computer storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the disaster hazard identification method according to any one of claims 1 to 7 are implemented.

Images