Hidden danger identification method in fire inspection based on multi-modal large model

Abstract

The invention relates to the field of hidden danger identification, in particular to a hidden danger identification method in fire inspection based on a multi-modal large model, which comprises the following steps of: 1, establishing the multi-modal large model; 2, model training; 3, fine adjustment of the model; 4, deploying and using; 5, displaying a result; according to the method, the multi-modal large model is constructed, and the image data and the corresponding text description collected in the inspection process are effectively utilized, so that the fire hazard safety problem is accurately identified, the hidden danger identification capability is enhanced, and the fire hazard identification accuracy is remarkably improved. According to the fusion analysis mode, the limitation of single-mode data is made up, and misjudgment and missed judgment caused by insufficient information are reduced.

Description

Technical Field

[0001] The present invention relates to the field of hidden danger identification, and in particular to a hidden danger identification method in fire inspection based on a multi-modal large model. Background Art

[0002] In the current field of fire safety management, traditional inspection methods mainly rely on manual experience and judgment, and inspectors discover safety hazards through direct observation of the scene. With the development of technology, some inspections have begun to use image recognition technology to assist in judgment, but these systems are often based on single-modal data (such as only using image information or only using text descriptions) for analysis, resulting in low accuracy of analysis. Therefore, it is necessary to propose a method for identifying hidden dangers in fire inspections based on a multimodal large model. Summary of the invention

[0003] The purpose of the present invention is to provide a method for identifying hidden dangers in fire inspections based on a multimodal large model, which can more accurately identify the content of safety hazards and the location of safety hazards, reduce false alarms or missed alarms, and solve existing technical defects and unmet technical requirements.

[0004] To achieve the above object, the present invention provides the following technical solution: a method for identifying hidden dangers in fire inspection based on a multimodal large model, comprising the following steps:

[0005] 1. Establish a multimodal large model: Based on the Transformer architecture and combined with VQ-VAE technology, build a multimodal large model that can process bimodal inputs of images and texts and output heat maps and hidden danger descriptions. The main network of the model is a unidirectional Transformer model. It should be explained that the VQ-VAE technology in this application is not part of the model.

[0006] 2. Model Training

[0007] 2.1) Collect sample data: collect a large amount of image data and corresponding text descriptions in fire inspection scenarios;

[0008] In this application, these data should cover various possible types of hazards, such as fire hazards, electrical hazards, blocked emergency exits, etc.

[0009] 2.2) Sample data preprocessing

[0010] 2.2.1) Image preprocessing: After scaling and normalizing the collected images, they are input into VQ-VAE to generate image token sequences;

[0011] 2.2.2) Text preprocessing: segment and encode the collected text descriptions and convert them into text token sequences;

[0012] 2.2.3) Input sequence generation: concatenate the image token sequence and the text token sequence to form a unified input sequence, and input this input sequence into the multimodal large model;

[0013] 2.2.4) Manual heat map weight value generation: Manually mark the hidden danger areas in the image and generate the corresponding heat map weight values;

[0014] 2.2.5) Model prediction heat map weight value generation;

[0015] 3. Model fine-tuning: Use the Low-Rank Adaptation method to fine-tune the model to obtain a fine-tuned multimodal large model;

[0016] 4. Deployment and use: During the actual inspection process, input the image data of the real scene and the text description of the corresponding scene into the fine-tuned multimodal large model to generate the actual heat map weight value and hidden danger description;

[0017] 5. Result display: The heat map weight values ​​are denormalized and mapped to 8-bit grayscale color values ​​to generate a visual heat map, which is then superimposed on the original image to intuitively display potential hidden danger areas. At the same time, the hidden danger description is output in text form.

[0018] It should be noted in this application that in step 5, the denormalization operation is: the decimal size range that can be represented by 8-bit binary is [0,255], and the size range of the heat map weight value is [0,1], so you only need to multiply the heat map weight value by 255 to get the grayscale color value.

[0019] In addition, in step 5: Mapping: Mapping using a fully connected layer is a common operation in neural networks, especially when processing feature vectors. Fully connected layers are often used to map the output of the previous layer to a new feature space. A fully connected layer usually consists of a weight matrix and a bias vector, and the dimension of the weight matrix determines the dimension of the input and output feature vectors. For example, if the dimension of the input feature vector is n, and we want to map it to an output feature vector of dimension m, then the dimension of the weight matrix should be m×n. During calculation, the fully connected layer multiplies the input feature vector by the weight matrix and adds the bias vector.

[0020] Overlay: Image overlay is an image processing technique that aims to combine two or more images into a new image. The overlay operation in this article only refers to the simple addition operation of the original image and the thermal value image at the pixel point.

[0021] Preferably, in step 1), the unidirectional Transformer model mainly includes 48 Transformer encoder layers, the hidden layer dimension is 2560, 40 attention heads, and a total of 4 billion parameters.

[0022] Preferably, in step 2.2.1), the specific contents of image preprocessing include:

[0023] In the step 2.2.1), the specific contents of image preprocessing include:

[0024] 2.2.1.1) Scaling: Limit the size of the collected images to 512×512;

[0025] In this application, it is important to emphasize that this zoomed content is artificially zoomed;

[0026] 2.2.1.2) Normalization: Linearly normalize each pixel value in the scaled image to between [0, 1];

[0027] 2.2.1.3) Token sequence generation: The latent feature z of the image is discretized into a 32×32 token sequence through the internal encoder;

[0028] It should be noted here that the latent feature z is the intermediate vector generated during the calculation process of the VQ-VAE internal encoder. Step 2.2.1.3) is to input the processed image into VQ-VAE to generate the content of the image token sequence.

[0029] 2.2.1.4) Establishment of image token sequence: Expand the 32×32 token sequence in step 2.2.1.3 in one dimension to obtain a 1024 image token sequence.

[0030] In this application, the image token sequence is obtained by discretization processing through VQ-VAE. VQ-VAE (Vector Quantized Variational Autoencoder) can be used for data dimension reduction, feature extraction, denoising and other tasks. It maps the input data to a low-dimensional hidden representation space through the structure of encoder and decoder, and then maps this representation back to the original data space to achieve data reconstruction.

[0031] Preferably, in step 2.2.2), the specific contents of text preprocessing include:

[0032] 2.2.2.1) Word segmentation: Use the SentencePiece model to segment the text;

[0033] 2.2.2.2) Encoding: Query the text dictionary according to the word segmentation content to obtain the index corresponding to each word segmentation;

[0034] 2.2.2.3) Establishment of text token sequence: Use the index set corresponding to all word segments as the text token sequence.

[0035] In this application, a sequence of text tokens is encoded by a text dictionary, which is also called a vocabulary. It is a set that maps different tokens to unique numbers. This number is usually called the index of the token. In natural language processing (NLP), tokens are the basic units of text, and text dictionaries enable computers to process and understand text. The value range of text tokens is from 8192-58192. In this model, the input text is first preprocessed and divided into tokens.

[0036] Preferably, in step 2.2.4), the content of the artificial heat map weight value generation is:

[0037] 2.2.4.1) Manually circle the potential danger areas in the image in advance;

[0038] 2.2.4.2) Create a two-dimensional matrix of all zeros with the same size as the original input image to store each pixel value of the heat map weight value;

[0039] 2.2.4.3) Establish coordinates: Use the top left corner of the image as the origin of the coordinate system, the right direction as the positive direction of the x-axis, and the downward direction as the positive direction of the y-axis to establish coordinates; in this application, it needs to be explained that the picture is the image, and the two have the same meaning.

[0040] 2.2.4.4) For the center of the hidden danger area, according to its radius and the preset Gaussian distribution standard deviation, the specific formula for calculating the Gaussian distribution value of each pixel position around the center of the circle is as follows:

[0041]

[0042] Among them, μ x and μ y is the coordinate of the center of the circle, σ is the radius of the area, x and y are the coordinates of the current pixel, and G(x,y) represents the thermal weight value at the (x,y) coordinate.

[0043] Preferably, the specific process of generating the prediction of the heat map weight value in step 2.2.5) is as follows:

[0044] The self-attention layer in the penultimate layer of the Transformer model is used as the feature vector for generating heat map weight values. Its dimension is the same as that of the hidden layer, which is 2560. It is mapped to 1024 dimensions after a one-dimensional linear transformation, and then a 64×64 weight image is generated through the VQ-VAE decoder. Finally, the heat map weight values ​​of the same size as the original image are generated by scaling.

[0045] Preferably, in step 3), the process of model fine-tuning is:

[0046] 3.1) Specify the target module: Determine the module that needs to be fine-tuned by Lora:

[0047] 3.2) Set fine-tuning parameters: Set the parameters of Lora fine-tuning according to the requirements of specific tasks;

[0048] 3.3) Training and validation: Use the fine-tuned model to train on the training set and validate on the validation set to evaluate the fine-tuning effect.

[0049] In this application, it should be noted that the training set and the validation set here are randomly extracted from the text description and image information during preprocessing to obtain the training set and the validation set.

[0050] The overall logic of fine-tuning can be understood as the model is trained on the training set, the output results are compared with the weight values ​​of the artificial heat map, the model is continuously improved, and then the validation set is used to verify whether the model is fine-tuned successfully.

[0051] Specifically: fine-tune the large model, including freezing the original weight W0 of the multimodal large model, inputting sample data into the model, and obtaining output. Perform negative log-likelihood loss on the model output and the annotated true result. Update the model weight W=W0+BA by minimizing the value of the negative log-likelihood function.

[0052] Lora fine-tuning principle:

[0053] Weight update: During the training process, the original weight W0 is updated to W = W0 + BA, where B and A are two low-rank matrices. By optimizing B and A, efficient fine-tuning of the original weight can be achieved.

[0054] Parameter merging: After training, B and A can be merged with the original weight W0 to obtain the updated weight W, thus avoiding the introduction of additional computation during the inference process.

[0055] In order to improve the performance of the multimodal large model on specific tasks, the present invention adopts the Lora (Low-Rank Adaptation) method to fine-tune the model. Lora is an efficient fine-tuning method that decomposes the weight update into the product of two low-rank matrices, thereby achieving efficient parameter updates without changing the original model structure.

[0056] Preferably, in step 4), the process of deployment and use is:

[0057] 4.1) During the actual inspection process, obtain image data of the real scene and text description of the corresponding scene;

[0058] 4.2) Process the acquired image data of the real scene and the text description of the corresponding scene according to the contents of steps 2.2.1) to 2.2.2) to obtain a new image token sequence and a new text token sequence;

[0059] 4.3) Generate a new input sequence using the new image token sequence and the new text token sequence obtained in step 4.2, and then input the new input sequence into the fine-tuned model;

[0060] The input sequence generation part here is the same as before, but the difference is that the input is into the fine-tuned model.

[0061] Specifically, in actual use, at the inspection site, the inspector takes a photo of the site where hidden dangers may exist, and fills in the scene description text in the photo;

[0062] Use the SentencePiece model to segment the scene description text, and use the text dictionary to obtain the text token sequence of the scene description (<text_1token> ...<text_n token> );

[0063] After resizing and normalizing the captured photos, they are input into the VQ-VAE encoder to obtain the image token sequence (<img_1token> ...<img_m token> );

[0064] Use text modal tags for " <text>< / text> " and image modal tag pair " ", concatenate the text token sequence and the image token sequence to get the fused token sequence" <text><text_1token>...<text_n token>< / text> <img_1token> ...<img_m token> ”.

[0065] 4.4) The fine-tuned model calculates the new input sequence to obtain new heat map weight values ​​and hidden danger descriptions.

[0066] The output calculation in step 4.4) is:

[0067] 4.4.1) Use the self-attention layer in the penultimate layer of the Transformer model as the feature vector for generating the heat map;

[0068] The self-attention layer in the penultimate layer of the Transformer model is used as the feature vector for generating heat map weight values. Its dimension is the same as that of the hidden layer, which is 2560. It is mapped to 1024 dimensions after passing through the fully connected layer, and then a 64×64 weight image is generated through the VQ-VAE decoder. Finally, the heat map weight values ​​of the same size as the original image are generated by scaling.

[0069] 4.4.2) Use the autoregressive decoder to convert the output layer of the Transformer model into a text description. The decoder gradually generates each character in the description based on the content of the feature vector of the last layer of the Transformer.

[0070] Specifically, the decoder starts with a specific start tag <start>Start by predicting the probability distribution of the next character step by step. Based on the probability distribution, select the most likely character as part of the description and add it to the current description. Repeat this process until the end tag " <end>"until.

[0071] The Transformer structure is a mainstream deep learning model, mainly used to process sequence data, such as natural language, time series, etc. It captures the dependencies between different positions in the sequence by introducing the self-attention mechanism, and greatly improves the training efficiency through parallel processing. The Transformer consists of two parts, the encoder and the decoder, and usually uses a stacked structure to build a multi-layer structure. The encoder is responsible for processing the input sequence, and the decoder is responsible for generating the output sequence. The encoder consists of a self-attention layer, a feedforward neural network layer, and contains residual connections and layer normalization operations. Among them, the self-attention layer is the core part of the encoder. It calculates the dot product of the query vector and the key vector to obtain the attention score, and then normalizes it through the softmax function to obtain the attention weight. The decoder is similar to the encoder, and also contains a self-attention layer and a feedforward neural network layer. In addition, it also contains an additional encoder-decoder attention layer to focus on the relevant parts of the input sequence when generating the output.

[0072] Preferably, in step 2.2.1.4) and step 2.2.2.2), the value range of the token is 8192-58192, and the value range of the image token is 0-8192.

[0073] Compared with the prior art, the present invention has the following beneficial effects:

[0074] 1. This application constructs a multimodal large model and effectively utilizes the image data and corresponding text descriptions collected during the inspection process to achieve accurate identification of fire hazard safety issues, enhance the ability to identify hidden dangers, and significantly improve the accuracy of fire hazard identification. This fusion analysis method makes up for the limitations of single-modal data and reduces misjudgments and missed judgments caused by insufficient information.

[0075] 2. This application uses a heat map to visually display potential hidden danger areas, allowing inspectors to quickly locate the problem, thereby greatly improving inspection efficiency and response speed. This visual feedback method not only enhances the intuitiveness of hidden danger identification, allowing inspectors to quickly locate the problem, enriching hidden danger description information, improving inspection efficiency and response speed, and helping inspectors better understand and process identification results. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 It is a schematic diagram of the overall process of the present invention;

[0077] Figure 2 This is a schematic diagram of the overall structure of Example 1 of the present invention; DETAILED DESCRIPTION

[0078] The following will be combined with the attached embodiment of the present invention Figure 1-2 , the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0079] See also Figure 1-2 , embodiments of the present invention:

[0080] Embodiment 1:

[0081] like Figure 1-2 As shown: A method for identifying hidden dangers in fire inspection based on a multimodal large model, characterized in that it includes the following steps:

[0082] 1. Build a multimodal large model: Based on the Transformer architecture and combined with VQ-VAE technology, build a multimodal large model that can process image and text bimodal inputs and output heat maps and hidden danger descriptions. The main network of the model is a unidirectional Transformer model;

[0083] In the step 1), the unidirectional Transformer model mainly includes 48 Transformer encoder layers, the size of the hidden layer dimension is 2560, 40 attention heads, and a total of 4 billion parameters.

[0084] 2. Model Training

[0085] 2.1) Collect sample data: collect a large amount of image data and corresponding text descriptions in fire inspection scenarios;

[0086] In this embodiment, these data should cover various possible hidden danger types, such as fire hazards, electrical hazards, blocked emergency exits, etc.

[0087] 2.2) Sample data preprocessing

[0088] 2.2.1) Image preprocessing: After scaling and normalizing the collected images, they are input into VQ-VAE to generate image token sequences;

[0089] The specific contents of image preprocessing include:

[0090] 2.2.1.1) Scaling: Limit the size of the collected images to 512×512;

[0091] 2.2.1.2) Normalization: Linearly normalize each pixel value in the scaled image to between [0, 1];

[0092] 2.2.1.3) Token sequence generation: The latent feature z of the image is discretized into a 32×32 token sequence through the internal encoder;

[0093] 2.2.1.4) Establishment of image token sequence: Expand the 32×32 token sequence in step 2.2.1.3 in one dimension to obtain a sequence of 1024 image tokens, where the value range of the image token is 0-8192.

[0094] 2.2.2) Text preprocessing: segment and encode the collected text descriptions and convert them into text token sequences;

[0095] In the step 2.2.2), the specific contents of the text preprocessing include:

[0096] 2.2.2.1) Word segmentation: Use the SentencePiece model to segment the text;

[0097] 2.2.2.2) Encoding: Query the text dictionary according to the word segmentation content to obtain the index corresponding to each word segmentation;

[0098] 2.2.2.3) Establishment of text token sequence: The index set corresponding to all word segments is used as the text token sequence, and the value range of the token is 8192-58192.

[0099] In this application, a sequence of text tokens is encoded by a text dictionary, which is also called a vocabulary. It is a set that maps different tokens to unique numbers. This number is usually called the index of the token. In natural language processing (NLP), tokens are the basic units of text, and text dictionaries enable computers to process and understand text. The value range of text tokens is from 8192-58192. In this model, the input text is first preprocessed and divided into tokens.

[0100] For example, the text input by the user is "The door of the safety exit uses the access control system", which is segmented into "safety", "exit", "of", "door", "use", "access control", and "system". The indexes obtained after querying the dictionary are 35872, 16584, 11548, 12596, 13595, 19763, and 24843. Therefore, the text token sequence is recorded as [35872, 16584, 1548, 2596, 13595, 19763, 24843].

[0101] 2.2.3) Input sequence generation: concatenate the image token sequence and the text token sequence to form a unified input sequence, and input this input sequence into the multimodal large model;

[0102] 2.2.4) Manual heat map weight value generation: Manually mark the hidden danger areas in the image and generate the corresponding heat map weight values;

[0103] The content of artificial heat map weight value generation is:

[0104] 2.2.4.1) Manually circle the potential danger areas in the image in advance;

[0105] 2.2.4.2) Create a two-dimensional matrix of all zeros with the same size as the original input image to store each pixel value of the heat map weight value;

[0106] 2.2.4.3) Establish coordinates: Take the top left corner of the image as the origin of the coordinate system, the right direction is the positive direction of the x-axis, and the downward direction is the positive direction of the y-axis to establish coordinates; (It should be noted that the image is the image data)

[0107] 2.2.4.4) For the center of the hidden danger area, according to its radius and the preset Gaussian distribution standard deviation, the specific formula for calculating the Gaussian distribution value of each pixel position around the center of the circle is as follows:

[0108]

[0109] Among them, μx and μy are the coordinates of the center of the circle, σ is the radius of the area, x and y are the coordinates of the current pixel, and G(x,y) represents the thermal weight value at the (x,y) coordinate.

[0110] 2.2.5) Model prediction heat map weight value generation;

[0111] The specific process of heat map weight value prediction generation is as follows:

[0112] The self-attention layer in the penultimate layer of the Transformer model is used as the feature vector for generating heat map weight values. Its dimension is the same as that of the hidden layer, which is 2560. It is mapped to 1024 dimensions after a one-dimensional linear transformation, and then a 64×64 weight image is generated through the VQ-VAE decoder. Finally, the heat map weight values ​​of the same size as the original image are generated by scaling.

[0113] 2. Model fine-tuning: Use the Low-Rank Adaptation method to fine-tune the model to obtain a fine-tuned multimodal large model;

[0114] The process of model fine-tuning is:

[0115] 3.1) Specify the target module: Determine the module that needs to be fine-tuned by Lora:

[0116] 3.2) Set fine-tuning parameters: Set the parameters of Lora fine-tuning according to the requirements of specific tasks;

[0117] 3.3) Training and Validation: Use the fine-tuned model to train on the training set and validate on the validation set to evaluate the fine-tuning effect.

[0118] Specifically: fine-tune the large model, including freezing the original weight W0 of the multimodal large model, inputting sample data into the model, and obtaining output. Perform negative log-likelihood loss on the model output and the annotated true result. Update the model weight W=W0+BA by minimizing the value of the negative log-likelihood function.

[0119] Lora fine-tuning principle:

[0120] Weight update: During the training process, the original weight W0 is updated to W = W0 + BA, where B and A are two low-rank matrices. By optimizing B and A, efficient fine-tuning of the original weight can be achieved.

[0121] Parameter merging: After training, B and A can be merged with the original weight W0 to obtain the updated weight W, thus avoiding the introduction of additional computation during the inference process.

[0122] In order to improve the performance of the multimodal large model on specific tasks, the present invention adopts the Lora (Low-Rank Adaptation) method to fine-tune the model. Lora is an efficient fine-tuning method that decomposes the weight update into the product of two low-rank matrices, thereby achieving efficient parameter updates without changing the original model structure.

[0123] 4. Deployment and use: During the actual inspection process, input the image data of the real scene and the text description of the corresponding scene into the fine-tuned multimodal large model to generate the actual heat map weight value and hidden danger description;

[0124] In step 4), the process of deployment and use is:

[0125] 4.1) During the inspection, the inspection personnel found suspected hidden dangers near the door of the on-site emergency exit. They took photos of the scene and recorded the scene description: the door of the emergency exit uses an access control system.

[0126] 4.2) Use the SentencePiece model to segment the scene description and obtain the segmentation results "safety", "exit", "of", "door", "use", "access control", and "system";

[0127] The encoding results of each word after segmentation are queried in the dictionary: 35872, 16584, 11548, 12596, 13595, 19763, 24843. The token sequence of the text description of the scene is

[0128] [35872,16584,1548,2596,13595,19763,24843].

[0129] The photo size is scaled to 512×512 and normalized to obtain a 512×512 two-dimensional matrix.

[0130]

[0131] The two-dimensional matrix is ​​input into the encoder of VQ-VAE to obtain a 32×32 discrete feature vector, which is then expanded in one dimension to obtain a 1024-dimensional feature vector, and its value is then mapped to [0,8192] (denormalization operation) to obtain the image token sequence [2097,3399,…,2826].

[0132] 4.3) Use the text modality tag pair and the image modality tag pair to concatenate the scene text description token sequence and the image token sequence to obtain:

[0133] [ <text> 35872,16584,1548,2596,13595,19763,24843< / text> 2097,3399,……,2826< / im g> ],in <text> 、< / text> , , the tags also have corresponding token values, which are 60002, 60003, 60006, and 60007. After replacement, the final token sequence is:

[0134] [60002,35872,16584,1548,2596,13595,19763,24843,60003,60006,2097,3399,……,2826,60007].

[0135] Input the token sequence into the Transformer model for calculation.

[0136] 4.4) Take the weight calculation result of the self-attention layer in the penultimate layer of the Transformer model, pass it through the fully connected layer to obtain a 1024-dimensional feature vector, and then merge its two-dimensional vector into a 32×32 matrix.

[0137] The 32×32 two-dimensional matrix is ​​passed through the VQ-VAE decoder to generate a heat map weight value of size 64×64;

[0138] Scale the 64×64 heat map weight value to the original image size and superimpose it on the original image to obtain the final hidden danger result map;

[0139] Take the last layer of the Transformer model and use the decoder to decode them one by one to get the token sequence of hidden danger descriptions. When encountering " <end>"When the token value corresponding to the tag is reached, the decoding stops and the token sequence obtained is [60000,35486,42598,……,14526,60001], where " <start>"The token value corresponding to the tag is 60000," <end>The token value corresponding to the " tag is 60001.

[0140] After querying the dictionary for the token sequence of hidden danger description, the word segmentation results of hidden danger description are: "building", "inside", "main", "evacuation", "passageway", "up", "", "stairwell", "of", "door", "and", "safety", "exit", "place", "setting", "access control", "system", ",", "impact", "personnel", "safety", "evacuation". After splicing together, it becomes: "Access control systems are set up on the main evacuation passages, doors in the stairwells and safety exits in the building, affecting the safe evacuation of personnel."

[0141] Comparative Example 1:

[0142] The difference between this comparative example and Example 1 is that: this comparative example uses manual fire inspection

[0143] The specific method is: manually use experience to judge whether there are hidden dangers in the scene. If there are hidden dangers, take photos on the spot and record the hidden danger description results in a paper hidden danger record form. Generally, photos and hidden danger record results are stored separately, and the two need to be integrated together later.

[0144] The results of manual fire inspections are: there may be missed judgments or inaccurate statements, such as the door at the exit of the safety passage cannot be locked.

[0145] To sum up, compared with Example 1, Example 1, first of all, during the process, the present application relies on model calculation, and manual inspections are required on site according to the inspection route. During the process, the present application saves time, outputs results quickly, has higher overall efficiency, and reduces the investment in labor costs. As for the hidden danger output results, first of all, a small number of inspectors did not find all areas with fire hazards. In this regard, the hidden danger identification results in Example 1 are more comprehensive and complete. Similarly, although some inspections in Example 1 also found the same number of fire hazard areas as in Example 1, different personnel's descriptions of different hidden danger areas cannot better express the specific location of the hidden dangers. The present application has specific pictures and annotations, as well as unified standard hidden danger identification descriptions. Obviously, compared with the results of Comparative Example 1, the output results of Example 1 are fundamentally more accurate and standardized.

[0146] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and the present invention can be implemented in other specific forms without departing from the spirit or basic features of the present invention. Therefore, no matter from which point of view, the embodiments should be regarded as exemplary and non-restrictive. The scope of the present invention is defined by the attached claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any figure mark in the claims should not be regarded as limiting the claims involved.

[0147] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other implementation modes that those skilled in the art can understand.< / end> < / start> < / end> < / end> < / start>

Claims

1. A method for identifying hidden dangers in fire inspection based on a multimodal large model, characterized in that: The following steps are involved:

1. Build a multimodal large model: Based on the Transformer architecture and combined with VQ-VAE technology, build a multimodal large model that can process image and text bimodal inputs and output heat maps and hidden danger descriptions. The main network of the model is a unidirectional Transformer model; 2. Model Training 2.1) Collect sample data: collect a large amount of image data and corresponding text descriptions in fire inspection scenarios; 2.2) Sample data preprocessing 2.2.1) Image preprocessing: After scaling and normalizing the collected images, they are input into VQ-VAE to generate image token sequences; 2.2.2) Text preprocessing: segment and encode the collected text descriptions and convert them into text token sequences; 2.2.3) Input sequence generation: concatenate the image token sequence and the text token sequence to form a unified input sequence, and input this input sequence into the multimodal large model; 2.2.4) Manual heat map weight value generation: Manually mark the hidden danger areas in the image and generate the corresponding heat map weight values; 2.2.5) Model prediction heat map weight value generation; 3. Model fine-tuning: Use the Low-Rank Adaptation method to fine-tune the model to obtain a fine-tuned multimodal large model; 4. Deployment and use: During the actual inspection process, input the image data of the real scene and the text description of the corresponding scene into the fine-tuned multimodal large model to generate the actual heat map weight value and hidden danger description; 5. Result display: The heat map weight values ​​are denormalized and mapped to 8-bit grayscale color values ​​to generate a visual heat map, which is then superimposed on the original image to intuitively display potential hidden danger areas. At the same time, the hidden danger description is output in text form.

2. According to claim 1, the method for identifying hidden dangers in fire inspection based on a multimodal large model is characterized in that: In the step 1), the unidirectional Transformer model mainly includes 48 Transformer encoder layers, the size of the hidden layer dimension is 2560, 40 attention heads, and a total of 4 billion parameters.

3. The method for identifying hidden dangers in fire inspection based on a multimodal large model according to claim 2 is characterized in that: In the step 2.2.1), the specific contents of image preprocessing include: 2.2.1.1) Scaling: Limit the size of the collected images to 512×512; 2.2.1.2) Normalization: Linearly normalize each pixel value in the scaled image to between [0, 1]; 2.2.1.3) Token sequence generation: The latent feature z of the image is discretized into a 32×32 token sequence through the internal encoder; 2.2.1.4) Establishment of image token sequence: Expand the 32×32 token sequence in step 2.2.1.3 in one dimension to obtain a 1024 image token sequence.

4. The method for identifying hidden dangers in fire inspection based on a multimodal large model according to claim 3 is characterized in that: In the step 2.2.2), the specific contents of the text preprocessing include: 2.2.2.1) Word segmentation: Use the SentencePiece model to segment the text; 2.2.2.2) Encoding: Query the text dictionary according to the word segmentation content to obtain the index corresponding to each word segmentation; 2.2.2.3) Establishment of text token sequence: Use the index set corresponding to all word segments as the text token sequence.

5. The method for identifying hidden dangers in fire inspection based on a multimodal large model according to claim 4 is characterized in that: In step 2.2.4), the content of the artificial heat map weight value generation is: 2.2.4.1) Manually circle the potential danger areas in the image in advance; 2.2.4.2) Create a two-dimensional matrix of all zeros with the same size as the original input image to store each pixel value of the heat map weight value; 2.2.4.3) Establish coordinates: Take the top left corner of the image as the origin of the coordinate system, the right direction as the positive direction of the x-axis, and the downward direction as the positive direction of the y-axis to establish the coordinates; 2.2.4.4) For the center of the hidden danger area, according to its radius and the preset Gaussian distribution standard deviation, the specific formula for calculating the Gaussian distribution value of each pixel position around the center of the circle is as follows: Among them, μ x and μ y is the coordinate of the center of the circle, σ is the radius of the area, x and y are the coordinates of the current pixel, and G(x,y) represents the thermal weight value at the (x,y) coordinate.

6. The method for identifying hidden dangers in fire inspection based on a multimodal large model according to claim 5 is characterized in that: The specific process of generating the prediction of the heat map weight value in step 2.2.5) is as follows: The self-attention layer in the penultimate layer of the Transformer model is used as the feature vector for generating heat map weight values. Its dimension is the same as that of the hidden layer, which is 2560. It is mapped to 1024 dimensions after a one-dimensional linear transformation, and then a 64×64 weight image is generated through the VQ-VAE decoder. Finally, the heat map weight values ​​of the same size as the original image are generated by scaling.

7. The method for identifying hidden dangers in fire inspection based on a multimodal large model according to claim 6 is characterized in that: In step 3), the process of model fine-tuning is as follows: 3.1) Specify the target module: Determine the module that needs to be fine-tuned by Lora: 3.2) Set fine-tuning parameters: Set the parameters of Lora fine-tuning according to the requirements of specific tasks; 3.3) Training and validation: Use the fine-tuned model to train on the training set and validate on the validation set to evaluate the fine-tuning effect.

8. The method for identifying hidden dangers in fire inspection based on a multimodal large model according to claim 7 is characterized in that: In step 4), the process of deployment and use is: 4.1) During the actual inspection process, obtain image data of the real scene and text description of the corresponding scene; 4.2) Process the acquired image data of the real scene and the text description of the corresponding scene according to the contents of steps 2.2.1) to 2.2.2) to obtain a new image token sequence and a new text token sequence; 4.3) Generate a new input sequence using the new image token sequence and the new text token sequence obtained in step 4.2, and then input the new input sequence into the fine-tuned model; 4.4) The fine-tuned model calculates the new input sequence to obtain new heat map weight values ​​and hidden danger descriptions.

9. The method for identifying hidden dangers in fire inspection based on a multimodal large model according to claim 8 is characterized in that: The output calculation in step 4.4) is: 4.4.1) Use the self-attention layer in the penultimate layer of the Transformer model as the feature vector for generating the heat map; 4.4.2) Use the autoregressive decoder to convert the output layer of the Transformer model into a text description. The decoder gradually generates each character in the description based on the content of the feature vector of the last layer of the Transformer.

10. The method for identifying hidden dangers in fire inspection based on a multimodal large model according to claim 9 is characterized in that: In step 2.2.1.4) and step 2.2.2.2), the value range of the token is 8192-58192, and the value range of the image token is 0-8192.

Images