Multi-stage field smoke and fire detection method based on lightweight multi-modal large model

Abstract

The invention relates to the field of artificial intelligence safety prevention and control, provides a multi-stage field smoke and fire detection method based on a lightweight multi-mode large model, and is mainly applied to field smoke and fire monitoring and prevention and control in a power transmission line scene. The invention aims to solve the problems of low precision and high false alarm rate in field smoke and fire detection, especially when processing interference factors of targets with different sizes and similar characteristics of smoke and fire. According to the main scheme, the method comprises the steps of constructing a detection data set and a classification data set, and training a yolov51 smoke and fire detection model and a tinylava smoke and fire classification model. In the prediction process, a trained yolov51 model is used for carrying out first-stage smoke and fire detection, and then a tinylava model is used for carrying out second-stage non-smoke and fire target filtering. In the training stage, a text-image pair training set is constructed through adaptive image cutting and text cues, and model fine tuning is performed by adopting a Lora method and a Deepspeed technology. The method is used for improving the accuracy and efficiency of field smoke and fire detection, and particularly has excellent performance in a field environment with limited resources.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence security control, and provides a multi-stage outdoor fireworks detection method based on a lightweight multimodal large model, which is mainly used in the field of monitoring and prevention of outdoor fireworks in power line scenarios. Background Art

[0002] In the field of outdoor fireworks detection, the traditional detection method is to determine whether there is a wildfire incident in the image through monitoring equipment image acquisition and manual inspection, so as to ensure environmental safety. With the continuous development of deep learning, various high-precision and high-efficiency models have gradually replaced traditional detection methods, providing a more intelligent, safe and efficient prevention and control mode.

[0003] Existing technical solutions include:

[0004] Directly detect outdoor fireworks targets using traditional deep learning target detection models

[0005] Use time-series-based deep learning methods to analyze video streams and obtain fireworks targets through frame differences

[0006] Use experience and prior information to plan the fireworks abnormal area in advance, and detect the fireworks target when it appears.

[0007] These solutions have the following problems:

[0008] Traditional deep learning detection methods: The field of view of outdoor fireworks varies greatly, and the targets present a variety of different sizes due to the shooting distance. Traditional deep learning is not ideal for small target detection and has a weak ability to capture subtle changes. In addition, clouds, fog, reflective lakes, rivers, roads, lights, and reflective walls with similar characteristics to smoke are prone to false detection, and the model's false alarm rate and missed alarm rate are relatively high.

[0009] Time-series-based methods: Time-series-based methods require video processing, making data acquisition more difficult, requiring high storage requirements, and requiring more manpower and material resources. In addition, video data has high redundancy, and there are problems such as occlusion, motion blur, video defocus, and strange target postures in the data. Time-series-based methods often do not have a clear advantage in efficiency.

[0010] Methods based on prior information: Methods based on prior information use relatively fixed scenes and cannot cope with situations where outdoor scenes vary greatly and targets are not fixed, so they have great limitations in application. Summary of the invention

[0011] The specific technical problem to be solved by the present invention is: in the field power transmission line scene, how to improve the detection accuracy of fireworks targets while reducing the false alarm rate, especially when dealing with various targets of different sizes caused by shooting distances, and interference factors such as clouds, fog, reflective water surfaces, lights, etc. that have similar characteristics to fireworks.

[0012] In order to solve the above technical problems, the present invention adopts the following technical means:

[0013] The present invention provides a multi-stage outdoor fireworks detection method based on a lightweight multimodal large model, characterized in that the method comprises a training process and a prediction process. In the training process, a detection data set and a text-image pair data set are constructed, and a yolov51 fireworks detection model and a tinyllava fireworks classification model are trained respectively. In the prediction process, the trained yolov51 fireworks detection model is used as a first-stage fireworks detection module for target prediction, and then the tinyllava fireworks classification model is used as a second-stage fireworks classification module for non-fireworks target filtering.

[0014] The training process includes the following steps:

[0015] Step a1: Obtain a large amount of fireworks image data from the Internet, power line surveillance cameras and public data sets, annotate and filter all image data, distinguish target labels with obvious smoke and flame features from non-target labels with suspected smoke and flame features. Suspected smoke and suspected flame targets include clouds, mist, reflective lakes, lights, etc., and obtain the target detection dataset;

[0016] Step a2: In the target detection training data set, the adaptive image cropping module is used to expand and crop all the labeled targets, and the background image that does not contain the target is randomly cropped to obtain the cropped classification data set, which includes five categories: smoke, flame, suspected cloud, suspected flame, and background;

[0017] Step a3: Use the classification dataset to construct text prompt words according to each type of target to form a text-image pair training set. The classification dataset contains cropped images and fixed question information Question and answer information Answer. The Question provides a basic attribute description of the category, including the cause of formation, general performance, and a question about the category to which the image belongs. The Answer provides an information description of the image and the category attribution of different targets.

[0018] Step a4: Use the detection data set to train the Yolov5-1 model to obtain the Yolov5-1 smoke detection model, and the detection categories are smoke and flame;

[0019] Step a5: Use the text-image pair training set to fine-tune the Tinyllava model. During the fine-tuning process, the Lora method is used to update the weights. At the same time, the Deepspeed method is used to reduce the dependence of the training process on hardware devices, and the fine-tuned model weights are obtained.

[0020] The purpose of this step is to extract target image data according to the pre-labeled fireworks category, extract easy-to-false-alarm data according to the pre-labeled suspected targets, and perform image extraction through the adaptive image cropping module, thereby increasing the proportion of target information in the image in order to be able to pass the classification model that pays more attention to the image content;

[0021] Step a2.1: The adaptive image cropping module first obtains the target's center point coordinates (x, y), length (h), width (w), category (class) and other information based on the detection results, and packages them into a dictionary to be processed;

[0022] Step a2.2: read the dictionary to be processed and locate the target in the original image according to the dictionary information;

[0023] Step a2.3: According to the h and w of the target information, calculate the proportion of the area of ​​the original target in the entire image. When it is lower than the threshold of 0.0009, it is identified as a small target and expanded cropping is performed. The cropping range is obtained by taking 4 times the maximum value of the length and width as the side length and the center point coordinates as the center. When it is higher than the threshold of 0.0009 and lower than 0.0100, it is identified as a medium target and ordinary cropping is performed. The cropping range is obtained by taking 2 times the maximum value of the length and width as the side length and the center point coordinates as the center. When it is higher than 0.0100, it is identified as a large target. The cropping range is obtained by taking the maximum value of the length and width as the side length and the center point coordinates as the center.

[0024] Step a2.4: Determine the image boundary according to the cropping range. For the part beyond the boundary, use the cropping method to fill the part of the space beyond the image with the internal information of the image to ensure the consistency of the aspect ratio of the input image. Figure 2 As shown, the gray part is the image that exceeds the limit, the blue part is the restored image, and finally the image is resized to 384*384;

[0025] Step a2.5: Save the cropped image and the absolute coordinates corresponding to the original image in the corresponding folder according to the category information. Through the above method, the problem of large size differences when directly resizing objects of different scales can be effectively reduced, and the context information is appropriately supplemented, which is conducive to improving the focus on important features when training the classification model.

[0026] Step a3: Use the classification dataset to construct text prompt words according to each type of target to form a text-image pair training set. The dataset contains cropped images and fixed question information Question and answer information Answer. The Question provides a basic attribute description of the category, including the cause of formation, general performance, and a question about the category to which the image belongs. The Answer provides information description of the image and the category attribution of different targets.

[0027] Step a4: Use the detection data set to train the Yolov5-1 model to obtain the Yolov5-1 smoke detection model, and the detection categories are smoke and flame;

[0028] Step a5: Use the text-image pair training set to fine-tune the Tinyllava model. During the fine-tuning process, the Lora method is used to update the weights. At the same time, the Deepspeed method is used to reduce the dependence of the training process on hardware devices, and the fine-tuned model weights are obtained.

[0029] Step a5.1: Configure the Lora parameter file based on the hardware system and Python library version;

[0030] Step a5.2: Configure Deepspeed training parameter file;

[0031] Step a5.3: Use the Lora method for training. Each time the training process is updated, the update formula is:

[0032] y=(W+ΔW)x+b

[0033] Among them, ΔW is a trainable increment, and its value is obtained by multiplying a low-rank matrix A and a residual matrix B. During the training process, ΔW is trained and the original weight W is frozen;

[0034] Step a5.4: Merge and transform the trained models to obtain the fine-tuned tinyllava-lora-smokefire classification model weights.

[0035] The inference phase consists of the following steps:

[0036] Step b1: collecting image data and related information through a monitoring camera of a power transmission service line to obtain an input image;

[0037] Step b2: pass the input image into the first stage detection module to obtain the detection box information;

[0038] Step b2.1: The detection module uses yolov51 as the basic structure, adopts the pre-trained yolov5-1 smoke detection model, and freezes the weights of the feature extraction layer and the prediction layer;

[0039] Step b2.2: Send the input image to the feature extraction layer to obtain image features, and make preliminary predictions through the prediction layer;

[0040] Step b2.3: ​​Using 0.2 as the score threshold, a large amount of detection box information is initially obtained through NMS.

[0041] Step b3: Using the same cropping method as described in steps a2.1-a2.4 of the training phase, input the input image and detection frame information into the adaptive cropping module to obtain the resized cropped image array and absolute coordinates, and save the cropped multiple image arrays as a numpy array list.

[0042] Step 4: Send the cropped array list and question information to the second-stage lightweight multimodal large model for classification to obtain the text description result.

[0043] Step b4.1: Input the cropped image into the SigLIP visual encoder module to encode the image features, extract key visual information, and obtain image feature embedding. The SigLIP visual encoder module;

[0044] Step b4.2: A two-layer MLP is used as a connector module to re-encode the generated image feature embeddings and adjust them to the text embedding space that matches the language model, thus achieving modal alignment of image features and language features.

[0045] Step b4.3: Convert the manually designed input through a specific prompt structure, format the input prompt into a structured sequence containing system instructions, user input, and assistant response, and then encode each part using Siglip's text encoder module to extract key semantic information and obtain semantic feature embedding;

[0046] Step b4.4: Initialize the large language model module based on the tinyllava-lora-smokefire classification model weights obtained through fine-tuning training;

[0047] Step b4.5: The initialized lora model will predict each token in the text output sequence based on the semantic feature embedding and visual feature embedding until the model generation end condition is met, at which time the text description result is obtained.

[0048] Step b5: Filter and screen the text description results. When there are smoke and flame target prompt words, the classification category of the cropped image is output. When there are no smoke and flame target prompt words, no information is output.

[0049] Step b6: splice the target center point coordinates, frame width and frame length, and classification category information in the detection frame information in step 2, and output the detection result. When the category information does not exist, filter out the detection frame.

[0050] Because the present invention adopts the above technical means, it has the following beneficial effects:

[0051] 1. The present invention combines the Yolov5-1 model with a lightweight multimodal large model to solve the problems of insufficient recall and precision of traditional target detection in outdoor fireworks scenes, achieves a detection effect with low false alarms and high recall, and significantly improves the detection performance.

[0052] 2. By introducing an adaptive image cropping strategy, the present invention effectively solves the problem of identifying targets of different sizes during detection, especially the detection of small targets. By intelligently adjusting the cropping ratio, the model's recognition ability for targets of various sizes is enhanced, thereby improving the comprehensiveness and accuracy of detection.

[0053] 3. The present invention adopts an innovative training strategy, combining the Lora method and DeepSpeed ​​technology, which reduces the training cost of multimodal large models and improves training efficiency. At the same time, it ensures the high accuracy of the model in classification and realizes efficient and accurate model training.

[0054] 4. By using the lightweight models Yolov5-1 and Tinyllava, the present invention successfully adapts to the challenge of limited resources in the wild environment, so that the constructed model can run effectively under limited computing resources, meeting the actual needs of outdoor smoke and fire detection and analysis.

[0055] 5. Based on the Yolov51 model, the present invention adds a multimodal lightweight large model as a post-classification model, which effectively solves the limitations of the single modality detection method when dealing with complex scenes. By combining the two modal information of text description and image details, the classification accuracy is significantly improved, and the overall detection performance is further enhanced. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is a simplified diagram of the process;

[0057] Figure 2 Schematic diagram of cutting and patching;

[0058] Figure 3 A simplified diagram of tinyllava's reasoning process. DETAILED DESCRIPTION

[0059] The following is a detailed description of the embodiments of the present invention. Although the present invention will be described and illustrated in conjunction with some specific embodiments, it should be noted that the present invention is not limited to these embodiments. On the contrary, modifications or equivalent substitutions made to the present invention should all be included in the scope of the claims of the present invention.

[0060] In addition, in order to better illustrate the present invention, numerous specific details are given in the following specific embodiments. It will be understood by those skilled in the art that the present invention can also be implemented without these specific details.

[0061] A multi-stage outdoor fireworks detection method based on a lightweight multimodal large model includes a training phase and a testing phase. In the training phase, the main purpose is to obtain the fine-tuning weights of the multimodal large model on the fireworks data, so as to use it in the model of this solution.

[0062] The training phase consists of the following steps:

[0063] Step a1: Obtain a large amount of fireworks image data from the Internet, power line surveillance cameras and public data sets, annotate and filter all image data, distinguish target labels with obvious smoke and flame features from non-target labels with suspected smoke and flame features. Suspected smoke and suspected flame targets include clouds, mist, reflective lakes, lights, etc., and obtain the target detection dataset;

[0064] Step a2: Use the adaptive image cropping module to expand and crop all labeled targets in the target detection training dataset, and randomly crop the background image that does not contain the target to obtain a cropped classification dataset, which includes five categories: smoke, flame, suspected cloud, suspected flame, and background.

[0065] The purpose of this step is to extract target image data according to the pre-labeled fireworks category, extract easy-to-false-alarm data according to the pre-labeled suspected targets, and perform image extraction through the adaptive image cropping module, thereby increasing the proportion of target information in the image in order to be able to pass the classification model that pays more attention to the image content;

[0066] Step a2.1: The adaptive image cropping module first obtains the target's center point coordinates (x, y), length (h), width (w), category (class) and other information based on the detection results, and packages them into a dictionary to be processed;

[0067] Step a2.2: read the dictionary to be processed and locate the target in the original image according to the dictionary information;

[0068] Step a2.3: According to the h and w of the target information, calculate the proportion of the area of ​​the original target in the entire image. When it is lower than the threshold of 0.0009, it is identified as a small target and expanded cropping is performed. The cropping range is obtained by taking 4 times the maximum value of the length and width as the side length and the center point coordinates as the center. When it is higher than the threshold of 0.0009 and lower than 0.0100, it is identified as a medium target and ordinary cropping is performed. The cropping range is obtained by taking 2 times the maximum value of the length and width as the side length and the center point coordinates as the center. When it is higher than 0.0100, it is identified as a large target. The cropping range is obtained by taking the maximum value of the length and width as the side length and the center point coordinates as the center.

[0069] Step a2.4: Determine the image boundary according to the cropping range. For the part beyond the boundary, use the cropping method to fill the part of the space beyond the image with the internal information of the image to ensure the aspect ratio of the input image is consistent, and then resize the image to 384*384;

[0070] Step a2.5: Save the cropped image and the absolute coordinates corresponding to the original image in the corresponding folder according to the category information.

[0071] Step a3: Use the classification data set to construct text prompt words according to each type of target to form a text-image pair training set. The image pair contains the cropped image and the corresponding question information Question and answer information Answer. The Question provides a basic attribute description of the category, including the cause of formation, general performance, and a question about the category to which the image belongs. The Answer provides information description of the image and the category it belongs to.

[0072] Step a4: Use the detection data set to train the Yolov5-1 model to obtain the Yolov5-1 smoke detection model, and the detection categories are smoke and flame;

[0073] Step a5: Use the text-image pair training set to fine-tune the Tinyllava model. During the fine-tuning process, the Lora method is used to update the weights. At the same time, the Deepspeed method is used to reduce the dependence of the training process on hardware devices, and the fine-tuned model weights are obtained.

[0074] Step a5.1: Configure the Lora parameter file based on the hardware system and Python library version;

[0075] Step a5.2: Configure Deepspeed training parameter file;

[0076] Step a5.3: Use the Lora method for training. Each time the training process is updated, the update formula is:

[0077] y=(W+ΔW)x+b

[0078] Among them, ΔW is a trainable increment, and its value is obtained by multiplying a low-rank matrix A and a residual matrix B. During the training process, ΔW is trained and the original weight W is frozen;

[0079] Step a5.4: Merge and transform the trained models to obtain the fine-tuned tinyllava-lora-smokefire model weights.

[0080] The inference phase consists of the following steps:

[0081] Step b1: collecting image data and related information through a monitoring camera of a power transmission service line to obtain an input image;

[0082] Step b2: pass the input image into the first stage detection module to obtain the detection box information;

[0083] Step b2.1: The detection module uses yolov51 as the basic structure, adopts the pre-trained yolov5-1 smoke detection model, and freezes the weights of the feature extraction layer and the prediction layer;

[0084] Step b2.2: Send the input image to the feature extraction layer to obtain image features, and make preliminary predictions through the prediction layer;

[0085] Step b2.3: ​​Using 0.2 as the score threshold, a large amount of detection box information is initially obtained through NMS.

[0086] Step b3: Using the same cropping method as described in steps a2.1-a2.4 of the training phase, input the input image and detection frame information into the adaptive cropping module to obtain the resized cropped image array and absolute coordinates, and save the cropped multiple image arrays as a numpy array list.

[0087] Step b4: Send the cropped array list and question information to the second-stage lightweight multimodal large model for classification to obtain the text description result.

[0088] Step b4.1: Input the cropped image into the SigLIP visual encoder module to encode the image features, extract key visual information, and obtain image feature embedding;

[0089] Step b4.2: A two-layer MLP is used as a connector module to re-encode the generated image feature embeddings and adjust them to the text embedding space that matches the language model, thus achieving modal alignment of image features and language features.

[0090] Step b4.3: Convert the manually designed input through a specific prompt structure, format the input prompt into a structured sequence containing system instructions, user input, and assistant response, and then encode each part using Siglip's text encoder module to extract key semantic information and obtain semantic feature embedding;

[0091] Step b4.4: Initialize the large language model module based on the tinyllava-lora-smokefire classification model weights obtained through fine-tuning training;

[0092] Step b4.5: The initialized large language model module will predict each token in the text output sequence based on the semantic feature embedding and visual feature embedding until the model generation end condition is met, at which time the text description result is obtained.

[0093] Step b5: Filter and screen the text description results. When there are smoke and flame target prompt words, the classification category of the cropped image is output. When there are no smoke and flame target prompt words, no information is output.

[0094] Step b6: splice the target center point coordinates, frame width and frame length, and classification category information in the detection frame information in step 2, and output the detection result. When the category information does not exist, filter out the detection frame.

[0095] This technical proposal has the following characteristics: After the detection model in the first stage detects the fireworks target, it inputs more spatial scene and cognitive prior information into the second stage classification multimodal large model through an adaptive image expansion method. The classification model uses information from both text and image modalities to classify the target, which greatly enriches the target feature information, thereby reducing the false detection of the target and improving the detection accuracy.

Claims

1. A multi-stage outdoor fireworks detection method based on a lightweight multimodal large model, characterized in that: The method includes a training process and a prediction process. In the training process, a detection data set and a classification data set are constructed, and the yolov51 fireworks detection model and the tinyllava fireworks classification model are trained respectively. In the prediction process, the trained yolov51 fireworks detection model is used as the first-stage fireworks detection module for target prediction, and then the tinlllava fireworks classification model is used as the second-stage fireworks classification module for non-fireworks target filtering.

2. The multi-stage outdoor fireworks detection method based on a lightweight multi-modal large model according to claim 1 is characterized in that: The training process includes the following steps: Step a1: Obtain a large amount of fireworks image data from the Internet, power line surveillance cameras and public data sets, annotate and filter all image data, distinguish target labels with obvious smoke and flame features from non-target labels with suspected smoke and flame features. Suspected smoke and suspected flame targets include clouds, mist, reflective lakes, lights, etc., and obtain the target detection dataset; Step a2: In the target detection training data set, the adaptive image cropping module is used to expand and crop all the labeled targets, and the background image that does not contain the target is randomly cropped to obtain the cropped classified image data, which includes five categories: smoke, flame, suspected cloud, suspected flame, and background; Step a3: Use the classified image data to construct text prompt words according to each type of target to form a text-image pair training set. The classified data set contains cropped images and fixed question information Question and answer information Answer. The Question provides a basic attribute description of the category, including the cause of formation, general performance, and a question about the category to which the image belongs. The Answer provides an information description of the image and the category attribution of different targets. Step a4: Use the detection data set to train the Yolov5-1 model to obtain the Yolov5-1 smoke detection model, and the detection categories are smoke and flame; Step a5: Use the text-image pair training set to fine-tune the Tinyllava model. During the fine-tuning process, the Lora method is used to update the weights. At the same time, the Deepspeed method is used to reduce the dependence of the training process on hardware devices, and the fine-tuned model weights are obtained.

3. The multi-stage outdoor fireworks detection method based on a lightweight multi-modal large model according to claim 2 is characterized in that: Step a2 includes the following steps: Step a2.1: The adaptive image cropping module first obtains the target's center point coordinates (x, y), length h, width w, and category class information based on the detection results, and packages them into a dictionary to be processed; Step a2.2: read the dictionary to be processed and locate the target in the original image according to the dictionary information; Step a2.3: According to the h and w of the target information, calculate the proportion of the area of ​​the original target in the entire image. When it is lower than the threshold of 0.0009, it is identified as a small target and expanded cropping is performed. The cropping range is obtained by taking 4 times the maximum value of the length and width as the side length and the center point coordinates as the center. When it is higher than the threshold of 0.0009 and lower than 0.0100, it is identified as a medium target and ordinary cropping is performed. The cropping range is obtained by taking 2 times the maximum value of the length and width as the side length and the center point coordinates as the center. When it is higher than 0.0100, it is identified as a large target. The cropping range is obtained by taking the maximum value of the length and width as the side length and the center point coordinates as the center. Step a2.4: Determine the image boundary according to the cropping range. For the part beyond the boundary, use the cutting and filling method to fill the part of the space beyond the image with the internal information of the image to ensure the consistency of the aspect ratio of the input image. Step a2.5: Save the cropped image and the absolute coordinates corresponding to the original image in the corresponding folder according to the category information.

4. A multi-stage outdoor fireworks detection method based on a lightweight multi-modal large model according to claim a2, characterized in that: Step a5 includes the following steps: Step a5.1: Configure the Lora training parameter file based on the hardware system and Python library version; Step a5.2: Configure Deepspeed parameter file; Step a5.3: Use the Lora method for training. Each time the training process is updated, the update formula is: y=(W+ΔW)x+b Among them, ΔW is a trainable increment, and its value is obtained by multiplying a low-rank matrix A and a residual matrix B. During the training process, ΔW is trained and the original weight W is frozen; Step a5.4: Merge and transform the trained models to obtain the fine-tuned tinyllava-lora-smokefire classification model weights.

5. The multi-stage outdoor fireworks detection method based on a lightweight multi-modal large model according to claim 3 is characterized in that: The inference phase consists of the following steps: Step b1: collecting image data and related information through a monitoring camera of a power transmission service line to obtain an input image; Step b2: passing the input image into the first-stage fire and smoke detection module to obtain the detection frame information; Step b3: Using the same cropping method as described in steps a2.1-a2.4 of the training phase, input the input image and detection frame information into the adaptive cropping module, obtain the resized cropped image array and absolute coordinates, and save the cropped multiple image arrays as a numpy array list; Step b4: Send the cropped array list and the preset question information to the second-stage fireworks classification module for classification to obtain a text description result; Step b5: Filter and screen the text description results. When there are smoke and flame target prompt words, output the classification category of the cropped image. When there are no smoke and flame target prompt words, no information is output. Step b6: splice the target center point coordinates, frame width and frame length, and classification category information in the detection frame information in step b2, and output the detection result. When the category information does not exist, filter out the detection frame.

6. The multi-stage outdoor fireworks detection method based on a lightweight multi-modal large model according to claim 5 is characterized in that: Step b2 includes the following steps: Step b2.1: The first-stage smoke and fire detection module uses yolov51 as the basic structure, adopts the pre-trained yolov5-1 smoke detection model, and freezes the weights of the feature extraction layer and the prediction layer; Step b2.2: Send the input image to the feature extraction layer to obtain image features, and make preliminary predictions through the prediction layer; Step b2.3: ​​Using 0.2 as the score threshold, a large amount of detection box information is initially obtained through NMS.

7. The multi-stage outdoor fireworks detection method based on a lightweight multi-modal large model according to claim 5 is characterized in that: Step b4 includes the following steps: Step 4.1: Input the cropped image into the SigLIP visual encoder module to encode the image features, extract key visual information, and obtain image feature embedding; Step 4.2: Use a two-layer MLP as a connector module to re-encode the generated image feature embeddings through the module and adjust them to the text embedding space that matches the language model to achieve modality alignment of image features and language features; Step 4.3: Convert the manually designed input through a specific prompt structure, format the input prompt into a structured sequence containing system instructions, user input, and assistant response, and then encode each part using SigLIP's text encoder module to extract key semantic information and obtain semantic feature embedding; Step 4.4: Initialize the large language model module based on the tinyllava-lora-smokefire classification model weights obtained through fine-tuning training; Step 4.5: The initialized large language model module will predict each token in the text output sequence based on the semantic feature embedding and visual feature embedding until the model generation end condition is met, at which time the text description result is obtained.

Images