Smoke mountain fire identification method, device, equipment, medium and program product
Patent Information
- Application Number
- CN202611125027.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-28
- Publication Date
- 2026-11-03
- Estimated Expiration
- 2046-07-28
Smart Images
Figure CN122637149B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of image recognition technology, and in particular to a method, apparatus, equipment, medium, and program product for identifying smoke and wildfires. Background Technology
[0002] Power transmission lines are the lifeblood of electrical energy transmission, and their safe operation is directly related to the stability of the power grid and the normal functioning of social and economic life. Transmission lines often traverse complex geographical environments such as mountainous areas and forests. These areas are densely vegetated and highly susceptible to wildfires in dry, hot, or windy weather. The high-temperature smoke from wildfires can reduce the insulation strength of the air, causing transmission lines to trip, short-circuit, or even break, resulting in widespread power outages and huge economic losses.
[0003] In traditional technology, images of power transmission lines are typically captured by cameras, and then back-end algorithms are used to analyze the images.
[0004] However, due to the complex environment and numerous interference factors in the transmission line corridor, current image detection models suffer from false alarms, resulting in low accuracy in smoke and wildfire identification. Summary of the Invention
[0005] Therefore, it is necessary to provide a smoke and wildfire identification method, device, equipment, medium, and program product that can improve the accuracy of smoke and wildfire identification in response to the above-mentioned technical problems.
[0006] Firstly, this application provides a method for identifying smoke and wildfires, the method comprising:
[0007] The target image is used to identify smoke and wildfires using a pre-set target detection model to determine the preliminary identification results.
[0008] If the initial identification result indicates that there is smoke and wildfire in the target image, then obtain the single image prompt word, and input the target image and the single image prompt word into the preset multimodal large model to obtain the advanced identification result of the target image;
[0009] If the advanced recognition result indicates that there is smoke and wildfire in the target image, then the historical image and multiple image prompts at the previous moment of the target image acquisition time are obtained, and the historical image, target image, and multiple image prompts are input into the multimodal large model to obtain the smoke and wildfire recognition result of the target image.
[0010] In one embodiment, obtaining a single image cue word includes:
[0011] Determine the target time period based on the data collection time;
[0012] Based on the target time period, query the prompt word database and determine the single image prompt word according to the query results. The prompt word database includes the correspondence between different time periods and prompt words.
[0013] In one embodiment, the training process of the multimodal large model includes:
[0014] A sample image dataset is generated based on ordinary sample images of the first ratio and difficult sample images of the second ratio. The difficult sample images are sample images that the object detection model misidentifies. The second ratio is greater than the first ratio.
[0015] Generate a multimodal sample image dataset based on the sample image dataset;
[0016] Annotate the multimodal sample image dataset to generate a sample dataset;
[0017] The initial multimodal large model is trained based on the sample dataset to obtain the multimodal large model.
[0018] In one embodiment, the multimodal sample image dataset includes first-type sample image data, second-type sample image data, third-type sample image data, and fourth-type sample image data. Based on the sample image dataset, a multimodal sample image dataset is generated, including:
[0019] The sample image dataset is divided into four categories to obtain the first candidate sample, the second candidate sample, the third candidate sample, and the fourth candidate sample;
[0020] The first candidate sample is used as the first type of sample image data;
[0021] The second candidate sample is input into the target detection model for detection, generating a second candidate sample with a detection box, and the second candidate sample with the detection box is used as the second type of sample image data.
[0022] Based on sample image data from multiple different scenarios in the third candidate samples, generate third type sample image data;
[0023] Based on the sample image data from multiple consecutive time points in the fourth candidate sample, fourth type sample image data is generated.
[0024] In one embodiment, the multimodal sample image dataset is labeled to generate a sample dataset, including:
[0025] For any multimodal sample image in the multimodal sample image dataset, obtain the pre-labeling results of multiple candidate large models annotating the multimodal sample image data;
[0026] Each pre-labeled result is scored;
[0027] Based on the highest-scoring pre-annotated results, generate annotation results for multimodal sample image data;
[0028] A sample dataset is generated based on the multimodal sample image data and the corresponding annotation results.
[0029] In one embodiment, an initial multimodal large model is trained based on a sample dataset to obtain a multimodal large model, including:
[0030] Freeze the backbone parameters of the initial multimodal large model and inject a low-rank decomposition matrix into the self-attention layer of the initial multimodal large model to obtain the fine-tuned initial multimodal large model;
[0031] The fine-tuned initial multimodal large model is trained based on the sample dataset and preset training configuration parameters to obtain the multimodal large model.
[0032] Secondly, this application also provides a smoke and wildfire identification device, the device comprising:
[0033] The recognition module is used to identify smoke and wildfires in target images using a preset target detection model and determine the preliminary recognition results.
[0034] The first input module is used to obtain a single image prompt if the preliminary recognition result indicates that there is smoke or wildfire in the target image, and input the target image and the single image prompt into a preset multimodal large model to obtain the advanced recognition result of the target image.
[0035] The second input module is used to obtain the historical image and multiple image prompts of the target image at the previous acquisition time if the advanced recognition result is that there is smoke and wildfire in the target image. The historical image, target image and multiple image prompts are then input into the multimodal large model to obtain the smoke and wildfire recognition result of the target image.
[0036] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described in any one of the first aspects above.
[0037] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any one of the first aspects above.
[0038] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in any one of the first aspects above.
[0039] The aforementioned smoke and wildfire identification method, device, equipment, medium, and program products first identify smoke and wildfire in the target image using a preset target detection model to determine the preliminary identification result. If the preliminary identification result indicates that smoke and wildfire exist in the target image, a single image prompt word is obtained, and the target image and the single image prompt word are input into a preset multimodal large model to obtain the advanced identification result of the target image. If the advanced identification result indicates that smoke and wildfire exist in the target image, the historical image and multi-image prompt words from the previous acquisition time of the target image are obtained, and the historical image, target image, and multi-image prompt words are input into the multimodal large model to obtain the smoke and wildfire identification result of the target image. In this way, after the target image is initially identified as smoke and wildfire by the target detection model, the target image is further identified based on the initial identification result using a multimodal large model to obtain an advanced identification result. If the advanced identification result still indicates that there is smoke and wildfire in the target image, the historical image from the previous moment is acquired, and the smoke and wildfire identification result is obtained by simultaneously detecting the historical image and the target image. In this way, through multiple identification processes, the misidentification phenomenon that may occur with a single detection method of the target detection model is avoided, and the accuracy of smoke and wildfire identification is improved. Attached Figure Description
[0040] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0041] Figure 1 This is an application environment diagram of a smoke and wildfire identification method in one embodiment;
[0042] Figure 2 This is a flowchart illustrating a smoke and wildfire identification method in one embodiment;
[0043] Figure 3 This is a flowchart illustrating the single-image prompt word determination step in one embodiment;
[0044] Figure 4 This is a flowchart illustrating the multimodal large model training steps in one embodiment;
[0045] Figure 5 This is a flowchart illustrating the sample dataset generation steps in one embodiment;
[0046] Figure 6 This is a flowchart illustrating the smoke and wildfire identification method in another embodiment;
[0047] Figure 7This is a structural block diagram of a smoke and wildfire detection device in one embodiment;
[0048] Figure 8 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation
[0049] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0050] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.
[0051] Traditional technologies typically involve capturing images of power transmission lines using cameras, followed by analysis of these images using backend algorithms. However, due to the complex environment along power transmission line corridors, visual features such as morning fog, clouds, factory chimney emissions, nighttime vehicle lights, and streetlight reflections are highly similar to wildfire smoke. Current image detection models suffer from false alarms, resulting in a large number of false alerts being received by monitoring centers, leading to low accuracy in identifying smoke and wildfires.
[0052] In view of this, this application provides a smoke and wildfire identification method. After the target image is initially identified by the target detection model, the target image is further identified based on the initial identification result using a multimodal large model to obtain the smoke and wildfire identification result. This avoids the misidentification or missed identification that may occur with the single detection method of the target detection model, and improves the accuracy of smoke and wildfire identification.
[0053] The smoke and wildfire identification method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, server 102 communicates with image acquisition device 104 via a network. Server 102 can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server providing cloud computing services. Image acquisition device 104 can be a camera, a dual-spectrum visible light and thermal imaging device, etc.
[0054] In one exemplary embodiment, such as Figure 2As shown, a method for identifying smoke and wildfires is provided, which can be applied to... Figure 1 Taking the server in the example, the explanation includes the following steps 201 to 203. Wherein:
[0055] Step 201: Use a preset target detection model to identify smoke and wildfire in the target image and determine the preliminary identification results.
[0056] The preset target detection model can be built based on target detection algorithms such as YOLO (You Only Look Once), Faster R-CNN (Faster Region-based Convolutional Neural Network), and DETR (Detection Transformer). The above target detection algorithms are trained using a large amount of single-frame labeled image data to obtain the target detection model, which is able to locate smoke and fire regions in the input image and output results such as the category and coordinates of the smoke and fire regions.
[0057] Optionally, the target image can be an image of the power transmission line at the current moment acquired by the image acquisition device. The target image is input into the target detection model for preliminary identification. After the target detection model extracts features from the target image, it outputs the preliminary identification results. The preliminary identification results can include the bounding boxes of areas in the target image that are suspected to have smoke and wildfires detected by the target detection model, as well as the coordinates, confidence level and category of the bounding boxes.
[0058] Step 202: If the preliminary identification result indicates that there is smoke and wildfire in the target image, then obtain the single image prompt word, and input the target image and the single image prompt word into the preset multimodal large model to obtain the advanced identification result of the target image.
[0059] The pre-defined multimodal large model can be trained based on an open-source general multimodal large model with strong visual encoding and long text reasoning capabilities. The open-source general multimodal large model can be Qwen-VL (QwenVision-Language Model) or other open-source VL (Vision-Language Model) series. These general multimodal large models have been pre-trained on large-scale general data and have good zero-shot recognition capabilities.
[0060] If the initial identification result of the target detection model indicates that there is smoke and wildfire in the target image, that is, the initial identification result includes a bounding box of a region suspected of having smoke and wildfire, in order to avoid misidentification by the target detection model, a multimodal large model is used to further detect the target image.
[0061] In multimodal large-scale modeling, prompt words serve as operational instructions, directing the model on how to analyze images and what results to output. Pre-stored single-image prompt words are retrieved from a database or configuration file. A single-image prompt word could be something like, "Is there a risk of wildfire in the image? Please analyze it in conjunction with vegetation, sky background, and smoke patterns." The target image and the single-image prompt words are then input into the multimodal large-scale model to obtain advanced recognition results for the target image.
[0062] Optionally, single-image prompts can be further categorized, and corresponding single-image prompts can be obtained based on the acquisition time or acquisition type of the target image.
[0063] Step 203: If the advanced recognition result is that there is smoke and wildfire in the target image, then obtain the historical image and multi-image prompt words of the previous moment of the target image acquisition time, and input the historical image, target image and multi-image prompt words into the multimodal large model to obtain the smoke and wildfire recognition result of the target image.
[0064] If the advanced recognition result obtained by the multimodal large model in the first instance of target image recognition still indicates the presence of smoke and wildfire in the target image, this is because the first recognition process of the multimodal large model detects a single frame image. A single frame image cannot determine the motion trend. To further improve recognition accuracy and avoid false recognition, historical images from the previous moment of the target image acquisition time are obtained. These historical images and the target image are then combined to form a multi-frame image, enabling motion vector detection similar to video analysis. Optionally, the historical images can be images acquired 5 minutes prior using the same image acquisition device as the target image.
[0065] For example, historical images and target images can be used as dual-image inputs or stitched together and then input into a preset multimodal large model. Simultaneously, multi-image prompts are obtained based on the input processing methods of the historical and target images. These multi-image prompts can be pre-stored in a database or configuration file. Different input processing methods for historical and target images can correspond to different identifiers, and corresponding multi-image prompts are obtained based on these identifiers.
[0066] Optionally, if the historical image and the target image are used as dual-image inputs in a multimodal large model, the multi-image prompts can be " Figure 1 This is the scene from the previous moment. Figure 2 This refers to the current scene. The object detection model is already in use. Figure 2The image shows a suspected smoke and wildfire area. Please compare the two images and analyze whether there are changes in motion vectors in this area, such as changes in the upward velocity, diffusion direction, and area growth rate. If the historical image and the target image are stitched together and then input into a preset multimodal large-scale model, taking a top-and-bottom stitching of the historical and target images as an example, the multi-image prompt could be: "The upper half of the image is the scene from the previous moment, and the lower half is the scene from the current moment. The target detection model has already selected a suspected smoke and wildfire area in the current moment's scene. Please compare the upper and lower images and analyze whether there are changes in motion vectors in this area, such as changes in the upward velocity, diffusion direction, and area growth rate."
[0067] The aforementioned historical images, target images, and multiple image prompts are input into a multimodal large model. Based on the output of the multimodal large model, the smoke and wildfire identification results of the target image are obtained. The smoke and wildfire identification results can include the conclusion and confidence level of whether smoke and wildfire exist in the target image, as well as a natural language image analysis report for reference by maintenance personnel.
[0068] If the preliminary or advanced identification results indicate that there is no smoke or wildfire in the target image, then the preliminary or advanced identification results will be taken as the smoke and wildfire identification results of the target image, meaning that there is no smoke or wildfire in the target image.
[0069] In the above embodiments, firstly, a preset target detection model is used to identify smoke and wildfire in the target image to determine a preliminary identification result. If the preliminary identification result indicates that smoke and wildfire exist in the target image, a single-image prompt is obtained, and the target image and the single-image prompt are input into a preset multimodal large model to obtain an advanced identification result for the target image. If the advanced identification result indicates that smoke and wildfire exist in the target image, the historical image and multi-image prompts from the previous acquisition time of the target image are obtained, and the historical image, target image, and multi-image prompts are input into the multimodal large model to obtain the smoke and wildfire identification result for the target image. In this way, after the target image is initially identified as having smoke and wildfire by the target detection model, the target image is further identified based on the preliminary identification result using the multimodal large model to obtain an advanced identification result. If the advanced identification result still indicates that smoke and wildfire exist in the target image, the historical image from the previous moment is obtained again, and the smoke and wildfire identification result is obtained by simultaneously detecting the historical image and the target image. In this way, through multiple identification processes, the misidentification phenomenon that may occur with a single detection method of the target detection model is avoided, and the accuracy of smoke and wildfire identification is improved.
[0070] When the advanced recognition result obtained by the multimodal large model through the first target image recognition still indicates the presence of smoke and wildfire in the target image, in order to avoid misidentification, historical images are acquired and simultaneously input into the multimodal large model for further judgment. Without increasing the bandwidth pressure of a large amount of video stream processing, motion vector detection similar to video analysis is realized, which can capture the dynamic process of smoke and wildfire occurrence, effectively filter out static background interference, and improve the spatiotemporal accuracy of smoke and wildfire recognition.
[0071] In one embodiment, a single image cue word is obtained, such as... Figure 3 As shown, it may include:
[0072] Step 301: Determine the target time period based on the collection time.
[0073] Because the interference features that may affect the identification of smoke and wildfires in target images may differ depending on the time of acquisition, for example, in the early morning, the main interference features in target images may be morning fog, dew reflections, etc. During the daytime, the main interference features may be smoke from burning fields, industrial emissions, etc. At night, the main interference features may be vehicle headlights, streetlights, village lights, moonlight reflections, etc.
[0074] Therefore, in order to improve the recognition accuracy of target images, guide the multimodal large model to focus on key discriminative features, determine the target time period based on the acquisition time of the target image, and thus obtain the corresponding prompt words according to the main interference features of the target time period.
[0075] For example, define a time segmentation function. :
[0076]
[0077] in, This refers to the data collection time.
[0078] Step 302: Query the prompt word database based on the target time period, and determine the single image prompt word based on the query results.
[0079] The prompt word database includes the correspondence between different time periods and prompt words. Different prompt words are pre-configured based on the interference characteristics of different time periods and stored in the prompt word database. After obtaining the target time period of the target image, the prompt word database is queried according to the target time period to determine the single-image prompt word corresponding to the target image.
[0080] For example, for the early morning period, in order to distinguish the interference features of the early morning period, strong semantically constrained prompts are used. That is, information such as domain knowledge rules, logical judgment conditions, or physical morphological features are added to the prompts to filter false alarms and improve the accuracy of multimodal large model recognition. For example, based on the interference features of the early morning period, the single image prompt could be: "The false alarms for morning fog are extremely high at this time. Pay attention to distinguishing between morning fog and smoke. Morning fog usually covers a wide area, is close to the ground, and has no obvious upward vector; wildfire smoke usually has a clear source and drifts upward. If it cannot be distinguished, it tends to be judged as morning fog."
[0081] For daytime periods, to maximize the multimodal large model's ability to find all relevant targets (i.e., improve recall), high-recall cue words are used. The goal is to ensure that every potential source of smoke or fire is detected by the multimodal large model. Based on the interference characteristics during the daytime, single-image cue words could be: "Pay close attention to gray and black smoke, noting the rising and spreading trends of the smoke. If the smoke source is located in a vegetation area and is spreading, it is considered a potential wildfire hazard."
[0082] For nighttime periods, negative constraint prompts can be used to guide the multimodal large model to ignore interference items and targets that do not need attention, thereby filtering out easily confused false targets. Based on the interference characteristics of nighttime periods, single-image prompts could be: "It is currently nighttime; please strictly exclude point-like strong light sources. If the target area is a fixed bright spot without accompanying smoke, it is determined to be light interference. Only when a clearly luminous fire point is detected and accompanied by smoke above is it determined to be a wildfire."
[0083] In the above embodiments, by analyzing the interference characteristics of different time periods, different types of prompt words are set, and the multimodal large model is guided to perform image recognition based on the different types of prompt words, thereby improving the accuracy of target image recognition while ensuring recall.
[0084] In one embodiment of this application, the training process of a multimodal large model is as follows: Figure 4 As shown, it includes:
[0085] Step 401: Generate a sample image dataset based on ordinary sample images of the first scale and difficult sample images of the second scale.
[0086] Among them, the difficult example sample images are those that the object detection model misidentifies, with the second proportion being greater than the first proportion. To reduce misidentification of smoke and wildfires, a difficult example mining sampling strategy is adopted when acquiring sample images. That is, a sample image dataset is generated based on the first proportion of ordinary sample images and the second proportion of difficult example sample images. Both ordinary sample images and difficult example sample images are obtained from the historical database of power transmission line monitoring. Difficult example sample images can include image data of scenes that are easily confused, such as morning fog obscuring objects, backlight halos, factory chimneys, smoke from burning fields, and vehicle exhaust.
[0087] For example, the total number of samples in the sample image dataset is N. The first proportion can be 30%, meaning 30% consists of typical, characteristic wildfire or non-wildfire ordinary sample images. These ordinary sample images are used to maintain the basic recognition capability of the multimodal large model. The second proportion can be 70%, meaning 70% consists of difficult sample images where the object detection model misidentifies, including misidentification or missed identification. The first and second proportions can be set according to actual needs, and this application does not impose any restrictions on them.
[0088] Optional, set For ordinary sample images, For difficult example sample images, The sample image dataset is constructed as follows:
[0089]
[0090] By generating a sample image dataset with a high proportion of difficult example images, the multimodal large model can focus on discriminative features and optimize decision boundaries during the fine-tuning training process when training based on the sample image dataset.
[0091] Step 402: Generate a multimodal sample image dataset based on the sample image dataset.
[0092] To enhance the understanding and anti-interference capabilities of multimodal large models in complex scenes, a multimodal sample image dataset is generated based on the sample image dataset.
[0093] For example, the multimodal sample image dataset includes first-type sample image data, second-type sample image data, third-type sample image data, and fourth-type sample image data; that is, the multimodal sample image dataset includes four types of image-text pairs, each type enhancing different reasoning abilities. Based on the sample image dataset, a multimodal sample image dataset is generated. First, the sample image dataset is divided into four categories, obtaining first-candidate samples, second-candidate samples, third-candidate samples, and fourth-candidate samples. Then, different types of multimodal sample images are generated according to the different types of candidate samples, including the following cases. Let the format of each sample image data in the sample image dataset be... Where I is the image input, Q is the prompt word, and A is the labeled answer.
[0094] In the first case, the first candidate sample is used as the first type of sample image data, and the input is the original image corresponding to the first candidate sample. The prompt Q1 could be, "Is there a risk of wildfire in the image? Please analyze it in conjunction with the vegetation, sky background, and smoke pattern." The first type of sample image data can be used to train the multimodal large model's full-image perception capability, avoiding over-reliance on local features and learning the global context.
[0095] In the second scenario, the second candidate sample is input into the target detection model for detection, generating a second candidate sample carrying a detection box. This second candidate sample with the detection box is then used as the second type of sample image data. The detection box can be superimposed on the original image corresponding to the second candidate sample. The rectangular detection box on At this point, the image input for the second type of sample image data can be represented as: Meanwhile, the prompt Q2 could be, "Is there a risk of wildfire in the area within the red box in the image? Please analyze the difference between the texture within the box and the surrounding environment to determine whether it is real smoke or interference." By using the second type of sample image data, the recall capability of the object detection model can be used as candidate regions to train the multimodal large model's ability to "binary classify and discriminate" candidate regions, with a focus on learning how to identify false positives.
[0096] The third scenario involves generating a third type of sample image data based on sample image data from multiple different scenes within the third candidate sample. Through multi-image comparison analysis, sample image data from multiple different scenes are simultaneously used as image input; for example, the input might be a combination of k images from different scenes. The prompt word Q3 is " Figure 1 The light is from streetlights at night, so there is no risk of wildfire. Figure 2 It was a significant wildfire that occurred at night, and the smoke is spreading and rising. Please compare and analyze. Figure 3Are there any potential risks of wildfires, and explain why? Third-type sample image data is used to improve the few-shot learning ability and analogical reasoning ability of multimodal large models, and the feature boundaries are strengthened by comparing positive and negative samples.
[0097] The fourth scenario involves generating fourth-type sample image data based on sample image data from multiple consecutive time points within the fourth candidate sample. Through continuous multi-frame dynamic analysis, m frames of sample image data from the same monitoring point at consecutive time points are used as image input, such as... The prompt Q4 could be: "These are three consecutive frames of the same scene. Please analyze whether a wildfire has occurred in this scene. Pay special attention to whether the smoke area has moved, expanded in size, or darkened in color, etc." Using this fourth type of sample image data, multimodal large-scale models can be endowed with spatiotemporal reasoning capabilities during training, utilizing dynamic change features to eliminate static interference such as clouds and fixed chimneys.
[0098] Optionally, the prompts corresponding to different types of sample image data can be pre-optimized based on expert experience and stored in a prompt database, and the corresponding prompts can be obtained according to the type of image input of the sample image data.
[0099] Step 403: Label the multimodal sample image dataset to generate a sample dataset.
[0100] After determining the image inputs and prompts in the multimodal sample image dataset, labeled answers are generated for each image input and prompt, thus obtaining the sample dataset.
[0101] Optionally, the annotations for the multimodal sample image dataset can be obtained based on the input port, or the multimodal sample image dataset can be annotated using other models to generate a sample dataset, such as... Figure 5 As shown, it may include:
[0102] Step 501: For any multimodal sample image data in the multimodal sample image dataset, obtain the pre-labeling results of multiple candidate large models annotating the multimodal sample image data.
[0103] For any multimodal sample image in the multimodal sample image dataset, call K candidate large models with different architectures to obtain a candidate large model set. The candidate large models can be open-source multimodal large models, and the selection can be made according to the actual situation; this application does not impose any restrictions on this. The input image I of the multimodal sample image data and the prompt words are input into each candidate large model to generate multiple pre-labeled results. By pre-labeling multiple candidate large models with different architectures, and leveraging the cognitive differences among these candidate large models, diverse pre-labeling results can be generated.
[0104] Step 502: Score each pre-labeled result.
[0105] For the pre-labeled results generated by each candidate large model, another multimodal large model can be called to score each pre-labeled result. Alternatively, each candidate large model can score each pre-labeled result, and then a weighted voting mechanism can be used to obtain the final score of each pre-labeled result.
[0106] For example, the above pre-annotation results The data is then input into each candidate large model, and each candidate large model scores the pre-labeled results. For example, taking candidate large models A, B, and C as an example, more candidate large models can be included. Model A's scores for each pre-labeled result are (Model A: 93, Model B: 85, Model C: 72), Model B's scores are (Model A: 78, Model B: 91, Model C: 86), and Model C's scores are (Model A: 84, Model B: 89, Model C: 92). Optionally, to avoid models naturally overestimating their own scores, cross-voting can be used, where each candidate large model can only score the pre-labeled results of other candidate large models besides itself.
[0107] Step 503: Based on the highest-scoring pre-annotated result, generate the annotation results for the multimodal sample image data.
[0108] Optionally, based on the scores of each candidate large model for each pre-labeled result, several candidates can be selected for each candidate large model, such as the two best candidate large models. The selected candidate large model is given 1 point, and the pre-labeled result of the candidate large model with the highest cumulative score is the pre-labeled result with the highest score.
[0109] Optionally, the voting mechanism can be as follows.
[0110]
[0111] in, The indicator function is used to select the candidate large model with the highest score through a voting mechanism.
[0112] Based on the voting results, Model B's pre-labeling result is the highest-scoring pre-labeling result, and it will be used as the labeling result for the multimodal sample image data. To further improve the accuracy of the labeling, the standard result of Model B can be reviewed by business experts in the field of transmission line operation and maintenance, who will conduct a final review of the vote-selected best pre-labeling result A_best. The review will focus on correcting logical errors and supplementing technical terms. The labeling result for the multimodal sample image data will be obtained based on the experts' corrections.
[0113] Step 504: Generate a sample dataset based on the multimodal sample image data and the corresponding annotation results.
[0114] Annotation results are generated for each multimodal sample image data in the multimodal sample image dataset. Based on each multimodal sample image data and the corresponding annotation results, a sample dataset is generated.
[0115] In the above embodiments, pre-labeled results are generated by candidate large models, and the model votes the pre-labeled results to select the pre-labeled result with the highest score as the final labeling result. This ensures the logical accuracy and semantic richness of the labeling results, reduces labeling noise, and improves the efficiency of sample dataset labeling.
[0116] Step 404: Train the initial multimodal large model based on the sample dataset to obtain the multimodal large model.
[0117] In this embodiment, a general multimodal large model is used as the initial multimodal large model. A parameter-efficient fine-tuning technique is employed to train the initial multimodal large model, thereby obtaining the final multimodal large model and enabling the infusion of knowledge from the power-related domain. The initial multimodal large model can be an open-source multimodal large model with strong visual encoding and long-text reasoning capabilities. To reduce training costs and prevent catastrophic forgetting, LoRA (Low-Rank Adaptation) technology is used for fine-tuning.
[0118] The backbone parameters of the initial multimodal large model are frozen, and a low-rank decomposition matrix is injected into the self-attention layer of the initial multimodal large model to obtain the fine-tuned initial multimodal large model.
[0119] The backbone parameters of the initial multimodal large model include the main weights W0 of the visual encoder and the large language model. A trainable low-rank decomposition matrix is injected into the self-attention layer of the large language model within the initial multimodal large model.
[0120] The fine-tuned initial multimodal large model is trained based on the sample dataset and preset training configuration parameters to obtain the multimodal large model.
[0121] The initial multimodal large model with the injected trainable low-rank decomposition matrix is trained based on the sample dataset and preset training configuration parameters. After training, the multimodal large model is obtained.
[0122] For example, for the weight matrix Express its update quantity as ,in , r is the rank, and .
[0123] The initial multimodal large model after fine-tuning is calculated during inference as follows:
[0124]
[0125] Where x is the model input vector and h is the model output vector. During training, only A and B are updated, and W0 holds the gradient at rest.
[0126] The loss function uses the standard cross-entropy loss, and the optimization objective is to maximize the probability of generating correct analytical text.
[0127]
[0128] in, The parameters for the LoRA module are A and B as mentioned above, L is the length of the generated text, I is the input image, and Q is the prompt word.
[0129] The training configuration parameters also include an optimizer, employing AdamW (Adaptive Moment Estimation with Weight Decay), with weight decay... The learning rate is The cosine annealing strategy is adopted, as shown in the following formula:
[0130]
[0131] in, The current learning rate, To minimize the learning rate, To achieve the maximum learning rate, This represents the current iteration step. This represents the maximum number of iterations.
[0132] Batch size is dynamically adjusted based on GPU memory capacity, in conjunction with gradient accumulation. Input resolution maintains the high resolution supported by the initial multimodal large model, such as 448x448, to preserve smoke texture details.
[0133] By constructing the sample dataset and training configuration parameters as described above, the LoRA module learns specific semantic mappings for power grid scenarios. For example, the semantic mapping of "grayish-white clumps" is "morning fog" in the early morning and "potential smoke" in the dry afternoon. After training, a multimodal large model is obtained.
[0134] In the above embodiments, low-rank adaptive technology is used to fine-tune the initial multimodal large model. Based on freezing the pre-trained backbone parameters, professional knowledge from the power transmission inspection domain is injected to achieve rapid alignment of the model's general inference capabilities with power grid business rules at low cost. LoRA technology requires only a small amount of GPU memory and computing resources to complete domain fine-tuning, eliminating the need to train the initial multimodal large model from scratch and reducing computational costs. Simultaneously, the model voting and labeling mechanism ensures the high quality of the sample dataset, resulting in a more accurate multimodal large model.
[0135] In embodiments of this application, a method for identifying smoke and wildfires is provided, including a model training stage and a model recognition stage, such as... Figure 6 As shown, it includes:
[0136] Step 601: Generate a sample image dataset based on ordinary sample images of the first scale and difficult sample images of the second scale.
[0137] Step 602: Generate a multimodal sample image dataset based on the sample image dataset.
[0138] Step 603: For any multimodal sample image data in the multimodal sample image dataset, obtain the pre-labeling results of multiple candidate large models annotating the multimodal sample image data.
[0139] Step 604: Score each pre-labeled result.
[0140] Step 605: Based on the highest-scoring pre-annotated result, generate the annotation results for the multimodal sample image data.
[0141] Step 606: Generate a sample dataset based on the multimodal sample image data and the corresponding annotation results.
[0142] Step 607: Train the initial multimodal large model based on the sample dataset to obtain the multimodal large model.
[0143] Step 608: Use a preset target detection model to identify smoke and wildfire in the target image and determine the preliminary identification results.
[0144] Step 609: If the preliminary identification result indicates that there is smoke and wildfire in the target image, then obtain a single image prompt word, and input the target image and the single image prompt word into a preset multimodal large model to obtain the advanced identification result of the target image.
[0145] Step 610: If the advanced recognition result is that there is smoke and wildfire in the target image, then obtain the historical image and multi-image prompt words of the previous moment of the target image acquisition time, and input the historical image, target image and multi-image prompt words into the multimodal large model to obtain the smoke and wildfire recognition result of the target image.
[0146] In the above embodiments, the logical reasoning capabilities of a multimodal large model are used for further identification, and different inputs correspond to different strategy prompts. Prompts for different time periods are applied based on the acquisition time of the target image, effectively distinguishing easily confused targets such as morning fog, lights, and chimneys. During the verification of the smoke and wildfire identification method proposed in this application using a test set, compared to traditional target detection algorithms, this method reduced the false alarm rate by more than 60% in the early morning and at night, significantly reducing the workload of manual verification in the monitoring center and allowing maintenance personnel to focus on handling real high-risk hazards.
[0147] By constructing a sample image dataset using a second proportion of difficult example images and employing LoRA fine-tuning technology, the multimodal large model learns most of the scenarios where traditional algorithms would make mistakes. It learns to understand input images from a semantic and environmental context perspective, thus exhibiting strong robustness in different environments. The multimodal large model generates natural language analysis reports, such as "excluding lights, as there is no accompanying smoke diffusion," providing interpretable evidence for alarms and assisting operations personnel in making more accurate decisions.
[0148] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.
[0149] Based on the same inventive concept, this application also provides a smoke and wildfire identification device for implementing the smoke and wildfire identification method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more embodiments of the smoke and wildfire identification device provided below can be found in the limitations of the smoke and wildfire identification method described above, and will not be repeated here.
[0150] In one exemplary embodiment, such as Figure 7 As shown, a smoke and wildfire identification device 700 is provided, comprising: an identification module, a first input module, and a second input module, wherein:
[0151] The recognition module is used to identify smoke and wildfires in target images using a preset target detection model and determine the preliminary recognition results.
[0152] The first input module is used to obtain a single image prompt if the preliminary recognition result indicates that there is smoke or wildfire in the target image, and input the target image and the single image prompt into a preset multimodal large model to obtain the advanced recognition result of the target image.
[0153] The second input module is used to obtain the historical image and multiple image prompts of the target image at the previous acquisition time if the advanced recognition result is that there is smoke and wildfire in the target image. The historical image, target image and multiple image prompts are then input into the multimodal large model to obtain the smoke and wildfire recognition result of the target image.
[0154] In one embodiment, the first input module is specifically used to determine a target time period based on the acquisition time; query a prompt word database based on the target time period; and determine a single image prompt word based on the query results. The prompt word database includes the correspondence between different time periods and prompt words.
[0155] In one embodiment, the device further includes a training module for generating a sample image dataset based on ordinary sample images at a first ratio and difficult sample images at a second ratio, wherein the difficult sample images are sample images that the object detection model misidentifies, and the second ratio is greater than the first ratio; and generating a multimodal sample image dataset based on the sample image dataset.
[0156] The multimodal sample image dataset is labeled to generate a sample dataset; the initial multimodal large model is trained based on the sample dataset to obtain the multimodal large model.
[0157] In one embodiment, the multimodal sample image dataset includes first type sample image data, second type sample image data, third type sample image data, and fourth type sample image data. The training module is specifically used to divide the sample image dataset into four categories to obtain first candidate samples, second candidate samples, third candidate samples, and fourth candidate samples; use the first candidate samples as first type sample image data; input the second candidate samples into the target detection model for detection to generate second candidate samples carrying detection boxes, and use the second candidate samples carrying detection boxes as second type sample image data; generate third type sample image data based on sample image data of multiple different scenes in the third candidate samples; and generate fourth type sample image data based on sample image data of multiple consecutive time moments in the fourth candidate samples.
[0158] In one embodiment, the training module is specifically used to obtain pre-annotation results of multiple candidate large models annotating the multimodal sample image data for any multimodal sample image data in the multimodal sample image dataset; score each pre-annotation result; generate annotation results of the multimodal sample image data based on the highest-scoring pre-annotation result; and generate a sample dataset based on each multimodal sample image data and the corresponding annotation results.
[0159] In one embodiment, the training module is specifically used to freeze the backbone parameters of the initial multimodal large model and inject a low-rank decomposition matrix into the self-attention layer of the initial multimodal large model to obtain a fine-tuned initial multimodal large model; and to train the fine-tuned initial multimodal large model based on the sample dataset and preset training configuration parameters to obtain the multimodal large model.
[0160] The modules in the aforementioned smoke and wildfire detection device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.
[0161] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 8 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores target image data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network. When executed by the processor, the computer program implements a smoke and wildfire identification method. The display unit of the computer device forms a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.
[0162] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0163] In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to perform the following steps: identifying smoke and wildfire in a target image using a preset target detection model to determine a preliminary identification result; if the preliminary identification result indicates that smoke and wildfire exist in the target image, then a single image prompt word is obtained, and the target image and the single image prompt word are input into a preset multimodal large model to obtain an advanced identification result of the target image; if the advanced identification result indicates that smoke and wildfire exist in the target image, then a historical image and multiple image prompt words from the previous acquisition time of the target image are obtained, and the historical image, the target image, and the multiple image prompt words are input into the multimodal large model to obtain the smoke and wildfire identification result of the target image.
[0164] In one embodiment, when the processor executes the computer program, it further performs the following steps: determining a target time period based on the acquisition time; querying a prompt word database based on the target time period; and determining a single image prompt word based on the query results. The prompt word database includes the correspondence between different time periods and prompt words.
[0165] In one embodiment, when the processor executes the computer program, it further performs the following steps: generating a sample image dataset based on ordinary sample images of a first ratio and difficult sample images of a second ratio, wherein the difficult sample images are sample images that the object detection model misidentifies, and the second ratio is greater than the first ratio; generating a multimodal sample image dataset based on the sample image dataset; labeling the multimodal sample image dataset to generate a sample dataset; and training an initial multimodal large model based on the sample dataset to obtain a multimodal large model.
[0166] In one embodiment, the multimodal sample image dataset includes first type sample image data, second type sample image data, third type sample image data, and fourth type sample image data. When the processor executes the computer program, it further implements the following steps: dividing the sample image dataset into four categories to obtain first candidate samples, second candidate samples, third candidate samples, and fourth candidate samples; using the first candidate samples as first type sample image data; inputting the second candidate samples into the target detection model for detection to generate second candidate samples carrying detection boxes, and using the second candidate samples carrying detection boxes as second type sample image data; generating third type sample image data based on sample image data of multiple different scenes in the third candidate samples; and generating fourth type sample image data based on sample image data of multiple consecutive time moments in the fourth candidate samples.
[0167] In one embodiment, when the processor executes the computer program, it further performs the following steps: for any multimodal sample image data in the multimodal sample image dataset, obtain the pre-annotation results of multiple candidate large models annotating the multimodal sample image data; score each pre-annotation result; generate the annotation result of the multimodal sample image data based on the pre-annotation result with the highest score; and generate a sample dataset based on each multimodal sample image data and the corresponding annotation result.
[0168] In one embodiment, when the processor executes the computer program, it further performs the following steps: freezing the backbone parameters of the initial multimodal large model and injecting a low-rank decomposition matrix into the self-attention layer of the initial multimodal large model to obtain a fine-tuned initial multimodal large model; training the fine-tuned initial multimodal large model based on the sample dataset and preset training configuration parameters to obtain a multimodal large model.
[0169] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, it performs the following steps: identifying smoke and wildfire in a target image using a preset target detection model to determine a preliminary identification result; if the preliminary identification result indicates that smoke and wildfire exist in the target image, a single image prompt word is obtained, and the target image and the single image prompt word are input into a preset multimodal large model to obtain an advanced identification result of the target image; if the advanced identification result indicates that smoke and wildfire exist in the target image, a historical image and multiple image prompt words from the previous acquisition time of the target image are obtained, and the historical image, the target image, and the multiple image prompt words are input into the multimodal large model to obtain a smoke and wildfire identification result of the target image.
[0170] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: determining a target time period based on the acquisition time; querying a prompt word database based on the target time period; and determining a single image prompt word based on the query results. The prompt word database includes the correspondence between different time periods and prompt words.
[0171] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: generating a sample image dataset based on ordinary sample images of a first proportion and difficult sample images of a second proportion, wherein the difficult sample images are sample images that the object detection model misidentifies, and the second proportion is greater than the first proportion; generating a multimodal sample image dataset based on the sample image dataset; labeling the multimodal sample image dataset to generate a sample dataset; and training an initial multimodal large model based on the sample dataset to obtain a multimodal large model.
[0172] In one embodiment, the multimodal sample image dataset includes first type sample image data, second type sample image data, third type sample image data, and fourth type sample image data. When the computer program is executed by the processor, it further performs the following steps: dividing the sample image dataset into four categories to obtain first candidate samples, second candidate samples, third candidate samples, and fourth candidate samples; using the first candidate samples as first type sample image data; inputting the second candidate samples into the target detection model for detection to generate second candidate samples carrying detection boxes, and using the second candidate samples carrying detection boxes as second type sample image data; generating third type sample image data based on sample image data of multiple different scenes in the third candidate samples; and generating fourth type sample image data based on sample image data of multiple consecutive time moments in the fourth candidate samples.
[0173] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: for any multimodal sample image data in the multimodal sample image dataset, obtain the pre-annotation results of multiple candidate large models annotating the multimodal sample image data; score each pre-annotation result; generate the annotation result of the multimodal sample image data based on the pre-annotation result with the highest score; and generate a sample dataset based on each multimodal sample image data and the corresponding annotation result.
[0174] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: freezing the backbone parameters of the initial multimodal large model and injecting a low-rank decomposition matrix into the self-attention layer of the initial multimodal large model to obtain a fine-tuned initial multimodal large model; training the fine-tuned initial multimodal large model based on the sample dataset and preset training configuration parameters to obtain a multimodal large model.
[0175] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps: identifying smoke and wildfire in a target image using a preset target detection model to determine a preliminary identification result; if the preliminary identification result indicates that smoke and wildfire exist in the target image, obtaining a single-image prompt word and inputting the target image and the single-image prompt word into a preset multimodal large model to obtain an advanced identification result for the target image; if the advanced identification result indicates that smoke and wildfire exist in the target image, obtaining a historical image and multiple-image prompt words from the previous acquisition time of the target image, and inputting the historical image, the target image, and the multiple-image prompt words into the multimodal large model to obtain a smoke and wildfire identification result for the target image.
[0176] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: determining a target time period based on the acquisition time; querying a prompt word database based on the target time period; and determining a single image prompt word based on the query results. The prompt word database includes the correspondence between different time periods and prompt words.
[0177] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: generating a sample image dataset based on ordinary sample images of a first proportion and difficult sample images of a second proportion, wherein the difficult sample images are sample images that the object detection model misidentifies, and the second proportion is greater than the first proportion; generating a multimodal sample image dataset based on the sample image dataset; labeling the multimodal sample image dataset to generate a sample dataset; and training an initial multimodal large model based on the sample dataset to obtain a multimodal large model.
[0178] In one embodiment, the multimodal sample image dataset includes first type sample image data, second type sample image data, third type sample image data, and fourth type sample image data. When the computer program is executed by the processor, it further performs the following steps: dividing the sample image dataset into four categories to obtain first candidate samples, second candidate samples, third candidate samples, and fourth candidate samples; using the first candidate samples as first type sample image data; inputting the second candidate samples into the target detection model for detection to generate second candidate samples carrying detection boxes, and using the second candidate samples carrying detection boxes as second type sample image data; generating third type sample image data based on sample image data of multiple different scenes in the third candidate samples; and generating fourth type sample image data based on sample image data of multiple consecutive time moments in the fourth candidate samples.
[0179] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: for any multimodal sample image data in the multimodal sample image dataset, obtain the pre-annotation results of multiple candidate large models annotating the multimodal sample image data; score each pre-annotation result; generate the annotation result of the multimodal sample image data based on the pre-annotation result with the highest score; and generate a sample dataset based on each multimodal sample image data and the corresponding annotation result.
[0180] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: freezing the backbone parameters of the initial multimodal large model and injecting a low-rank decomposition matrix into the self-attention layer of the initial multimodal large model to obtain a fine-tuned initial multimodal large model; training the fine-tuned initial multimodal large model based on the sample dataset and preset training configuration parameters to obtain a multimodal large model.
[0181] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.
[0182] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.
[0183] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.
[0184] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.
Claims
1. A method for identifying smoke and wildfires, characterized in that, The method includes: The target image is used to identify smoke and wildfires using a pre-set target detection model to determine the preliminary identification results. If the preliminary identification result indicates that there is smoke or wildfire in the target image, then a single image prompt word is obtained, and the target image and the single image prompt word are input into a preset multimodal large model to obtain the advanced identification result of the target image; If the advanced recognition result indicates that there is smoke and wildfire in the target image, then the historical image and multiple image prompts of the target image at the previous acquisition time are obtained, and the historical image, the target image, and the multiple image prompts are input into the multimodal large model to obtain the smoke and wildfire recognition result of the target image; the multiple image prompts are used to indicate whether there are motion vector changes in the suspected smoke and wildfire area; The acquisition of single-image prompts includes: The target time period is determined based on the data collection time. Based on the target time period, a prompt word database is queried, and the single image prompt word is determined according to the query results. The prompt word database includes the correspondence between different time periods and prompt words.
2. The method according to claim 1, characterized in that, The training process of the multimodal large model includes: A sample image dataset is generated based on ordinary sample images of a first ratio and difficult sample images of a second ratio, wherein the difficult sample images are sample images that the target detection model misidentifies, and the second ratio is greater than the first ratio. Generate a multimodal sample image dataset based on the sample image dataset; The multimodal sample image dataset is labeled to generate a sample dataset; The initial multimodal large model is trained based on the sample dataset to obtain the multimodal large model.
3. The method according to claim 2, characterized in that, The multimodal sample image dataset includes first-type sample image data, second-type sample image data, third-type sample image data, and fourth-type sample image data. Generating the multimodal sample image dataset based on the sample image dataset includes: The sample image dataset is divided into four categories to obtain the first candidate sample, the second candidate sample, the third candidate sample, and the fourth candidate sample; The first candidate sample is used as the first type of sample image data; The second candidate sample is input into the target detection model for detection, generating a second candidate sample with a detection box, and the second candidate sample with the detection box is used as the second type of sample image data. The third type of sample image data is generated based on sample image data of multiple different scenes in the third candidate samples; The fourth type of sample image data is generated based on the sample image data at multiple consecutive time points in the fourth candidate sample.
4. The method according to claim 2, characterized in that, The step of annotating the multimodal sample image dataset to generate a sample dataset includes: For any multimodal sample image data in the multimodal sample image dataset, obtain the pre-labeling results of multiple candidate large models annotating the multimodal sample image data; Each of the pre-labeled results is scored; Based on the highest-scoring pre-annotation result, the annotation result of the multimodal sample image data is generated; The sample dataset is generated based on the multimodal sample image data and the corresponding annotation results.
5. The method according to claim 2, characterized in that, The process of training the initial multimodal large model based on the sample dataset to obtain the multimodal large model includes: Freeze the backbone parameters of the initial multimodal large model and inject a low-rank decomposition matrix into the self-attention layer of the initial multimodal large model to obtain the fine-tuned initial multimodal large model; The fine-tuned initial multimodal large model is trained based on the sample dataset and preset training configuration parameters to obtain the multimodal large model.
6. A smoke and wildfire detection device, characterized in that, The device includes: The recognition module is used to identify smoke and wildfires in target images using a preset target detection model and determine the preliminary recognition results. The first input module is used to obtain a single image prompt word if the preliminary recognition result is that there is smoke and wildfire in the target image, and input the target image and the single image prompt word into a preset multimodal large model to obtain the advanced recognition result of the target image; The second input module is used to obtain the historical image and multiple image prompts of the target image at the previous acquisition time if the advanced recognition result is that there is smoke and wildfire in the target image, and input the historical image, the target image, and the multiple image prompts into the multimodal large model to obtain the smoke and wildfire recognition result of the target image; the multiple image prompts are used to indicate whether there are motion vector changes in the suspected smoke and wildfire area; The first input module is specifically used to determine a target time period based on the acquisition time; query a prompt word database based on the target time period; and determine the single image prompt word based on the query results. The prompt word database includes the correspondence between different time periods and prompt words.
7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.
8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.
9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.
Citation Information
Patent Citations
Smoke and fire detection method and device and electronic equipment
CN118135187A
Forest fire identification method and device for power transmission corridor, electronic equipment and storage medium
CN120612609A