Thought chain-based multi-modal forest fire target detection method and system

Abstract

The application discloses a kind of multi-modal forest fire target detection method and system based on thought chain, belong to forestry intelligent monitoring technical field.It includes the following steps: collecting and pre-processing paired visible light and thermal infrared forestry scene image, after data labeling, pre-training large visual language model is trained using supervised fine-tuning, and the basic perception model is constructed;Composite reward function is constructed, and the reasoning process of the basic perception model is optimized using group strategy optimization reinforcement learning algorithm;By inputting the preset thought chain prompt to the optimized basic perception model, the model synchronously generates structured natural language text.Compared with the prior art, the application has the advantages that the "black box" model is converted into an interpretable and traceable decision report by the thought chain, greatly enhancing the credibility of the system.

Description

Technical Field

[0001] This invention relates to the field of intelligent forestry monitoring technology, and more specifically, to a multimodal forest fire target detection method and system based on thought chain. Background Technology

[0002] Modern forestry management and disaster prevention heavily rely on target detection technology, which plays a crucial role in early warning of forest fires, monitoring of illegal logging activities, pest and disease monitoring, and wildlife protection. Traditional solutions often rely on single visible light cameras, but are limited by low-visibility scenarios such as nighttime, heavy fog, and forest shadows, making them prone to missed detections and false alarms.

[0003] To overcome the limitations of visible light imaging, multimodal monitoring schemes combining visible light and thermal infrared imagers have gradually become a research hotspot. Thermal infrared sensors capture the thermal radiation of targets, can penetrate optical interference, and achieve all-weather monitoring, making them particularly suitable for fire detection and nighttime target identification. However, existing technologies still suffer from two major bottlenecks. First, there is insufficient robustness in weak alignment scenarios. Existing multimodal fusion algorithms (whether feature-level or decision-level fusion) heavily rely on the ideal assumption of pixel-level precise alignment of multimodal images. However, in actual forestry monitoring scenarios, factors such as installation errors, equipment vibration, optical distortion, and thermal drift lead to unavoidable spatial 'weak alignment' problems between visible light and thermal infrared images. The fusion mechanisms of existing methods cannot effectively adapt to and handle such spatial deviations, resulting in target positioning offsets, feature mismatches, and a significant decrease in detection performance in real complex environments. Second, there is the lack of transparency in the decision-making process. Whether it is a traditional deep learning network or an emerging large-scale visual language model, its internal decision-making mechanism is a 'black box' that users cannot see. When the model issues a fire alarm, managers cannot trace the basis of its judgment (whether it's smoke, a hotspot, or a combination of both); when no alarm is issued, it's impossible to assess whether there is no risk or the model has missed a warning. This lack of decision-making logic makes it difficult for the system to establish the necessary level of trust in high-reliability safety applications such as forestry, and also poses a significant challenge to error detection and targeted optimization of the model.

[0004] In summary, existing technologies struggle to effectively address the weak spatial alignment problem of multimodal images in real forestry environments, resulting in insufficient robustness of detection models. Furthermore, the opaque decision-making process of these models fails to meet the application requirements of high reliability and traceability in forestry monitoring. Therefore, developing a multimodal forestry target detection technology that can adapt to weakly aligned scenarios and whose decision-making process is completely transparent and interpretable has become a core technical challenge that urgently needs to be overcome in this field. Summary of the Invention

[0005] To address the aforementioned problems in existing technologies, this invention provides a multimodal forest fire target detection method and system based on thought chain. By introducing thought chain prompts to drive the model to synchronously generate structured reasoning text during the detection process, the internal decision-making process of traditional models is transformed into a readable and verifiable natural language description, achieving transparency and traceability of decision logic. Simultaneously, through multimodal image annotation design adapted to weak alignment scenarios, construction of a composite reward function that integrates weak alignment robustness, and group policy optimization reinforcement learning algorithm, the model is endowed with adaptive capabilities for weak alignment scenarios in multimodal image space.

[0006] The objective of this invention is achieved through the following technical solutions.

[0007] The summary section of this application is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description section below. This summary section is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0008] As a first aspect of this application, this application provides a multimodal forest fire target detection method based on thought chain, comprising the following steps:

[0009] Step S1: Collect pairs of visible light and thermal infrared forestry scene images and annotate the data to train a pre-trained large visual language model using supervised fine-tuning, so that the large visual language model has the basic mapping ability from multimodal images to target descriptive text, and build a basic perception model. Step S2: Construct a composite reward function that includes detection accuracy, interpretation quality, and weak alignment robustness, and use a group policy optimization reinforcement learning algorithm to optimize the reasoning process of the basic perception model; Step S3: By inputting preset thought chain prompts into the optimized basic perception model, the model is driven to generate structured natural language text simultaneously when performing target detection. The natural language text includes independent evidence extraction, cross-modal association verification, fusion decision and interpretation. Step S4: Deploy the optimized basic perception model on the online inference server, receive the real-time multimodal image stream from the front end, perform the transparent detection inference process, thereby obtaining the forestry target detection results and their decision interpretation, and push the information containing the target detection results and decision interpretation text to the forestry monitoring platform.

[0010] Furthermore, in step S1, the basic perception model construction process specifically includes: By deploying front-end acquisition devices in the forestry environment, timestamp-aligned visible light and thermal infrared image pairs are collected simultaneously, and after preprocessing, a multimodal forestry dataset is constructed. Key forestry targets within images in a multimodal forestry dataset are labeled, with the labeling information including the target category and its bounding box in each modality of the image. We selected a large visual language model with multimodal dialogue capabilities as the basic framework and constructed a multimodal encoder-decoder architecture that includes a visual encoder, a feature projection layer, and a large language model decoder. The labeled data is converted into a dialogue format, and the selected model is trained using a full-parameter fine-tuning method until the model's loss on the validation set converges, thus obtaining the basic perception model.

[0011] Furthermore, in step S1, data annotation includes: Boundary boxes are labeled for the same target in pairs of visible light and thermal infrared images. The center point coordinates of the labeled boundary boxes for the same target in the visible light and thermal infrared images are allowed to be less than or equal to 15 pixels apart, so as to simulate slight registration errors in real-world scenes.

[0012] Furthermore, in step S1, the large visual language model adopts a multimodal encoder-decoder architecture, including a visual encoder, a feature projection layer, and a large language model decoder. The visual encoder extracts and stitches together the deep visual features from the visible light and thermal infrared images respectively; the feature projection layer projects the stitched visual features onto the text feature space; and the large language model decoder receives the projected visual features and the user's thought chain prompts to generate structured text.

[0013] Furthermore, in step S2, the group policy optimization reinforcement learning algorithm specifically includes: For an input case, the basic perception model generates multiple inference text candidate outputs; Each output is automatically scored using a composite reward function; Calculate the normalized advantage of each output relative to the within-group reward statistic; By updating the model parameters by maximizing the objective function of group policy optimization, the model is made to tend to generate inference text with higher scores.

[0014] Furthermore, the composite reward function is composed of a weighted average of detection accuracy reward, interpretation quality reward, and weak alignment robustness reward; The accuracy reward evaluation assesses the intersection-union ratio (IoU) of the final detection result with the true label; the quality reward evaluation assesses whether the output text follows a predefined structured paradigm, including whether it contains elements used to identify the reasoning process. <think>Labels and identification of test results <answer>Tags: Weak alignment, robustness, reward guarantee, model, final answer, consistency with reasoning process in thought chain.

[0015] Furthermore, in step S3, the thought chain prompting model generates reasoning step by step according to the following logic: Independent evidence extraction: Analyze target features in visible light images and thermal infrared images respectively; Cross-modal association verification: verifying the spatial proximity and feature consistency of evidence from different modalities; Fusion Decision Interpretation: Determines targets based on correlation results and outputs interpretations.

[0016] As a second aspect of this application, this application also provides a multimodal forestry target detection system, comprising: Front-end data acquisition equipment is used to simultaneously acquire visible light and thermal infrared images; The offline training server is equipped with a basic perception model building module and a group policy-based inference process optimization module, which are used to perform initial training and deep logic optimization of the basic perception model. The online inference server, equipped with a model inference module and a forestry monitoring platform, is used to load the optimized basic perception model, process real-time image streams, perform transparent detection inference, and visualize the target detection results and corresponding decision explanation text.

[0017] Furthermore, the front-end data acquisition equipment is a multimodal gimbal camera that integrates visible light and long-wave infrared sensors and is deployed on a fixed watchtower or drone, and ensures that the timestamp difference of the image pairs is less than 20 milliseconds through GPS second pulse signals.

[0018] Furthermore, the inference process optimization module based on group policy optimization in the offline training server uses a composite reward function and a group policy optimization algorithm to perform reinforcement learning optimization on the inference process of the basic perception model. The composite reward function includes detection accuracy, interpretation quality and weak alignment robustness. The model reasoning module in the online reasoning server drives the model to generate structured text that includes independent evidence extraction, cross-modal association verification, and fusion decision-making and interpretation by inputting preset thought chain prompts; The forestry monitoring platform in the online inference server is used to receive and visualize structured text.

[0019] Compared with the prior art, the advantages of this invention are: (1) By inputting preset thought chain prompts into the optimized basic perception model, the model can be driven to generate structured natural language text step by step. After the model is deployed on the online inference server, it can output the forestry target detection results and corresponding decision explanations in a synchronous manner, thereby making the decision-making process of the model verifiable and traceable. (2) When annotating data, the same target is allowed to have a deviation of no more than 15 pixels at the center point of the bounding box in visible light and thermal infrared images, and the composite reward function includes a weak alignment robustness reward, which can guide the model to actively verify the spatial proximity and feature consistency of different modal evidence during inference, so as to achieve stable response to the weak alignment problem in actual monitoring without relying on pixel-level precise alignment images. (3) The group strategy optimization reinforcement learning algorithm is used to optimize the reasoning process of the basic perception model. After scoring by the composite reward function and calculating the normalized advantage, the parameters are updated, which makes the model better at integrating weak and complementary cues from visible light and thermal infrared images, thereby improving the probability of identifying early targets of forest fires. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the overall process of a transparent multimodal forestry scene target detection method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the logical architecture of a transparent multimodal forestry scene target detection system provided in an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating the detailed logical flow of reasoning and interpretation generated by the thought chain reasoning module in this embodiment of the invention. Detailed Implementation

[0021] To enable those skilled in the art to more fully understand the technical solution of the present invention, the embodiments of the present invention will be described in detail below in conjunction with a specific application scenario.

[0022] like Figures 1 to 3 As shown, the multimodal forest fire target detection method based on thought chain of the present invention includes the following steps: S1. Constructing a basic perception model: This step aims to build a basic perceptual model for initially understanding multimodal images and generating target descriptions. This process is completed offline and includes the following sub-steps: S101. Constructing a multimodal forestry dataset First, a multimodal forestry dataset specifically designed for forestry target detection is constructed. The multimodal dataset consists of paired, time-synchronized visible light and thermal infrared images, covering diverse forestry scenes and targets.

[0023] This embodiment uses a multimodal pan-tilt camera, integrating visible light and long-wave infrared sensors, deployed on a fixed observation tower as the front-end acquisition device to collect multimodal forestry data. Hardware synchronization is achieved using GPS second pulse signals, ensuring that the timestamp difference between each pair of visible light and thermal infrared images is less than 20 milliseconds. The data collection activities covered the critical fire prevention periods in spring and autumn, as well as different meteorological conditions (such as early morning, afternoon, and heavy fog) to ensure data diversity and representativeness. The final result is a raw dataset containing several image pairs. The raw dataset covers various key forestry targets, such as early smoke, open fire spots, illegal logging vehicles and patrol personnel, as well as a large amount of forest background and easily confused interference (such as mountain morning fog, hot rocks, and wildlife).

[0024] In one specific embodiment, multimodal PTZ cameras, integrating visible light sensors (1920×1080 resolution, 30fps) and long-wave infrared sensors (640×512 resolution, thermal sensitivity ≤50mK), are deployed on top of multiple fixed watchtowers for data acquisition. A hardware-level GPS second pulse signal synchronization mechanism ensures that the timestamp difference between each set of acquired visible light and thermal infrared image pairs is less than 20 milliseconds. The acquisition scenarios cover all four seasons of 2024, including key time points such as early morning, afternoon, and evening during the spring fire prevention period (March-May) and autumn fire prevention period (September-November), as well as special meteorological conditions such as heavy fog and light snow, acquiring data including... Multimodal forestry data consisting of 62,500 valid image pairs, including 5,000 sets of early smoke generated by burning damp leaves within safe control zones, simulating the faint, transparent smoke at the beginning of a forest fire; 3,000 sets of open fire points, simulated at night by alcohol lamps or small campfires; 2,000 sets of illegal logging vehicles, including pickup trucks and excavators partially covered by camouflage materials such as canvas and branches; 1,500 sets of patrol personnel, including patrolmen active during the day and night, with nighttime primarily relying on thermal infrared signal characteristics; and 51,000 sets of background / interference items, including a large number of normal forest backgrounds, as well as easily confused interferences such as mountain morning mist, exposed hot rocks, and live animals (deer, wild boars).

[0025] Subsequently, the original dataset undergoes preprocessing including cleaning and standardization, specifically including: The Laplacian variance of each image is calculated using the OpenCV library (with a threshold of 100), automatically removing blurry images caused by camera shake or defocus. This step removes approximately 2% of the data.

[0026] By using the camera SDK to perform non-uniformity correction and temperature linearization on thermal infrared images, pixel values ​​are converted into accurate Celsius temperature scales, providing the model with physically meaningful temperature information.

[0027] An image interpolation algorithm (such as bicubic interpolation) is used to upsample the low-resolution thermal infrared image to the same size as the visible light image (1920×1080), aligning it with the visible light image size to achieve spatial dimension alignment in subsequent processing.

[0028] Specifically, the image pairs in the preprocessed original dataset are labeled according to a predefined labeling specification. The core of the labeling process is to use the image labeling tool LabelImg to create bounding boxes on both images of the same target in each image pair. The labeling information is tailored to each key forestry target (including early smoke, open fire points, illegal logging vehicles, and patrol personnel), drawing bounding boxes on both the visible light and thermal infrared images. To simulate realistic weak alignment scenarios, a maximum deviation of 15 pixels between the center points of the bounding boxes of the same target in the two modal images is allowed. Finally, a labeled multimodal forestry dataset containing various target categories and background interference is generated.

[0029] In one specific embodiment, the constructed multimodal forestry dataset includes 5,000 sets of early smoke, 3,000 sets of open flame ignition points, 2,000 sets of illegal logging vehicles, 1,500 sets of patrol personnel, and 51,000 sets of background / interference items.

[0030] S102, Model Selection and Construction In this invention, a large visual language model pre-trained on general multimodal data is used, and it is fine-tuned under supervision by a constructed multimodal forestry dataset.

[0031] Specifically, this embodiment uses the Qwen2.5-VL large-scale visual language model as the basic framework, and its multimodal encoder-decoder architecture specifically includes: Visual encoder: The Vision Transformer built into Qwen2.5-VL is used as the encoder to extract deep visual features from the input visible light image and thermal infrared image respectively. After extracting the deep visual features from the visible light and thermal infrared images, the two feature sequences are concatenated.

[0032] Feature projection layer: Composed of a two-layer multilayer perceptron, responsible for projecting the visual features output by the visual encoder onto a text feature space compatible with large visual language models.

[0033] Large-scale language model decoder: Using the language model built into the large-scale visual language model, it is responsible for receiving the visual features mapped by the feature projection layer, and combining them with the text prompts input by the user (thought chain prompts in subsequent steps) to generate structured target descriptions and inference text.

[0034] This yields an untrained model architecture that is structurally well-suited for handling multimodal inputs, preparing it for the next step of supervised fine-tuning.

[0035] S103, Supervision and Fine-tuning Based on a multimodal forestry dataset, the constructed model architecture is trained on a training server using the multimodal forestry dataset. Specifically, this includes: Training data preparation: The bounding boxes (including target category names and bounding box coordinates) and corresponding image file paths of the multimodal forestry dataset are converted into a structured dialogue format for supervised fine-tuning of large-scale visual language models. This format clearly defines the text structure of user commands and model responses.

[0036] Training environment and hyperparameter settings: Training was conducted on a server configured with four NVIDIA L40 GPUs. The training hyperparameters included: using the AdamW optimizer, setting the initial learning rate to 1e-6, using cosine annealing for learning rate scheduling, and setting the global batch size to 8. The training method was full parameter fine-tuning.

[0037] Training execution and termination: The multimodal forestry dataset is divided into training, validation, and test sets in an 8:1:1 ratio. During training, the model's performance on the validation set is continuously monitored. When the validation set loss no longer decreases significantly for three consecutive training epochs (the absolute value of the decrease is less than 1e-5), an early stopping mechanism is triggered, terminating the training.

[0038] Final output: Save the model weights that perform best on the validation set throughout the entire training process. The final output model is the basic perception model required by this invention. This model can perform basic target recognition and description on input visible light-thermal infrared image pairs, but its decision-making process is still a "black box" and needs to be optimized through step S2.

[0039] In a specific application scenario, the training server runs on Ubuntu 20.04, uses PyTorch 2.4 as the deep learning framework, and utilizes the SWIFT framework for training management. The entire supervised fine-tuning process lasts approximately 8 hours, ultimately yielding a stable basic perception model that can be used for subsequent steps.

[0040] In a specific application scenario, the training data is organized as JSON files. Each piece of training data is a JSON object, where the "images" field contains ordered paths to visible light and thermal infrared image files, and the "messages" field contains a list of conversations, consisting of the following roles and contents: The "system" role defines the basic behavior of the model (e.g., "content": "You are a professional forestry monitoring assistant.").

[0041] The "user" role contains standardized text instructions to guide the model in performing detection tasks (e.g., "content": "Please detect and describe key forestry targets in this visible light image and its corresponding thermal infrared image."). During actual loading, the text... The placeholder will be replaced by the image data of the corresponding path in the "images" field.

[0042] The "assistant" role contains the expected response text that conforms to predefined specifications. This text is a structured JSON string containing all detected target information.

[0043] S2. Optimization of the reasoning process based on group policy optimization This step is an offline, specialized optimization phase based on the basic perception model obtained in step S1. By introducing reinforcement learning algorithms, the reasoning logic of the basic perception model is guided to more closely resemble the rigorous thinking of human experts, thereby enhancing the reliability and transparency of the final decision.

[0044] S201. Establishing Optimization Objectives The optimization target in this step is the text generated by the basic perception model after receiving the thought chain prompts. The optimization goal is to make the reasoning process meet the requirements of the following three dimensions: Accuracy: The reasoning process should support correct detection results.

[0045] Logicality: The reasoning should be clear in steps, logically coherent, and conform to the established process of "independent evidence extraction - cross-modal correlation verification - fusion decision interpretation".

[0046] Robustness: When faced with weakly aligned data or interfering information, the inference process should remain stable and avoid logical contradictions (i.e., "reward cracking", such as the target described in the inference not corresponding to the final detection box).

[0047] To achieve the optimization objective, this invention employs the Group Policy Optimization (GRPO) algorithm. GRPO is an efficient reinforcement learning algorithm whose core idea is to allow the model to generate multiple candidate outputs to form a "group" for the same input, and then use a composite reward function to automatically score and compare the outputs of this "group," thereby guiding the model to update its parameters in the direction of obtaining higher rewards.

[0048] S202, Optimize Data Preparation From the test set obtained in step S1, 1000 cases containing complex scenarios are selected, such as image pairs with weak target signals, strong interference, or significant weak alignment.

[0049] For each selected case, a high-quality thought chain text conforming to the logic of "independent evidence extraction - cross-modal association verification - fusion decision interpretation" is provided as an optimization reference. This text is generated based on a pre-defined reasoning logic specification, and its content details the complete reasoning chain from extracting evidence from bimodal images, performing cross-modal association verification, and finally arriving at the detection conclusion.

[0050] S203, Reward Function Design To automatically and comprehensively evaluate the quality of the thought chains generated by the model, this step defines a composite reward function. This function comprehensively evaluates performance across the following three dimensions through a weighted summation: Detection accuracy reward: Evaluates the detection results output by the model (i.e., <answer>The intersection-over-union ratio (IoU) between the bounding boxes in the part and the ground truth bounding boxes. This ensures that the fundamental goal of optimization remains detection accuracy.

[0051] Explanation of quality reward: Evaluates whether the entire text output by the model strictly adheres to a predefined structured format (e.g., it must contain...). <think>and <answer>(The tag is complete, and the internal logic steps are intact).

[0052] Weakly aligned robust rewards: evaluating the model's reasoning process ( <think>(Partial) and the final answer ( <answer>Consistency among parts aims to prevent "reward-cracking" behavior in the model (i.e., fabricating a reasonable reasoning process but arriving at an incorrect answer, or providing a correct answer but with confused reasoning logic). Specifically, the algorithm checks whether the bounding boxes in the final answer can be found in the bounding boxes. <think>Reasonable support is found in the visible light and thermal infrared evidence frames described in part, ensuring the self-consistency of the reasoning.

[0053] In this embodiment, the weights of the three reward components are set to 0.5, 0.2 and 0.3, respectively.

[0054] In a specific embodiment, the composite reward function is expressed as:

[0055] in, The model input consists of a pair of visible light and thermal infrared images and corresponding thought chain cues. For model output, it refers to the model's output for... The generated complete text contains <think>(Reasoning process) and <answer>The test results consist of two parts. The reward function is for detecting accuracy; for In <answer>Partial content, namely the final detection results (such as bounding boxes and categories). For the true label of visible light images, To explain the quality reward function, For weakly aligned robust reward functions, For the true label of thermal infrared image, , , represents the weighting coefficient. Wherein, =0.5, =0.2, =0.3.

[0056] For a pair of acquired visible light images and thermal infrared images, examine <answer>The reward is the intersection-union ratio (IUU) of the bounding boxes in the code and the ground truth bounding boxes annotated by experts. This ensures that the ultimate goal of optimization is still accurate detection. Check if the output format follows the predefined structured output paradigm. If it does, the reward is 1; otherwise, the reward is 0. To ensure the consistency of the model's reasoning process, a greedy algorithm is used to search for visible light and thermal imaging bounding boxes in the thought chain for the model's final answer. This determines whether the model has encountered a reward-breaking behavior. If a reward-breaking behavior occurs, 0 points are awarded; otherwise, 1 point is awarded.

[0057] S204, Execution Group Policy Optimization Training This step employs the Group Policy Optimization (GRPO) algorithm, utilizing optimized data and a compound reward function to train the basic perceptual model through reinforcement learning. The policy optimization algorithm is a reinforcement learning method that updates the model's policy by comparing the performance of multiple output samples within the same group. Its specific execution flow is as follows: Training environment and configuration: The server environment and training framework from S1 are used, with an initial learning rate of 5e-7 and a cosine annealing learning rate scheduling strategy. The global batch size is set to 4. On the offline training server, the training environment from S1 is also used. The GRPO hyperparameters are set, with an initial learning rate of 5e-7 and a cosine annealing learning rate scheduling strategy. The global batch size is set to 4. The number of candidate outputs generated per group is G=8.

[0058] Optimization Iterative Process: For each input case q in the optimization dataset (containing a pair of images and a thought chain cue), the following process is executed: Generate candidate groups: Use the current base perception model to generate G distinct inference text candidate outputs {o1, o2, ..., o8}.

[0059] Calculating Rewards and Advantages: Each output within the group is scored using a composite reward function to obtain a scalar reward value. Then, the normalized advantage Ai of each output relative to the group's reward statistic is calculated using the following formula: ; in, For the first The normalization advantage of each candidate output; Indicates the first in the candidate output group A reasoning text; For the first candidate outputs The scalar reward value calculated based on the composite reward function; and These are used to calculate the mean and standard deviation of the rewards within the group, respectively. It is a small positive number.

[0060] Finally, by maximizing the following objective function To update the model parameters to be optimized : ; in, For trainable parameters, Let be the mathematical expectation, representing the expectation of all samples taken from the data distribution. Find the average. Importance sampling ratio, ; The model strategy before the update. For the updated model strategy, For the clipping function, This is a clipping range hyperparameter used to limit the maximum step size of a single model update. KL divergence is used to measure... With more The differences between them This is a hyperparameter for controlling the strength of the KL divergence penalty term.

[0061] Model parameter update: Based on the calculated normalization advantage, the model parameters are updated by maximizing the GRPO objective function. The composite reward function encourages high reward output while ensuring that the update step size of the base perception model is moderate and does not deviate too far from the original model through pruning and KL divergence constraints, thus maintaining training stability.

[0062] Optimization Termination: Iterative optimization continues. After each iteration, the average reward value of the model output is calculated on an independent validation set. When the average reward value no longer shows a significant improvement, the optimization is considered to have converged, and training is terminated. The final saved basic perception model is the basic perception model that has undergone deep logic optimization and can be used for final deployment.

[0063] S3, Transparent Detection Reasoning Based on Thinking Chain This step defines the online inference method of the basic perception model optimized by S2. Through designed thought chain prompts, the model is driven to synchronously output its internal decision-making process in structured natural language text when performing target detection tasks, thereby achieving transparency in the detection process and results. The aim is to deploy the high-quality inference model produced in step S2 in real-world application environments, process real-time data, and output transparent reports that combine detection results with decision-making basis.

[0064] Specifically, front-end data acquisition equipment deployed at forestry sites (such as multimodal pan-tilt cameras on lookout towers) simultaneously captures a pair of new, real-time visible light and thermal infrared images. This image pair is then transmitted in real-time to a back-end online inference server via a data transmission network (such as a 5G network).

[0065] The basic perception model optimized by S2 is loaded. Subsequently, the received real-time image pairs are combined with the preset, fixed thought chain prompt text and fed into the basic perception model.

[0066] The received real-time image pairs and thought chain prompts are combined and input into the loaded model to perform a forward computation. Based on the bimodal features extracted by the visual encoder, the basic perception model generates a complete and structured natural language text as output.

[0067] The natural language text contains a structured natural language description of the process and results of each reasoning step, including independent evidence extraction, cross-modal association verification, fusion decision-making, and interpretation.

[0068] Specifically, a thought chain prompt is a predefined natural language instruction whose core function is to guide the model to follow a specific reasoning framework. A thought chain prompt combines received real-time image pairs with a pre-defined, structured natural language prompt, i.e., the thought chain prompt. An example of the core content of a thought chain prompt is as follows: Please analyze this visible light image and its corresponding thermal infrared image. Please strictly follow the three-step logic of 'independent evidence extraction -> cross-modal association verification -> fusion decision interpretation' in your analysis. Your final output must be strictly contained within the following two tags: First, in <think>Within the tags, elaborate on your three-step reasoning process; secondly, in <answer>Within the tag, the final detection results are output in JSON format. The thought chain prompts explicit instructions to generate structured outputs based on the perception model. The separation of the reasoning process from the detection results is the key to achieving transparency.

[0069] The natural language text output by the basic perception model consists of two core parts: Reasoning process: located at <think>Within the tags, this section elaborates on the three-step logic strictly following the prompts: Independent evidence extraction: describes which potential target features and their locations were identified in visible light and thermal infrared images, respectively.

[0070] Cross-modal correlation verification: Spatial proximity (e.g., calculating coordinate distance and comparing it with tolerance threshold) and feature consistency (e.g., physical correlation between morphology and heat source features) between different modal evidence were analyzed.

[0071] Fusion Decision Interpretation: Based on the correlation results, comprehensively judge the target category and explain the logical chain of the final decision, while explaining how to integrate bimodal information to correct the positioning.

[0072] This section represents the generated thought process chain, detailing the complete decision-making logic. Its organization aligns with the optimization objective in step S2, clearly demonstrating each step and reasoning from discovering evidence in the bimodal image, verifying correlations, to the final judgment.

[0073] Test results: Located in <answer>Within the tag, this section outputs the final target category and the coordinates of the merged bounding box in a structured data format (such as JSON).

[0074] The structured text generated by the basic perception model, which includes the reasoning process and detection results, is output as the final result of this reasoning. Because the basic perception model has been optimized by S2 group policy, the reasoning process text it generates is significantly better than the unoptimized model in terms of logical rigor, ability to identify distractors, and handling of weak alignment.

[0075] In a specific application scenario, sensors deployed in forest lookout towers transmit real-time image streams via a 5G network to an online inference server at the backend of the monitoring center. Upon receiving the images, the inference module on the server automatically adds preset prompts, such as: "Please analyze this visible light image and its corresponding thermal infrared image. Please strictly follow the three-step logic of 'independent evidence extraction -> cross-modal association verification -> fusion decision interpretation' to generate a report containing a detailed decision-making process and final results." The model responds to the request within milliseconds, generating a complete report, as exemplified in the specification, containing detailed evidence IDs, spatial relationship analysis, and interference removal logic.

[0076] The thought chain prompts defined in step S3 serve as fixed inference templates and are invoked during real-time inference in step S4, ensuring that the detection process for each frame of image follows the same interpretation logic, thereby providing a complete basis for decision-making while outputting detection results.

[0077] S4. Forestry Target Detection and Interpretation Output This step deploys the transparent detection and inference capabilities implemented in S3 into a real forestry monitoring environment. Its core purpose is to drive the basic perception model to infer from real-time input multimodal images through pre-set thought chain prompts, and simultaneously output a transparent report that includes both detection results and the complete decision-making process, thereby transforming the "black box" model into an interpretable and traceable decision support system.

[0078] S401, System Deployment and Service Startup The basic perception model, optimized by S2 and verified by S3, is deployed on the online inference server of the forestry monitoring center and loaded into the model inference module. After deployment, the relevant streaming services are started. Front-end data acquisition devices (such as lookout tower cameras) continuously transmit real-time, synchronous visible light and thermal infrared video streams to the server via standard streaming media protocols (such as RTSP).

[0079] S402, Real-time Data Processing and Inference The real-time video stream received by the server is deframed into a continuous sequence of images. For each pair of time-aligned visible light and thermal infrared images, the system automatically performs the following operations: preprocessing (such as format conversion and size normalization); feeding the image pair into the loaded basic perception model; and triggering and executing a transparent detection inference process based on thought chains.

[0080] S403, Results Push and Visualization The structured text generated after the basic perception model inference is completed (containing <think>Reasoning process and <answer>A complete report of the detection results is pushed to the front-end forestry monitoring platform via a real-time communication protocol (such as WebSocket). The forestry monitoring platform then parses and visualizes the received results. Target Alarm and Location: Platform Automatic Analysis <answer>In some cases, detected targets (such as "early smoke") and their bounding boxes are highlighted in the original video footage, and can trigger audio and visual alarms.

[0081] Interactive query for decision interpretation: The platform provides an interactive interface where users can click on any marked target to expand an information panel that fully displays the data generated by the model. <think>The detailed reasoning text within the tag clearly answers the question "Why is this considered a fire?" Its logic follows a chain of "independent evidence extraction - cross-modal correlation verification - fusion decision interpretation." This text allows for a quick understanding of the model's judgment basis, effectively identifying false alarms (e.g., the model itself has identified high-temperature rocks and interpreted them as interference), confirming real threats (e.g., the model simultaneously reports visible smoke and abnormal heat sources and explains the correlation logic), and making accurate decisions based on this highly credible information, thus rapidly initiating the corresponding emergency response procedures.

[0082] like Figure 2 As shown, a multimodal forestry target detection system based on thought chain for implementing the aforementioned method according to the present invention includes a front-end data acquisition device 201, a data transmission network 202, an offline training server 203, and an online inference server 206.

[0083] The front-end data acquisition device 201 is a multimodal gimbal camera deployed on forest lookout towers or drones. It integrates a visible light sensor and a thermal infrared sensor to simultaneously acquire image pairs of visible light and thermal infrared images in forestry scenarios, and ensures the temporal consistency of the image pairs through a hardware synchronization mechanism.

[0084] The data transmission network 202 serves as a wired or wireless communication channel (such as 5G or fiber optic network) connecting the front-end data acquisition device 201 and the offline training server 203, and is responsible for transmitting the image data acquired by the front-end data acquisition device 201 to the offline training server 203 in real time or near real time.

[0085] The offline training server 203 is a high-performance computing server or server cluster deployed in a data center, used to perform computationally intensive model training and optimization tasks. It is connected to the front-end data acquisition device 201 via the data transmission network 202.

[0086] The basic perception model building module 204 and the group policy-based inference process optimization module 205 are deployed and run on the offline training server 203.

[0087] The basic perception model building module 204 is configured to perform the aforementioned method step S1, and is responsible for supervised fine-tuning of the pre-trained large visual language model to build a basic perception model with multimodal image understanding capabilities.

[0088] The group policy optimization-based inference process optimization module 205 is configured to execute the aforementioned method step S2, and is responsible for using the group policy optimization algorithm and composite reward function to perform reinforcement learning optimization on the inference logic of the basic perception model.

[0089] The online inference server 206 is a service unit designed for real-time applications. It can be located in the cloud or a local data center and is responsible for providing high-concurrency, low-latency inference services. The model inference module 207 and the forestry monitoring platform 208 are deployed and run on the online inference server 206.

[0090] The model inference module 207 is configured to execute the aforementioned method step S3, receive real-time image pairs from the front-end data acquisition device 201, and drive the model to perform transparent detection inference based on the thought chain, generating structured text containing the inference process and detection results. The structured text includes a structured natural language description of the process and results of each inference step, including independent evidence extraction, cross-modal association verification, fusion decision-making, and interpretation.

[0091] The forestry monitoring platform 208 is deployed on an online inference server or a standalone web server, and is usually presented in the form of a web application.

[0092] The forestry monitoring platform 208 is configured to execute the aforementioned method step S4, which is used to receive and visualize the target detection results and corresponding complete decision explanation text pushed by the model inference module 207, and provide an interactive interface for forestry managers to monitor and make decisions.

[0093] The invention and its embodiments have been described above illustratively. This description is not restrictive, and the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. The accompanying drawings are only one embodiment of the invention, and the actual structure is not limited thereto. No reference numerals in the claims should limit the scope of the claims. Therefore, if a person skilled in the art is inspired by this description and designs a similar structure and embodiment without departing from the spirit of the invention, such design should fall within the scope of protection of this patent. Furthermore, the word "comprising" does not exclude other elements or steps, and the word "a" preceding an element does not exclude the inclusion of "a plurality" of that element. Multiple elements stated in the product claims may also be implemented by a single element through software or hardware. The terms "first," "second," etc., are used to indicate names and do not indicate any specific order.< / think> < / answer> < / answer> < / think> < / answer> < / think> < / answer> < / think> < / answer> < / answer> < / answer> < / think> < / think> < / answer> < / think> < / answer> < / think> < / answer> < / answer> < / think>

Claims

1. A multimodal forest fire target detection method based on thought chain, comprising the following steps: Step S1: Collect timestamp-aligned multimodal images and annotate the data to form training data for supervised fine-tuning of a large visual language model pre-trained on general multimodal data in forestry scenarios, thus obtaining a basic perception model; the multimodal images include visible light forestry scene images and thermal infrared forestry scene images; the basic perception model has the basic mapping capability from multimodal images to target descriptive text; The data annotation information includes the bounding box coordinates and target category of the same target on the multimodal image, and the positional deviation of the center point of the bounding box of the same target on the multimodal image is not greater than a preset value; Step S2: Construct a composite reward function that can be used to evaluate detection accuracy, interpretation quality, and weak alignment robustness, and optimize the inference process of the basic perception model; Step S3: By inputting preset thought chain prompts into the optimized basic perception model, the basic perception model is driven to generate structured natural language text step by step according to preset logic. The natural language text contains a structured natural language description of the process and results of each reasoning step, including independent evidence extraction, cross-modal association verification, fusion decision and interpretation. Step S4: Deploy the optimized basic perception model on the online inference server, receive the real-time multimodal image stream from the front end, and perform a transparent detection inference process based on the thought chain prompts to obtain the forestry target detection results and their decision interpretation.

2. The multimodal forest fire target detection method based on thought chain as described in claim 1, characterized in that: In step S1, the Qwen2.5-VL model with multimodal dialogue capabilities is selected as the basic framework to construct a multimodal encoder-decoder architecture that includes a visual encoder, a feature projection layer, and a large language model decoder. The visual encoder is used to extract deep visual features from the multimodal image and stitch them together. The feature projection layer projects the stitched visual features onto the text feature space. The large language model decoder receives the projected visual features and the user's thought chain prompts to generate structured text. During the training process of the Qwen2.5-VL model: The labeled data is converted into a structured dialogue format, and the Qwen2.5-VL model is trained using a full-parameter fine-tuning method until the loss of the model on the validation set converges, thus obtaining the basic perception model.

3. The multimodal forest fire target detection method based on thought chain as described in claim 1, characterized in that: The reasoning process of the basic perception model is optimized by using the composite reward function and a group policy optimization reinforcement learning algorithm.

4. The multimodal forest fire target detection method based on thought chain as described in claim 3, characterized in that: In step S2, the group policy optimization reinforcement learning algorithm specifically includes: For the training set in the training dataset, the basic perception model generates multiple inference text candidate outputs; The composite reward function is used to automatically score each candidate output of the inference text; Calculate the normalized advantage of each inference text candidate output score relative to the within-group reward statistic; By maximizing the objective function of the group policy optimization, which includes normalization advantage and KL divergence constraints, the model parameters are updated to make the model tend to generate inference text with higher scores, until the average reward value of the model on the validation set in the training data converges.

5. The multimodal forest fire target detection method based on thought chain as described in claim 4, characterized in that: The composite reward function is composed of a weighted average of detection accuracy reward, interpretation quality reward, and weak alignment robustness reward; The accuracy reward evaluation assesses the intersection-union ratio (IoU) of the final detection result with the true label; the quality reward evaluation assesses whether the output text follows a predefined structured paradigm, including whether it contains elements used to identify the reasoning process. <think>Labels and identification of test results <answer> Tags: Weak alignment, robustness, reward guarantee, model, final answer, consistency with reasoning process in thought chain.< / answer> < / think> 6. The multimodal forest fire target detection method based on thought chain as described in claim 1, characterized in that: In step S3, the thought chain prompt is used to guide the optimized basic perception model to perform detection reasoning step by step according to the reasoning analysis logic of independent evidence extraction, cross-modal association verification, and fusion decision and interpretation, and to generate reasoning text based on the reasoning process; The model operations corresponding to each reasoning and analysis logic are as follows: Independent evidence extraction: Analyze target features in visible light images and thermal infrared images respectively; Cross-modal association verification: verifying the spatial proximity and feature consistency of evidence from different modalities; Fusion Decision Interpretation: Determines targets based on correlation results and outputs interpretations.

7. The multimodal forest fire target detection method based on thought chain as described in claim 1, characterized in that: In step S1, when acquiring paired visible light and thermal infrared forestry scene images, the acquisition is triggered by a GPS second pulse signal to ensure that the timestamp difference between the acquired visible light image and thermal infrared image is less than 20 milliseconds.

8. A multimodal forest fire target detection system based on thought chain for implementing the method of any one of claims 1-7, characterized in that, include: Front-end data acquisition equipment is used to simultaneously acquire pairs of visible light and thermal infrared forestry scene images; The offline training server is deployed with: The basic perception model construction module is used to annotate the paired images to construct a training dataset, wherein the coordinates of the center points of the annotated bounding boxes of the same target in the two modal images are allowed to deviate within a preset pixel range; and based on this dataset, the large visual language model pre-trained on general multimodal data is optimized through supervised fine-tuning to obtain a basic perception model with multimodal image understanding capabilities. The reasoning process optimization module is used to construct a composite reward function and employ a group policy optimization reinforcement learning algorithm to optimize the reasoning logic of the basic perception model. Online inference server, deployed with: The model inference module is used to load the optimized basic perception model, receive real-time multimodal image streams from the front-end data acquisition device, and drive the model to generate outputs containing target detection results and structured decision explanation text by inputting preset thought chain prompts into the model. The structured decision explanation text contains a structured natural language description of the process and results of each inference step, including independent evidence extraction, cross-modal association verification, and fusion decision and explanation. The forestry monitoring platform is used to receive and visualize the target detection results and corresponding decision explanation text output by the model inference module.

9. The multimodal forest fire target detection system based on thought chain as described in claim 8, characterized in that: The front-end data acquisition device is a multimodal gimbal camera that integrates visible light and long-wave infrared sensors and is deployed on a fixed watchtower or drone.

10. An electronic device, characterized in that, Includes one or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the multimodal forest fire target detection method based on the thought chain as described in any one of claims 1 to 11.

Images