Mountain fire monitoring method and system based on multi-modal large model and program product

Abstract

The invention belongs to the technical field of forest fire monitoring, and particularly discloses a forest fire monitoring method and system based on a multi-modal large model and a program product, a convolutional neural network and a Transform network are combined, the local feature extraction capability of the convolutional neural network and the global context modeling advantage of the Transform network can be integrated, the data deep analysis capability of the model is enhanced, and the forest fire monitoring efficiency is improved. Comprising image analysis and natural language processing so as to effectively cope with complex challenges difficult to overcome by a traditional single network structure, and is suitable for mountain fire monitoring scenes requiring fine information and macroscopicity. While low resource occupation and efficient reasoning are maintained, global and local details can be taken into consideration, an efficient solution is provided for mountain fire disaster monitoring in an actual application scene, and complex monitoring scenes such as far and near mountain fire and daytime and night can be coped with by taking fine information and macroscopic information into consideration.

Description

Technical Field

[0001] The present invention belongs to the technical field of wildfire monitoring, and in particular relates to a wildfire monitoring method, system and program product based on a multimodal large model. Background Art

[0002] Intelligent wildfire monitoring through machine learning models is the main research direction of current wildfire monitoring technology. At present, large models in the industry's production environment are mainly based on global feature extraction, and small models are mainly based on local feature extraction. The progress made by the industry in multimodal large models shows that multimodal large models have demonstrated excellent visual understanding and reasoning capabilities in various tasks such as visual question answering and reference understanding, but they still have defects in details and require a large amount of hardware environment during deployment. In actual production environments, large models lack strong performance in detail content, and small models (such as convolutional networks) used in the industry also perform unsatisfactory in global image understanding. Although small models require fewer hardware resources, they are difficult to perform satisfactorily when dealing with overall complex environments. Therefore, there is an urgent need to use a more efficient and accurate model to effectively respond to the complex challenges of wildfire monitoring. Summary of the invention

[0003] The purpose of the present invention is to provide a forest fire monitoring method, system and program product based on a multimodal large model to solve the above-mentioned problems existing in the prior art.

[0004] In order to achieve the above object, the present invention adopts the following technical solutions: First, a wildfire monitoring method based on a multimodal large model is provided, including: Acquire multimodal wildfire monitoring information, wherein the multimodal wildfire monitoring information includes wildfire monitoring images and wildfire disaster prompt text; Inputting the multimodal wildfire monitoring information into a trained multimodal large model for multimodal wildfire analysis to obtain corresponding wildfire analysis results, wherein the multimodal large model includes a CNN convolution branch and a Transformer semantic branch, wherein the CNN convolution branch is used for local feature extraction, and the Transformer semantic branch is used for global information extraction, and backbone feature alignment and shared decoder head alignment are performed between the CNN convolution branch and the Transformer semantic branch through a SIAM semantic information alignment module; Determine whether an abnormal wildfire situation occurs based on the wildfire analysis results, and output corresponding wildfire alarm information when an abnormal wildfire situation is determined to occur.

[0005] In one possible design, before inputting the multimodal wildfire monitoring information into the trained multimodal large model for multimodal wildfire analysis, the method further includes: Constructing a large multimodal model, the large multimodal model includes a CNN convolution branch and a Transformer semantic branch, the CNN convolution branch is used for local feature extraction, the Transformer semantic branch is used for global information extraction, and the CNN convolution branch and the Transformer semantic branch are aligned with backbone features and shared decoder heads through a SIAM semantic information alignment module; A training set is used to train the multimodal large model to obtain a trained multimodal large model, wherein the training set includes a number of wildfire monitoring image samples and wildfire disaster warning text samples marked with corresponding classification labels, the CNN convolution branch is used to input the wildfire monitoring image samples, and the Transformer semantic branch is used to input the wildfire monitoring image samples and wildfire disaster warning text samples.

[0006] In one possible design, the CNN convolution branch includes a first Stem convolution layer, several Conv Block convolution blocks, several Conv2Former Block convolution blocks and a first Decoder decoder connected in sequence.

[0007] In one possible design, the Transformer semantic branch includes a second Stem convolutional layer, a plurality of Transformer Block modules and a second Decoder decoder connected in sequence.

[0008] In one possible design, the SIAM semantic information alignment module includes a backbone feature alignment module and a shared decoder head alignment module. The backbone feature alignment module is used to align the feature representations of the CNN convolution branch and the Transformer semantic branch, and the shared decoder head alignment module is used to align the features of the CNN convolution branch and the Transformer semantic branch in the decoding stage of the first Decoder decoder and the second Decoder decoder.

[0009] In a possible design, the multimodal large model uses the CWD Loss function as the loss function for aligning the feature representations of the CNN convolution branch and the Transformer semantic branch. The expression of the CWD Loss function is:

[0010] Among them, L CWD Represents the CWD Loss function, C represents the number of channels of the feature map of the CNN convolution branch and the Transformer semantic branch, Characterize the feature map of the ith channel of the CNN convolution branch, The feature map representing the i-th channel of the Transformer semantic branch, Characterize the two-norm operation; The Semantic Loss loss function is used as the feature alignment loss function between the CNN convolution branch and the Transformer semantic branch in the decoding stage. The expression of the Semantic Loss loss function is:

[0011] Among them, L SA Represents the Semantic Loss loss function, H represents the height of the feature map of the CNN convolution branch and the Transformer semantic branch, W represents the width of the feature map of the CNN convolution branch and the Transformer semantic branch, h∈H, w∈W, c∈C, O CNN Characterize the output characteristics of the first Decoder decoder, O Trans Characterize the output characteristics of the second Decoder; The total loss function of the multimodal large model is

[0012] Among them, L total is the total loss function, L CE is the classification cross entropy loss function, λ is the weight coefficient of the CWD Loss loss function, and γ is the weight coefficient of the Semantic Loss loss function.

[0013] In one possible design, during the training of the multimodal large model, the method further includes: The alignment effect of the SIAM semantic information alignment module is sampled, and the parameters of the Conv2FormerBlock convolution block and the two weight coefficients λ and γ are adjusted according to the alignment effect.

[0014] In a second aspect, a wildfire monitoring system based on a multimodal large model is provided, comprising an information acquisition unit, a wildfire analysis unit and a result determination unit, wherein: An information acquisition unit, used to acquire multimodal wildfire monitoring information, wherein the multimodal wildfire monitoring information includes wildfire monitoring images and wildfire disaster prompt text; A wildfire analysis unit, used for inputting multimodal wildfire monitoring information into a trained multimodal large model for multimodal wildfire analysis to obtain corresponding wildfire analysis results, wherein the multimodal large model includes a CNN convolution branch and a Transformer semantic branch, wherein the CNN convolution branch is used for local feature extraction, and the Transformer semantic branch is used for global information extraction, and backbone feature alignment and shared decoder head alignment are performed between the CNN convolution branch and the Transformer semantic branch through a SIAM semantic information alignment module; The result determination unit is used to determine whether an abnormal wildfire situation occurs according to the wildfire analysis result, and output corresponding wildfire alarm information when it is determined that an abnormal wildfire situation occurs.

[0015] Thirdly, we provide a wildfire monitoring system based on a multimodal large model, including: A memory for storing instructions; A processor is used to read the instructions stored in the memory and execute any one of the methods described in the first aspect according to the instructions.

[0016] In a fourth aspect, a computer-readable storage medium is provided, wherein instructions are stored on the computer-readable storage medium, and when the instructions are executed on a computer, the computer executes any one of the methods described in the first aspect. In addition, a computer program product is provided, and when the computer program product is executed on a computer, the computer executes any one of the methods described in the first aspect.

[0017] Beneficial effects: By combining the convolutional neural network with the Transformer network, the present invention can integrate the local feature extraction capability of the convolutional neural network and the global context modeling advantage of the Transformer, and enhance the model's ability to deeply analyze data, including image analysis and natural language processing, so as to effectively cope with complex challenges that are difficult to overcome with traditional single network structures, and is suitable for wildfire monitoring scenarios that require both delicate information and macro-equality. While maintaining low resource usage and efficient reasoning, the present invention can achieve a balance between global and local details, providing an efficient solution for wildfire disaster monitoring in actual application scenarios. By taking into account both delicate information and macro-equality, it can cope with complex monitoring scenarios such as near and far wildfires, day and night. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 This is a schematic diagram of the steps of the method in Example 1 of the present invention; Figure 2 Schematic diagram of the architecture of the multimodal large model in Example 1 of the present invention; Figure 3 Schematic diagram of the system structure in Example 3 of the present invention. DETAILED DESCRIPTION

[0020] It should be noted that the description of these embodiments is intended to help understand the present invention, but does not constitute a limitation of the present invention. The specific structures and functional details disclosed herein are only intended to describe exemplary embodiments of the present invention. However, the present invention can be embodied in a number of alternative forms, and should not be construed as being limited to the embodiments set forth herein.

[0021] It should be understood that, unless otherwise clearly specified and limited, the term "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the embodiments can be understood according to specific circumstances.

[0022] In the following description, certain details are provided to facilitate a complete understanding of the example embodiments. However, it will be appreciated by those of ordinary skill in the art that the example embodiments may be implemented without these certain details. For example, devices may be shown in block diagrams to avoid obscuring the examples with unnecessary details. In other embodiments, well-known processes, structures, and techniques may not be shown in unnecessary detail to avoid obscuring the embodiments.

[0023] Embodiment 1: This embodiment provides a wildfire monitoring method based on a multimodal large model, which can be applied to corresponding wildfire monitoring terminals, such as Figure 1 As shown, the method comprises the following steps: S1. Acquire multimodal wildfire monitoring information, wherein the multimodal wildfire monitoring information includes wildfire monitoring images and wildfire disaster prompt texts.

[0024] In specific implementation, when performing wildfire monitoring, it is necessary to obtain multimodal wildfire monitoring information, which includes wildfire monitoring images and wildfire disaster prompt text. Among them, wildfire monitoring images can be captured from multi-dimensional visual data such as satellite images and drone video images, and the wildfire disaster prompt text can be input meteorological data, topographic information, and other prompt words related to wildfire disaster monitoring.

[0025] S2. Input the multimodal wildfire monitoring information into a trained multimodal large model for multimodal wildfire analysis to obtain corresponding wildfire analysis results. The multimodal large model includes a CNN convolution branch and a Transformer semantic branch. The CNN convolution branch is used for local feature extraction, and the Transformer semantic branch is used for global information extraction. The CNN convolution branch and the Transformer semantic branch are used for backbone feature alignment and shared decoder head alignment through the SIAM semantic information alignment module.

[0026] In specific implementation, pre-build Figure 2 The multimodal large model shown in the figure comprises a CNN convolution branch and a Transformer semantic branch; the CNN convolution branch is used for local feature extraction, including a first Stem convolution layer, a plurality of Conv Block convolution blocks, a plurality of Conv2Former Block convolution blocks and a first Decoder decoder connected in sequence; the Transformer semantic branch is used for global information extraction, including a second Stem convolution layer, a plurality of Transformer Block modules and a second Decoder decoder connected in sequence; and the CNN convolution branch and the Transformer semantic branch are aligned with backbone features and shared decoder heads through a SIAM semantic information alignment module, and the SIAM semantic information alignment module includes a backbone feature alignment module (i.e., corresponding to Figure 2 BackboneFeature module and BA module in the decoder) and shared decoder head alignment module (i.e. corresponding Figure 2 The backbone feature alignment module is used to align the feature representations of the CNN convolution branch and the Transformer semantic branch, and the shared decoder head alignment module is used to align the features of the CNN convolution branch and the Transformer semantic branch in the decoding stage of the first Decoder decoder and the second Decoder decoder.

[0027] Among them, the first Stem convolution layer and the second Stem convolution layer are the initial convolution layers of the two branches, which are used to perform preliminary convolution processing on the input information; the Conv Block convolution block includes a convolution layer, an activation function, and a pooling layer; the Conv2FormerBlock convolution block is a module that combines the characteristics of CNN and Transformer. While maintaining the convolution characteristics, it can learn the semantic representation ability of Transformer; the Transformer Block module is used to capture long-distance global context information; the Decoder decoder is used to restore the feature map to the original resolution and generate the final segmentation result. The backbone feature alignment module can align the feature representations of the CNN branch and the Transformer branch, including feature map size adjustment, channel number adjustment, and feature alignment; the shared decoder head alignment module is used to align the features of the CNN branch and the transformer branch during the decoding stage. Figure 2 In the backbone feature alignment module shown in the figure: Projection is a projection operation block, which is used to adjust the size and number of channels of the feature map so that it can be aligned with the target feature in the spatial and channel dimensions. Projection ensures that the feature maps of different branches can match during the alignment and fusion process. Semantic Alignment Loss is used to align the feature representations from the CNN branch and the Transformer branch during training. This loss function focuses on the semantic information of the aligned features rather than the spatial information, thereby ensuring that the single-branch CNN can capture rich semantic information during inference. W (width, Width) represents the horizontal size of the image or feature map; H (height, Height) represents the vertical size of the image or feature map; C (number of channels, Channels) represents the number of channels of the image or feature map. H_c×W_c×C_c represents that in the decoder part, the size of the feature map is H_c×W_c and the number of channels is C_c. H×W×C_t represents that the size of the feature map after alignment and projection is H×W and the number of channels is C_t.

[0028] The multimodal large model uses the CWD Loss function as the loss function for aligning the feature representations of the CNN convolution branch and the Transformer semantic branch. The expression of the CWD Loss function is:

[0029] Among them, L CWD Represents the CWD Loss function, C represents the number of channels of the feature map of the CNN convolution branch and the Transformer semantic branch, Characterize the feature map of the ith channel of the CNN convolution branch, The feature map representing the i-th channel of the Transformer semantic branch, Characterize the two-norm operation; The Semantic Loss loss function is used as the feature alignment loss function between the CNN convolution branch and the Transformer semantic branch in the decoding stage. The expression of the Semantic Loss loss function is:

[0030] Among them, L SA Represents the Semantic Loss loss function, H represents the height of the feature map of the CNN convolution branch and the Transformer semantic branch, W represents the width of the feature map of the CNN convolution branch and the Transformer semantic branch, h∈H, w∈W, c∈C, O CNN Characterize the output characteristics of the first Decoder decoder, O Trans Characterize the output characteristics of the second Decoder; The total loss function of the multimodal large model is

[0031] Among them, L total is the total loss function, L CE is the classification cross entropy loss function, λ is the weight coefficient of the CWD Loss loss function, and γ is the weight coefficient of the Semantic Loss loss function.

[0032] The backbone feature alignment module ensures the consistency of features by adjusting and projecting the feature maps so that the feature maps can be aligned at different resolutions. Set the projection matrix to P, and align the features of the CNN branch and the Transformer branch to F CNN and F Trans , then the expression for aligning the features of the CNN branch and the Transformer branch is

[0033]

[0034] The shared decoder head alignment module is used to align the features of the CNN branch and the Transformer branch during the decoding phase, ensuring that the decoder can process the aligned feature maps when generating the final segmentation result. The output of the decoder is O aligned , then the expression for alignment through the shared decoder header is

[0035] .

[0036] After constructing the multimodal large model, the multimodal large model can be trained using a training set to obtain a trained multimodal large model, wherein the training set includes a number of wildfire monitoring image samples and wildfire disaster warning text samples marked with corresponding classification labels, the CNN convolution branch is used to input wildfire monitoring image samples, and the Transformer semantic branch is used to input wildfire monitoring image samples and wildfire disaster warning text samples. During the training process, the input data passes through the CNN branch and the Transformer branch at the same time, wherein the Transformer branch inputs text and image data, and the CNN branch inputs image data, and uses the standard task loss to optimize the performance of the model for specific tasks, and adds CWD Loss to prompt the CNN branch to learn the semantic feature representation of the Transformer branch. In the training iteration, the weight parameters of the model are continuously adjusted, and the CNN branch is optimized to simulate the semantic information learned by the Transformer branch, while ensuring the overall performance of the model. At the same time, the alignment effect of the SIAM semantic information alignment module is sampled regularly during the training process to evaluate the consistency of the features of the CNN branch and the Transformer branch through visualization or quantitative indicators. Then, the parameters of the Conv2Former Block convolution block and the two weight coefficients λ and γ are adjusted according to the alignment effect to achieve the best semantic information transfer and task performance.

[0037] After the training optimization is completed and the optimal multimodal large model is obtained, the model can be used for reasoning and application. The acquired multimodal wildfire monitoring information can be input into the trained multimodal large model for multimodal wildfire analysis. Through the cross-modal projector, these separated image information and text information are integrated into a unified representation to accurately understand the complex image information, complete the analysis of wildfire disasters through reasoning, and obtain the corresponding wildfire analysis results.

[0038] S3. Determine whether an abnormal wildfire situation occurs based on the wildfire analysis result, and output corresponding wildfire alarm information when it is determined that an abnormal wildfire situation occurs.

[0039] During specific implementation, in the actual reasoning application process, it can be determined whether an abnormal wildfire situation occurs based on the wildfire analysis results of the multimodal large model. If an abnormal wildfire situation is determined to occur, the terminal can output the corresponding wildfire alarm information for abnormal alarm.

[0040] The method of this embodiment combines the convolutional neural network with the Transformer network to integrate the local feature extraction capability of the convolutional neural network and the global context modeling advantage of the Transformer, and enhances the model's ability to deeply analyze data, including image analysis and natural language processing, so as to effectively cope with complex challenges that are difficult to overcome with traditional single network structures, and is suitable for wildfire monitoring scenarios that require both delicate information and macro-equality. While maintaining low resource usage and efficient reasoning, this technical solution can achieve a balance between global and local details, providing an efficient solution for wildfire disaster monitoring in actual application scenarios. By taking into account both delicate information and macro-equality information, it can cope with complex monitoring scenarios such as near and far wildfires, day and night.

[0041] Embodiment 2: This embodiment provides a wildfire monitoring system based on a multimodal large model, including an information acquisition unit, a wildfire analysis unit and a result determination unit, wherein: An information acquisition unit, used to acquire multimodal wildfire monitoring information, wherein the multimodal wildfire monitoring information includes wildfire monitoring images and wildfire disaster prompt text; A wildfire analysis unit, used for inputting multimodal wildfire monitoring information into a trained multimodal large model for multimodal wildfire analysis to obtain corresponding wildfire analysis results, wherein the multimodal large model includes a CNN convolution branch and a Transformer semantic branch, wherein the CNN convolution branch is used for local feature extraction, and the Transformer semantic branch is used for global information extraction, and backbone feature alignment and shared decoder head alignment are performed between the CNN convolution branch and the Transformer semantic branch through a SIAM semantic information alignment module; The result determination unit is used to determine whether an abnormal wildfire situation occurs according to the wildfire analysis result, and output corresponding wildfire alarm information when it is determined that an abnormal wildfire situation occurs.

[0042] Embodiment 3: This embodiment provides a forest fire monitoring system based on a multi-modal large model, such as Figure 3 As shown, at the hardware level, it includes: Data interface, used to establish data connection between the processor and the external data terminal; A memory for storing instructions; The processor is used to read the instructions stored in the memory and execute the wildfire monitoring method based on the multimodal large model in Example 1 according to the instructions.

[0043] Optionally, the system further includes an internal bus, through which the processor, the memory and the data interface can be interconnected, and the internal bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc.

[0044] The memory may include, but is not limited to, random access memory (RAM), read only memory (ROM), flash memory, first input first output (FIFO) and / or first in last out (FILO), etc. The processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0045] Embodiment 4: This embodiment provides a computer-readable storage medium, on which instructions are stored. When the instructions are executed on a computer, the computer executes the wildfire monitoring method based on a multimodal large model in Embodiment 1. The computer-readable storage medium refers to a carrier for storing data, which may include but is not limited to a floppy disk, an optical disk, a hard disk, a flash memory, a USB flash drive, and / or a memory stick, etc. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0046] This embodiment also provides a computer program product, which, when executed on a computer, executes the wildfire monitoring method based on a multimodal large model in Embodiment 1. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0047] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A wildfire monitoring method based on a multimodal large model, characterized in that: include: Acquire multimodal wildfire monitoring information, wherein the multimodal wildfire monitoring information includes wildfire monitoring images and wildfire disaster prompt text; Inputting the multimodal wildfire monitoring information into a trained multimodal large model for multimodal wildfire analysis to obtain corresponding wildfire analysis results, wherein the multimodal large model includes a CNN convolution branch and a Transformer semantic branch, wherein the CNN convolution branch is used for local feature extraction, and the Transformer semantic branch is used for global information extraction, and backbone feature alignment and shared decoder head alignment are performed between the CNN convolution branch and the Transformer semantic branch through a SIAM semantic information alignment module; Determine whether an abnormal wildfire situation occurs based on the wildfire analysis results, and output corresponding wildfire alarm information when an abnormal wildfire situation is determined to occur.

2. The method for monitoring wildfires based on a multimodal large model according to claim 1, characterized in that: Before inputting the multimodal wildfire monitoring information into the trained multimodal large model for multimodal wildfire analysis, the method further includes: Constructing a large multimodal model, the large multimodal model includes a CNN convolution branch and a Transformer semantic branch, the CNN convolution branch is used for local feature extraction, the Transformer semantic branch is used for global information extraction, and the CNN convolution branch and the Transformer semantic branch are aligned with backbone features and shared decoder heads through a SIAM semantic information alignment module; A training set is used to train the multimodal large model to obtain a trained multimodal large model, wherein the training set includes a number of wildfire monitoring image samples and wildfire disaster warning text samples marked with corresponding classification labels, the CNN convolution branch is used to input the wildfire monitoring image samples, and the Transformer semantic branch is used to input the wildfire monitoring image samples and wildfire disaster warning text samples.

3. The method for monitoring wildfires based on a multimodal large model according to claim 2, characterized in that: The CNN convolution branch includes a first Stem convolution layer, a plurality of Conv Block convolution blocks, a plurality of Conv2Former Block convolution blocks and a first Decoder decoder which are connected in sequence.

4. The method for monitoring wildfires based on a multimodal large model according to claim 3, characterized in that: The Transformer semantic branch includes a second Stem convolutional layer, a plurality of Transformer Block modules and a second Decoder decoder connected in sequence.

5. The method for monitoring wildfires based on a multimodal large model according to claim 4, characterized in that: The SIAM semantic information alignment module includes a backbone feature alignment module and a shared decoder head alignment module. The backbone feature alignment module is used to align the feature representations of the CNN convolution branch and the Transformer semantic branch. The shared decoder head alignment module is used to align the features of the CNN convolution branch and the Transformer semantic branch during the decoding stage of the first Decoder decoder and the second Decoder decoder.

6. The method for monitoring wildfires based on a multimodal large model according to claim 5, characterized in that: The multimodal large model uses the CWD Loss function as the loss function for aligning the feature representations of the CNN convolution branch and the Transformer semantic branch. The expression of the CWD Loss function is: Among them, L CWD Represents the CWD Loss function, C represents the number of channels of the feature map of the CNN convolution branch and the Transformer semantic branch, Characterize the feature map of the ith channel of the CNN convolution branch, The feature map representing the i-th channel of the Transformer semantic branch, Characterize the two-norm operation; The Semantic Loss loss function is used as the feature alignment loss function between the CNN convolution branch and the Transformer semantic branch in the decoding stage. The expression of the Semantic Loss loss function is: Among them, L SA Represents the Semantic Loss loss function, H represents the height of the feature map of the CNN convolution branch and the Transformer semantic branch, W represents the width of the feature map of the CNN convolution branch and the Transformer semantic branch, h∈H, w∈W, c∈C, O CNN Characterize the output characteristics of the first Decoder decoder, O Trans Characterize the output characteristics of the second Decoder; The total loss function of the multimodal large model is Among them, L total is the total loss function, L CE is the classification cross entropy loss function, λ is the weight coefficient of the CWD Loss loss function, and γ is the weight coefficient of the Semantic Loss loss function.

7. The method for monitoring wildfires based on a multimodal large model according to claim 6, characterized in that: In the process of training the multimodal large model, the method further includes: The alignment effect of the SIAM semantic information alignment module is sampled, and the parameters of the Conv2Former Block convolution block and the two weight coefficients λ and γ are adjusted according to the alignment effect.

8. A forest fire monitoring system based on a multimodal large model, characterized in that: It includes an information acquisition unit, a wildfire analysis unit and a result determination unit, wherein: An information acquisition unit, used to acquire multimodal wildfire monitoring information, wherein the multimodal wildfire monitoring information includes wildfire monitoring images and wildfire disaster prompt text; A wildfire analysis unit, used for inputting multimodal wildfire monitoring information into a trained multimodal large model for multimodal wildfire analysis to obtain corresponding wildfire analysis results, wherein the multimodal large model includes a CNN convolution branch and a Transformer semantic branch, wherein the CNN convolution branch is used for local feature extraction, and the Transformer semantic branch is used for global information extraction, and backbone feature alignment and shared decoder head alignment are performed between the CNN convolution branch and the Transformer semantic branch through a SIAM semantic information alignment module; The result determination unit is used to determine whether an abnormal wildfire situation occurs according to the wildfire analysis result, and output corresponding wildfire alarm information when it is determined that an abnormal wildfire situation occurs.

9. A forest fire monitoring system based on a multimodal large model, characterized in that: include: A memory for storing instructions; A processor is used to read the instructions stored in the memory and execute the wildfire monitoring method based on the multimodal large model according to any one of claims 1-7.

10. A computer program product, characterized in that When the computer program product is run on a computer, the wildfire monitoring method based on a multimodal large model described in any one of claims 1 to 7 is executed.

Images