A knowledge- and vision-driven method and apparatus for flame image recognition and analysis

Abstract

This invention discloses a knowledge- and vision-driven method and apparatus for flame image recognition and analysis, relating to the field of image recognition technology. The method includes the following steps: First, flame knowledge graph data is collected. Based on this data, a convolutional neural network is established, using the corresponding flame image as input and entity embedding vectors as labels for training, forming an entity embedding vector prediction model. The flame image is segmented, flame visual feature parameters are extracted, and embedding and spatial expansion are performed to generate feature data training tensors. A ViT network model is used to train the feature data to extract comprehensive features. The flame image to be identified is acquired, the hazard emergency coefficient is calculated using the model, and compared with a threshold to generate a corresponding processing scheme. The results are then output as text. This effectively combines visual features and knowledge graphs to improve the accuracy and intelligence of flame image recognition and emergency response.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image recognition, and particularly relates to a knowledge and vision jointly driven flame image recognition analysis method and device. BACKGROUND

[0002] In the current digital era, the efficiency of fire prevention and emergency disposal is crucial. As a semantic network, the knowledge graph stores a large amount of rich knowledge about flames, including the generation mechanism of flames, the flame characteristics of different burning substances, the development law of fire in various fire scenes, and the matching fire fighting strategies. For example, it records key information such as "flames caused by electrical faults usually have electric sparks and dark smoke", "in forest fires, the fire is affected by wind direction and terrain, and spreads in a fan shape along the wind direction".

[0003] The ViT model has strong feature extraction capability and high computational efficiency in the field of image analysis by virtue of the Transformer architecture. However, the traditional ViT model is limited to the pixel information of the image itself when processing images, and cannot make use of the large amount of valuable flame-related knowledge in the knowledge graph. This results in the inability to deeply understand the semantic information contained in the flame in the image when facing complex fire image scenes, making it difficult to accurately judge the development trend of the flame, such as the growth rate of the fire in the future, the direction of spread, etc., and unable to give targeted fire fighting methods based on the image content.

[0004] In the prior art, the patent for invention with the authorization announcement number CN109711345B discloses a flame image recognition method, device and storage medium, which comprises the following steps: determining a suspected flame area in a collected image; extracting a texture feature image of the suspected flame area using a filter; extracting a local binary feature of the texture feature image based on a local binary pattern; and determining whether the suspected flame area exists based on the local binary feature using a support vector machine (SVM) classifier. This method improves the flame recognition efficiency and accuracy by extracting the texture feature image and further local binary feature, but this scheme can only identify whether the flame exists, and does not describe the development law of the fire, the characteristic parameters of the flame and the danger level of the flame, thus reducing the practicality and effectiveness of the recognition system.

[0005] The above information disclosed in the background section is only used to enhance the understanding of the background of the present disclosure, and therefore it can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0006] The present application aims to provide a knowledge and vision combined driving flame image recognition analysis method and device to solve the problems in the background art.

[0007] To achieve the above-mentioned purpose, the present application provides the following technical solutions:

[0008] A knowledge and vision combined driving flame image recognition analysis method, the specific steps include:

[0009] Obtain a training set and a flame image to be identified, the training set includes multiple groups of trained flame images and the corresponding flame knowledge graph, perform information fusion on the flame knowledge graph to obtain a corresponding entity embedding vector, use the trained flame image as a feature input, and use the corresponding entity embedding vector as a feature output to train a convolutional neural network, obtain an entity embedding vector evaluation model, and use the entity embedding vector evaluation model to perform feature recognition on the flame image to be identified to obtain a corresponding entity embedding vector;

[0010] Segment the trained flame image and the flame image to be identified to obtain a plurality of image blocks of the same size, obtain the flame visual feature parameters of each image block and embed them into the corresponding image block to obtain a three-dimensional image block embedding tensor corresponding to each flame image, and perform a tensor expansion operation on the entity embedding vector corresponding to each flame image in space to obtain a three-dimensional entity information embedding tensor corresponding to each flame image;

[0011] Splice the three-dimensional image block embedding tensor and the three-dimensional entity information embedding tensor corresponding to each flame image to obtain a corresponding feature data training tensor, label the feature data training tensor corresponding to the trained flame image as a first tensor, and label the feature data training tensor corresponding to the flame image to be identified as a second tensor;

[0012] Use the first tensor as a feature input, and use the corresponding relationship of the entity in the trained flame image as a feature output to train a ViT network model, and use the trained ViT network model to extract features from the second tensor to obtain the corresponding relationship between entities in the flame image to be identified;

[0013] Model and analyze the second tensor to obtain a dangerous emergency coefficient of the flame image to be identified, compare the dangerous emergency coefficient with a dangerous judgment threshold, determine a processing mode according to the comparison result, convert the corresponding relationship of the entity in the flame image to be identified and the processing mode into a language text, and output the language text.

[0014] Furthermore, a number of flame knowledge graph data are collected. The flame knowledge graph data is graph structure data, which includes a node set and an edge set. The node set covers all entities and concepts, and the edge set represents the correspondence between entities. The correspondence between entities is represented by vectors, specifically the connection vectors between the fire source location, fire growth rate, spread direction angle, and quantity of flammable materials. The fusion of the collected flame knowledge graph data information specifically refers to fusing the entity information of each node in the node set with the information of adjacent nodes to obtain an entity embedding vector. The adjacent nodes are four adjacent nodes, specifically including the upper, lower, left, and right adjacent nodes. The entity information of each node in the node set includes the fire source location, fire growth rate, spread direction angle, and quantity of flammable materials.

[0015] Based on the collected flame knowledge graph data, a convolutional neural network is established. This convolutional neural network consists of an input layer, convolutional layers, pooling layers, fully connected layers, and an output layer. The activation function in the convolutional layers is... function, The specific expression of the function is:

[0016]

[0017] in, Indicates the first Each corresponding convolutional layer Then it means the first In the corresponding convolutional layer, the th The first feature map There are eigenvalues; among which For the index of the convolutional layer, For the index of training sample images, The index of the feature values ​​in the training sample image;

[0018] For the fully connected layer, the number of neurons in the fully connected layer is set to 32, the initial neural network learning rate is set to 0.001, and the number of training rounds is 100.

[0019] The trained entity embedding vector prediction model takes a flame image as input and outputs node entity information within the corresponding node set and the entity embedding vector fused from the corresponding nodes. Specifically, the entity embedding vector is a one-dimensional vector. , dimension .

[0020] Furthermore, the flame images within the flame knowledge graph data are segmented into several image blocks of the same size. The specific formula used to calculate the number of segmented image blocks is as follows:

[0021]

[0022] wherein, is the number of segmented image blocks, and are the height and width of the input flame image, respectively, and are the height and width of the segmented image blocks, respectively;

[0023] Based on the obtained flame visual feature parameters, an embedding operation is performed on each segmented image block to obtain a three-dimensional image block embedding tensor, wherein the three-dimensional image block embedding tensor is specifically represented as wherein is the number of flame visual feature parameter channels, and the flame visual feature parameters include flame color gray value, flame contour area, and smoke color RGB value.

[0024] Further, according to the obtained entity embedding vector, a tensor expansion operation is performed on the entity embedding vector in space to obtain a three-dimensional entity information embedding tensor, wherein the specific manner of the tensor expansion operation is that the relationship vector between the corresponding entities is repeated h times in the height direction and w times in the width direction, so that the entity relationship embedding is completely aligned with the image block embedding in the spatial layout. The expanded entity relationship embedding tensor changes from a one-dimensional vector to a three-dimensional entity information embedding tensor, wherein the formula according to which the three-dimensional entity information embedding tensor is specifically calculated is:

[0025]

[0026] wherein, is the three-dimensional entity information embedding tensor, is an all-1 matrix of , represents a tensor outer product operation, is the entity embedding vector; the three-dimensional entity information embedding tensor is ;

[0027] The obtained three-dimensional image block embedding tensor and the three-dimensional entity information embedding tensor are spliced to obtain a feature data training tensor, wherein the specific logic according to which the two tensors are spliced is that the two tensors are connected in the channel dimension, that is, the C channels of the image block embedding and the D channels of the expanded three-dimensional entity information embedding tensor are spliced together in order to form a new tensor. After splicing, the feature data training tensor Z is obtained, and the dimension is .

[0028] Furthermore, based on the obtained first tensor, a ViT network model is established. The training of the ViT network model specifically includes constructing a relation-aware attention mask by embedding the 3D entity information within the first tensor Z into the tensor. This relation-aware attention mask is constructed using both channel attention and spatial attention. The formula used to calculate channel attention is:

[0029]

[0030] In the formula, For channel attention, For activation function, This represents the confidence level of the correspondence in the knowledge graph. This indicates a global average pooling operation. This is the channel weight matrix. This is the bias vector for channel attention;

[0031] The formula used to calculate spatial attention is as follows:

[0032]

[0033] In the formula, For spatial attention, These are the weights of the convolutional layer. This represents the convolution operation. and These represent average pooling and max pooling operations, respectively. ( ); ( This indicates that the results of average pooling and max pooling are concatenated along the channel dimension. This is the bias vector for spatial attention;

[0034] Among them, the confidence level of the correspondence in the knowledge graph This is obtained by fusing entity relevance, semantic consistency of relations, and global reasoning information through a multilayer perceptron. The specific formula used is as follows:

[0035]

[0036] In the formula, This represents a multi-layer sensing fusion operation. It is the confidence level of entity relevance. It is the confidence level of relational semantic consistency. For global reasoning confidence, For the head entity, A relation is used to describe the association between a head entity and a tail entity. It is a tail entity;

[0037] A relation-aware attention mask is generated based on channel attention and spatial attention. This mask is then used to dynamically adjust the first input tensor, resulting in an adaptive feature data training tensor. The formula used to calculate this adaptive feature data training tensor is as follows:

[0038]

[0039] In the formula, To train tensors for adaptive feature data, Relationship-aware attention mask;

[0040] The logic behind establishing the ViT network model based on the first tensor and generating the comprehensive feature extraction model is as follows: the first tensor is used as the input of the ViT network model, and the correspondence between entities in the edge set of the corresponding nodes in the flame knowledge graph data is used as the label to train the ViT network model and obtain the comprehensive feature extraction model.

[0041] Furthermore, the hazard emergency coefficient is calculated based on the second tensor, and the specific formula for calculating the hazard emergency coefficient is as follows:

[0042]

[0043] In the formula, For the risk emergency response factor, This is a predicted rate of fire growth. This is the predicted value for the flame outline area. The minimum angle between the direction of flame spread and the residential area and the factory. This is a predicted value for the quantity of flammable materials. The characteristic value of smoke color. This is the weighting coefficient for the sum of the predicted fire growth rate and the predicted flame outline area. , and These are the weighting coefficients for the predicted quantity of flammable materials, the characteristic value of smoke color, and the direction of flame spread, respectively, relative to the minimum angle between residential areas and factories. and , , and All are greater than 0;

[0044] Among the smoke color feature values The formula used for the calculation is:

[0045]

[0046] In the formula, , and These are the red, green, and blue channel values ​​for the smoke color, respectively.

[0047] Among them, the minimum angle of flame spread direction relative to residential areas and factories. The formula used for the calculation is:

[0048]

[0049] In the formula, The angle of the direction of flame spread. The direction angle of the nearest residential area. The direction angle is the nearest factory.

[0050] Furthermore, the hazard emergency response coefficient is compared with the hazard judgment threshold, and corresponding processing methods are generated based on different comparison results. The specific judgment logic is as follows:

[0051] when If the fire is deemed to be of high danger, evacuate the crowd and contact the relevant authorities to take appropriate firefighting measures based on the location of the fire.

[0052] when If the current fire hazard level is judged to be "good," it means that the current fire will not cause a large-scale fire, and firefighting can be carried out independently based on the location of the fire source.

[0053] when When the current flame is deemed to be of poor danger level, it means that the flame will quickly extinguish and cannot spread.

[0054] in The hazard assessment threshold is determined by adjusting the concentration of volatile organic compounds and the grayscale value of the flame color at the image acquisition location. The formula used for the calculation is:

[0055]

[0056] In the formula, This is the initial value for the danger assessment threshold. The concentration of volatile organic compounds at the location where the flame image was acquired is shown. This is a reference value for the concentration of organic compounds. This represents the grayscale value of the flame color.

[0057] The logic behind training a tensor based on feature data of flame images from known flame knowledge graph data to establish a ViT network model and generate a comprehensive feature extraction model is as follows: the tensor of feature data training and the corresponding flame images are used as inputs to the ViT network model, and the correspondence between nodes in the edge set corresponding to the embedded 3D entity information is used as labels to train the ViT network model and obtain the comprehensive feature extraction model.

[0058] This invention also provides a knowledge- and vision-driven flame image recognition and analysis device, which is used to execute the above-described knowledge- and vision-driven flame image recognition and analysis method, including:

[0059] The entity relationship extraction module is used to acquire a training set and flame images to be identified. The training set includes multiple sets of trained flame images and flame knowledge graphs mapped to them. Information fusion is performed on the flame knowledge graph to obtain the corresponding entity embedding vectors. The training flame images are used as feature inputs and the corresponding entity embedding vectors are used as feature outputs to train a convolutional neural network to obtain an entity embedding vector evaluation model. The entity embedding vector evaluation model is used to perform feature recognition on the flame images to be identified to obtain the corresponding entity embedding vectors.

[0060] The training data processing module is used to segment the training flame image and the flame image to be identified into several image blocks of the same size, obtain the flame visual feature parameters of each image block and embed them into the corresponding image block to obtain the three-dimensional image block embedding tensor corresponding to each flame image, and perform tensor expansion operation on the entity embedding vector corresponding to each flame image in space to obtain the three-dimensional entity information embedding tensor corresponding to each flame image.

[0061] The relation tensor fusion and stitching module is used to stitch together the 3D image block embedding tensor and the 3D entity information embedding tensor corresponding to each flame image to obtain the corresponding feature data training tensor. The feature data training tensor corresponding to the trained flame image is labeled as the first tensor, and the feature data training tensor corresponding to the flame image to be identified is labeled as the second tensor.

[0062] The network model training module is used to train the ViT network model by taking the first tensor as the feature input and the corresponding relationship between entities in the flame image as the feature output. The trained ViT network model is then used to extract features from the second tensor to obtain the correspondence between entities in the flame image to be identified.

[0063] The language text generation module is used to model and analyze the second tensor to obtain the hazard emergency coefficient of the flame image to be identified, compare the hazard emergency coefficient with the hazard judgment threshold, determine the processing method based on the comparison result, and output the correspondence and processing method of entities in the flame image to be identified as language text.

[0064] Compared with the prior art, the beneficial effects of the present invention are:

[0065] First, by collecting flame knowledge graph data and combining it with the vectors between flame images and their corresponding entities, rich contextual information is provided to the convolutional neural network. This not only improves the model's ability to recognize complex features in flame images but also enhances its sensitivity to dynamic changes in flames. Training the convolutional neural network allows flame recognition to go beyond surface features, delving deeper to uncover important information such as the location of the fire source, the rate of fire growth, and the direction of spread, providing foundational data for subsequent hazard assessment. Second, by segmenting flame images and extracting visual feature parameters, a 3D image block embedding tensor and a 3D entity information embedding tensor are constructed and concatenated, effectively combining image features with relational features. This allows the model to more comprehensively consider the visual attributes of flames and their intrinsic relationships when analyzing flame images to be identified. Finally, the comprehensive feature extraction model obtained through training can quickly analyze flame images to be identified in real-time environments, calculate hazard emergency coefficients, and compare them with set hazard judgment thresholds. This improves the intelligence level of fire emergency response. By outputting the predicted feature values ​​and corresponding processing methods in a textual format, the fire monitoring and emergency response process becomes more efficient and clear. Attached Figure Description

[0066] Figure 1 This is a schematic diagram of the overall method flow of the present invention;

[0067] Figure 2 This is a schematic diagram of the segmentation structure of a flame image;

[0068] Figure 3 This is a schematic diagram of flame region extraction within a flame image;

[0069] Figure 4 Schematic diagram of the optimized edge of the flame area;

[0070] Figure 5 This is a schematic diagram for smoke recognition and extraction from flame images;

[0071] Figure 6 The feature image after segmentation and stitching of the flame image;

[0072] Figure 7 This is a schematic diagram of the overall device structure of the present invention. Detailed Implementation

[0073] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments.

[0074] It should be noted that, unless otherwise defined, the technical or scientific terms used in this invention should have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0075] Example:

[0076] Please see Figures 1-6 The present invention provides a technical solution:

[0077] A knowledge- and vision-driven method for flame image recognition and analysis includes the following steps:

[0078] Step 1: Obtain a training set and flame images to be identified. The training set includes multiple sets of trained flame images and flame knowledge graphs mapped to them. Information fusion is performed on the flame knowledge graphs to obtain corresponding entity embedding vectors. The trained flame images are used as feature inputs, and the corresponding entity embedding vectors are used as feature outputs to train a convolutional neural network to obtain an entity embedding vector evaluation model. The entity embedding vector evaluation model is used to perform feature recognition on the flame images to be identified to obtain the corresponding entity embedding vectors.

[0079] Several flame knowledge graph data sets are collected. These data sets include node sets and edge sets. The node sets cover all entities and concepts, such as "electrical equipment," "fire source," and "flame" in the flame-related knowledge graph. The edge sets represent the correspondences between entities, such as "electrical fault - causes - flame" and "fire source - located near - flammable material." Each node has an initial feature vector. The entity information of each node within the node set is fused with the information of its neighboring nodes. The neighboring nodes are four-way adjacent nodes, specifically the nodes directly connected to the node above, below, left, and right. For each node, its set of neighboring nodes is extracted, and the node's entity information is converted into a numerical feature vector. One-hot encoding, word embedding, or other vectorization techniques can be used to represent the node's attribute information. A suitable fusion strategy is selected; common methods include weighted summation or concatenation to obtain the entity embedding vector.

[0080] The entity embedding vectors have a unified dimension, and each vector integrates information about the node itself and its neighboring nodes, containing rich entity relationship information. For example, the entity relationship embedding vector of the "fire source" node integrates information from related nodes such as "flammable material" and "flame," which can reflect the relationship between them.

[0081] Several flame knowledge graph data are collected. The flame knowledge graph data is graph structure data, which includes a node set and an edge set. The node set covers all entities and concepts, and the edge set represents the correspondence between entities. The correspondence between entities is represented by vectors, specifically the connection vectors between the fire source location, fire growth rate, spread direction angle, and quantity of flammable materials. The fusion of the collected flame knowledge graph data information specifically refers to fusing the entity information of each node in the node set with the information of the adjacent nodes to obtain an entity embedding vector. The adjacent nodes are four adjacent nodes, specifically including the upper, lower, left, and right adjacent nodes. The entity information of each node in the node set includes the fire source location, fire growth rate, spread direction angle, and quantity of flammable materials.

[0082] Based on the collected flame knowledge graph data, a convolutional neural network is established. This convolutional neural network consists of an input layer, convolutional layers, pooling layers, fully connected layers, and an output layer. The activation function in the convolutional layers is... function, The specific expression of the function is:

[0083]

[0084] in, Indicates the first Each corresponding convolutional layer Then it means the first In the corresponding convolutional layer, the th The first feature map There are eigenvalues; among which For the index of the convolutional layer, For the index of training sample images, The index of the feature values ​​in the training sample image;

[0085] For the fully connected layer, the number of neurons in the fully connected layer is set to 32, the initial neural network learning rate is set to 0.001, and the number of training rounds is 100.

[0086] The trained entity embedding vector prediction model takes a flame image as input and outputs node entity information within the corresponding node set and the entity embedding vector fused from the corresponding nodes. Specifically, the entity embedding vector is a one-dimensional vector. , dimension .

[0087] The output is the updated feature vector of each node after processing by the convolutional neural network, i.e., the entity embedding. These vectors have a uniform dimension, and each vector integrates information from the node itself and its neighboring nodes, containing rich entity relationship information. For example, the entity relationship embedding vector of the "fire source" node integrates information from related nodes such as "flammable material" and "flame," which can reflect the relationship between them and provide prior relationship knowledge from a knowledge graph perspective for subsequent integration with the ViT model input.

[0088] Step 2: Segment the trained flame image and the flame image to be identified into several image blocks of the same size. Obtain the flame visual feature parameters of each image block and embed them into the corresponding image block to obtain the 3D image block embedding tensor corresponding to each flame image. Perform tensor expansion operation on the entity embedding vector corresponding to each flame image in space to obtain the 3D entity information embedding tensor corresponding to each flame image.

[0089] To achieve spatial collaborative reasoning between image regions and knowledge relationships, the flame images within the flame knowledge graph data are segmented into several image blocks of the same size. The specific formula used to calculate the number of segmented image blocks is as follows:

[0090]

[0091] In the formula, The number of image patches to be segmented. and The height and width of the input flame image are respectively. and These are the height and width of the segmented image block, respectively;

[0092] Based on the obtained flame visual feature parameters, an embedding operation is performed on each segmented image block to obtain a 3D image block embedding tensor, which is specifically represented as follows: ,in The number of channels for flame visual feature parameters defines the spatial size of image patches and describes their spatial layout. These flame visual feature parameters include the flame color grayscale value, the flame outline area, and the smoke color RGB value. The specific methods for obtaining these parameters are as follows: The image is converted from the RGB color space to HSV or another suitable color space for flame detection to more easily distinguish between flame and background. Color thresholding or edge detection algorithms, such as Canny edge detection, are used to extract the flame region. A threshold can be set based on typical flame colors, such as orange and red. The extracted flame region image is converted to a grayscale image, and the average grayscale value of the flame region is calculated as the flame color grayscale value. Contour detection algorithms, such as the findContours method in OpenCV, are used to identify the contour of the flame region, and the area of ​​the extracted contour is calculated, usually in pixels. Similar to flame region extraction, color thresholding is used to extract the smoke region. Alternatively, regions with lower grayscale values ​​can be used to represent smoke, and the average RGB value of the extracted smoke region is calculated as the smoke color RGB value.

[0093] Based on the obtained entity embedding vectors, a tensor expansion operation is performed on the entity embedding vectors in space to obtain a 3D entity information embedding tensor. The specific method of the tensor expansion operation is as follows: the relationship vectors between corresponding entities are repeated h times in the height direction and w times in the width direction, so that the entity relationship embedding is completely aligned with the image patch embedding in spatial layout. The expanded entity relationship embedding tensor transforms from a one-dimensional vector into a 3D entity information embedding tensor. The formula used for the specific calculation of the 3D entity information embedding tensor is as follows:

[0094]

[0095] In the formula, Embed tensors to 3D entity information. for A matrix of all 1s This represents the tensor outer product operation. Let be the entity embedding vector; let be the 3D entity information embedding tensor. This completes the expansion of entity relationships embedded in the spatial dimension.

[0096] Step 3: Concatenate the 3D image block embedding tensor and the 3D entity information embedding tensor corresponding to each flame image to obtain the corresponding feature data training tensor. Label the feature data training tensor corresponding to the trained flame image as the first tensor, and label the feature data training tensor corresponding to the flame image to be identified as the second tensor.

[0097] The obtained 3D image patch embedding tensor and 3D entity information embedding tensor are concatenated to obtain the feature data training tensor. The specific logic for concatenating the two tensors is as follows: the two tensors are connected along the channel dimension. That is, the C channels of the image patch embedding and the D channels of the expanded entity relationship embedding are concatenated sequentially to form a new tensor. After concatenation, the feature data training tensor Z with dimension Z is obtained. .

[0098] In this new tensor, the first C channels retain the visual feature information of the image patch itself, such as the grayscale value of the flame color, the area of ​​the flame outline, and the RGB value of the smoke color. The D channels that are then concatenated incorporate entity relationship information from the knowledge graph, such as the relationship between different fire sources and flames, like "an electrical fault caused a fire," and the relationship between the spread of fire and the surrounding environment, like "the spread of a forest fire is affected by wind direction," and other semantic information.

[0099] The feature data training tensor serves as the input to the relation-guided attention mechanism unit, possessing both visual features of the flame image patches and entity information from the knowledge graph. In subsequent model processing, this rich information provides strong support for the model to deeply understand flame images, accurately analyze flame features, and predict fire development trends, such as fire growth rate and spread direction.

[0100] The specific method for obtaining the second tensor is the same as the method described above, and will not be repeated here.

[0101] The spliced ​​feature data training tensor not only contains the visual features of the flame image patches but also incorporates entity association information from the knowledge graph. Training the ViT network model with this feature data training tensor allows for a more comprehensive understanding of the context of the flame image, enhancing the expressiveness of the features. By splicing along the channel dimension, the model receives input from multiple sources, which helps it learn more complex patterns and relationships during training, thereby improving its classification and prediction capabilities. Specifically, understanding the flame features while also grasping their relationship with the environment helps in a comprehensive assessment of fire development.

[0102] If the flame image patch embedding tensor and the entity information embedding tensor are input separately, the model may fail to capture the potential relationship between these two types of information. Specifically, the information cannot be linked, and the model may fail to learn the correlation between specific flame features and specific events such as "electrical faults." Furthermore, separate inputs may cause the model to rely solely on either visual features or semantic information, thus limiting the model's expressive power and learning effectiveness. Especially in complex flame scenes, a single feature is insufficient for accurate judgment.

[0103] Step 4: Use the first tensor as the feature input and the corresponding relationship between entities in the flame image to be trained as the feature output to train the ViT network model. Use the trained ViT network model to extract features from the second tensor to obtain the correspondence between entities in the flame image to be identified.

[0104] While multi-head attention in the ViT network model can effectively extract global dependencies of image patches, it relies solely on pixel-level features and lacks an understanding of semantic relationships within the knowledge graph, such as the causal relationship of "fire source - ignition - flame". Therefore, channel and spatial attention are introduced to bridge the gap between the semantics of the knowledge graph and the visual features of the image, guiding the model to focus on channel features and spatial regions related to entity relationships. Entity relationships in the knowledge graph, such as "forest fire - affected by wind direction", need to be mapped to specific channels and spatial locations in the image through an attention mechanism. Multi-head attention alone cannot directly associate the semantics of the knowledge graph; additional channel and spatial attention are required to encode the relational information of the knowledge graph into features.

[0105] Based on the obtained first tensor, a ViT network model is established. Training the ViT network model specifically includes constructing a relation-aware attention mask by embedding 3D entity information within the first tensor Z into a tensor. This relation-aware attention mask is constructed using both channel attention and spatial attention. The formula for calculating channel attention is as follows:

[0106]

[0107] In the formula, For channel attention, For activation function, This represents the confidence level of the correspondence in the knowledge graph. This indicates a global average pooling operation. This is the channel weight matrix. This is the bias vector for channel attention;

[0108] The formula used to calculate spatial attention is as follows:

[0109]

[0110] In the formula, For spatial attention, These are the weights of the convolutional layer. This represents the convolution operation. and These represent average pooling and max pooling operations, respectively. ( ); ( This indicates that the results of average pooling and max pooling are concatenated along the channel dimension. This is the bias vector for spatial attention;

[0111] Among them, the confidence level of the correspondence in the knowledge graph This is obtained by fusing entity relevance, semantic consistency of relations, and global reasoning information through a multilayer perceptron. The specific formula used is as follows:

[0112]

[0113] In the formula, This represents a multi-layer sensing fusion operation. It is the confidence level of entity relevance. It is the confidence level of relational semantic consistency. For global reasoning confidence, For the head entity, A relation is used to describe the association between a head entity and a tail entity. It is a tail entity;

[0114] A relation-aware attention mask is generated based on channel attention and spatial attention. This mask is then used to dynamically adjust the first input tensor, resulting in an adaptive feature data training tensor. The formula used to calculate this adaptive feature data training tensor is as follows:

[0115]

[0116] In the formula, To train tensors for adaptive feature data, Relationship-aware attention mask;

[0117] The logic behind establishing the ViT network model based on the first tensor and generating the comprehensive feature extraction model is as follows: the first tensor is used as the input of the ViT network model, and the correspondence between entities in the edge set of the corresponding nodes in the flame knowledge graph data is used as the label to train the ViT network model and obtain the comprehensive feature extraction model.

[0118] The correspondence between the entities is specifically defined as: adaptive feature data training tensor The input and output features are made to have the same shape. In the task of recognizing a factory fire scene, the knowledge graph contains relationships such as "electrical equipment - location - workshop". When the multi-head attention mechanism of the ViT network model is faced with an image containing a factory fire scene, it first identifies entities related to the fire in the image, such as burning electrical equipment, burning flammable chemical storage areas, and surrounding fire-fighting facilities. Then, for the "electrical equipment - location - workshop" relationship, the attention mask guides the model to focus on the flame and smoke diffusion areas around the electrical equipment in the workshop, as well as the connection between the electrical equipment and the workshop building structure, because these areas may play an important role in determining key information such as the cause of the fire and the direction of fire spread.

[0119] Step 5: Model and analyze the second tensor to obtain the hazard emergency coefficient of the flame image to be identified, compare the hazard emergency coefficient with the hazard judgment threshold, determine the processing method based on the comparison result, and translate the correspondence of entities in the flame image to be identified and the processing method into language text and output it.

[0120] The hazard emergency coefficient is calculated based on the second tensor, and the specific formula for calculating the hazard emergency coefficient is as follows:

[0121]

[0122] In the formula, For the risk emergency response factor, This is a predicted rate of fire growth. This is the predicted value for the flame outline area. The minimum angle between the direction of flame spread and the residential area and the factory. This is a predicted value for the quantity of flammable materials. The characteristic value of smoke color. This is the weighting coefficient for the sum of the predicted fire growth rate and the predicted flame outline area. , and These are the weighting coefficients for the predicted quantity of flammable materials, the characteristic value of smoke color, and the direction of flame spread, respectively, relative to the minimum angle between residential areas and factories. and , , and All are greater than 0.

[0123] It should be noted that the emergency response coefficient is... The hazard level is used to indicate the degree of danger of a flame within an image to be identified, where the hazard emergency factor is... The higher the value, the more dangerous the flame.

[0124] The rate of fire growth directly reflects how quickly a fire spreads; the faster the growth rate, the greater the threat the fire poses to the surrounding environment. Therefore, the predicted fire growth rate... With risk emergency coefficient Proportional, through the square The form emphasizes the danger of rapid fire growth, which can cause widespread fire damage in a short period of time. Therefore, the accelerated growth of a fire exponentially increases its danger.

[0125] The flame outline area represents the area occupied by a flame in space; a larger flame area indicates a greater potential threat. A larger flame area typically means the release of more heat and smoke, potentially leading to wider damage and higher risk. Therefore, increasing the flame outline area directly increases the hazard response factor; hence, the predicted value of the flame outline area... With risk emergency coefficient Proportional.

[0126] The angle at which a flame spreads determines its potential threat to a specific area, such as a residential area or factory. If the angle between the flame's direction of propagation and these areas is small—that is, close to the flame being directly towards these areas—the danger increases. Therefore, the minimum angle between the flame's direction of propagation and residential areas or factories is crucial. With risk emergency coefficient Inversely proportional, through By characterizing the effect, the square root form can gradually amplify this effect without causing an overly drastic one, thus ensuring the smoothness and rationality of the hazard assessment.

[0127] The quantity of flammable materials is closely related to the speed and intensity of fire spread. A larger quantity increases the potential damage and the greater the likelihood of a large-scale fire. Therefore, the predicted quantity of flammable materials... With risk emergency coefficient Proportional, through The denominator indicates that as the quantity increases, the risk factor will increase at a faster rate.

[0128] Among the smoke color feature values The formula used for the calculation is:

[0129]

[0130] In the formula, , and These are the red, green, and blue channel values ​​for the smoke color, respectively.

[0131] It should be noted that Used to represent the vividness of smoke color, where smoke color feature values The higher the value, the more vibrant the smoke color. More vibrant smoke often indicates the release of toxic gases, such as carbon monoxide and cyanide, which pose a direct threat to life. Therefore, the vibrancy of smoke color can serve as an important indicator for assessing the hazard of a fire scene. With risk emergency coefficient Proportional, logarithmic function This can effectively smooth out this effect, preventing excessive fluctuations in the risk factor within a small range of eigenvalues, while moderately enhancing the risk assessment when the eigenvalues ​​are high.

[0132] Among them, the minimum angle of flame spread direction relative to residential areas and factories. The formula used for the calculation is:

[0133]

[0134] In the formula, The angle of the direction of flame spread. The direction angle of the nearest residential area. This represents the direction angle of the nearest factory. North is the positive direction, and east is the positive direction for determining the angles in each direction.

[0135] The rate of fire spread is one of the most critical factors in assessing fire hazard. A faster spread means the fire can expand more widely in a short time, posing a greater threat. The area of ​​the flame is equally important, representing the extent of the fire in space. A larger flame area releases more heat and causes greater potential damage. Both the flame area and the rate of fire spread directly affect the degree of hazard. The quantity of flammable material directly affects the intensity and spread rate of the fire. A larger quantity of flammable material increases the risk and potential damage, but this doesn't reflect the degree of hazard in real time. Changes in the direction of flame spread can affect the threat of a fire to a specific area, but its impact is usually less direct than the quantity of flammable material. While smoke color can provide information about the nature of the fire, it is typically used as a supplementary indicator in fire assessment. Changes in smoke color may reflect incomplete combustion, but its impact on hazard is less direct than the direction of flame spread. Therefore, setting... and , , and All are greater than 0.

[0136] The hazard emergency response coefficient is compared with the hazard assessment threshold, and corresponding processing methods are generated based on different comparison results. The specific judgment logic is as follows:

[0137] when If the fire is deemed to be of high danger, evacuate the crowd and contact the relevant authorities to take appropriate firefighting measures based on the location of the fire.

[0138] when If the current fire hazard level is judged to be "good," it means that the current fire will not cause a large-scale fire, and firefighting can be carried out independently based on the location of the fire source.

[0139] when When the current flame is deemed to be of poor danger level, it means that the flame will quickly extinguish and cannot spread.

[0140] in The hazard assessment threshold is determined by adjusting the concentration of volatile organic compounds and the grayscale value of the flame color at the image acquisition location. The formula used for the calculation is:

[0141]

[0142] In the formula, This is the initial value for the danger assessment threshold. The concentration of volatile organic compounds at the location where the flame image was acquired is shown. This is a reference value for the concentration of organic compounds. This represents the grayscale value of the flame color.

[0143] Volatile organic compounds, such as benzene, toluene, and formaldehyde, can cause headaches, eye and throat irritation, nausea, and other symptoms if their concentrations are high, seriously affecting human health and safety. They can also react with nitrogen oxides to form ozone, leading to urban air pollution. Therefore, the higher the concentration of volatile organic compounds at the location where the flame image is to be identified, the lower the initial value of the hazard judgment threshold should be to prevent the hazard of the flame from being missed.

[0144] The grayscale value of flame color increases with increasing temperature, displaying an increasingly brighter color; therefore, the grayscale value of flame color... Risk assessment threshold Inversely proportional, through the logarithmic function It can appropriately enhance hazard assessment, preventing the underestimation of the danger of flames while avoiding misjudgment.

[0145] There are various methods for determining the concentration of volatile organic compounds (VOCs) at the location where the flame image is acquired, including gas chromatography, mass spectrometry, and sensor technology. Reference values ​​for VOC concentrations are provided. Configure based on expert experience.

[0146] The predicted feature values ​​and corresponding processing methods are transcribed into natural language text. Templates or text generation models can be used, allowing the model to generate text describing the fire source location, surrounding flammable materials, flame spread rate, spread direction angle, flame visual feature parameters, and corresponding response methods based on the recognized image content and the corresponding knowledge graph. This text generation model is developed and trained based on generative models in natural language processing. Data preparation: Training the language generation module requires collecting a large amount of fire-related text data, including fire accident reports, professional fire protection literature, and fire case analyses. Descriptions of the fire source location, surrounding flammable materials, flame spread rate, spread direction angle, flame visual feature parameters, and response methods are extracted from these texts. This text data is then associated and labeled with corresponding flame image data to form a training dataset. The text generation model is trained to output text describing the fire source location, surrounding flammable materials, flame spread rate, spread direction angle, flame visual feature parameters, and response methods. Specifically, the T5 model can be used. The T5 model is a model that transforms all language processing tasks into text generation tasks, possessing strong versatility and flexibility. The XLNet model combines the advantages of the T5 model and incorporates the characteristics of an autoregressive model, resulting in excellent performance in text generation.

[0147] Please see Figure 7 The present invention also provides a knowledge- and vision-driven flame image recognition and analysis device, which is used to execute the above-described knowledge- and vision-driven flame image recognition and analysis method, including:

[0148] The entity relationship extraction module is used to acquire a training set and flame images to be identified. The training set includes multiple sets of trained flame images and flame knowledge graphs mapped to them. Information fusion is performed on the flame knowledge graph to obtain the corresponding entity embedding vectors. The training flame images are used as feature inputs and the corresponding entity embedding vectors are used as feature outputs to train a convolutional neural network to obtain an entity embedding vector evaluation model. The entity embedding vector evaluation model is used to perform feature recognition on the flame images to be identified to obtain the corresponding entity embedding vectors.

[0149] The training data processing module is used to segment the training flame image and the flame image to be identified into several image blocks of the same size, obtain the flame visual feature parameters of each image block and embed them into the corresponding image block to obtain the three-dimensional image block embedding tensor corresponding to each flame image, and perform tensor expansion operation on the entity embedding vector corresponding to each flame image in space to obtain the three-dimensional entity information embedding tensor corresponding to each flame image.

[0150] The relation tensor fusion and stitching module is used to stitch together the 3D image block embedding tensor and the 3D entity information embedding tensor corresponding to each flame image to obtain the corresponding feature data training tensor. The feature data training tensor corresponding to the trained flame image is labeled as the first tensor, and the feature data training tensor corresponding to the flame image to be identified is labeled as the second tensor.

[0151] The network model training module is used to train the ViT network model by taking the first tensor as the feature input and the corresponding relationship between entities in the flame image as the feature output. The trained ViT network model is then used to extract features from the second tensor to obtain the correspondence between entities in the flame image to be identified.

[0152] The language text generation module is used to model and analyze the second tensor to obtain the hazard emergency coefficient of the flame image to be identified, compare the hazard emergency coefficient with the hazard judgment threshold, determine the processing method based on the comparison result, and output the correspondence and processing method of entities in the flame image to be identified as language text.

[0153] The above formulas are all dimensionless calculations. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters in the formulas are set by those skilled in the art according to the actual situation.

[0154] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented in software, the above embodiments can be implemented, in whole or in part, as a computer program product. Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution.

[0155] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0156] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A method for flame image recognition analysis driven by knowledge and vision, characterized in that, The specific steps include: Obtain a training set and a flame image to be identified, the training set including multiple groups of trained flame images and flame knowledge graphs mapped thereto, performing information fusion on the flame knowledge graphs to obtain corresponding entity embedding vectors, training a convolutional neural network with the trained flame images as feature inputs and the corresponding entity embedding vectors as feature outputs, obtaining an entity embedding vector evaluation model, and using the entity embedding vector evaluation model to perform feature recognition on the flame image to be identified to obtain a corresponding entity embedding vector; Segment the trained flame images and the flame image to be identified to obtain a plurality of image blocks of the same size, obtain flame visual feature parameters of each image block and embed them into the corresponding image block to obtain a three-dimensional image block embedding tensor corresponding to each flame image, and perform tensor expansion on the entity embedding vector corresponding to each flame image in space to obtain a three-dimensional entity information embedding tensor corresponding to each flame image; Splice the three-dimensional image block embedding tensor and the three-dimensional entity information embedding tensor corresponding to each flame image to obtain a corresponding feature data training tensor, and label the feature data training tensor corresponding to the trained flame image as a first tensor and the feature data training tensor corresponding to the flame image to be identified as a second tensor; Train a ViT network model with the first tensor as a feature input and the corresponding relationship of entities in the trained flame image as a feature output, use the trained ViT network model to extract features from the second tensor to obtain the corresponding relationship of entities in the flame image to be identified; Model and analyze the second tensor to obtain a danger emergency coefficient of the flame image to be identified, compare the danger emergency coefficient with a danger judgment threshold, determine a processing mode according to the comparison result, and output the corresponding relationship of entities in the flame image to be identified and the processing mode in a language text.

2. The method according to claim 1, wherein: A plurality of flame knowledge graph data are collected, the flame knowledge graph data being graph structure data, the flame knowledge graph data including a node set and an edge set, the node set covering all entities and concepts, and the edge set representing the corresponding relationship between entities, the corresponding relationship between entities being represented by a vector, specifically a connection relationship vector between a fire source location, a fire growth rate, a spread direction angle and a flammable material quantity, wherein the information fusion of the collected flame knowledge graph data specifically refers to fusing the entity information of each node in the node set with the entity information of adjacent nodes to obtain an entity embedding vector, the adjacent nodes being four adjacent nodes, specifically including the adjacent nodes above, below, left and right, and the entity information of each node in the node set including the fire source location, the fire growth rate, the spread direction angle and the flammable material quantity; Based on the collected flame knowledge graph data, a convolutional neural network is established, wherein the convolutional neural network is composed of an input layer, a convolutional layer, a pooling layer, a fully connected layer and an output layer, and the activation function in the convolutional layer is function, The specific expression of the function is: wherein, represents the i-th corresponding convolutional layer, represents the i-th corresponding convolutional layer, represents the i-th feature value of the i-th feature map in the i-th corresponding convolutional layer; wherein is an index of the convolutional layer, is an index of the training sample image, is an index of the feature value in the training sample image;​​​ For the fully connected layer, the number of fully connected layer neurons is set to 32, the initial neural network learning rate is set to 0.001, and the training round number is 100; The entity embedding vector prediction model input is a flame image, and the output is node entity information in a corresponding node set and an entity embedding vector fused after corresponding nodes, and the entity embedding vector is a one-dimensional vector , and the dimension is .

3. The method of claim 2, wherein the method further comprises: Segment the flame images in the flame knowledge graph data to obtain a plurality of image blocks of the same size, and the number of segmented image blocks is calculated according to the formula: wherein is the number of segmented image blocks, and are the height and width of the input flame image, respectively, and are the height and width of the segmented image blocks, respectively. Based on the obtained flame visual feature parameters, an embedding operation is performed on each segmented image block to obtain a three-dimensional image block embedding tensor, wherein the three-dimensional image block embedding tensor is specifically represented as wherein is the number of flame visual feature parameter channels, and the flame visual feature parameters include flame color gray value, flame contour area, and smoke color RGB value.

4. The method of claim 3, wherein: According to the obtained entity embedding vector, a tensor expansion operation is performed on the entity embedding vector in space to obtain a three-dimensional entity information embedding tensor, wherein the specific manner of the tensor expansion operation is: repeating the relationship vector between corresponding entities h times in the height direction and w times in the width direction, so that the entity relationship embedding is completely aligned with the image block embedding in the spatial layout, and the expanded entity relationship embedding tensor changes from a one-dimensional vector to a three-dimensional entity information embedding tensor, wherein the formula for calculating the three-dimensional entity information embedding tensor is: wherein, is a three-dimensional entity information embedding tensor, is a is an all-ones matrix, denotes a tensor outer product operation, is an entity embedding vector; the three-dimensional entity information embedding tensor is ; The obtained three-dimensional image block embedding tensor and three-dimensional entity information embedding tensor are spliced to obtain a feature data training tensor, wherein the specific logic for splicing the two tensors is: connecting the two tensors in the channel dimension, that is, splicing the C channels of the image block embedding and the D channels of the expanded three-dimensional entity information embedding tensor in order to form a new tensor, and after splicing, the feature data training tensor with a dimension of is obtained.

5. The method of claim 4, wherein: Based on the obtained first tensor, a ViT network model is established, wherein the training of the ViT network model specifically includes constructing a relationship-aware attention mask according to the three-dimensional entity information embedding tensor in the first tensor Z, wherein the relationship-aware attention mask is composed of channel attention and spatial attention, wherein the formula for calculating the channel attention is: wherein, is a channel attention, is an activation function, is a confidence of a corresponding relation in a knowledge graph, denotes a global average pooling operation, is a channel weight matrix, is a bias vector of the channel attention. wherein the formula for calculating the spatial attention is: wherein, is a spatial attention, is a weight of a convolution layer, denotes a convolution operation, and denote average pooling and max pooling operation, respectively, ( ); ( ) denotes concatenating the results of average pooling and max pooling in the channel dimension, is a bias vector of the spatial attention; Wherein, the confidence of the corresponding relationship in the knowledge graph The entity correlation, the relationship semantic consistency and the global inference information are fused by the multilayer perceptron, and the specific formula is: In the formula, represents a multi-layer perception fusion operation, is an entity correlation confidence, is a relationship semantic consistency confidence, is a global reasoning confidence, is a head entity, is a relationship, used to describe the association between the head entity and the tail entity, is a tail entity; The relationship-aware attention mask is generated according to the channel attention and the spatial attention, and the input first tensor is dynamically adjusted through the relationship-aware attention mask to obtain an adaptive feature data training tensor, wherein the formula for calculating the adaptive feature data training tensor is: wherein is the adaptive feature data training tensor, is the relationship-aware attention mask; wherein based on the first tensor, a ViT network model is established, and the specific logic for generating a comprehensive feature extraction model is: taking the first tensor as the input of the ViT network model, taking the corresponding relationship between the corresponding nodes in the edge set in the flame knowledge graph data as the label, training the ViT network model to obtain a comprehensive feature extraction model.

6. The method of claim 1, wherein: The dangerous emergency coefficient is calculated according to the second tensor, wherein the specific formula for calculating the dangerous emergency coefficient is: wherein, is a dangerous emergency coefficient, is a fire growth rate prediction value, is a flame profile area prediction value, is a minimum angle of flame spread direction to residential area and factory, is a flammable material quantity prediction value, is a smoke color characteristic value, is a weight coefficient of the sum of the fire growth rate prediction value and the flame profile area prediction value, , and are weight coefficients of the flammable material quantity prediction value, the smoke color characteristic value and the minimum angle of flame spread direction to residential area and factory, respectively, wherein and , , and are all greater than 0. wherein the smoke color characteristic value The formula on which the calculation is based is: wherein , and are the smoke color red, green and blue channel values, respectively. where the minimum angle of the flame spread direction to the residential area and the factory The formula for the calculation is: wherein is the angle of the direction of the fire spread, is the angle of the direction of the nearest residential area, is the angle of the direction of the nearest factory.

7. The method of claim 6, wherein: The dangerous emergency coefficient is compared with the dangerous judgment threshold, and according to different comparison results, a corresponding processing mode is generated, wherein the specific judgment logic is: When the current fire danger level is high, leave the evacuation crowd and contact the relevant departments, and take appropriate fire extinguishing measures according to the fire source location; When the current flame is determined to be safe, indicating that the current flame will not cause a large-scale fire and can be extinguished by itself according to the location of the fire source; When the current flame is determined to be poor, indicating that the current flame will quickly extinguish and cannot spread. wherein is a danger determination threshold value, which is corrected by the volatile organic compound concentration and the flame color gray value at the flame image acquisition site to be recognized, wherein the danger determination threshold value The formula for calculation is: In the formula, is a dangerous judgment threshold initial value, is a volatile organic compound concentration at a flame image collection position to be identified, is an organic compound concentration reference value, is a flame color gray value.

8. A knowledge and vision driven flame image recognition analysis apparatus, characterized by: The knowledge and vision jointly driven flame image recognition analysis device is used to execute the knowledge and vision jointly driven flame image recognition analysis method of any one of claims 1-7, comprising: An entity relationship extraction module is configured to obtain a training set and a flame image to be identified, wherein the training set includes multiple groups of training flame images and flame knowledge graphs mapped thereto, and the flame knowledge graphs are fused to obtain corresponding entity embedding vectors. The convolutional neural network is trained with the training flame images as feature inputs and the corresponding entity embedding vectors as feature outputs to obtain an entity embedding vector evaluation model. The entity embedding vector evaluation model is used to identify the features of the flame image to be identified to obtain corresponding entity embedding vectors. A training data processing module is configured to divide the training flame images and the flame image to be identified into a plurality of image blocks of the same size, obtain flame visual feature parameters of each image block, and embed the flame visual feature parameters into the corresponding image block to obtain a three-dimensional image block embedding tensor corresponding to each flame image. The entity embedding vector corresponding to each flame image is subjected to a tensor expansion operation in space to obtain a three-dimensional entity information embedding tensor corresponding to each flame image. The relationship tensor fusion splicing module is configured to splice the three-dimensional image block embedding tensor corresponding to each flame image and the three-dimensional entity information embedding tensor to obtain a corresponding feature data training tensor, calibrate the feature data training tensor corresponding to the trained flame image as a first tensor, and calibrate the feature data training tensor corresponding to the flame image to be identified as a second tensor. The network model training module is configured to train the ViT network model by taking the first tensor as a feature input and taking the corresponding relationship of the entity in the trained flame image as a feature output, and to use the trained ViT network model to perform feature extraction on the second tensor to obtain the corresponding relationship between the entities in the flame image to be identified. The language text generation module is configured to model and analyze the second tensor to obtain a danger emergency coefficient of the flame image to be identified, compare the danger emergency coefficient with a danger judgment threshold, determine a processing mode according to a comparison result, and output the corresponding relationship of the entities in the flame image to be identified and the processing mode in a language text.

Images