Ship fire multi-dimensional identification and classification method

Abstract

The invention belongs to the technical field of ship fire analysis, and particularly relates to a ship fire multi-dimensional identification and classification method. Comprising the following steps: continuously sampling fire images; preprocessing image data in the sampling sequence, and determining a spatial scale unified standard; the generalization index of the identification and analysis network is improved by carrying out necessary data enhancement processing on the sampling sequence; establishing a fire image recognition network based on element feature expression ability; dynamic and static fusion features obtained after network learning are obtained, a full-connection classification module E carries out training classification on the fire image samples by considering the dynamic and static fusion features of the samples, and a final fire image classification result is obtained. The fire recognition neural network can pay attention to expression ability of element features in same-level and same-kind fire images, key channel control in corresponding feature maps of the fire images is realized by using a global convolution attention mechanism, and classification and recognition of the fire images are realized by creating dynamic and static fusion features.

Description

Technical Field

[0001] The present invention belongs to the technical field of ship fire analysis, and particularly relates to a multi-dimensional recognition and classification method for ship fires. Background Technique

[0002] In recent years, with the development of visual analysis technology, methods for classifying and analyzing fire status information based on image feature recognition and classification counting have developed rapidly. Its basic process includes fire sample data collection, temporal sampling analysis of videos or images, data processing of image information, recognition and analysis of fire characteristics, etc. However, in the field of ship fire analysis, there are still many problems in the application of the above methods. First, due to the complex and changeable fire status information inside the ship, during the process of classifying and analyzing the fire category and the fire severity level, etc., the same-level and same-class fire image data usually have different image features according to their combustion status, fuel category, etc., resulting in large differences between the same-level and same-class fire images. During the process of recognizing and analyzing fire images, it is impossible to align and analyze them accurately, and it is easy to make classification and grading errors. Second, due to the differences in the ship's own structural design and the differences in the external environmental characteristics of the ship when the fire occurs, the characteristics expressed by the fire images of different ship fire incidents and the fire incidents that occur in different regions and at different times on the same ship also have differences. The current visual classification method relies to a large extent on the common visual features in the fire images for analysis, and has insufficient ability to identify the difference information between individual samples. Summary of the Invention

[0003] The purpose of the present invention is that the multi-dimensional recognition and classification method for ship fires in this application is to solve the problem of low efficiency and poor accuracy in fire image recognition in the complex situation of current ship fires. By optimizing the design of the dynamic and static multi-dimensional features of ship fires, the analysis of the difference in the fusion feature attributes in the same-class images is improved to achieve a better fire image classification and grading effect.

[0004] To achieve the above purpose, the present invention adopts the following technical solutions.

[0005] A multi-dimensional recognition and classification method for ship fires includes the following steps:

[0006] Step 1. Sampling and processing of fire images

[0007] For the i-th original image sampling sequence X based on temporal expression i ∈X A.H.W , where A represents the number of frames in the image sampling sequence, and H and W respectively refer to the pixel height and pixel width of the image;

[0008] Based on the continuous sampling method, randomly select the frame moment t0 ∈ [0, T i-T c , continuously collect T c frames of images starting from t0 as a sampling sequence, and the sampling sequence can be expressed as: X s = X i (t0 → t0 + T c );

[0009] where T i is the number of frames of the i-th original image, and T c is the number of sampling frames;

[0010] Step 2. Preprocessing of image data in the sampling sequence

[0011] For the pixel coordinates [x, y] of any reference feature in the fire image T , where x and y are the pixel coordinates of the centroid of the reference feature in the fire image, and its coordinates after spatial scale transformation are [x′, y′] T ;

[0012] For the entire sampling sequence, determining the best unified standard of spatial scale is an optimal solution problem, which can be expressed as: arg min||ARg + Δ - g′|| Frobenius ;

[0013] And

[0014] where ||*|| Frobenius refers to the Frobenius norm; x v , y v are the pixel coordinates of the centroid of the v-th reference feature in the fire image of the sampling sequence, where x′ v , y′ v are the pixel coordinates of the centroid of the v-th reference feature in the transformed image, and V is the total number of reference features;

[0015] Solve the above optimal solution to determine the unified standard of spatial transformation;

[0016] Step 3. Enhancement processing of the sampling sequence

[0017] Through necessary data enhancement processing of the sampling sequence, the generalization index of the recognition and analysis network is improved;

[0018] Step 4. Fire image recognition based on the expression ability of feature features

[0019] Establish a fire image recognition network based on the expression ability of feature features. The network sequentially includes a residual module A, a channel feature control module B, a spatial feature fusion module C, and a fully connected classification module E;

[0020] The residual module A is composed of two ResNet residual blocks. The residual module A obtains the sampling sequence and extracts the spatial features of the fire image through the residual blocks;

[0021] The channel feature control module B is composed of a transformation convolution module and a global convolutional attention module. The transformation convolution module is used to transform the input spatial feature map. Specifically, it means using a 2D spatial kernel convolution to perform convolution processing on each layer of the input feature map, and outputting a new feature map, which can be expressed as: F(X→U):

[0022] where X∈X A.H.W is the input feature map, U∈X C.H.W is the output feature map, V refers to the learned filter kernel, and V c refers to the parameters of the c-th learned filter, is the 2D spatial convolution kernel, and * represents the convolution operation;

[0023] The global convolutional attention module is used to extract the multi-channel attention weights of the spatial features. Specifically, it means enhancing the feature channel ability by generating video frame weight information considering the global channel attention weights, which can be expressed as:

[0024] where sig(.) is the Sigmoid function, Re(.) is the ReLU activation function, W1 and W2 are the weight matrices of the first and second fully connected layers, Z = ∑(U c ×W c ) is the channel descriptor obtained through global convolution, W c is the weight of the c-th channel of the global convolution kernel, U c is the tensor of the c-th channel of the input feature U, and Z c is the tensor of the c-th channel of the feature channel descriptor, is the element-wise multiplication;

[0025] The spatial feature fusion module C is used to fuse the short-term spatial dynamic features and static features, obtain the dynamic information of the fire pixel image, and optimize the dynamic and static feature fusion process according to the change of static features. Specifically:

[0026] For each sample, after passing through the residual module A and the channel feature control module B, T c static features E = {E j} are extracted from the video frame, where j = 1, 2, 3...T c ;

[0027] The spatial feature fusion module C, according to the static features E j and E j+1Calculate the feature difference between the previous and current moments to obtain the dynamic feature D; and further extract features from the static feature E and the dynamic feature D respectively. Specifically:

[0028] For the static feature E, first encode the static feature E according to the spatial position through a single-layer convolutional network to obtain the spatial sequence P, which can be expressed as:

[0029] where represents a single-layer convolutional operation, M f is the convolutional kernel of the single-layer convolution, N f is the bias of the single-layer convolution;

[0030] Define the position information sequence Q of the static feature E, and combine the spatial sequence P and the position encoding sequence Q to obtain the joint input sequence B = P + Q; use the input sequence B as the input and learn through the Transformer network;

[0031] For the dynamic feature D, further extract its features through a convolutional neural network, which can be expressed as:

[0032] where represents the convolution operation, M l is the convolutional kernel of the l-th convolution operation, N l is the bias of the l-th convolution operation, and l refers to the serial number of the convolution operation;

[0033] Obtain the static feature B″ obtained after learning through the Transformer network, and fuse it with the dynamic feature D′ of the previous and next frames to obtain the dynamic-static fusion feature, which can be expressed as: TZ i = B″ i + 0.5(D′ i-1 + D′ i );

[0034] where B″ i refers to the static feature of the i-th frame of the sample, and D′ i are the dynamic features of the i-th frame of the sample respectively;

[0035] where TZ1 = B″1 + D′1,

[0036] The fully connected classification module E trains and classifies the fire image samples by considering the dynamic-static fusion features of the samples to obtain the final fire image classification result.

[0037] The spatial transformation process of the further improvement or preferred implementation manner of the foregoing ship fire multi-dimensional recognition and classification method can be expressed as:

[0038]

[0039] where a x refers to the horizontal axis scaling ratio during spatial transformation, and a y refers to the vertical axis scaling ratio during spatial transformation, θ refers to the rotation angle during spatial transformation, and Δ x refers to the horizontal axis translation distance during spatial transformation, and Δ y refers to the vertical axis translation distance during spatial transformation.

[0040] For further improvement or preferred implementation of the aforementioned multi-dimensional recognition and classification method for ship fires, the following methods are adopted for data augmentation of images:

[0041] A. Randomly synchronously rotate all images in the sample;

[0042] B. Randomly adjust the brightness of all images in the sample;

[0043] C. Randomly intercept a random area of a given area from all images in the sample and scale it to the size of the original image.

[0044] For further improvement or preferred implementation of the aforementioned multi-dimensional recognition and classification method for ship fires, the l-th layer of the Transformer network can be expressed as:

[0045] B′ l = MultiHeadSelfAttention(LayerNormalization(B l-1 )) + B l-1

[0046] B l = FeedForwardNetwork(LayerNormalization(B′ l )) + B l

[0047] where LayerNormalization(.) represents the normalization layer, MultiHeadSelfAttention(.) represents the multi-head self-attention layer, and FeedForwardNetwork(.) represents the feed-forward network layer; B l is the output of the l-th layer and also the input of the l+1-th layer.

[0048] For further improvement or preferred implementation of the aforementioned multi-dimensional recognition and classification method for ship fires, as a preferred method, the convolution kernel size of the single-layer convolution is 2*2; the stride of the convolution operation when k = 1 is 2.

[0049] The beneficial effects are as follows:

[0050] Combined with the current fire recognition scheme, this application provides an analysis and processing scheme that considers the feature expression ability of elements in fire image information, enabling the fire recognition neural network to focus on the feature expression ability of elements in fire images of the same level and type, and using the global convolutional attention mechanism to achieve key channel control in the corresponding feature map of the fire image, suppressing irrelevant channel features, strengthening element channel features, and creating dynamic and static fusion features by combining the dynamic information and static features of adjacent frames of the image to achieve the classification and recognition of fire images. Brief Description of the Drawings

[0051] Figure 1 is a schematic flowchart of a multi-dimensional recognition and classification method for ship fires;

[0052] Figure 2 is a schematic structural diagram of a residual module and a channel feature control module. Detailed Embodiment

[0053] The following describes the present invention in detail with reference to specific embodiments.

[0054] To solve the problem that the difference between fire sample images of the same level and type in the background technology is too large to accurately complete hierarchical classification. This application combines the current fire recognition scheme, optimizes the feature expression ability of elements in fire image information, uses the global convolutional attention mechanism to achieve channel feature control, and combines static features to create dynamic and static fusion features to further improve the accuracy of fire image classification and recognition.

[0055] The following describes the specific scheme with reference to specific embodiments:

[0056] Step 1. Sampling and processing of fire images

[0057] Since there are large differences in the frame rate, duration, and number of frames of the original fire images, and there is a large amount of information redundancy in adjacent frame images, in order to unify the input data and extract effective data, it is necessary to sample the original fire images. Specifically:

[0058] For the i-th original image sampling sequence X based on temporal expression i ∈X A.H.W , where A represents the number of frames in the image sampling sequence, and H and W respectively refer to the pixel height and pixel width of the image;

[0059] Based on the continuous sampling method, randomly select the frame moment t0 ∈ [0, T i -T c , and continuously collect T c frames of images starting from t0 as the sampling sequence, and the sampling sequence can be expressed as: X s = X i (t0 → t0 + T c )

[0060] where T i is the number of frames of the i-th original image, and T c is the number of sampled frames;

[0061] Step 2. Preprocessing of the image data in the sampling sequence

[0062] Due to differences in factors such as the camera's viewing angle range, distance, and angle, the spatial scales in the fire images are not consistent, and a large amount of irrelevant background information is included. To avoid the interference of the above factors, the images in the sampling sequence should be subjected to homogenization processing. First, based on the same structural elements or reference elements in the ship fire images as the reference features, with the fire occurrence area as the reserved area, the reserved area is extracted through reference feature detection to obtain an image that only retains fire information, and a spatial scale transformation is performed based on the reference features so that the fire images have the same spatial scale. Specifically:

[0063] For the pixel coordinates [x, y] of any reference feature in the fire image T , where x and y are the pixel coordinates of the centroid of the reference feature in the fire image. Assuming that its coordinates after spatial scale transformation are [x′, y′] T , then its spatial transformation process can be expressed as:

[0064]

[0065] where a x refers to the horizontal axis scaling ratio during spatial transformation, and a y refers to the vertical axis scaling ratio during spatial transformation, θ refers to the rotation angle during spatial transformation, and Δ x refers to the horizontal axis translation distance during spatial transformation, and Δ y refers to the vertical axis translation distance during spatial transformation;

[0066] For the entire sampling sequence, determining the optimal spatial scale unification standard is an optimal solution problem, which can be expressed as: arg min||ARg+Δ - g′|| Frobenius

[0067] and

[0068] where ||*|| Frobenius refers to the Frobenius norm; x v , y v are the pixel coordinates of the centroid of the v-th reference feature in the fire images of the sampling sequence, where x′ v , y′ v are the pixel coordinates of the centroid of the v-th reference feature in the transformed image, and V is the total number of reference features;

[0069] Solve the above optimal solution to determine the unified standard for spatial transformation;

[0070] Step 3: Sampling sequence enhancement processing

[0071] Based on Step 2, initial data for recognition and analysis can be obtained. However, due to the influence of the network's own fitting properties, overfitting may occur during the recognition training process, affecting the network quality. Therefore, it is necessary to perform necessary data enhancement processing on the sampling sequence to improve the generalization index of the recognition and analysis network;

[0072] The following methods are mainly used for data enhancement of images:

[0073] A. Randomly rotate all images in the sample synchronously;

[0074] B. Randomly adjust the brightness of all images in the sample;

[0075] C. Randomly intercept a random area of a given area from all images in the sample and scale it to the size of the original image;

[0076] Step 4: Fire image recognition based on the feature expression ability of features

[0077] In this step, a fire image recognition network based on the feature expression ability of features is established. The network sequentially includes a residual module A, a channel feature control module B, a spatial feature fusion module C, and a fully connected classification module E. The basic structures and uses of each module are described below.

[0078] The residual module A is composed of two ResNet residual blocks. The residual module A obtains the sampling sequence and extracts the spatial features of the fire image through the residual blocks;

[0079] The channel feature control module B is composed of a transformation convolution module and a global convolution attention module. The transformation convolution module is used to transform the input spatial feature map. Specifically, it means using a 2D spatial kernel convolution to perform convolution processing on each layer of the input feature map to output a new feature map, which can be expressed as: F(X→U):

[0080] where X ∈ X A.H.W is the input feature map, U ∈ X C.H.W is the output feature map, V refers to the learned filter kernel, V c is the parameter of the learned c-th filter, is the 2D spatial convolution kernel, and * represents the convolution operation;

[0081] Global Convolution Attention Module, which is used to extract multi-channel attention weights of spatial features. Specifically, it refers to enhancing the feature channel ability by generating video frame weight information considering global channel attention weights, and can be expressed as:

[0082] where sig(.) is the Sigmoid function, Re(.) is the ReLU activation function, W1 and W2 refer to the weight matrices of the first and second fully connected layers, Z = ∑(U c ×W c ) refers to the channel descriptor obtained through global convolution, W c is the weight of the c-th channel of the global convolution kernel, U c refers to the tensor of the c-th channel of the input feature U, Z c refers to the tensor of the c-th channel of the feature channel descriptor, refers to element-wise multiplication;

[0083] Spatial Feature Fusion Module C is used to fuse short-term spatial dynamic features and static features, obtain the dynamic information of the fire pixel image, and optimize the dynamic and static feature fusion process according to the change of static features. Specifically:

[0084] For each sample, after passing through the residual module A and the channel feature control module B, T c static features E = {E j} are extracted from the video frame, where j = 1, 2, 3...T c ;

[0085] The Spatial Feature Fusion Module C calculates the feature difference between the previous and current moments based on the static features E j and E j+1 of adjacent frames to obtain the dynamic feature D; and further feature extraction is performed on the static feature E and the dynamic feature D respectively. Specifically:

[0086] For the static feature E, first, the static feature E is encoded according to the spatial position through a single-layer convolutional network to obtain the spatial sequence P, which can be expressed as:

[0087] where F Mf.Nf (.) represents the single-layer convolutional operation, M f is the convolutional kernel of the single-layer convolution, and N f is the bias of the single-layer convolution;

[0088] Define the position information sequence Q of the static feature E, and combine the spatial sequence P and the position encoding sequence Q to obtain the joint input sequence B = P + Q; use the Transformer network to learn with the input sequence B as the input, where the l-th layer of the Transformer network can be expressed as:

[0089] B′ l = MultiHeadSelfAttention(LayerNormalization(B l-1 )) + B l-1

[0090] B l = FeedForwardNetwork(LayerNormalization(B′ l )) + B l

[0091] where LayerNormalization(.) represents the normalization layer, MultiHeadSelfAttention(.) represents the multi-head self-attention layer, and FeedForwardNetwork(.) represents the feed-forward network layer; B l is the output of the l-th layer and also the input of the (l + 1)-th layer;

[0092] For the dynamic feature D, its features are further extracted through a convolutional neural network, which can be expressed as:

[0093] where represents the convolution operation, M l is the convolution kernel of the l-th convolution operation, and N l is the bias of the l-th convolution operation, and l refers to the serial number of the convolution operation;

[0094] As a preferred method, the size of the convolution kernel of a single-layer convolution is 2 * 2; the stride of the convolution operation when k = 1 is 2;

[0095] Obtain the static feature B″ obtained after learning through the Transformer network, and fuse it with the dynamic features D′ of the previous and next frames to obtain the dynamic-static fusion feature, which can be expressed as: TZ i = B″ i + 0.5(D′ i-1 + D′ i ) ;

[0096] where B″ i refers to the static feature of the i-th frame of the sample, and D′ i are the dynamic features of the i-th frame of the sample respectively;

[0097] where TZ1 = B″1 + D′1,

[0098] The fully connected classification module E trains and classifies the fire image samples by considering the dynamic-static fusion features of the samples, and obtains the final fire image classification result.

[0099] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than limiting the protection scope of the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced without departing from the essence and scope of the technical solutions of the present invention.

Claims

1. A multi-dimensional identification and classification method for ship fires, characterized in that: The steps include: Step 1: Sampling and processing of fire images For the i-th original image sampling sequence X based on time series expression i ∈X A.H.W , where A represents the number of frames in the image sampling sequence, H and W refer to the pixel height and pixel width of the image respectively; Based on the continuous sampling method, the frame time t0∈[0,T i -T c ], starting from t0, continuously collect T c The frame image is used as a sampling sequence, and the sampling sequence can be expressed as: s =X i (t0→t0+T c ); Where T i is the number of frames of the i-th original image, T c is the number of sampling frames; Step 2: Preprocessing of image data in the sampling sequence For any reference feature in the fire image, the pixel coordinates [x,y] T , where x, y are the pixel coordinates of the centroid of the reference feature in the fire image, and its coordinates after spatial scale transformation are [x′, y′] T ; For the entire sampling sequence, determining the best spatial scale uniform standard is an optimal solution problem, which can be expressed as: argmin||ARg+Δ-g′|| Frobenius ; and where ||*|| Frobenius refers to the Frobenius norm; x v ,y v is the pixel coordinate of the centroid of the vth benchmark feature in the fire image of the sampling sequence, where x v ′,y v ′ is the pixel coordinate of the centroid of the vth fiducial feature in the transformed image, and V is the total number of fiducial features; Solve the above optimal solution and determine the unified standard of space transformation; Step 3: Sampling sequence enhancement processing By performing necessary data enhancement processing on the sampling sequence, the generalization index of the recognition analysis network can be improved; Step 4: Fire image recognition based on element feature expression capabilities Establishing a fire image recognition network based on the expression ability of element features, the network sequentially includes a residual module A, a channel feature control module B, a spatial feature fusion module C, and a fully connected classification module E; The residual module A is composed of two ResNet residual blocks, and the residual module A obtains the sampling sequence and extracts the spatial features of the fire image through the residual blocks; The channel feature control module B consists of a transformation convolution module and a global convolution attention module. The transformation convolution module is used to transform the input spatial feature map. Specifically, it uses a 2D spatial kernel convolution to perform convolution processing on each layer of the input feature map and outputs a new feature map, which can be expressed as: F(X→U): where X∈X A.H.W is the input feature map, U∈X C.H.W is the output feature map, V refers to the learned filter kernel, V c refers to the learned parameters of the cth filter, refers to the 2D spatial convolution kernel, and * indicates the convolution operation; The global convolutional attention module is used to extract the multi-channel attention weights of spatial features. Specifically, it generates video frame weight information considering the global channel attention weights to enhance the feature channel capabilities, which can be expressed as: Where sig(.) is the Sigmoid function, Re(.) is the ReLU activation function, W1 and W2 refer to the weight matrices of the first and second fully connected layers, and Z = ∑(U c ×W c ) refers to the channel descriptor obtained after global convolution, W c is the weight of the cth channel of the global convolution kernel, U c Refers to the tensor of the cth channel of the input feature U, Z c refers to the tensor of the c-th channel of the feature channel descriptor, refers to element-wise multiplication; The spatial feature fusion module C is used to fuse short-term spatial dynamic features and static features, obtain the dynamic information of the fire pixel image, and optimize the dynamic and static feature fusion process according to the changes of static features. Specifically: For each sample, T in the video frame is extracted through the residual module A and the channel feature control module B. c Static features E = {E j }, j = 1, 2, 3...T c ; The spatial feature fusion module is based on C and the static features E of adjacent frames. j and E j+1 The feature difference between the previous and next moments is calculated to obtain the dynamic feature D; and further feature extraction is performed on the static feature E and the dynamic feature D respectively. Specifically: For the static feature E, firstly, the static feature E is encoded according to the spatial position through a single-layer convolutional network to obtain the spatial sequence P, which can be expressed as: in Represents a single-layer convolution operation, M f is the convolution kernel of a single layer of convolution, N f is the bias of a single layer of convolution; Define the position information sequence Q of the static feature E, combine the spatial sequence P and the position encoding sequence Q to obtain the joint input sequence B = P + Q; use the input sequence B as input to learn using the Transformer network; For the dynamic feature D, the convolutional neural network is used to further extract its features, which can be expressed as: in represents the convolution operation, M l is the convolution kernel of the lth convolution operation, N l is the bias of the lth convolution operation, where l refers to the sequence number of the convolution operation; Obtain the static feature B″ obtained after Transformer network learning, and fuse it with the dynamic feature D′ of the previous and next frames to obtain the dynamic and static fusion feature, which can be expressed as: TZ i =B i ″+0.5(D i ' -1 +D i ′); Among them B i ″ refers to the static features of the sample frame i, D i ′ are the dynamic features of the i-th frame of the sample; Where TZ1=B1″+D1′, The fully connected classification module E trains and classifies fire image samples by considering the dynamic and static fusion features of the samples to obtain the final fire image classification results.

2. The multi-dimensional identification and classification method for ship fires according to claim 1 is characterized in that: The spatial transformation process can be expressed as: where a x It refers to the horizontal axis scaling ratio during space transformation, a y refers to the vertical axis scaling ratio during spatial transformation, θ refers to the rotation angle during spatial transformation, Δ x Refers to the horizontal translation distance during space transformation, Δ y It refers to the longitudinal translation distance during space transformation.

3. The multi-dimensional identification and classification method for ship fires according to claim 1 is characterized in that: The following methods are used to enhance the image data: A. Randomly rotate all images in the sample synchronously; B. Randomly adjust the brightness of all images in the sample; C. Randomly extract a random region of a given area from all images in the sample and scale it to the size of the original image.

4. The multi-dimensional identification and classification method for ship fires according to claim 1 is characterized in that: The lth layer of the Transformer network can be expressed as: B l ′=MultiHeadSelfAttention(LayerNormalization(B l-1 ))+B l-1 B l =FeedForwardNetwork(LayerNormalization(B l ′))+B l Among them, LayerNormalization(.) represents the normalization layer, MultiHeadSelfAttention(.) represents the multi-head self-attention layer, and FeedForwardNetwork(.) represents the feedforward network layer; B l It is the output of the lth layer and also the input of the l+1th layer.

5. The multi-dimensional identification and classification method for ship fires according to claim 1 is characterized in that: As a preferred method, the convolution kernel size of a single-layer convolution is 2*2; the step size of the convolution operation when k=1 is 2.

Images