A multi-dimensional identification and classification method for ship fire

Abstract

The application belongs to the technical field of ship fire analysis, and particularly relates to a ship fire multi-dimensional identification and classification method. The method comprises the following steps: continuously sampling fire images; pre-processing image data in the sampling sequence to determine a unified spatial scale standard; improving the generalization index of the identification and analysis network through necessary data enhancement processing on the sampling sequence; establishing a fire image identification network based on the element feature expression capability; obtaining dynamic and static fusion features obtained after network learning; and training and classifying fire image samples through a full-connection classification module E by considering the dynamic and static fusion features of the samples, to obtain the final fire image classification result. The fire identification neural network can pay attention to the expression capability of element features in the same level and same type fire images, and utilize a global convolution attention mechanism to realize key channel control in the corresponding feature map of the fire image, to create dynamic and static fusion features to realize the classification and identification of the fire image.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of ship fire analysis, and particularly relates to a ship fire multi-dimensional identification and classification method. BACKGROUND

[0002] In recent years, with the development of visual analysis technology, the method of fire state information classification analysis based on image feature recognition and classification counting has developed rapidly. The basic process includes fire sample data collection, video or image time sequence sampling analysis, image information data processing, fire feature recognition analysis, etc. However, in the field of ship fire analysis, there are still many problems in the use of the above method. First, due to the complex and changeable fire state information in the ship, during the classification and classification analysis process of fire categories and fire degree levels, the same level and same type of fire image data will have different image features according to their burning state, fuel category, etc., resulting in large differences between the same level and same type of fire image. During the identification and analysis process of the fire image, it is difficult to accurately analyze and process, and errors in classification and classification are prone to occur. Second, due to the difference in the structure design of the ship itself and the difference in the external environmental characteristics of the ship when the fire occurs, the features expressed by the fire images of different ship fire events and the fire events occurring in different areas and at different times of the same ship are different. The current visual classification method relies on the common visual features in the fire image for analysis, and the recognition ability for the difference information between individual samples is insufficient. SUMMARY

[0003] The purpose of the present application is to solve the problem of low recognition efficiency and poor accuracy of fire images in the complex situation of ship fire by optimizing the design of dynamic and static multi-dimensional features of ship fire, improving the fusion feature attribute difference analysis in the same type of image, and achieving better classification and grading effect of fire images.

[0004] To achieve the above purpose, the technical scheme is as follows.

[0005] A ship fire multi-dimensional identification and classification method, comprising the following steps:

[0006] Step one, sampling and processing of fire image

[0007] For the i-th original image sampling sequence X expressed based on time sequence i ∈X A.H.W Wherein A represents the frame number of the image sampling sequence, H and W respectively represent the pixel height and pixel width of the image;

[0008] Based on the continuous sampling method, a frame time t0∈[0,T i-T c ], continuously collect T starting from t0 c A frame image is a sampling sequence, and the sampling sequence can be expressed as: X s =X i (t0→t0+T c );

[0009] Where T i Let T be the frame number of the i-th original image. c This represents the number of sampling frames.

[0010] Step 2: Preprocessing of image data in the sampling sequence

[0011] For any reference feature pixel coordinates [x,y] in a fire image T , where x, y are the pixel coordinates of the reference feature centroid in the fire image, and their coordinates after spatial scale transformation are [x′, y′]. T ;

[0012] Determining the optimal spatial scale standard for the entire sampling sequence is an optimization problem, which can be expressed as: arg min||ARg+Δ-g′|| Frobenius ;

[0013] and

[0014] Where ||*|| Frobenius This refers to the Frobenius norm; x v ,y v Let x′ be the pixel coordinates of the v-th reference feature centroid in the sampled sequence fire image, where x′ is the reference feature centroid. v ,y′ v Let V be the pixel coordinates of the centroid of the v-th reference feature in the transformed image, where V is the total number of reference features;

[0015] Solve for the above optimal solution to determine a unified standard for spatial transformation;

[0016] Step 3: Sample Sequence Enhancement Processing

[0017] By performing necessary data augmentation on the sampled sequences, the generalization index of the identification and analysis network can be improved.

[0018] Step 4: Fire Image Recognition Based on Feature Representation Capabilities

[0019] A fire image recognition network based on the ability to express element features is established. The network includes, in sequence, a residual module A, a channel feature control module B, a spatial feature fusion module C, and a fully connected classification module E.

[0020] The residual module A is composed of two ResNet residual blocks, which obtains the sampling sequence and extracts the spatial features of the fire image through the residual block;

[0021] The channel feature control module B is composed of a transformation convolution module and a global convolution attention module, wherein the transformation convolution module is used to transform the input spatial feature map, specifically, a 2D spatial kernel convolution is used to convolve each layer of the input feature map, and a new feature map is output, which can be represented as: F(X→U):

[0022] Wherein X∈X A.H.W is the input feature map, U∈X C.H.W is the output feature map, V is the learned filter kernel, V c is the learned filter parameter of the cth, is a 2D spatial convolution kernel, and * represents convolution operation;

[0023] The global convolution attention module is used to extract the multi-channel attention weight of the spatial feature, specifically, the video frame weight information considering the global channel attention weight is generated to enhance the feature channel capability, which can be represented as:

[0024] Wherein sig(.) is the Sigmoid function, Re(.) is the ReLU activation function, W1 and W2 are the first and second fully connected layer weight matrices, Z=∑(U c ×W c ) is the channel descriptor obtained by global convolution, W c is the weight of the cth channel of the global convolution kernel, U c is the tensor of the cth channel of the input feature U, Z c is the tensor of the cth channel of the feature channel descriptor, is the element-wise multiplication;

[0025] The spatial feature fusion module C is used to fuse the short-time spatial dynamic feature and static feature, obtain the dynamic information of the fire pixel image, and optimize the dynamic and static feature fusion process according to the static feature change, specifically:

[0026] For each sample, T c static features E={E j} are extracted in the video frame through the residual module A and the channel feature control module B, j=1, 2, 3...T c ;

[0027] The spatial feature fusion module C obtains the static features E j and E j+1The feature difference between the before and after time is calculated to obtain a dynamic feature D; and the static feature E and the dynamic feature D are further extracted, specifically:

[0028] For the static feature E, the static feature E is first encoded according to the spatial position by a single-layer convolutional network to obtain a spatial sequence P, which can be represented as:

[0029] Wherein represents a single-layer convolution operation, M f is a convolution kernel of the single-layer convolution, N f is a bias quantity of the single-layer convolution;

[0030] The position information sequence Q of the static feature E is defined, and the spatial sequence P and the position encoding sequence Q are combined to obtain a joint input sequence B=P+Q; the input sequence B is input into the Transformer network for learning;

[0031] For the dynamic feature D, the feature is further extracted by a convolutional neural network, which can be represented as:

[0032] Wherein represents a convolution operation, M l is a convolution kernel of the lth convolution operation, N l is a bias quantity of the lth convolution operation, and l is the serial number of the convolution operation;

[0033] The static feature B'' obtained after learning by the Transformer network is obtained, and the dynamic feature D' before and after the frame is fused to obtain a dynamic static fusion feature, which can be represented as: TZ i =B'' i +0.5(D' i-1 +D' i );

[0034] Wherein B'' i is the static feature of the i-th frame of the sample, D' i is the dynamic feature of the i-th frame of the sample, respectively;

[0035] Wherein TZ1=B''1+D'1,

[0036] The full-connection classification module E is trained and classified by considering the dynamic static fusion feature of the sample to obtain the final fire image classification result.

[0037] The further improved or preferred embodiment of the ship fire multi-dimensional recognition classification method is that the spatial transformation process can be represented as:

[0038]

[0039] wherein a x is the horizontal axis scaling ratio during spatial transformation, a y is the vertical axis scaling ratio during spatial transformation, and theta is the rotation angle during spatial transformation, Delta x is the horizontal axis translation distance during spatial transformation, and Delta y is the vertical axis translation distance during spatial transformation.

[0040] The further improvement or preferred embodiment of the aforementioned ship fire multi-dimensional recognition classification method adopts the following methods for data enhancement of the image:

[0041] A. Randomly synchronously rotate all images in the sample;

[0042] B. Randomly adjust the brightness of all images in the sample;

[0043] C. Randomly cut a random area of a given area from all images in the sample, and scale it to the size of the original image.

[0044] The lth layer of the Transformer network in the further improvement or preferred embodiment of the aforementioned ship fire multi-dimensional recognition classification method can be represented as:

[0045] B′ l = MultiHeadSelfAttention(LayerNormalization(B l-1 ))+B l-1

[0046] B l = FeedForwardNetwork(LayerNormalization(B′ l ))+B l

[0047] wherein LayerNormalization(.) represents a normalization layer, MultiHeadSelfAttention(.) represents a multi-head self-attention layer, and FeedForwardNetwork(.) represents a feedforward network layer; B l is the output of the lth layer, and is also the input of the (l+1)th layer.

[0048] As a preferred embodiment, the further improvement or preferred embodiment of the aforementioned ship fire multi-dimensional recognition classification method has a convolution kernel size of 2*2 for a single-layer convolution, and a step size of 2 for the convolution operation when k=1.

[0049] The beneficial effects are:

[0050] The application provides an analysis and processing scheme considering the element feature expression capability in fire image information, so that the fire recognition neural network can pay attention to the element feature expression capability in the same level and same type fire image, and uses a global convolution attention mechanism to realize key channel control in the corresponding feature map of the fire image, suppresses irrelevant channel features, strengthens element channel features, and creates dynamic and static fusion features by combining dynamic information of adjacent image frames with static features to realize classification and recognition of the fire image. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 is a flowchart of a multi-dimensional recognition and classification method for ship fire;

[0052] Figure 2 is a structural diagram of a residual module and a channel feature control module. DETAILED DESCRIPTION

[0053] The application will be described in detail below in combination with specific embodiments.

[0054] To solve the problem that the same level and same type fire sample image has too large difference and cannot accurately complete hierarchical classification, the application combines the current fire recognition scheme, optimizes the element feature expression capability in the fire image information, uses a global convolution attention mechanism to realize channel feature control, and creates dynamic and static fusion features by combining static features to further improve the accuracy of classification and recognition of the fire image.

[0055] The specific scheme will be described below in combination with specific embodiments.

[0056] Step one, sampling processing of fire image

[0057] Since the frame rate, time length and frame number of the original fire image all have large differences, and there is a large information redundancy in adjacent frames, in order to unify the input data and extract effective data, the original fire image needs to be sampled, specifically:

[0058] For the i-th original image sampling sequence X based on time sequence expression i ∈X A.H.W , where A represents the frame number of the image sampling sequence, H and W represent the pixel height and pixel width of the image respectively;

[0059] Based on the continuous sampling method, a frame time t0∈[0,T i -T c ] is randomly selected, T c frame images are continuously collected from t0 as a sampling sequence, and the sampling sequence can be expressed as: X s = X i (t0→t0+T c )

[0060] Where T i Let T be the frame number of the i-th original image. c This represents the number of sampling frames.

[0061] Step 2: Preprocessing of image data in the sampling sequence

[0062] Due to differences in camera field of view, distance, angle, and other factors, the spatial scale of fire images is inconsistent, and they also contain a large amount of irrelevant background information. To avoid interference from these factors, the images in the sampling sequence should be homogenized. First, based on the same structural elements or reference elements in the ship fire images as reference features, and with the fire occurrence area as the retained area, the retained area is extracted through reference feature detection to obtain an image that retains only fire information. Then, spatial scale transformation is performed based on the reference features to make the fire images have the same spatial scale. Specifically:

[0063] For any reference feature pixel coordinates [x,y] in a fire image T , where x, y are the pixel coordinates of the reference feature centroid in the fire image, assuming their coordinates after spatial scale transformation are [x′, y′]. T Then its spatial transformation process can be expressed as:

[0064]

[0065] Where a x This refers to the horizontal scaling ratio during spatial transformation, a y θ refers to the scaling ratio of the vertical axis during spatial transformation, θ refers to the rotation angle during spatial transformation, and Δ x Δ refers to the horizontal translation distance during spatial transformation. y This refers to the vertical axis translation distance during spatial transformation;

[0066] Determining the optimal spatial scale standard for the entire sampling sequence is an optimization problem, which can be expressed as: arg min||ARg+Δ-g′|| Frobenius

[0067] and

[0068] Where ||*|| Frobenius This refers to the Frobenius norm; x v ,y v Let x′ be the pixel coordinates of the v-th reference feature centroid in the sampled sequence fire image, where x′ is the reference feature centroid. v ,y′ v Let V be the pixel coordinates of the centroid of the v-th reference feature in the transformed image, where V is the total number of reference features;

[0069] Solving the above optimal solution, determining the unified standard of spatial transformation;

[0070] Step three, sample sequence enhancement processing

[0071] Based on step two, initial data can be obtained for identification analysis, but due to the influence of network fitting properties, overfitting may occur during the identification training process, which affects the quality of the network. Therefore, necessary data enhancement processing is needed for the sample sequence to improve the generalization index of the identification analysis network.

[0072] The following methods are mainly used for data enhancement of images:

[0073] A, randomly rotate all images in the sample synchronously;

[0074] B, randomly adjust the brightness of all images in the sample;

[0075] C, randomly cut a random area of a given area from all images in the sample, and scale it to the size of the original image;

[0076] Step four, fire image identification based on element feature expression ability

[0077] This step establishes a fire image identification network based on the expression ability of element features. The network includes residual module A, channel feature control module B, spatial feature fusion module C, and full connection classification module E in turn. The basic structure and purpose of each module are described below.

[0078] Residual module A is composed of two ResNet residual blocks. The residual module A obtains the sample sequence and extracts the spatial features of the fire image through the residual block;

[0079] The channel feature control module B is composed of a transformation convolution module and a global convolution attention module. The transformation convolution module is used to convert the input spatial feature map. Specifically, a 2D spatial kernel convolution is used to convolve each layer of the input feature map, outputting a new feature map, which can be represented as: F(X→U):

[0080] Where X∈X A.H.W is the input feature map, U∈X C.H.W is the output feature map, V is the learned filter kernel, V c is the learned parameter of the cth filter, is a 2D spatial convolution kernel, and * represents convolution operation;

[0081] A global convolution attention module is used to extract multi-channel attention weights of spatial features, specifically, video frame weight information considering global channel attention weights is generated to enhance feature channel capability, which can be expressed as:

[0082] wherein sig(.) is a Sigmoid function, Re(.) is a ReLU activation function, W1 and W2 refer to the first and second fully connected layer weight matrices, Z =∑(U c ×W c ) refers to a channel descriptor obtained through global convolution, W c is a weight of the cth channel of the global convolution kernel, U c refers to a tensor of the cth channel of the input feature U, Z c refers to a tensor of the cth channel of the feature channel descriptor, is an element-wise multiplication;

[0083] The spatial feature fusion module C is used to fuse short-time spatial dynamic features and static features, obtain dynamic information of the fire pixel image, and optimize the dynamic and static feature fusion process according to the static feature change, specifically:

[0084] For each sample, T c static features E = {E j} are extracted from the video frame through the residual module A and the channel feature control module B, j = 1, 2, 3... T c ;

[0085] The spatial feature fusion module calculates the feature difference between the before and after time according to the static features E j and E j+1 of adjacent frames, and obtains dynamic features D; and further feature extraction is performed on the static features E and the dynamic features D, specifically:

[0086] For the static features E, first, the spatial sequence P is obtained by encoding the static features E according to the spatial position through a single-layer convolutional network, which can be expressed as:

[0087] wherein F Mf.Nf (.) represents a single-layer convolution operation, M f is a convolution kernel of the single-layer convolution, and N f is a bias quantity of the single-layer convolution;

[0088] The position information sequence Q of the static features E is defined, and the joint input sequence B = P + Q is obtained by combining the spatial sequence P and the position encoding sequence Q; the input sequence B is used as input to learn by using a Transformer network, wherein the lth layer of the Transformer network can be expressed as:

[0089] B′ l = MultiHeadSelfAttention(LayerNormalization(B l-1 ))+ B l-1

[0090] B l = FeedForwardNetwork(LayerNormalization(B′ l ))+ B l

[0091] wherein LayerNormalization(.) represents a normalization layer, MultiHeadSelfAttention(.) represents a multi-head self-attention layer, and FeedForwardNetwork(.) represents a feedforward network layer; B l is the output of the lth layer and also the input of the (l+1)th layer;

[0092] For the dynamic feature D, further feature extraction is performed on the feature thereof by a convolutional neural network, which can be represented as:

[0093] wherein represents a convolution operation, M l is a convolution kernel of the lth convolution operation, N l is a bias quantity of the lth convolution operation, and l refers to the serial number of the convolution operation;

[0094] As a preferred mode, the convolution kernel size of the single-layer convolution is 2*2; and the step length of the convolution operation when k=1 is 2.

[0095] The static feature B" obtained after learning by the Transformer network is acquired, and the dynamic and static fusion feature is obtained by fusing the static feature B" and the dynamic features D' before and after the frame, which can be represented as: TZ i = B" i + 0.5(D' i-1 + D' i );

[0096] wherein B" i is the static feature of the i-th frame of the sample, and D' i are the dynamic features of the i-th frame of the sample, respectively.

[0097] wherein TZ1 = B"1 + D'1,

[0098] The full-connection classification module E performs training and classification on the fire image sample by considering the dynamic and static fusion feature of the sample, and obtains the final fire image classification result.

[0099] It should be pointed out finally that the above embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit the protection scope of the present application. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or equivalently replaced without departing from the essence and scope of the technical solutions of the present application.

Claims

1. A multi-dimensional recognition classification method for ship fire, characterized in that, Comprising the following steps: Step one, sampling processing of fire image For the first original image sample sequence based on the timing table expression , wherein represents the number of frames of the image sample sequence, respectively refer to the pixel height and the pixel width of the image. Based on the continuous sampling method, randomly select frame time From Start continuous acquisition Frame image as a sampling sequence, the sampling sequence can be expressed as: ; wherein is the frame number of the original video, is the sampling frame number; Step two, pre-processing of image data in the sampling sequence pixel coordinates of any reference feature in the fire image wherein is the pixel coordinates of the reference feature centroid in the fire image, and its coordinates after spatial scaling is ; For the whole sampling sequence, the determination of the best spatial scale uniform criterion is an optimal solution problem, which can be expressed as: ; and , , , , ; wherein refers to norm; is the pixel coordinate of the centroid of the is the pixel coordinate of the centroid of the is the total number of reference features; Solving the above optimal solution to determine the unified standard of spatial transformation; Step three, sampling sequence enhancement processing Through data enhancement processing on the sampling sequence, the generalization index of the recognition analysis network is improved; Step four, fire image recognition based on element feature expression ability A fire image recognition network based on element feature expression ability is established, and the network sequentially comprises a residual module A, a channel feature control module B, a spatial feature fusion module C, and a full connection classification module E; The residual module A is composed of two ResNet residual blocks, and the residual module A obtains the sampling sequence and extracts the spatial features of the fire image through the residual blocks; The channel feature control module B is composed of a transformation convolution module and a global convolution attention module, wherein the transformation convolution module is used for transforming the input spatial feature map, specifically, a 2D spatial kernel convolution is used to convolve each layer of the input feature map, and a new feature map is output, which can be expressed as: ; wherein is the input feature map, is the output feature map, denotes a learned filter kernel, denotes a learned parameter of the c-th filter, denotes a 2D spatial convolution kernel, * denotes a convolution operation; A global convolution attention module is used to extract multi-channel attention weights of spatial features, specifically, video frame weight information considering global channel attention weights is generated to enhance feature channel capability, which can be expressed as: ; wherein is a function, is an activation function, refers to the first and second fully connected layer weight matrices, refers to the channel descriptors obtained by global convolution, is the weight of the global convolution kernel for the channel, refers to the input feature tensor for the channel, is the tensor of the feature channel descriptor for the channel, is an element-wise multiplication; The spatial feature fusion module C is used for fusing short-time spatial dynamic features and static features, obtaining dynamic information of the fire pixel image, and optimizing the dynamic and static feature fusion process according to the static feature change, specifically: For each sample, static features in the video frame are extracted after passing through the residual module A and the channel feature control module B ;​​ The spatial feature fusion module C fuses the static features of the adjacent frames according to the spatial features of the adjacent frames The feature difference between the previous and subsequent moments is calculated to obtain the dynamic features ; and further feature extraction is performed on the static features and the dynamic features , specifically: For static features , the static features are first encoded by a single-layer convolutional network to obtain spatial sequences according to spatial positions, which can be represented as: ; wherein denotes a single-layer convolution operation, is a convolution kernel of the single-layer convolution, is a bias quantity of the single-layer convolution; Define static features Location information sequence , spatial sequence and positional encoding sequence Combined to obtain the joint input sequence ; Input sequence As input utilization Learning online; For dynamic features , further feature extraction is performed on the features thereof by a convolutional neural network, which can be expressed as: ; wherein denotes a convolution operation, is a convolution kernel of the th convolution operation, is a bias of the th convolution operation, is a serial number of the convolution operation; Acquisition process Static features obtained after network learning , and the dynamic features of the previous and subsequent frames Fusion of dynamic and static features, which can be represented as: ; wherein refers to the static feature of the sample at the frame, respectively, the dynamic feature of the sample at the frame; wherein , ; The full connection classification module E trains and classifies the fire image sample by considering the sample dynamic and static fusion features, and obtains the final fire image classification result.

2. The method of claim 1, wherein, The spatial transformation process can be represented as: ， wherein is the horizontal axis scaling ratio when the spatial transformation is applied, is the vertical axis scaling ratio when the spatial transformation is applied, is the rotation angle when the spatial transformation is applied, is the horizontal axis translation distance when the spatial transformation is applied, is the vertical axis translation distance when the spatial transformation is applied.

3. The method of claim 1, wherein, The following methods are used for data enhancement of the image: A, randomly synchronously rotating all images in the sample; B, randomly adjusting the brightness of all images in the sample; C, randomly cutting a random area of a given area from all images in the sample, and scaling it to the size of the original image.

4. The method of claim 1, wherein, The The first The layer can be represented as: ， wherein denotes a normalization layer, denotes a multi-head self-attention layer, denotes a feed-forward network layer; is the output of the layer, and is also the input of the layer.

5. The method of claim 1, wherein, The convolution kernel size of the single-layer convolution is 2*2; The step size of the convolution operation at the time is 2.

Images