Smoke semantic segmentation method and system based on reliable query and prototype driving

Abstract

The invention relates to the field of image segmentation, and provides a smoke semantic segmentation method and system based on reliable query and prototype driving, a smoke semantic segmentation network is designed, and a backbone sub-network obtains basic features of multiple different scales so as to obtain semantic information of multiple scales; coordinates, enabling components of the same category on a potential manifold, in features of the classification branch network are gathered as much as possible so as to obtain a feature prototype more beneficial to smoke component classification, attention in a prediction foreground area is highlighted by the segmentation branch network according to a reliable target query extraction module, more accurate target position priori is provided for each query feature, and the accuracy of smoke component classification is improved. According to the smoke image segmentation method based on the local-global interaction module, fine-grained target positioning is promoted, the features of multiple scales obtained by the backbone sub-network are processed in a progressive mode according to the local-global interaction module so as to mine richer details, and the accuracy of smoke image segmentation is greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of image segmentation, and in particular to a smoke semantic segmentation method and system based on reliable query and prototype drive. Background Art

[0002] With the rapid development of computer vision technology, image segmentation has gradually been applied to various security monitoring scenarios. Among them, fire, as a disaster with extremely high occurrence frequency and serious threat to social security, has naturally become a key security monitoring scenario for the application of image segmentation technology.

[0003] In the existing technology, fire monitoring is still mostly achieved by means of manual image judgment and smoke detection. The workload of manual investigation and monitoring is extremely large and the cost is extremely high, so image segmentation technology has gradually been paid attention to. However, the application scenarios of fire monitoring are relatively complex and there are many interference conditions. In addition to the common changes in light and dark conditions and the influence of weather factors, there are also many misjudged targets with similar shapes in smoke, such as clouds and haze. The color, texture, shape and transparency of the smoke itself vary greatly, especially the boundaries of the smoke are difficult to accurately define, resulting in the accuracy and versatility of smoke image segmentation being difficult to meet the current requirements of fire safety monitoring.

[0004] Therefore, how to design an image segmentation method to improve the accuracy of smoke image segmentation is an urgent problem to be solved. Summary of the invention

[0005] Based on this, the purpose of the present invention is to provide a smoke semantic segmentation method and system based on reliable query and prototype driving. By designing a smoke semantic segmentation network, including a backbone subnetwork, a classification branch network, a segmentation branch network and a prediction module, the backbone subnetwork obtains basic features of multiple scales to obtain semantic information of multiple scales. The classification branch network features make the coordinates of components of the same category on the potential manifold as aggregated as possible, while making components of different categories as dispersed as possible to obtain feature prototypes that are more conducive to the classification of smoke components. The segmentation branch network highlights the attention in the predicted foreground area according to the reliable target query extraction module, rather than focusing on the entire feature map, providing a richer and more accurate target position prior for each query feature, greatly promoting fine-grained target positioning, and processing the features of multiple scales obtained by the backbone subnetwork in a progressive manner according to the local-global interaction module to mine richer details, thereby effectively performing local and global relationship aggregation. The present invention greatly improves the accuracy of smoke image segmentation.

[0006] The present invention proposes a smoke semantic segmentation method based on reliable query and prototype drive, comprising: Acquire a single-frame smoke image in real time and perform preprocessing to input the preprocessed smoke image into a smoke semantic segmentation network, wherein the smoke semantic segmentation network includes a backbone sub-network, a classification branch network, a segmentation branch network, and a prediction module; The backbone sub-network obtains a plurality of basic features of different scales, wherein the basic features include low-order basic features, medium-order basic features and high-order basic features, inputs all the basic features into the segmentation branch network, and inputs the high-order basic features into the classification branch network; The classification branch network deeply classifies the high-order basic features based on the popular space to obtain a plurality of feature prototypes, the feature prototypes are in single correspondence with the high-order basic features, and the feature prototypes are input into the segmentation branch network; The segmentation branch network performs feature enhancement on the high-order basic features according to the reliable target query extraction module to obtain reliable target query features, performs feature enhancement on the low-order basic features, the mid-order basic features and the feature prototype according to the local-global interaction module to obtain detail enhancement features, and inputs the reliable target query features and the detail enhancement features into the prediction module; The prediction module obtains a final prediction result according to the reliable target query feature and the detail enhancement feature, and performs correction and optimization according to the smoke focus loss function.

[0007] In summary, according to the above-mentioned smoke semantic segmentation method based on reliable query and prototype drive, a smoke semantic segmentation network is designed, including a backbone subnetwork, a classification branch network, a segmentation branch network and a prediction module. The backbone subnetwork obtains basic features of multiple scales to obtain semantic information of multiple scales. The classification branch network features make the coordinates of components of the same category on the potential manifold as aggregated as possible, and at the same time make components of different categories as dispersed as possible to obtain feature prototypes that are more conducive to the classification of smoke components. The segmentation branch network highlights the attention in the predicted foreground area according to the reliable target query extraction module, rather than focusing on the entire feature map, and provides a richer and more accurate target position prior for each query feature, which greatly promotes fine-grained target positioning. According to the local-global interaction module, the features of multiple scales obtained by the backbone subnetwork are processed in a progressive manner to mine richer details, thereby effectively performing local and global relationship aggregation. The present invention greatly improves the accuracy of smoke image segmentation. Specifically, a single-frame smoke image is acquired in real time and preprocessed, so that the preprocessed smoke image is input into a smoke semantic segmentation network, the smoke semantic segmentation network includes a backbone sub-network, a classification branch network, a segmentation branch network and a prediction module, the backbone sub-network acquires basic features of multiple scales, the basic features include low-order basic features, mid-order basic features and high-order basic features, all the basic features are input into the segmentation branch network, the high-order basic features are input into the classification branch network to acquire semantic information of multiple scales, the classification branch network performs deep classification on the high-order basic features based on the popular space to acquire multiple feature prototypes, the feature prototypes are in single correspondence with the high-order basic features, the feature prototypes are input into the segmentation branch network The invention discloses a method for segmenting smoke images by using a segmentation branch network, and obtains a feature prototype that is more conducive to the classification of smoke components. The segmentation branch network performs feature enhancement on the high-order basic features according to the reliable target query extraction module to obtain reliable target query features. The low-order basic features, the mid-order basic features and the feature prototype are feature enhanced according to the local-global interaction module to obtain detail enhancement features. The reliable target query features and the detail enhancement features are input into the prediction module, and richer and more accurate target position priors are obtained while also mining richer details. The prediction module obtains the final prediction result according to the reliable target query features and the detail enhancement features, and performs correction and optimization according to the smoke focusing loss function. The present invention greatly improves the accuracy of smoke image segmentation.

[0008] Furthermore, the classification branch network performs deep classification on the high-order basic features based on the popular space to obtain a plurality of feature prototypes, which specifically includes: The classification branch network is based on a symmetric positive definite matrix of a Riemann manifold, and the classification branch network includes a Gaussian aggregation layer, a nonlinear transformation layer, a BN block, a pooling block, a global pooling block, a fully connected layer, and a prediction layer; The Gaussian aggregation layer obtains Gaussian effective information of high-order basic features to output a symmetric positive definite matrix corresponding to the Gaussian effective information to the nonlinear transformation layer, and the nonlinear transformation layer then performs eigenvalue decomposition and nonlinear transformation. The specific algorithm of the nonlinear transformation layer is as follows: , , in, represents the output of the nonlinear transformation layer, represents the input of the nonlinear transformation layer, represents the eigenvector obtained by eigenvalue decomposition, represents the eigenvalue matrix obtained by eigenvalue decomposition, represents the nonlinear transformation function, , T represents transpose, Represents the horizontal coordinate of the pixel point, Indicates the vertical coordinate of the pixel; The BN block and the pooling block map the symmetric positive definite matrix from the Riemann manifold to the Euclidean space according to the logarithmic function, and then remap the symmetric positive definite matrix back to the manifold space through the exponential function. The global pooling block performs global average pooling to obtain global context prior information. The specific algorithms of the BN block, the pooling block and the global pooling block are as follows: , , , in, Indicates the output, Indicates input, represents a BN block, represents the pooling block, represents the global pooling block, , and They represent batch normalization, average pooling, and global average pooling operations in Euclidean space, respectively. represents the exponential function, represents the logarithmic function; The specific algorithms of the fully connected layer and the prediction layer are as follows: , in, Represents the prediction result output by the classification network, represents the input of the fully connected layer, represents the fully connected layer, represents a nonlinear activation function; The feature prototype is obtained according to the outputs of the pooling block and the global pooling block, and the probability of containing a smoke target in the image is obtained according to the prediction result output by the classification network.

[0009] Furthermore, the segmentation branch network performs feature enhancement on the high-order basic features according to the reliable target query extraction module to obtain the reliable target query features, which specifically includes: The reliable target query extraction module includes an attention enhancement block and a region enhancement mapping block; The attention enhancement module performs feature weighting processing on the high-order basic features, wherein the feature weighting processing includes channel relationship weighting processing and spatial position weighting processing, and the spatial position weighting processing includes horizontal position weighting processing and vertical position weighting processing; The specific algorithm of the attention enhancement block is as follows: , , , , , in, represents high-order basic features, represents the inverse bottleneck layer, which includes three nonlinear activation fully connected layers, represents the learnable weight tensor at different angles, Indicates a flip operation. represents a channel-by-channel addition operation, represents the feature dimension, Indicates the height, width, and number of channels of the feature, represents the activation function, and represents the horizontal position weighted vector and the vertical position weighted vector, represents the channel relationship weighted vector, Represents the output features of the attention enhancement module; The output features of the attention enhancement module are input into the region enhancement mapping block to obtain reliable target query features.

[0010] Furthermore, the region enhancement mapping block specifically includes: The region enhancement mapping block obtains reliable target query features based on the output features of the attention enhancement module. The specific algorithm for obtaining reliable target query features is as follows: , , , in, represents reliable target query features, represents the activation function, is a dropout operation with a probability of 0.3, represents the fully connected layer, represents the output of the intermediate layer after dimension transformation, Indicates that b × c × h × w The size of the tensor is converted to b ×1×( c × h × w ), represents average pooling with a sampling rate of 2, represents batch normalization, represents the output after 1×1 convolution, represents 1×1 convolution, Represents the output features of the attention enhancement module unfolded into a two-dimensional tensor; The obtained reliable target query features are then post-processed, and the post-processing algorithm is as follows: , , , , in, represents the key matrix involved in the attention calculation, Represents the value matrix involved in the attention calculation, represents the attention calculation, represents the reliable target query features of the final output, T represents transpose, Represents matrix multiplication.

[0011] Further, the step of performing feature enhancement on the low-order basic features, the mid-order basic features and the feature prototype according to the local-global interaction module to obtain detail enhancement features specifically includes: The local-global interaction module includes the inter-domain attention learning block and the NL block; The specific algorithm of the inter-domain attention learning block is as follows: , , , in, Indicates that Copy 64 times, Represents a class prototype tensor, the size of which is b × c × h / 8× w / 8, represents the key matrix and the value matrix, express Divided into 64 sub-regions of equal size, represents the input feature of the inter-domain attention learning block, and the size of the input feature is b × c × h × w , represents the attention calculation, represents the activation function, represents the output of the inter-domain attention learning block, Indicates that 64× b × c × h / 8× w / 8The size of the tensor is converted to b × c × h × w , represents the Hadamard product; The specific algorithm of the NL block is as follows: , , in, represents the query matrix, represents the input of the NL block, Will Mapped to the corresponding query matrix Q , the bond matrix K Sum Matrix V , represents 1×1 convolution, represents the output of the NL block, Represents the key matrix K The length of the vector, T represents transpose, The object number representing the convolution function service; Detail enhancement features are obtained based on the outputs of the inter-domain attention learning block and the NL block.

[0012] Furthermore, the step of obtaining the final prediction result according to the reliable target query feature and the detail enhancement feature by the prediction module specifically includes: The prediction module includes an upsampling convolution module, a convolution module, and a sigmoid nonlinear activation layer; The upsampling convolution module includes a 2x bilinear interpolation upsampling layer, a 3×3 convolution layer, a BN layer, and a ReLu layer; The convolution module includes a 3×3 convolution layer, a BN layer and a ReLu layer; The prediction module maps reliable target query features and detail enhancement features from feature space to semantic space to obtain final segmentation results.

[0013] Furthermore, the step of correcting and optimizing according to the smoke focusing loss function specifically includes: Correction and optimization are performed according to the smoke focusing loss function, and the specific algorithm of the smoke focusing loss function is as follows: , , , , , , , , in, represents the smoke focusing loss function, and represents the weight of foreground and background loss distribution, represents prospect loss, represents the background loss, express The number of foreground pixels, express The number of foreground pixels, express The number of background pixels, express The number of background pixels, The segmentation label representing the final segmentation result after flattening, represents the final segmentation result after flattening and binarization, represents a vector of all 1s, represents transpose, represents the flattening operation, The segmentation label representing the final segmentation result, represents the final segmentation result of binarization, represents the final segmentation result, represents the feature position coordinates, Indicates the feature height of the final segmentation result, Represents the feature width of the final segmentation result, Represents a positive integer.

[0014] The present invention proposes a smoke semantic segmentation system based on reliable query and prototype drive, comprising: A preprocessing module, used for acquiring a single-frame smoke image in real time and performing preprocessing, so as to input the preprocessed smoke image into a smoke semantic segmentation network, wherein the smoke semantic segmentation network includes a backbone sub-network, a classification branch network, a segmentation branch network and a prediction module; A basic feature extraction module is used for the backbone sub-network to obtain a plurality of basic features of different scales, wherein the basic features include low-order basic features, medium-order basic features and high-order basic features, and all the basic features are input into the segmentation branch network, and the high-order basic features are input into the classification branch network; A classification module, used for the classification branch network to perform deep classification on the high-order basic features based on the popular space to obtain a plurality of feature prototypes, the feature prototypes having a single correspondence with the high-order basic features, and inputting the feature prototypes into the segmentation branch network; A segmentation module, used for the segmentation branch network to perform feature enhancement on the high-order basic features according to the reliable target query extraction module to obtain reliable target query features, perform feature enhancement on the low-order basic features, the mid-order basic features and the feature prototype according to the local-global interaction module to obtain detail enhancement features, and input the reliable target query features and the detail enhancement features into a prediction module; A prediction optimization module is used for the prediction module to obtain a final prediction result according to the reliable target query feature and the detail enhancement feature, and to perform correction and optimization according to the smoke focusing loss function.

[0015] The present invention also provides a storage medium, wherein the storage medium stores one or more programs, and when the programs are executed by a processor, the smoke semantic segmentation method based on reliable query and prototype drive as described above is implemented.

[0016] The present invention also provides a computer device, the computer device comprising a memory and a processor, wherein: The memory is used to store computer programs; When the processor is used to execute the computer program stored in the memory, the smoke semantic segmentation method based on reliable query and prototype drive as described above is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A flowchart of a smoke semantic segmentation method based on reliable query and prototype drive proposed in the first embodiment of the present invention; Figure 2 A flowchart of a smoke semantic segmentation method based on reliable query and prototype drive proposed in the second embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of a smoke semantic segmentation system based on reliable query and prototype drive proposed in the third embodiment of the present invention; Figure 4 This is a structural flow chart of the smoke semantic segmentation network proposed in the first embodiment of the present invention.

[0018] The following specific implementation manner will further illustrate the present invention in conjunction with the above-mentioned drawings. DETAILED DESCRIPTION

[0019] In order to facilitate understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. Several embodiments of the present invention are provided in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive.

[0020] It should be noted that when an element is referred to as being "fixed to" another element, it may be directly on the other element or there may be a central element. When an element is considered to be "connected to" another element, it may be directly connected to the other element or there may be a central element at the same time. The terms "vertical", "horizontal", "left", "right" and similar expressions used herein are for illustrative purposes only.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.

[0022] See also Figure 1 , which is a flow chart of a smoke semantic segmentation method based on reliable query and prototype drive proposed in the first embodiment of the present invention, the smoke semantic segmentation method based on reliable query and prototype drive comprises steps S01 to S05, wherein: Step S01: acquiring a single-frame smoke image in real time and preprocessing it, so as to input the preprocessed smoke image into a smoke semantic segmentation network; It should be noted that the smoke image input smoke semantic segmentation network in this embodiment includes a backbone sub-network, a classification branch network, a segmentation branch network and a prediction module. For the specific structure, please refer to Figure 4 .

[0023] Step S02: The backbone sub-network obtains multiple basic features of different scales, including low-order basic features, medium-order basic features and high-order basic features, inputs all basic features into the segmentation branch network, and inputs the high-order basic features into the classification branch network; Step S03: the classification branch network deeply classifies the high-order basic features based on the popular space to obtain multiple feature prototypes, and inputs the feature prototypes into the segmentation branch network; It should be noted that in this embodiment, the classification branch network is based on a symmetric positive definite matrix of a Riemann manifold, and the classification branch network includes a Gaussian aggregation layer, a nonlinear transformation layer, a BN block, a pooling block, a global pooling block, a fully connected layer and a prediction layer; The Gaussian aggregation layer obtains Gaussian effective information of high-order basic features to output a symmetric positive definite matrix corresponding to the Gaussian effective information to the nonlinear transformation layer, and the nonlinear transformation layer then performs eigenvalue decomposition and nonlinear transformation. The specific algorithm of the nonlinear transformation layer is as follows: , , in, represents the output of the nonlinear transformation layer, represents the input of the nonlinear transformation layer, represents the eigenvector obtained by eigenvalue decomposition, represents the eigenvalue matrix obtained by eigenvalue decomposition, represents the nonlinear transformation function, , T represents transpose, Represents the horizontal coordinate of the pixel point, Indicates the vertical coordinate of the pixel; The BN block and the pooling block map the symmetric positive definite matrix from the Riemann manifold to the Euclidean space according to the logarithmic function, and then remap the symmetric positive definite matrix back to the manifold space through the exponential function. The global pooling block performs global average pooling to obtain global context prior information. The specific algorithms of the BN block, the pooling block and the global pooling block are as follows: , , , in, Indicates the output, Indicates input, represents a BN block, represents the pooling block, represents the global pooling block, , and They represent batch normalization, average pooling, and global average pooling operations in Euclidean space, respectively. represents the exponential function, represents the logarithmic function; The specific algorithms of the fully connected layer and the prediction layer are as follows: , in, Represents the prediction result output by the classification network, represents the input of the fully connected layer, represents the fully connected layer, represents a nonlinear activation function; The feature prototype is obtained according to the outputs of the pooling block and the global pooling block, and the probability of containing a smoke target in the image is obtained according to the prediction result output by the classification network.

[0024] Step S04: the segmentation branch network performs feature enhancement on high-order basic features according to the reliable target query extraction module to obtain reliable target query features, and performs feature enhancement on low-order basic features, mid-order basic features and feature prototypes according to the local-global interaction module to obtain detail enhancement features, and inputs the reliable target query features and detail enhancement features into the prediction module; It should be noted that the reliable target query extraction module in this embodiment includes an attention enhancement block and a region enhancement mapping block; The attention enhancement module performs feature weighting processing on the high-order basic features, wherein the feature weighting processing includes channel relationship weighting processing and spatial position weighting processing, and the spatial position weighting processing includes horizontal position weighting processing and vertical position weighting processing; The specific algorithm of the attention enhancement block is as follows: , , , , , in, represents high-order basic features, represents the inverse bottleneck layer, which includes three nonlinear activation fully connected layers, represents the learnable weight tensor at different angles, Indicates a flip operation. represents a channel-by-channel addition operation, represents the feature dimension, Indicates the height, width, and number of channels of the feature, represents the activation function, and represents the horizontal position weighted vector and the vertical position weighted vector, represents the channel relationship weighted vector, Represents the output features of the attention enhancement module; Inputting the output features of the attention enhancement module into the region enhancement mapping block to obtain reliable target query features; In this embodiment, the region enhancement mapping block obtains reliable target query features based on the output features of the attention enhancement module. The specific algorithm for obtaining the reliable target query features is as follows: , , , in, represents reliable target query features, represents the activation function, is a dropout operation with a probability of 0.3, represents the fully connected layer, represents the output of the intermediate layer after dimension transformation, Indicates that b × c × h × w The size of the tensor is converted to b ×1×( c × h × w ), represents average pooling with a sampling rate of 2, represents batch normalization, represents the output after 1×1 convolution, represents 1×1 convolution, Represents the output features of the attention enhancement module unfolded into a two-dimensional tensor; The obtained reliable target query features are then post-processed, and the post-processing algorithm is as follows: , , , , in, represents the key matrix involved in the attention calculation, Represents the value matrix involved in the attention calculation, represents the attention calculation, represents the reliable target query features of the final output, T represents transpose, Represents matrix multiplication; In this embodiment, the local-global interaction module includes an inter-domain attention learning block and an NL block; The specific algorithm of the inter-domain attention learning block is as follows: , , , in, Indicates that Copy 64 times, Represents a class prototype tensor, the size of which is b × c × h / 8× w / 8, represents the key matrix and the value matrix, express Divided into 64 sub-regions of equal size, represents the input feature of the inter-domain attention learning block, and the size of the input feature is b × c × h × w , represents the attention calculation, represents the activation function, represents the output of the inter-domain attention learning block, Indicates that 64× b × c × h / 8× w / 8The size of the tensor is converted to b × c × h × w , represents the Hadamard product; The specific algorithm of the NL block is as follows: , , in, represents the query matrix, represents the input of the NL block, Will Mapped to the corresponding query matrix Q , the bond matrix K Sum Matrix V , represents 1×1 convolution, represents the output of the NL block, Represents the key matrix K The length of the vector, T represents transpose, The object number representing the convolution function service; Detail enhancement features are obtained based on the outputs of the inter-domain attention learning block and the NL block.

[0025] Step S05: the prediction module obtains the final prediction result according to the reliable target query feature and the detail enhancement feature, and performs correction and optimization according to the smoke focus loss function; It should be noted that the prediction module in this embodiment includes an upsampling convolution module, a convolution module and a sigmoid nonlinear activation layer; The upsampling convolution module includes a 2x bilinear interpolation upsampling layer, a 3×3 convolution layer, a BN layer, and a ReLu layer; The convolution module includes a 3×3 convolution layer, a BN layer and a ReLu layer; The prediction module maps reliable target query features and detail enhancement features from feature space to semantic space to obtain a final segmentation result; In this embodiment, the correction and optimization are performed according to the smoke focusing loss function, and the specific algorithm of the smoke focusing loss function is as follows: , , , , , , , , in, represents the smoke focusing loss function, and represents the weight of foreground and background loss distribution, represents prospect loss, represents the background loss, express The number of foreground pixels, express The number of foreground pixels, express The number of background pixels, express The number of background pixels, The segmentation label representing the final segmentation result after flattening, represents the final segmentation result after flattening and binarization, represents a vector of all 1s, represents transpose, represents the flattening operation, The segmentation label representing the final segmentation result, represents the final segmentation result of binarization, represents the final segmentation result, represents the feature position coordinates, Indicates the feature height of the final segmentation result, Represents the feature width of the final segmentation result, Represents a positive integer.

[0026] In summary, according to the above-mentioned smoke semantic segmentation method based on reliable query and prototype drive, a smoke semantic segmentation network is designed, including a backbone subnetwork, a classification branch network, a segmentation branch network and a prediction module. The backbone subnetwork obtains basic features of multiple scales to obtain semantic information of multiple scales. The classification branch network features make the coordinates of components of the same category on the potential manifold as aggregated as possible, and at the same time make components of different categories as dispersed as possible to obtain feature prototypes that are more conducive to the classification of smoke components. The segmentation branch network highlights the attention in the predicted foreground area according to the reliable target query extraction module, rather than focusing on the entire feature map, and provides a richer and more accurate target position prior for each query feature, which greatly promotes fine-grained target positioning. According to the local-global interaction module, the features of multiple scales obtained by the backbone subnetwork are processed in a progressive manner to mine richer details, thereby effectively performing local and global relationship aggregation. The present invention greatly improves the accuracy of smoke image segmentation. Specifically, a single-frame smoke image is acquired in real time and preprocessed, so that the preprocessed smoke image is input into a smoke semantic segmentation network, the smoke semantic segmentation network includes a backbone sub-network, a classification branch network, a segmentation branch network and a prediction module, the backbone sub-network acquires basic features of multiple scales, the basic features include low-order basic features, mid-order basic features and high-order basic features, all the basic features are input into the segmentation branch network, the high-order basic features are input into the classification branch network to acquire semantic information of multiple scales, the classification branch network performs deep classification on the high-order basic features based on the popular space to acquire multiple feature prototypes, the feature prototypes are in single correspondence with the high-order basic features, the feature prototypes are input into the segmentation branch network The invention discloses a method for segmenting smoke images by using a segmentation branch network, and obtains a feature prototype that is more conducive to the classification of smoke components. The segmentation branch network performs feature enhancement on the high-order basic features according to the reliable target query extraction module to obtain reliable target query features. The low-order basic features, the mid-order basic features and the feature prototype are feature enhanced according to the local-global interaction module to obtain detail enhancement features. The reliable target query features and the detail enhancement features are input into the prediction module, and richer and more accurate target position priors are obtained while also mining richer details. The prediction module obtains the final prediction result according to the reliable target query features and the detail enhancement features, and performs correction and optimization according to the smoke focusing loss function. The present invention greatly improves the accuracy of smoke image segmentation.

[0027] See also Figure 2 , which is a flow chart of a smoke semantic segmentation method based on reliable query and prototype drive proposed in the second embodiment of the present invention, the smoke semantic segmentation method based on reliable query and prototype drive comprises steps S11 to S16, wherein: Step S11: acquiring a single-frame smoke image in real time and performing preprocessing to input the preprocessed smoke image into a smoke semantic segmentation network; Step S12: The backbone sub-network obtains multiple basic features of different scales, including low-order basic features, medium-order basic features and high-order basic features, inputs all basic features into the segmentation branch network, and inputs the high-order basic features into the classification branch network; Step S13: The Gaussian aggregation layer of the classification branch network obtains Gaussian effective information of high-order basic features to output a symmetric positive definite matrix corresponding to the Gaussian effective information to the nonlinear transformation layer, and the nonlinear transformation layer then performs eigenvalue decomposition and nonlinear transformation. The BN block and the pooling block map the symmetric positive definite matrix from the Riemann manifold to the Euclidean space according to the logarithmic function, and then remap the symmetric positive definite matrix back to the manifold space through the exponential function. The global pooling block performs global average pooling to obtain global context prior information, and obtains the feature prototype according to the output of the pooling block and the global pooling block. The probability of containing a smoke target in the image is obtained according to the prediction result output by the classification network; Step S14: the reliable target query extraction module includes an attention enhancement block and a region enhancement mapping block. The attention enhancement module performs feature weighting processing on high-order basic features, and inputs the output features of the attention enhancement module into the region enhancement mapping block to obtain reliable target query features. Step S15: the local-global interaction module includes an inter-domain attention learning block and an NL block, and obtains detail enhancement features according to the outputs of the inter-domain attention learning block and the NL block; Step S16: The prediction module maps the reliable target query features and detail enhancement features from the feature space to the semantic space to obtain the final segmentation result, and performs correction and optimization according to the smoke focusing loss function.

[0028] In summary, according to the above-mentioned smoke semantic segmentation method based on reliable query and prototype drive, a smoke semantic segmentation network is designed, including a backbone subnetwork, a classification branch network, a segmentation branch network and a prediction module. The backbone subnetwork obtains basic features of multiple scales to obtain semantic information of multiple scales. The classification branch network features make the coordinates of components of the same category on the potential manifold as aggregated as possible, and at the same time make components of different categories as dispersed as possible to obtain feature prototypes that are more conducive to the classification of smoke components. The segmentation branch network highlights the attention in the predicted foreground area according to the reliable target query extraction module, rather than focusing on the entire feature map, and provides a richer and more accurate target position prior for each query feature, which greatly promotes fine-grained target positioning. According to the local-global interaction module, the features of multiple scales obtained by the backbone subnetwork are processed in a progressive manner to mine richer details, thereby effectively performing local and global relationship aggregation. The present invention greatly improves the accuracy of smoke image segmentation. Specifically, a single-frame smoke image is acquired in real time and preprocessed, so that the preprocessed smoke image is input into a smoke semantic segmentation network, the smoke semantic segmentation network includes a backbone sub-network, a classification branch network, a segmentation branch network and a prediction module, the backbone sub-network acquires basic features of multiple scales, the basic features include low-order basic features, mid-order basic features and high-order basic features, all the basic features are input into the segmentation branch network, the high-order basic features are input into the classification branch network to acquire semantic information of multiple scales, the classification branch network performs deep classification on the high-order basic features based on the popular space to acquire multiple feature prototypes, the feature prototypes are in single correspondence with the high-order basic features, the feature prototypes are input into the segmentation branch network The invention discloses a method for segmenting smoke images by using a segmentation branch network, and obtains a feature prototype that is more conducive to the classification of smoke components. The segmentation branch network performs feature enhancement on the high-order basic features according to the reliable target query extraction module to obtain reliable target query features. The low-order basic features, the mid-order basic features and the feature prototype are feature enhanced according to the local-global interaction module to obtain detail enhancement features. The reliable target query features and the detail enhancement features are input into the prediction module, and richer and more accurate target position priors are obtained while also mining richer details. The prediction module obtains the final prediction result according to the reliable target query features and the detail enhancement features, and performs correction and optimization according to the smoke focusing loss function. The present invention greatly improves the accuracy of smoke image segmentation.

[0029] See also Figure 3 , which is a schematic diagram of the structure of a smoke semantic segmentation system based on reliable query and prototype drive proposed in the third embodiment of the present invention, the system includes: A preprocessing module 10 is used to acquire a single-frame smoke image in real time and perform preprocessing to input the preprocessed smoke image into a smoke semantic segmentation network, wherein the smoke semantic segmentation network includes a backbone subnetwork, a classification branch network, a segmentation branch network and a prediction module; A basic feature extraction module 20 is used for the backbone sub-network to obtain a plurality of basic features of different scales, wherein the basic features include low-order basic features, medium-order basic features and high-order basic features, and input all the basic features into the segmentation branch network, and input the high-order basic features into the classification branch network; A classification module 30, configured for the classification branch network to perform deep classification on the high-order basic features based on the popular space to obtain a plurality of feature prototypes, wherein the feature prototypes correspond to the high-order basic features in a single manner, and input the feature prototypes into the segmentation branch network; A segmentation module 40, configured for the segmentation branch network to perform feature enhancement on the high-order basic features according to the reliable target query extraction module to obtain reliable target query features, perform feature enhancement on the low-order basic features, the mid-order basic features and the feature prototype according to the local-global interaction module to obtain detail enhancement features, and input the reliable target query features and the detail enhancement features into a prediction module; The prediction optimization module 50 is used for the prediction module to obtain the final prediction result according to the reliable target query feature and the detail enhancement feature, and to perform correction and optimization according to the smoke focus loss function.

[0030] The present invention also proposes a computer storage medium on which one or more programs are stored. When the programs are executed by a processor, the smoke semantic segmentation method based on reliable query and prototype drive is implemented.

[0031] The present invention also proposes a computer device, comprising a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to implement the above-mentioned smoke semantic segmentation method based on reliable query and prototype drive.

[0032] Those skilled in the art will appreciate that the logic and / or steps represented in the flowchart or otherwise described herein, for example, may be considered as an ordered list of executable instructions for implementing logical functions, and may be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For purposes of this specification, "computer-readable medium" may be any device that can contain storage, communication, propagation or transmission of a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0033] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0034] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or a combination thereof: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0035] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0036] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A smoke semantic segmentation method based on reliable query and prototype drive, characterized in that: include: Acquire a single-frame smoke image in real time and perform preprocessing to input the preprocessed smoke image into a smoke semantic segmentation network, wherein the smoke semantic segmentation network includes a backbone sub-network, a classification branch network, a segmentation branch network, and a prediction module; The backbone sub-network obtains a plurality of basic features of different scales, wherein the basic features include low-order basic features, medium-order basic features and high-order basic features, inputs all the basic features into the segmentation branch network, and inputs the high-order basic features into the classification branch network; The classification branch network deeply classifies the high-order basic features based on the popular space to obtain a plurality of feature prototypes, the feature prototypes are in single correspondence with the high-order basic features, and the feature prototypes are input into the segmentation branch network; The segmentation branch network performs feature enhancement on the high-order basic features according to the reliable target query extraction module to obtain reliable target query features, performs feature enhancement on the low-order basic features, the mid-order basic features and the feature prototype according to the local-global interaction module to obtain detail enhancement features, and inputs the reliable target query features and the detail enhancement features into the prediction module; The prediction module obtains a final prediction result according to the reliable target query feature and the detail enhancement feature, and performs correction and optimization according to the smoke focus loss function.

2. The smoke semantic segmentation method based on reliable query and prototype drive according to claim 1 is characterized in that: The classification branch network deeply classifies the high-order basic features based on the popular space to obtain a plurality of feature prototypes, specifically including: The classification branch network is based on a symmetric positive definite matrix of a Riemann manifold, and the classification branch network includes a Gaussian aggregation layer, a nonlinear transformation layer, a BN block, a pooling block, a global pooling block, a fully connected layer, and a prediction layer; The Gaussian aggregation layer obtains Gaussian effective information of high-order basic features to output a symmetric positive definite matrix corresponding to the Gaussian effective information to the nonlinear transformation layer, and the nonlinear transformation layer then performs eigenvalue decomposition and nonlinear transformation. The specific algorithm of the nonlinear transformation layer is as follows: ， ， in, represents the output of the nonlinear transformation layer, represents the input of the nonlinear transformation layer, represents the eigenvector obtained by eigenvalue decomposition, represents the eigenvalue matrix obtained by eigenvalue decomposition, represents the nonlinear transformation function, , T represents transpose, Represents the horizontal coordinate of the pixel point, Indicates the vertical coordinate of the pixel; The BN block and the pooling block map the symmetric positive definite matrix from the Riemann manifold to the Euclidean space according to the logarithmic function, and then remap the symmetric positive definite matrix back to the manifold space through the exponential function. The global pooling block performs global average pooling to obtain global context prior information. The specific algorithms of the BN block, the pooling block and the global pooling block are as follows: ， ， ， in, Indicates the output, Indicates input, represents a BN block, represents the pooling block, represents the global pooling block, , and They represent batch normalization, average pooling, and global average pooling operations in Euclidean space, respectively. represents the exponential function, represents the logarithmic function; The specific algorithms of the fully connected layer and the prediction layer are as follows: ， in, Represents the prediction result output by the classification network, represents the input of the fully connected layer, represents the fully connected layer, represents a nonlinear activation function; The feature prototype is obtained according to the outputs of the pooling block and the global pooling block, and the probability of containing a smoke target in the image is obtained according to the prediction result output by the classification network.

3. The smoke semantic segmentation method based on reliable query and prototype drive according to claim 1 is characterized in that: The segmentation branch network performs feature enhancement on the high-order basic features according to the reliable target query extraction module to obtain the reliable target query features, specifically comprising: The reliable target query extraction module includes an attention enhancement block and a region enhancement mapping block; The attention enhancement module performs feature weighting processing on the high-order basic features, wherein the feature weighting processing includes channel relationship weighting processing and spatial position weighting processing, and the spatial position weighting processing includes horizontal position weighting processing and vertical position weighting processing; The specific algorithm of the attention enhancement block is as follows: ， ， ， ， ， in, represents high-order basic features, represents the inverse bottleneck layer, which includes three nonlinear activation fully connected layers, represents the learnable weight tensor at different angles, Indicates a flip operation. represents a channel-by-channel addition operation, represents the feature dimension, Indicates the height, width, and number of channels of the feature, represents the activation function, and represents the horizontal position weighted vector and the vertical position weighted vector, represents the channel relationship weighted vector, Represents the output features of the attention enhancement module; The output features of the attention enhancement module are input into the region enhancement mapping block to obtain reliable target query features.

4. The smoke semantic segmentation method based on reliable query and prototype drive according to claim 3 is characterized in that: The regional enhancement mapping block specifically includes: The region enhancement mapping block obtains reliable target query features based on the output features of the attention enhancement module. The specific algorithm for obtaining reliable target query features is as follows: ， ， ， in, represents reliable target query features, represents the activation function, is a dropout operation with a probability of 0.3, represents the fully connected layer, represents the output of the intermediate layer after dimension transformation, Indicates that b × c × h × w The size of the tensor is converted to b ×1×( c × h × w ), represents average pooling with a sampling rate of 2, represents batch normalization, represents the output after 1×1 convolution, represents 1×1 convolution, Represents the output features of the attention enhancement module unfolded into a two-dimensional tensor; The obtained reliable target query features are then post-processed, and the post-processing algorithm is as follows: ， ， ， ， in, represents the key matrix involved in the attention calculation, Represents the value matrix involved in the attention calculation, represents the attention calculation, represents the reliable target query features of the final output, T represents transpose, Represents matrix multiplication.

5. The smoke semantic segmentation method based on reliable query and prototype drive according to claim 1 is characterized in that: The step of performing feature enhancement on the low-order basic features, the mid-order basic features and the feature prototype according to the local-global interaction module to obtain detail enhancement features specifically includes: The local-global interaction module includes the inter-domain attention learning block and the NL block; The specific algorithm of the inter-domain attention learning block is as follows: ， ， ， in, Indicates that Copy 64 times, Represents a class prototype tensor, the size of which is b × c × h / 8× w / 8, represents the key matrix and the value matrix, express Divided into 64 sub-regions of equal size, represents the input feature of the inter-domain attention learning block, and the size of the input feature is b × c × h × w , represents the attention calculation, represents the activation function, represents the output of the inter-domain attention learning block, Indicates that 64× b × c × h / 8× w / 8The size of the tensor is converted to b × c × h × w , represents the Hadamard product; The specific algorithm of the NL block is as follows: ， ， in, represents the query matrix, represents the input of the NL block, Will Mapped to the corresponding query matrix Q , the bond matrix K Sum Matrix V , represents 1×1 convolution, represents the output of the NL block, Represents the key matrix K The length of the vector, T represents transpose, The object label representing the convolution function service; Detail enhancement features are obtained based on the outputs of the inter-domain attention learning block and the NL block.

6. The smoke semantic segmentation method based on reliable query and prototype drive according to claim 1, characterized in that: The step of obtaining the final prediction result according to the reliable target query feature and the detail enhancement feature by the prediction module specifically includes: The prediction module includes an upsampling convolution module, a convolution module, and a sigmoid nonlinear activation layer; The upsampling convolution module includes a 2x bilinear interpolation upsampling layer, a 3×3 convolution layer, a BN layer, and a ReLu layer; The convolution module includes a 3×3 convolution layer, a BN layer and a ReLu layer; The prediction module maps reliable target query features and detail enhancement features from feature space to semantic space to obtain final segmentation results.

7. The smoke semantic segmentation method based on reliable query and prototype drive according to claim 1 is characterized in that: The step of correcting and optimizing according to the smoke focusing loss function specifically includes: Correction and optimization are performed according to the smoke focusing loss function, and the specific algorithm of the smoke focusing loss function is as follows: ， ， ， ， ， ， ， ， in, represents the smoke focusing loss function, and represents the weight of foreground and background loss distribution, represents prospect loss, represents the background loss, express The number of foreground pixels, express The number of foreground pixels, express The number of background pixels, express The number of background pixels, The segmentation label representing the final segmentation result after flattening, represents the final segmentation result after flattening and binarization, represents a vector of all 1s, represents transpose, represents the flattening operation, The segmentation label representing the final segmentation result, Represents the final segmentation result of binarization, represents the final segmentation result, represents the feature position coordinates, Indicates the feature height of the final segmentation result, Represents the feature width of the final segmentation result, Represents a positive integer.

8. A smoke semantic segmentation system based on reliable query and prototype drive, characterized in that: include: A preprocessing module, used for acquiring a single-frame smoke image in real time and performing preprocessing, so as to input the preprocessed smoke image into a smoke semantic segmentation network, wherein the smoke semantic segmentation network includes a backbone sub-network, a classification branch network, a segmentation branch network and a prediction module; A basic feature extraction module is used for the backbone sub-network to obtain a plurality of basic features of different scales, wherein the basic features include low-order basic features, medium-order basic features and high-order basic features, and all the basic features are input into the segmentation branch network, and the high-order basic features are input into the classification branch network; A classification module, used for the classification branch network to perform deep classification on the high-order basic features based on the popular space to obtain a plurality of feature prototypes, the feature prototypes having a single correspondence with the high-order basic features, and inputting the feature prototypes into the segmentation branch network; A segmentation module, used for the segmentation branch network to perform feature enhancement on the high-order basic features according to the reliable target query extraction module to obtain reliable target query features, perform feature enhancement on the low-order basic features, the mid-order basic features and the feature prototype according to the local-global interaction module to obtain detail enhancement features, and input the reliable target query features and the detail enhancement features into a prediction module; A prediction optimization module is used for the prediction module to obtain a final prediction result according to the reliable target query feature and the detail enhancement feature, and to perform correction and optimization according to the smoke focusing loss function.

9. A storage medium, characterized in that: The storage medium stores one or more programs, which, when executed by a processor, implement the smoke semantic segmentation method based on reliable query and prototype drive as described in any one of claims 1 to 7.

10. A computer device, characterized in that: The computer device comprises a memory and a processor, wherein: The memory is used to store computer programs; When the processor is used to execute the computer program stored in the memory, the smoke semantic segmentation method based on reliable query and prototype drive described in any one of claims 1 to 7 is implemented.

Images