Infrared image segmentation model training method and infrared image segmentation method

Abstract

The present invention relates to the field of image processing technology, and provides an infrared image segmentation model training method and an infrared image segmentation method. The training method introduces an auxiliary segmentation model which takes a high-frequency image block as input and has the same structure and shared parameters as an initial infrared image segmentation model, so as to assist in training the initial infrared image segmentation model, thereby improving the target infrared image segmentation model's ability to capture high-frequency information in infrared images, and further improving the segmentation effect and segmentation accuracy of the target infrared image segmentation model on infrared images.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an infrared image segmentation model training method and an infrared image segmentation method. Background Art

[0002] Compared with visible light imaging, infrared imaging has great advantages in adverse imaging conditions (low light, glare, etc.) and adverse weather conditions (rain, snow, fog, haze, etc.). Therefore, infrared imaging has been widely valued in applications such as nighttime assisted driving and drone monitoring systems. Infrared image segmentation is one of the key technologies of infrared-based visual perception systems. However, problems such as low resolution of infrared images and lack of texture information lead to low accuracy of current infrared image segmentation algorithms, which are difficult to meet actual needs.

[0003] Thanks to the attention mechanism's ability to model long-distance relationships, attention-based visual backbone networks (such as ViT and Swin) have powerful feature expression capabilities. Image segmentation networks based on the attention mechanism (such as SegFormer) have achieved significant improvements in natural image segmentation tasks compared to convolution-based image segmentation networks (such as DeeplabV3+).

[0004] However, since the attention mechanism tends to capture low-frequency information and ignore high-frequency information, and infrared images have low resolution and lack of texture information, the high-frequency information in infrared images is even weaker. Therefore, the problem that the attention mechanism is difficult to extract high-frequency information is more prominent for infrared images, resulting in poor segmentation effect and low segmentation accuracy of infrared images. Summary of the invention

[0005] The present invention provides an infrared image segmentation model training method and an infrared image segmentation method, which are used to solve the defects of the image segmentation method in the prior art that it is difficult to effectively capture high-frequency information on infrared images, resulting in poor segmentation effect and low segmentation accuracy.

[0006] The present invention provides an infrared image segmentation model training method, comprising:

[0007] Acquire an infrared image sample and segmentation annotation information corresponding to the infrared image sample, input the infrared image sample into an initial infrared image segmentation model, and obtain a first segmentation category;

[0008] Based on the high-frequency information image in the infrared image sample, extract a high-frequency image block from the infrared image sample, and input the high-frequency image block into the auxiliary segmentation model to obtain a second segmentation category; the initial infrared image segmentation model and the auxiliary segmentation model have the same structure and share parameters;

[0009] Calculate the model loss based on the segmentation annotation information, the first segmentation category, the second segmentation category, the first encoding information obtained by encoding the infrared image sample by the initial infrared image segmentation model, and the second encoding information obtained by encoding the high-frequency image block by the auxiliary segmentation model;

[0010] Based on the model loss, the initial infrared image segmentation model and the auxiliary segmentation model are synchronously trained, and the trained initial infrared image segmentation model is used as the target infrared image segmentation model.

[0011] According to an infrared image segmentation model training method provided by the present invention, the initial infrared image segmentation model is constructed based on SegFormer;

[0012] The auxiliary segmentation model is built based on sparse SegFormer;

[0013] The sparse SegFormer is obtained by replacing the convolution operation in the SegFormer with a sparse convolution operation and canceling the downsampling operation in the coding block of the SegFormer.

[0014] According to an infrared image segmentation model training method provided by the present invention, the sparse convolution operation includes:

[0015] Corresponding the input feature sequence to the coordinates in the image coordinate system to obtain an input feature map; the element values ​​of other positions in the input feature map except the position corresponding to the feature sequence are all 0;

[0016] In the non-zero position of the input feature map, an input image block of a preset size is extracted, the convolution kernel of the convolution operation is transformed, and the input image block and the transpose of the transformed convolution kernel are matrix multiplied to obtain the output result of the sparse convolution operation.

[0017] According to an infrared image segmentation model training method provided by the present invention, the high-frequency image blocks are extracted from the infrared image samples based on the high-frequency information images in the infrared image samples, comprising:

[0018] Adjusting the high-frequency information image to a spare image having the same size as the infrared image sample, extracting spare image blocks of a preset size from the spare image, and calculating an L2 norm map corresponding to each spare image block;

[0019] Binarization and expansion processing are performed on the L2 norm map to obtain a binary expansion map;

[0020] A sample image block of the preset size is extracted from the infrared image sample, the binary expansion map is used as a mask, and a sample image block corresponding to a non-zero value is extracted as the high-frequency image block.

[0021] According to an infrared image segmentation model training method provided by the present invention, the first coded information and the second coded information both include multiple extraction stages;

[0022] The calculating of the model loss based on the segmentation annotation information, the first segmentation category, the second segmentation category, the first encoding information obtained by encoding the infrared image sample by the initial infrared image segmentation model, and the second encoding information obtained by encoding the high-frequency image block by the auxiliary segmentation model comprises:

[0023] Calculating a first segmentation loss based on the segmentation label information and the first segmentation category;

[0024] Calculating a second segmentation loss based on the segmentation label information and the second segmentation category;

[0025] Based on the first encoding information and the second encoding information obtained in each extraction stage, calculating the knowledge distillation loss of each extraction stage;

[0026] The model loss is calculated based on the first segmentation loss, the second segmentation loss, and the knowledge distillation loss.

[0027] According to an infrared image segmentation model training method provided by the present invention, the second segmentation loss is calculated based on the segmentation annotation information and the second segmentation category, including:

[0028] Determining segmentation annotation information corresponding to the high-frequency image block based on the binary expansion map;

[0029] The second segmentation loss is calculated based on the segmentation annotation information corresponding to the high-frequency image block and the second segmentation category.

[0030] According to an infrared image segmentation model training method provided by the present invention, the high-frequency information image is extracted based on the following steps:

[0031] Determining the high-frequency components of the infrared image samples based on a two-dimensional discrete wavelet transform algorithm;

[0032] The high-frequency components are combined to obtain the high-frequency information image.

[0033] The present invention also provides an infrared image segmentation method, comprising:

[0034] Acquire the infrared image to be segmented;

[0035] Inputting the infrared image to be segmented into a target infrared image segmentation model to obtain a segmentation category of the infrared image to be segmented output by the target infrared image segmentation model;

[0036] Wherein, the target infrared image segmentation model is trained based on the above-mentioned infrared image segmentation model training method.

[0037] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the infrared image segmentation model training method or the infrared image segmentation method as described above is implemented.

[0038] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the infrared image segmentation model training method or infrared image segmentation method as described in any one of the above is implemented.

[0039] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the infrared image segmentation model training method or the infrared image segmentation method as described above is implemented.

[0040] The infrared image segmentation model training method and the infrared image segmentation method provided by the present invention introduce an auxiliary segmentation model which takes high-frequency image blocks as input and has the same structure and shared parameters as the initial infrared image segmentation model in the training method, so as to assist in the training of the initial infrared image segmentation model. This can improve the target infrared image segmentation model's ability to capture high-frequency information in infrared images, thereby improving the segmentation effect and segmentation accuracy of the target infrared image segmentation model for infrared images. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on the drawings in the following description without creative work.

[0042] Figure 1 It is a flow chart of the infrared image segmentation model training method provided by the present invention;

[0043] Figure 2 It is a structural schematic diagram of SegFormer and sparse SegFormer in the infrared image segmentation model training method provided by the present invention;

[0044] Figure 3It is a structural schematic diagram of the Sparse Block in the infrared image segmentation model training method provided by the present invention;

[0045] Figure 4 It is a schematic diagram of the process of extracting high-frequency image blocks in the infrared image segmentation model training method provided by the present invention;

[0046] Figure 5 It is a flow chart of the infrared image segmentation method provided by the present invention;

[0047] Figure 6 It is a structural schematic diagram of the infrared image segmentation model training device provided by the present invention;

[0048] Figure 7 It is a structural schematic diagram of the infrared image segmentation device provided by the present invention;

[0049] Figure 8 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] The term "first" or "second" in the specification and claims of the present invention may include one or more of the features explicitly or implicitly. In the description of the invention, unless otherwise specified, "plurality" means two or more. In addition, "and / or" in the specification and claims means at least one of the connected objects, and the character " / " generally means that the objects connected before and after are in an "or" relationship.

[0052] Figure 1 FIG. 1 is a flow chart of an infrared image segmentation model training method provided in an embodiment of the present invention, such as Figure 1 As shown, the training method includes:

[0053] S11, obtaining an infrared image sample and segmentation annotation information corresponding to the infrared image sample, inputting the infrared image sample into an initial infrared image segmentation model, and obtaining a first segmentation category;

[0054] S12, based on the high-frequency information image in the infrared image sample, extracting a high-frequency image block from the infrared image sample, and inputting the high-frequency image block into an auxiliary segmentation model to obtain a second segmentation category; the initial infrared image segmentation model and the auxiliary segmentation model have the same structure and share parameters;

[0055] S13, calculating a model loss based on the segmentation annotation information, the first segmentation category, the second segmentation category, first encoding information obtained by encoding the infrared image sample by the initial infrared image segmentation model, and second encoding information obtained by encoding the high-frequency image block by the auxiliary segmentation model;

[0056] S14, based on the model loss, synchronously training the initial infrared image segmentation model and the auxiliary segmentation model, and using the trained initial infrared image segmentation model as the target infrared image segmentation model.

[0057] Specifically, the infrared image segmentation model training method provided in the embodiment of the present invention is executed by an infrared image segmentation model training device, which can be configured in a computer. The computer can be a local computer or a cloud computer. The local computer can be a computer, a tablet, etc., which is not specifically limited here.

[0058] First, execute step S11 to obtain infrared image samples and segmentation annotation information corresponding to the infrared image samples. The infrared image samples can be determined by the following steps:

[0059] 1) Collect infrared video clips through the vehicle-mounted infrared sensor, set the frame rate to 30Hz, and set the resolution to 640x512. The infrared video clips can cover a variety of scenes such as intersections, streets, residential areas, sidewalks, and playgrounds.

[0060] 2) Sample one frame every 10 frames in the infrared video clip to obtain an infrared image sample. All infrared image samples form an original unlabeled image set. The number of infrared image samples can be set as needed and is not specifically limited here.

[0061] The segmentation annotation information corresponding to the infrared image sample can be obtained by annotating the target in the infrared image sample through LabelMe. Among them, the boundary of the target can be annotated with a polygon, and the polygon annotation can be converted into a mask annotation. The background class corresponds to a value of 0, and other target classes are represented by different non-zero integers. Targets can include people, cars, bicycles, motorcycles, tricycles, vans, etc.

[0062] The infrared image samples may include training samples and test samples. The training samples and test samples may be divided in a ratio of 8:2. All training samples may constitute a training set for model training, and all test samples may constitute a test set for evaluating model performance.

[0063] Afterwards, the infrared image sample can be input into the initial infrared image segmentation model, and the initial infrared image segmentation model outputs the first segmentation category. The initial infrared image segmentation model can be constructed based on a neural network, for example, can be constructed based on a network such as Transformer, SegFormer, etc.

[0064] Then, step S12 is performed to extract high-frequency image blocks from the infrared image samples using the high-frequency information images in the infrared image samples. The high-frequency information images can be extracted from the infrared image samples in advance and stored in the hard disk. When extracting high-frequency image blocks from the infrared image samples, they can be directly searched from the hard disk, which can improve the extraction efficiency of high-frequency image blocks, reduce the amount of calculation, and reduce the calculation cost.

[0065] When extracting high-frequency information images from infrared image samples, the Sobel operator, Laplcian operator, etc. can be used to extract the high-frequency information images in the infrared image samples, or the low-frequency components in the infrared image samples can be extracted first, and then the high-frequency information images can be obtained by subtracting the low-frequency components from the infrared image samples. The high-frequency information images can also be extracted by edge detection algorithms.

[0066] When extracting high-frequency image blocks from infrared image samples, the high-frequency information image can be used as a reference to extract high-frequency image blocks from the infrared image samples with a first specified step length. The first specified step length and the size of the high-frequency image blocks can be set as needed. For example, the first specified step length is set to s 1 =4, the size of the high-frequency image block is set to k×k, and the value of k can be 7 or other values.

[0067] High-frequency image patches can be obtained by tensor X∈R M×(C×k×k) , C=3 is the number of channels, and M is the number of non-zero elements in the high-frequency image block.

[0068] After that, the high-frequency image blocks are input into the auxiliary segmentation model to obtain the second segmentation category; the auxiliary segmentation model has the same structure as the initial infrared image segmentation model and shares parameters. The difference is that the initial infrared image segmentation model is used to process infrared image samples, and the auxiliary segmentation model is used to process high-frequency image blocks. If the initial infrared image segmentation model is built based on SegFormer, the auxiliary segmentation model can be built based on SegFormer or on a new network obtained by modifying the operations in SegFormer without changing the network structure and structural parameters of SegFormer.

[0069] Thereafter, step S13 is executed to calculate the model loss by using the segmentation annotation information, the first segmentation category, the second segmentation category, the first encoding information obtained by encoding the infrared image sample by the initial infrared image segmentation model, and the second encoding information obtained by encoding the high-frequency image block by the auxiliary segmentation model.

[0070] Among them, both the initial infrared image segmentation model and the auxiliary segmentation model can include encoders. The encoder in the initial infrared image segmentation model can encode infrared image samples to obtain first encoding information, and the encoder in the auxiliary segmentation model can encode high-frequency image blocks to obtain second encoding information.

[0071] The first segmentation loss corresponding to the initial infrared image segmentation model can be calculated using the segmentation annotation information and the first segmentation category, and the second segmentation loss corresponding to the auxiliary segmentation model can be calculated using the segmentation annotation information and the second segmentation category. The second segmentation loss can be understood as a mask segmentation loss.

[0072] The knowledge distillation loss can be calculated using the first encoding information and the second encoding information.

[0073] Afterwards, the first segmentation loss, the second segmentation loss, and the knowledge distillation loss can be weighted and summed to calculate the model loss.

[0074] Finally, step S14 is executed, and the initial infrared image segmentation model and the auxiliary segmentation model can be synchronously iterated and trained by using the model loss, and the structural parameters of the initial infrared image segmentation model and the auxiliary segmentation model can be updated until the model loss converges or reaches a preset number of iterations, and the trained initial infrared image segmentation model and the trained auxiliary segmentation model are obtained, and the trained initial infrared image segmentation model is used as the target infrared image segmentation model. The target infrared image segmentation model can be used for infrared image segmentation with high-frequency information enhancement in the subsequent process.

[0075] In the embodiment of the present invention, the AdamW optimizer can be used for training, the learning rate is 1e-6, the batch size of the image is 16, and the number of iterations is 16000.

[0076] In the embodiment of the present invention, the iterative training process can be implemented in code using the MMSegmentation framework. The initial infrared image segmentation model and the auxiliary segmentation model can be initialized using parameters pre-trained on ImageNet. The size of the input image of the initial infrared image segmentation model and the auxiliary segmentation model can both be 640×640.

[0077] In the embodiment of the present invention, forward reasoning can be performed on the test set to predict the segmentation category of each test sample, and the mIoU is used to evaluate the segmentation accuracy of the target infrared image segmentation model.

[0078] The infrared image segmentation model training method provided in the embodiment of the present invention introduces an auxiliary segmentation model that takes high-frequency image blocks as input and has the same structure and shared parameters as the initial infrared image segmentation model, to assist in training the initial infrared image segmentation model. This can improve the target infrared image segmentation model's ability to capture high-frequency information in infrared images, thereby improving the target infrared image segmentation model's segmentation effect and segmentation accuracy for infrared images.

[0079] On the basis of the above embodiment, the initial infrared image segmentation model is constructed based on SegFormer;

[0080] The auxiliary segmentation model is built based on sparse SegFormer;

[0081] The sparse SegFormer is obtained by replacing the convolution operation in the SegFormer with a sparse convolution operation and canceling the downsampling operation in the coding block of the SegFormer.

[0082] Specifically, in the embodiment of the present invention, the initial infrared image segmentation model can be constructed by SegFormer, and the auxiliary segmentation model can be constructed by Sparse SegFormer.

[0083] like Figure 2 As shown, the encoder of SegFormer includes a block embedding (PatchEmbed) layer and four encoding blocks (Block), and the decoder of SegFormer includes two multilayer perceptrons (Multilayer Perceptron, MLP).

[0084] The input of the PatchEmbed layer is an infrared image sample, and the size of the infrared image sample can be H×W, where H is the height of the infrared image sample and W is the width of the infrared image sample. The output of the block embedding layer passes through the first encoding block (Block1) and enters the first extraction stage to obtain the first encoding information of the first scale, which can be C 1 The number of output channels of Block1.

[0085] The first coded information of the first scale passes through the second coding block (Block2) and enters the second extraction stage to obtain the first coded information of the second scale. The second scale can be C 2 The number of output channels of Block2.

[0086] The first coded information of the second scale passes through the third coding block (Block3) and enters the third extraction stage to obtain the first coded information of the third scale. The third scale can be C 3 The number of output channels of Block3.

[0087] The first coded information of the third scale passes through the fourth coding block (Block4) and enters the fourth extraction stage to obtain the first coded information of the fourth scale. The fourth scale can be C 4 The number of output channels of Block4.

[0088] The first coded information of the four scales are input into the first multi-layer perceptron (MLP1), and the first decoded information of the fifth scale can be obtained by MLP1. The fifth scale can be

[0089] The first decoded information of the fifth scale can be obtained by the second multi-layer perceptron (MLP2) to obtain the first decoded information of the sixth scale, and the sixth scale can be N cls is the number of the first segmentation categories.

[0090] Since the PatchEmbed layer and each Block contain convolution operations (Conv), Conv is replaced with sparse convolution operations (SpConv), and the downsampling operations in each Block of SegFormer are canceled to obtain SparseSegFormer.

[0091] After the Conv in the PatchEmbed layer is replaced with SpConv, it can become a sparse block embedding (SparsePatchEmbed) layer, and after the Conv in the Block is replaced with SpConv and the downsampling operation is cancelled, it can become a sparse coding block (Sparse Block). Therefore, the encoder of Sparse SegFormer can include a Sparse PatchEmbed layer and four Sparse Blocks, and the decoder of Sparse SegFormer is the same as that of SegFormer, both of which include two MLPs.

[0092] The input of the Sparse PatchEmbed layer is a high-frequency image block. The Sparse PatchEmbed layer uses sparse convolution operations and normalization operations to transform the high-frequency image blocks to obtain an input sequence, namely LN(SpConv(X)). Among them, LN is a layer normalization operation. Unlike the PatchEmbed layer in SegFormer, which directly processes the input image, the input high-frequency image block cannot restore the two-dimensional grid structure of the infrared image sample. The dense convolution calculation method in SegFormer will still bring redundant calculations, while the use of sparse convolution operations can reduce the amount of calculation.

[0093] The output of the Sparse PatchEmbed layer passes through the first sparse coding block (Sparse Block1) and enters the first extraction stage to obtain the second coding information of the seventh scale, which can be C 1 The number of output channels of Sparse Block1 is the same as the number of output channels of Block1.

[0094] The second coded information of the seventh scale passes through the second sparse coding block (Sparse Block2) and enters the second extraction stage to obtain the second coded information of the eighth scale. The eighth scale can be C 2 The number of output channels of Sparse Block2 is the same as the number of output channels of Block2.

[0095] The second coded information of the eighth scale passes through the third sparse coding block (Sparse Block3) and enters the third extraction stage to obtain the second coded information of the ninth scale. The ninth scale can be c 3 The number of output channels of Sparse Block3 is the same as the number of output channels of Block3.

[0096] The second coded information of the ninth scale passes through the fourth sparse coding block (Sparse Block4) and enters the fourth extraction stage to obtain the second coded information of the tenth scale. The tenth scale can be c 4 The number of output channels of Sparse Block4 is the same as the number of output channels of Block4.

[0097] The second encoded information of the four scales is input into the first multi-layer perceptron (MLP1), and the second decoded information of the eleventh scale can be obtained by the first multi-layer perceptron. The eleventh scale can be

[0098] The second decoded information of the eleventh scale can be obtained by the second multi-layer perceptron (MLP2) to obtain the second decoded information of the twelfth scale. The twelfth scale can be N cls is the number of the second segmentation categories, which is the same as the number of the first segmentation categories.

[0099] In an embodiment of the present invention, the introduction of a sparse block embedding layer can transform the sparse high-frequency image blocks into an input sequence at the input end of the auxiliary segmentation model, or reduce the length of the sequence in the middle layer of the auxiliary segmentation model, so as to facilitate calculation. In each stage of the auxiliary segmentation model, the introduction of sparse coding blocks can also effectively process the input sparse high-frequency image blocks. Therefore, constructing an auxiliary segmentation model based on sparse SegFormer can effectively process sparse high-frequency image blocks, reduce the amount of calculation, improve the efficiency of obtaining the second segmentation category, and thus improve the training rate of the infrared image segmentation model.

[0100] Based on the above embodiment, the sparse convolution operation includes:

[0101] Corresponding the input feature sequence to the coordinates in the image coordinate system to obtain an input feature map; the element values ​​of other positions in the input feature map except the position corresponding to the feature sequence are all 0;

[0102] In the non-zero position of the input feature map, an input image block of a preset size is extracted, the convolution kernel of the convolution operation is transformed, and the input image block and the transpose of the transformed convolution kernel are matrix multiplied to obtain the output result of the sparse convolution operation.

[0103] Specifically, the calculation process of SpConv may include: for the input feature sequence, if X 0 ∈R M×C , where M and C represent the length and number of channels of the feature sequence respectively.

[0104] X 0Each feature in the image corresponds to the coordinates in the image coordinate system, and the input feature map can be restored Where s is the current step size. The elements in are defined as:

[0105]

[0106] Among them, (i m , j m ) is the spatial coordinate corresponding to the mth feature, for The i m Row, j m Column, c channel element value, X 0 [m, c] is the value of the element in the mth row and cth column of X. The values ​​in other positions are defined as 0.

[0107] exist The non-zero position of can be taken out according to the step size 2 3 × 3 size input image block, represented as a tensor

[0108] If the convolution kernel of the convolution operation is W, the Reshape operation can be used to transform W. The transformed convolution kernel can be expressed as Reshape(W), and its transpose is Reshape(W). T .

[0109] The input image block is matrix-multiplied with the transposed convolution kernel to obtain the output of the sparse convolution operation, i.e., Y Reshape(W) T .

[0110] In the embodiment of the present invention, a sparse convolution operation is introduced into the auxiliary segmentation model, which can reduce the amount of calculation and improve the calculation efficiency.

[0111] Based on the above embodiments, Figure 3 The structural diagram of Sparse Block is given. Each SparseBlock contains N self-attention layers, N FFN layers and a SparsePatchEmbed layer, in which the self-attention layers and FFN layers are used alternately.

[0112] In the self-attention layer, the input sequence X 1 ∈R M×C First, after layer normalization (LN), we get LN(X 1 ). LN(X 1 )After linear transformation, we get the query sequence Q∈R M×C :

[0113] Q=Linear(LN(X1 ))

[0114] For LN(X 1 ) Perform SpConv, LN, and Linear in sequence to obtain the key K∈R M×C The sum V∈R M×C :

[0115] K=Linear(LN(spConv(LN(X 1 ))))

[0116] V=Linear(LN(spConv(LN(X 1 ))))

[0117] The output of the Q, K, V self-attention layer is expressed as:

[0118]

[0119] Among them, MatMul represents matrix-matrix multiplication, and Softmax is the soft maximization function.

[0120] In the FFN layer, including LN, Linear, SpConv, Act, Linear, it is formally expressed as:

[0121] FFN(x)=Linear(Act(SpConv((Linear(LN(X 1 ))))))

[0122] Among them, Act represents a nonlinear activation function.

[0123] On the basis of the above embodiment, the step of extracting a high-frequency image block from the infrared image sample based on the high-frequency information image in the infrared image sample includes:

[0124] Adjusting the high-frequency information image to a spare image having the same size as the infrared image sample, extracting spare image blocks of a preset size from the spare image, and calculating an L2 norm map corresponding to each spare image block;

[0125] Binarizing the L2 norm image to obtain a binary expansion image;

[0126] A sample image block of the preset size is extracted from the infrared image sample, the binary expansion map is used as a mask, and a sample image block corresponding to a non-zero value is extracted as the high-frequency image block.

[0127] Specifically, for each infrared image sample, read the corresponding high-frequency information image. Then follow the steps below:

[0128] Step 1: Figure 4 In ①, by interpolating the high-frequency information image, a spare image with the same size as the infrared image sample is obtained, which is represented by I dwt ∈R H×W×C (C=3) Extract I according to step size s=4 dwt All candidate image blocks of size k×k in , where k×k is the preset size and the value of k can be 7, and the tensor is obtained

[0129] After that, the L2 norm of each candidate image block of size C×k×k is calculated to form an L2 norm graph

[0130] Step 2: If Figure 4 ② in the figure, find the first K maximum values ​​in the L2 norm graph L, set the positions corresponding to these values ​​to 1, and set the other positions to 0 to obtain the initial binary graph In order to balance accuracy and training speed, K can be set to 256. In addition, the adaptive threshold method can also be used to binarize the L2 norm image to obtain the initial binary image B.

[0131] Step 3: If Figure 4 ③, perform expansion operation on B to obtain a binary expansion map

[0132] Step 4: Figure 4 ④, extract infrared image samples I∈R according to the step size s=4 H×W×C All sample image blocks of size k×k in the tensor are obtained

[0133] Use B dilate As a mask, extract the sample image block corresponding to the non-zero value, that is, get the high-frequency image block, represented by X∈R M×(C×k×k) .

[0134] Through the above steps, X contains high-frequency information. Figure 4 As shown, it is mainly manifested in corners and edges.

[0135] In the embodiment of the present invention, by means of the L2 norm map and the mask extraction operation, the obtained high-frequency image block can be made more accurate and have higher precision.

[0136] Based on the above embodiment, the first coded information and the second coded information both include multiple extraction stages;

[0137] The calculating of the model loss based on the segmentation annotation information, the first segmentation category, the second segmentation category, the first encoding information obtained by encoding the infrared image sample by the initial infrared image segmentation model, and the second encoding information obtained by encoding the high-frequency image block by the auxiliary segmentation model comprises:

[0138] Calculating a first segmentation loss based on the segmentation label information and the first segmentation category;

[0139] Calculating a second segmentation loss based on the segmentation label information and the second segmentation category;

[0140] Based on the first encoding information and the second encoding information obtained in each extraction stage, calculating the knowledge distillation loss of each extraction stage;

[0141] The model loss is calculated based on the first segmentation loss, the second segmentation loss, and the knowledge distillation loss.

[0142] Specifically, Figure 2 As shown, the first coding information and the second coding information both include multiple scales, for example, both may include 4 scales.

[0143] When calculating the model loss, the first segmentation loss is calculated using the segmentation label information and the first segmentation category, and the second segmentation loss is calculated using the segmentation label information and the second segmentation category.

[0144] Then, the knowledge distillation loss of each extraction stage is calculated using the first encoding information and the second encoding information of the extraction stage.

[0145] Assume that the output sequence of the nth Block of SegFormer is X dense , the output sequence of the nth Sparse Block of SparseSegFormer is X sparse The knowledge distillation loss L at the nth extraction stage is rel It can be expressed as:

[0146]

[0147] Among them, l is the Huber loss function, d is the distance function, X spars [i,:] represents X sparse The i-th row, X sparse [j,:] represents X sparse The jth row, m i For X dense Zhong and X sparse The index corresponding to the i-th row in m j For X dense Zhong and X sparseThe index corresponding to the j-th row in .

[0148] Here, the distance function d may be a Euclidean distance function.

[0149] The knowledge distillation loss requires SegFormer and Sparse SegFormer to maintain the pairwise feature similarity on high-frequency image blocks as much as possible, thereby introducing the high-frequency related knowledge captured by Sparse SegFormer into SegFormer, enabling SegFormer to capture high-frequency information.

[0150] Finally, the model loss is calculated by weighted summing the first segmentation loss, the second segmentation loss, and the knowledge distillation loss of each extraction stage.

[0151] In the embodiment of the present invention, the knowledge distillation loss is introduced to enable the target infrared image segmentation model to have the ability to capture high-frequency information.

[0152] On the basis of the above embodiment, the calculating the second segmentation loss based on the segmentation labeling information and the second segmentation category includes:

[0153] Determining segmentation annotation information corresponding to the high-frequency image block based on the binary expansion map;

[0154] The second segmentation loss is calculated based on the segmentation annotation information corresponding to the high-frequency image block and the second segmentation category.

[0155] Specifically, since Sparse SegFormer only processes high-frequency image blocks, it can only predict the second segmentation category of high-frequency image blocks. Therefore, when calculating the second segmentation loss, the segmentation annotation information corresponding to each high-frequency image block can be found through the binary expansion map, and then the second segmentation loss can be calculated based on the segmentation annotation information and the second segmentation category.

[0156] In the embodiment of the present invention, the segmentation annotation information corresponding to the high-frequency image block can be quickly determined through the binary expansion map, thereby improving the calculation efficiency of the second segmentation loss.

[0157] Based on the above embodiment, the high-frequency information image is extracted based on the following steps:

[0158] Determining the high-frequency components of the infrared image samples based on a two-dimensional discrete wavelet transform algorithm;

[0159] The high-frequency components are combined to obtain the high-frequency information image.

[0160] Specifically, when extracting high-frequency information images, the frequency components of infrared image samples can be extracted by a two-dimensional discrete wavelet transform (DWT) algorithm. Each frequency component may include LL, HL, LH, and HH, where LL is a low-frequency component, which is an approximate image, and HL, LH, and HH are three high-frequency components, which are horizontal detail images, vertical detail images, and diagonal detail images, respectively.

[0161] Thereafter, the three high-frequency components of HL, LH, and HH are combined to form a pseudo-color image, which is a high-frequency information image.

[0162] Afterwards, the high-frequency information image can be saved to the hard disk so that it can be read during the model training process, which can improve the data loading speed.

[0163] like Figure 5 As shown, based on the above embodiment, an infrared image segmentation method is also provided in an embodiment of the present invention, and the method includes:

[0164] S21, obtaining the infrared image to be segmented;

[0165] S22, inputting the infrared image to be segmented into a target infrared image segmentation model to obtain a segmentation category of the infrared image to be segmented output by the target infrared image segmentation model;

[0166] The target infrared image segmentation model is trained based on the infrared image segmentation model training method provided in the above embodiments.

[0167] Specifically, the infrared image segmentation method provided in the embodiment of the present invention is executed by an infrared image segmentation device, which can be configured in a computer. The computer can be a local computer or a cloud computer. The local computer can be a computer, a tablet, etc., which is not specifically limited here.

[0168] First, step S21 is performed to obtain an infrared image to be segmented. The infrared image to be segmented can be of the same type as the infrared image sample to ensure the segmentation effect of the target infrared image segmentation model.

[0169] Thereafter, step S22 is executed to input the infrared image to be segmented into the target infrared image segmentation model, and the segmentation category of the infrared image to be segmented is determined by the target infrared image segmentation model.

[0170] Here, the target infrared image segmentation model applied may be obtained by training through the infrared image segmentation model training method provided in the above embodiments, so that the segmentation category of the infrared image to be segmented is more accurate and the segmentation accuracy is higher.

[0171] like Figure 6 As shown, based on the above embodiment, an infrared image segmentation model training device is provided in an embodiment of the present invention, comprising:

[0172] A first segmentation module 61 is used to obtain infrared image samples and segmentation annotation information corresponding to the infrared image samples, input the infrared image samples into an initial infrared image segmentation model, and obtain a first segmentation category;

[0173] A second segmentation module 62 is used to extract high-frequency image blocks from the infrared image samples based on the high-frequency information images in the infrared image samples, and input the high-frequency image blocks into the auxiliary segmentation model to obtain a second segmentation category; the initial infrared image segmentation model and the auxiliary segmentation model have the same structure and share parameters;

[0174] A loss calculation module 63, configured to calculate a model loss based on the segmentation annotation information, the first segmentation category, the second segmentation category, the first encoding information obtained by encoding the infrared image sample by the initial infrared image segmentation model, and the second encoding information obtained by encoding the high-frequency image block by the auxiliary segmentation model;

[0175] The synchronous training module 64 is used to synchronously train the initial infrared image segmentation model and the auxiliary segmentation model based on the model loss, and use the trained initial infrared image segmentation model as the target infrared image segmentation model.

[0176] On the basis of the above-mentioned embodiment, in the infrared image segmentation model training device provided in the embodiment of the present invention, the initial infrared image segmentation model is constructed based on SegFormer;

[0177] The auxiliary segmentation model is built based on sparse SegFormer;

[0178] The sparse SegFormer is obtained by replacing the convolution operation in the SegFormer with a sparse convolution operation and canceling the downsampling operation in the coding block of the SegFormer.

[0179] On the basis of the above embodiment, in the infrared image segmentation model training device provided in the embodiment of the present invention, the sparse convolution operation includes:

[0180] Corresponding the input feature sequence to the coordinates in the image coordinate system to obtain an input feature map; the element values ​​of other positions in the input feature map except the position corresponding to the feature sequence are all 0;

[0181] In the non-zero position of the input feature map, an input image block of a preset size is extracted, the convolution kernel of the convolution operation is transformed, and the input image block and the transpose of the transformed convolution kernel are matrix multiplied to obtain the output result of the sparse convolution operation.

[0182] On the basis of the above-mentioned embodiment, in the infrared image segmentation model training device provided in the embodiment of the present invention, the second segmentation module is specifically used for:

[0183] Adjusting the high-frequency information image to a spare image having the same size as the infrared image sample, extracting spare image blocks of a preset size from the spare image, and calculating an L2 norm map corresponding to each spare image block;

[0184] Binarization and expansion processing are performed on the L2 norm map to obtain a binary expansion map;

[0185] A sample image block of the preset size is extracted from the infrared image sample, the binary expansion map is used as a mask, and a sample image block corresponding to a non-zero value is extracted as the high-frequency image block.

[0186] On the basis of the above-mentioned embodiment, in the infrared image segmentation model training device provided in the embodiment of the present invention, the first coding information and the second coding information both include multiple extraction stages; the loss calculation module is specifically used for:

[0187] Calculating a first segmentation loss based on the segmentation label information and the first segmentation category;

[0188] Calculating a second segmentation loss based on the segmentation label information and the second segmentation category;

[0189] Based on the first encoding information and the second encoding information obtained in each extraction stage, calculating the knowledge distillation loss of each extraction stage;

[0190] The model loss is calculated based on the first segmentation loss, the second segmentation loss, and the knowledge distillation loss.

[0191] On the basis of the above-mentioned embodiment, in the infrared image segmentation model training device provided in the embodiment of the present invention, the loss calculation module is further specifically used for:

[0192] Determining segmentation annotation information corresponding to the high-frequency image block based on the binary expansion map;

[0193] The second segmentation loss is calculated based on the segmentation annotation information corresponding to the high-frequency image block and the second segmentation category.

[0194] On the basis of the above embodiment, the infrared image segmentation model training device provided in the embodiment of the present invention further includes a high-frequency information image extraction module for:

[0195] Determining the high-frequency components of the infrared image samples based on a two-dimensional discrete wavelet transform algorithm;

[0196] The high-frequency components are combined to obtain the high-frequency information image.

[0197] Specifically, the functions of each module in the infrared image segmentation model training device provided in the embodiment of the present invention correspond one-to-one to the operating procedures of each step in the above-mentioned infrared image segmentation model training method embodiment, and the effects achieved are also consistent. Please refer to the above-mentioned embodiment for details, and no further details will be given in the embodiment of the present invention.

[0198] like Figure 7 As shown, based on the above embodiment, an infrared image segmentation device is provided in an embodiment of the present invention, comprising:

[0199] An image acquisition module 71 is used to acquire the infrared image to be segmented;

[0200] An image segmentation module 72, used for inputting the infrared image to be segmented into a target infrared image segmentation model to obtain a segmentation category of the infrared image to be segmented output by the target infrared image segmentation model;

[0201] The target infrared image segmentation model is trained based on the infrared image segmentation model training method provided in the above embodiments.

[0202] Specifically, the functions of each module in the infrared image segmentation device provided in the embodiment of the present invention correspond one-to-one to the operation flow of each step in the above-mentioned infrared image segmentation method embodiment, and the effects achieved are also consistent. Please refer to the above-mentioned embodiment for details, which will not be repeated in the embodiment of the present invention.

[0203] Figure 8 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 8 As shown, the electronic device may include: a processor (Processor) 810, a communication interface (Communications Interface) 820, a memory (Memory) 830 and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 can call the logic instructions in the memory 830 to execute the infrared image segmentation model training method or the infrared image segmentation method provided in the above embodiments.

[0204] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0205] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the infrared image segmentation model training method or infrared image segmentation method provided in the above embodiments.

[0206] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the infrared image segmentation model training method or infrared image segmentation method provided in the above-mentioned embodiments.

[0207] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0208] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0209] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for training an infrared image segmentation model, characterized in that: include: Acquire an infrared image sample and segmentation annotation information corresponding to the infrared image sample, input the infrared image sample into an initial infrared image segmentation model, and obtain a first segmentation category; Based on the high-frequency information image in the infrared image sample, extract a high-frequency image block from the infrared image sample, and input the high-frequency image block into the auxiliary segmentation model to obtain a second segmentation category; the initial infrared image segmentation model and the auxiliary segmentation model have the same structure and share parameters; Calculate the model loss based on the segmentation annotation information, the first segmentation category, the second segmentation category, the first encoding information obtained by encoding the infrared image sample by the initial infrared image segmentation model, and the second encoding information obtained by encoding the high-frequency image block by the auxiliary segmentation model; Based on the model loss, the initial infrared image segmentation model and the auxiliary segmentation model are synchronously trained, and the trained initial infrared image segmentation model is used as the target infrared image segmentation model; The initial infrared image segmentation model is constructed based on SegFormer; The auxiliary segmentation model is built based on sparse SegFormer; The sparse SegFormer is obtained by replacing the convolution operation in the SegFormer with a sparse convolution operation and canceling the downsampling operation in the coding block of the SegFormer.

2. The infrared image segmentation model training method according to claim 1, characterized in that: The sparse convolution operation includes: Corresponding the input feature sequence to the coordinates in the image coordinate system to obtain an input feature map; the element values ​​of other positions in the input feature map except the position corresponding to the feature sequence are all 0; In the non-zero position of the input feature map, an input image block of a preset size is extracted, the convolution kernel of the convolution operation is transformed, and the input image block and the transpose of the transformed convolution kernel are matrix multiplied to obtain the output result of the sparse convolution operation.

3. The infrared image segmentation model training method according to claim 1 or 2, characterized in that: The extracting high-frequency image blocks from the infrared image samples based on the high-frequency information images in the infrared image samples comprises: Adjusting the high-frequency information image to a spare image having the same size as the infrared image sample, extracting spare image blocks of a preset size from the spare image, and calculating an L2 norm map corresponding to each spare image block; Binarization and expansion processing are performed on the L2 norm map to obtain a binary expansion map; A sample image block of the preset size is extracted from the infrared image sample, the binary expansion map is used as a mask, and a sample image block corresponding to a non-zero value is extracted as the high-frequency image block.

4. The infrared image segmentation model training method according to claim 3, characterized in that: The first encoded information and the second encoded information each include a plurality of extraction stages; The calculating of the model loss based on the segmentation annotation information, the first segmentation category, the second segmentation category, the first encoding information obtained by encoding the infrared image sample by the initial infrared image segmentation model, and the second encoding information obtained by encoding the high-frequency image block by the auxiliary segmentation model comprises: Calculating a first segmentation loss based on the segmentation label information and the first segmentation category; Calculating a second segmentation loss based on the segmentation label information and the second segmentation category; Based on the first encoding information and the second encoding information obtained in each extraction stage, calculating the knowledge distillation loss of each extraction stage; The model loss is calculated based on the first segmentation loss, the second segmentation loss, and the knowledge distillation loss.

5. The infrared image segmentation model training method according to claim 4, characterized in that: The calculating the second segmentation loss based on the segmentation label information and the second segmentation category includes: Determining segmentation annotation information corresponding to the high-frequency image block based on the binary expansion map; The second segmentation loss is calculated based on the segmentation annotation information corresponding to the high-frequency image block and the second segmentation category.

6. The infrared image segmentation model training method according to claim 1 or 2, characterized in that: The high-frequency information image is extracted based on the following steps: Determining the high-frequency components of the infrared image samples based on a two-dimensional discrete wavelet transform algorithm; The high-frequency components are combined to obtain the high-frequency information image.

7. An infrared image segmentation method, characterized in that: include: Acquire the infrared image to be segmented; Inputting the infrared image to be segmented into a target infrared image segmentation model to obtain a segmentation category of the infrared image to be segmented output by the target infrared image segmentation model; The target infrared image segmentation model is trained based on the infrared image segmentation model training method according to any one of claims 1 to 6.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the infrared image segmentation model training method as described in any one of claims 1 to 6, or the infrared image segmentation method as described in claim 7 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the infrared image segmentation model training method as described in any one of claims 1 to 6, or the infrared image segmentation method as described in claim 7 is implemented.

Images