Model training methods and apparatus, image segmentation methods and apparatus

Abstract

This application relates to a model training method and apparatus, and an image segmentation method and apparatus, applied in the field of image processing technology. The model training method includes: acquiring multiple sets of sample images and corresponding sample segmentation images for each set of sample images; extracting M-scale infrared and visible light basic features from the sample infrared and visible light images respectively, and fusing these features to obtain M-scale fused features; extracting boundary features from the fused features to obtain predicted boundary features; aggregating the fused features and predicted boundary features to obtain predicted segmentation features; obtaining a boundary loss value based on the actual and predicted boundary features extracted from the sample segmentation images; obtaining a segmentation loss value based on the actual and predicted segmentation features extracted from the sample segmentation images; and training an image segmentation model based on the weighted average of the boundary loss value and the segmentation loss value. This application can improve the accuracy of image segmentation.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a model training method and apparatus, and an image segmentation method and apparatus. Background Technology

[0002] Image segmentation is one of the four fundamental tasks of computer vision (classification, localization, detection, and segmentation), and a basic component of scene understanding, ensuring the reliable operation of autonomous vehicles in real-world urban environments. Due to the significant challenges in data acquisition and alignment, as well as the difficulty in designing effective multimodal fusion methods, current image segmentation methods primarily rely on visible light images. This severely limits the ability of autonomous vehicles to perceive their surroundings. Furthermore, it increases safety risks for road users in extreme weather conditions.

[0003] In related technologies, a scene segmentation model with a dual-path feature space adaptive module can be constructed to convert infrared thermal features into the visible light feature space and fuse the features of the two, thereby effectively utilizing visible light and infrared thermal image information for scene segmentation. However, this approach uses two encoders to process the infrared and visible light images separately, and a single decoder to simultaneously supervise segmentation and boundary checks. This approach piles different levels of information together, which not only leads to training difficulties but also causes the network to favor visible light image features during feature fusion under supervision, ultimately resulting in lower image segmentation accuracy. Summary of the Invention

[0004] To address the aforementioned technical problems, this application provides a model training method and apparatus, an image segmentation method and apparatus, an electronic device, a storage medium, and a computer program product.

[0005] According to a first aspect of this application, an image segmentation model training method is provided, comprising:

[0006] Acquire multiple sets of sample images and the corresponding sample segmentation images for each set of sample images. Each set of sample images includes: an infrared light image and a visible light image of a sample with the same scale in the same scene.

[0007] For each set of sample images, extract M infrared light basic features and M visible light basic features at different scales from the sample infrared light image and the sample visible light image, respectively, where M is a positive integer;

[0008] The infrared light basic features and the visible light basic features of the same scale are fused to obtain M fused features;

[0009] Boundary features are extracted from the fused features at the M scales to obtain predicted boundary features at the same scale as the original infrared and visible light images of the sample.

[0010] The fusion features of the M scales and the predicted boundary features are aggregated to obtain predicted segmentation features with the same original scale as the sample infrared light image and the sample visible light image;

[0011] The actual boundary features are extracted from the sample segmentation image, and the boundary loss value is obtained based on the actual boundary features, the predicted boundary features, and the preset boundary loss function;

[0012] The actual segmentation features are extracted from the sample segmentation image, and the segmentation loss value is obtained based on the actual segmentation features, the predicted segmentation features, and the preset boundary loss function;

[0013] The boundary loss value and the segmentation loss value are weighted and averaged to obtain the overall loss value, and an image segmentation model is trained based on the overall loss value.

[0014] Optionally, the step of extracting M-scale infrared fundamental features and M-scale visible fundamental features from the sample infrared image and the sample visible image, respectively, includes:

[0015] The first convolutional layer of the VGG model is used to extract features from the sample infrared light image and the sample visible light image respectively, to obtain the basic infrared light features and basic visible light features at the same scale as the original scale of the sample infrared light image and the sample visible light image;

[0016] The sample infrared image and the sample visible image are processed by a residual network to obtain M-1 infrared and visible basic features with successively decreasing scales. The largest scale among the M-1 scales is smaller than the original scale of the sample infrared image and the sample visible image.

[0017] Optionally, the infrared light basic feature and the visible light basic feature at each of the M scales are feature vectors of the same dimension, wherein the infrared light basic feature is an infrared light feature vector and the visible light basic feature is a visible light feature vector;

[0018] The process of fusing the infrared and visible light fundamental features of the same scale to obtain M fused features includes:

[0019] The common feature vector is obtained by multiplying the elements at the same position in the infrared light feature vector and the visible light feature vector.

[0020] The sum of the infrared light feature vector, the visible light feature vector, and the common feature vector is determined as the fusion feature of the scale.

[0021] Optionally, the step of extracting boundary features from the fused features at the M scales to obtain predicted boundary features at the same scale as the original infrared and visible light images of the sample includes:

[0022] The fused features of M scales are sorted from largest to smallest scale. The residual of the fused feature of the first scale is processed to obtain the boundary features.

[0023] For the fusion features of each remaining scale, the fusion features of the scale are upsampled to the original scale of the sample infrared light image and the sample visible light image, and feature extraction is performed to obtain the first channel feature; the boundary features corresponding to the output of the fusion features of the previous scale are subjected to residual processing to obtain the second channel feature; based on the first channel feature and the second channel feature, the boundary features corresponding to the fusion features of the scale are obtained.

[0024] The boundary features corresponding to the fusion features of M scales are concatenated to obtain concatenated boundary features, and the concatenated boundary features are then convolved to obtain predicted boundary features with the same scale as the original infrared light image and the visible light image of the sample.

[0025] Optionally, the step of aggregating the fused features of the M scales and the predicted boundary features to obtain predicted segmentation features with the same original scale as the sample infrared light image and the sample visible light image includes:

[0026] The predicted boundary features are downsampled to obtain M scaled downsampled boundary features;

[0027] The fusion features of M scales are sorted in ascending order of scale. The fusion feature of the first scale is concatenated with the downsampled boundary feature of the same scale as the fusion feature. Based on the obtained basic boundary concatenation feature, the integrated feature corresponding to the fusion feature of the first scale is determined.

[0028] For the remaining fusion features at each scale, the fusion features at that scale, the downsampled boundary features at the same scale as the fusion features at that scale, and the integrated features corresponding to the fusion features before that scale are concatenated. Based on the obtained basic boundary concatenation features, the integrated features corresponding to the fusion features at that scale are determined.

[0029] The integrated feature corresponding to the fusion feature of the last scale is determined as the prediction segmentation feature with the same original scale as the sample infrared light image and the sample visible light image.

[0030] Optionally, extracting actual boundary features from the sample segmentation image includes:

[0031] According to formula (1), the single-channel sample segmentation image is converted into a one-hot encoding with the number of channels equal to the number of segmentation categories:

[0032] OnehotMask[SegLabel[i,j],i,j]=1 (1)

[0033] Where OnehotMask is the one-hot encoding of shape (N, H, W), where N represents the number of segmentation categories, H and W represent the height and width of the sample segmentation image, respectively, and the initial value can be set to 0; SegLabel[i, j] represents the pixel category of the i-th row and j-th column in the sample segmentation image; SegLabel is the label data corresponding to the sample segmentation image of shape (H, W);

[0034] According to formula (2), the one-hot encoding is converted into the label data corresponding to the boundary image:

[0035] Distance n =R(OnehotMask[n,:,:])+R(1-OnehotMask[n,:,:])

[0036]

[0037]

[0038] Where R(·) represents the distance operator from a non-zero point in the segmented image to the nearest pixel with a value of 0, n represents the nth channel in OnehotMask encoding, and Distance n [i, j] represents the distance from the pixel at row i and column j in the segmented image of the nth category to the nearest pixel with a value of 0. If this distance is greater than the radius, then the pixel is not an edge point. n If [i, j] is assigned a value of 0, and this distance is less than or equal to radius, then this pixel is an edge point. n [i, j] are assigned the value 1.

[0039] Optionally, based on the actual segmentation features, the predicted segmentation features, and the preset boundary loss function, a segmentation loss value is obtained, including:

[0040] According to the following public notice (3):

[0041] L 分割全局 =-W n2 ΣB t (1-At ) μ log(A t (3)

[0042] Determine the global loss value L for segmentation 分割全局 ;

[0043] According to the following public notice (4):

[0044]

[0045] Determine the local loss value L for segmentation 分割局部 Among them, A t B represents the class prediction probability of the pixel value at position t in the sample infrared image and the sample visible light image. t W represents the actual probability of the class of the pixel value at position t in the segmented image, μ is the weighting exponent coefficient, and W n2 Represents the cross-entropy weighting coefficient for category n2; EdgeArea t This represents the pixel value at position t in the boundary image. A pixel value of 1 indicates that the position belongs to the region near the boundary, and a pixel value of 0 indicates that the position does not belong to the region near the boundary.

[0046] The segmentation loss value is determined based on the global segmentation loss value and the local segmentation loss value.

[0047] According to a second aspect of this application, an image segmentation method is provided, comprising:

[0048] Acquire the infrared light image and the visible light image to be processed in the same scene;

[0049] The infrared light image to be processed and the visible light image to be processed are input into a pre-trained image segmentation model to obtain segmented images corresponding to the infrared light image to be processed and the visible light image to be processed, wherein the image segmentation model is trained and generated based on the method described in the first aspect.

[0050] According to a third aspect of this application, an image segmentation model training apparatus is provided, comprising:

[0051] The sample data acquisition module is used to acquire multiple sets of sample images and the sample segmentation images corresponding to each set of sample images. Each set of sample images includes: an infrared light image and a visible light image of the sample with the same scale in the same scene.

[0052] The basic feature extraction module is used to extract M infrared light basic features and M visible light basic features at the scale from the sample infrared light image and the sample visible light image in each group of sample images, respectively, where M is a positive integer;

[0053] The feature fusion module is used to fuse the infrared light basic features and the visible light basic features of the same scale to obtain M fused features;

[0054] The boundary feature extraction module is used to extract boundary features from the fused features at the M scales to obtain predicted boundary features with the same scale as the original scale of the sample infrared light image and the sample visible light image.

[0055] The feature aggregation module is used to aggregate the fused features of the M scales and the predicted boundary features to obtain predicted segmentation features with the same original scale as the sample infrared light image and the sample visible light image;

[0056] The boundary loss value determination module is used to extract actual boundary features from the sample segmentation image and obtain the boundary loss value based on the actual boundary features, the predicted boundary features and the preset boundary loss function.

[0057] The segmentation loss value determination module is used to extract actual segmentation features from the sample segmentation image, and obtain the segmentation loss value based on the actual segmentation features, the predicted segmentation features, and the preset boundary loss function;

[0058] The model training module is used to perform a weighted average of the boundary loss value and the segmentation loss value to obtain the overall loss value, and to train and generate an image segmentation model based on the overall loss value.

[0059] Optionally, the basic feature extraction module is specifically used to extract features from the sample infrared light image and the sample visible light image in each group of sample images using the first convolutional layer of the VGG model, respectively, to obtain infrared light basic features and visible light basic features at the same first scale as the original scale of the sample infrared light image and the sample visible light image; and to process the sample infrared light image and the sample visible light image respectively through a residual network to obtain infrared light basic features and visible light basic features at M-1 scales that decrease sequentially, wherein the largest scale among the M-1 scales is smaller than the original scale of the sample infrared light image and the sample visible light image.

[0060] Optionally, the infrared light basic feature and the visible light basic feature at each of the M scales are feature vectors of the same dimension, wherein the infrared light basic feature is an infrared light feature vector and the visible light basic feature is a visible light feature vector;

[0061] The feature fusion module is specifically used to multiply the elements at the same position in the infrared light feature vector and the visible light feature vector to obtain a common feature vector; and to determine the sum of the infrared light feature vector, the visible light feature vector and the common feature vector as the fusion feature of the scale.

[0062] Optionally, the boundary feature extraction module is specifically used to sort the fusion features of M scales in descending order of scale, perform residual processing on the fusion features of the first scale to obtain boundary features; for the remaining fusion features of each scale, upsample the fusion features of the scale to the original scale of the sample infrared light image and the sample visible light image, and perform feature extraction to obtain the first channel feature; perform residual processing on the boundary features output corresponding to the fusion features of the previous scale to obtain the second channel feature, and obtain the boundary features corresponding to the fusion features of the scale based on the first channel feature and the second channel feature; concatenate the boundary features corresponding to the fusion features of the M scales to obtain the concatenated boundary features, and perform convolution processing on the concatenated boundary features to obtain the predicted boundary features with the same original scale as the sample infrared light image and the sample visible light image.

[0063] Optionally, the feature aggregation module is specifically used to downsample the predicted boundary features to obtain downsampled boundary features at M scales; sort the fusion features at the M scales in ascending order of scale; concatenate the fusion feature at the first scale with the downsampled boundary features at the same scale as the fusion feature; determine the integrated feature corresponding to the fusion feature at the first scale based on the obtained basic boundary concatenation feature; for each remaining scale of fusion features, concatenate the fusion feature at that scale, the downsampled boundary features at the same scale as the fusion feature at that scale, and the integrated feature corresponding to the fusion features before that scale; determine the integrated feature corresponding to the fusion feature at that scale based on the obtained basic boundary concatenation feature; and determine the integrated feature corresponding to the fusion feature at the last scale as the predicted segmentation feature at the same scale as the original scale of the sample infrared light image and the sample visible light image.

[0064] Optionally, the boundary loss value determination module is specifically used to extract actual boundary features from the sample segmentation image through the following steps:

[0065] According to formula (1), the single-channel sample segmentation image is converted into a one-hot encoding with the number of channels equal to the number of segmentation categories:

[0066] OnehotMask[SegLabel[i,j],i,j]=1 (1)

[0067] Where OnehotMask is the one-hot encoding of shape (N,H,W), where N represents the number of segmentation categories, H and W represent the height and width of the sample segmentation image, respectively, and the initial value can be set to 0; SegLabel[i,j] represents the pixel category of the i-th row and j-th column in the sample segmentation image; SegLabel is the label data corresponding to the sample segmentation image of shape (H,W);

[0068] According to formula (2), the one-hot encoding is converted into the label data corresponding to the boundary image:

[0069] Distance n =R(OnehotMask[n,:,:])+R(1-OnehotMask[n,:,:])

[0070]

[0071]

[0072] Where R(·) represents the distance operator from a non-zero point in the segmented image to the nearest pixel with a value of 0, n represents the nth channel in OnehotMask encoding, and Distance n [i,j] represents the distance from the pixel at row i and column j in the segmented image of the nth category to the nearest pixel with a value of 0. If this distance is greater than the radius, then the pixel is not an edge point. n If the distance [i,j] is set to 0, then the pixel is an edge point if the distance is less than or equal to the radius. n [i,j] is assigned the value 1.

[0073] Optionally, the segmentation loss determination module is used to extract actual segmentation features from the sample segmentation image, according to the following formula (3):

[0074] L 分割全局 =-W n2 ∑B t (1-A t ) μ log(A t (3)

[0075] Determine the global loss value L for segmentation 分割全局 ;

[0076] According to the following public notice (4):

[0077]

[0078] Determine the local loss value L for segmentation 分割局部 Among them, A tB represents the class prediction probability of the pixel value at position t in the sample infrared image and the sample visible light image. t W represents the actual probability of the class of the pixel value at position t in the segmented image, μ is the weighting exponent coefficient, and W n2 Represents the cross-entropy weighting coefficient for category n2; EdgeArea t The pixel value at position t in the boundary image represents the location. A pixel value of 1 indicates that the location belongs to the region near the boundary, and a pixel value of 0 indicates that the location does not belong to the region near the boundary. The segmentation loss value is determined based on the global segmentation loss value and the local segmentation loss value.

[0079] According to a fourth aspect of this application, an image segmentation apparatus is provided, comprising:

[0080] The image acquisition module is used to acquire infrared light images and visible light images to be processed in the same scene.

[0081] An image segmentation module is used to input the infrared light image to be processed and the visible light image to be processed into a pre-trained image segmentation model to obtain segmented images corresponding to the infrared light image to be processed and the visible light image to be processed, wherein the image segmentation model is trained and generated based on the method described in the first aspect.

[0082] According to a fifth aspect of this application, an electronic device is provided, comprising: a processor configured to execute a computer program stored in a memory, wherein the computer program, when executed by the processor, implements the method described in the first or second aspect.

[0083] According to a sixth aspect of this application, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method described in the first or second aspect.

[0084] According to a seventh aspect of this application, a computer program product is provided that, when the computer program product is run on a computer, causes the computer to perform the method described in the first or second aspect.

[0085] The technical solution provided in this application has the following advantages compared with the prior art:

[0086] In training the image segmentation model, multiple scales of infrared and visible light basic features are extracted, and features of the same scale are fused. This complements the dissimilar features of the two images and enhances the common features between them, preventing bias in the neural network during supervision and improving the accuracy of the trained image segmentation model. Furthermore, since boundaries and segmentation are not semantic information at the same level, two decoding networks are used to calculate predicted boundary features and predicted segmentation features separately, and boundary loss and segmentation loss values ​​are calculated separately. These two values ​​are then weighted and averaged to obtain the overall loss value, which is used to train the image segmentation model. By separately supervising boundaries and segmentation, processing information at different levels together avoids training difficulties, reduces the training cycle, and improves the accuracy of the trained image segmentation model. Thus, using the image segmentation model to segment images improves the accuracy of image segmentation. Attached Figure Description

[0087] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0088] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0089] Figure 1 This is a flowchart of an image segmentation model training method in an embodiment of this application;

[0090] Figure 2A This is a schematic diagram of residual attention based on infrared light fundamental features in the embodiments of this application;

[0091] Figure 2B This is a schematic diagram of residual attention based on visible light fundamental features in the embodiments of this application;

[0092] Figure 3A A schematic diagram of a dense block;

[0093] Figure 3B This is a schematic diagram of a residual block;

[0094] Figure 3C A schematic diagram illustrating the fusion and transmission of features between the same and opposite sexes;

[0095] Figure 4A This is a flowchart of a boundary feature extraction method in an embodiment of this application;

[0096] Figure 4B This is a schematic diagram of boundary feature extraction in an embodiment of this application;

[0097] Figure 5 This is a flowchart of a method for aggregating fused features and predicted boundary features in an embodiment of this application;

[0098] Figure 6 This is a schematic diagram of the structure of the improved Inception module in an embodiment of this application;

[0099] Figure 7 This is a schematic diagram of an image segmentation model training method in an embodiment of this application;

[0100] Figure 8 This is a flowchart of an image segmentation method in an embodiment of this application;

[0101] Figure 9 This is a schematic diagram of the structure of an image segmentation model training device in an embodiment of this application;

[0102] Figure 10 This is a schematic diagram of the structure of an image segmentation device in an embodiment of this application. Detailed Implementation

[0103] To better understand the above-mentioned objectives, features, and advantages of this application, the solution of this application will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0104] Many specific details are set forth in the following description in order to provide a full understanding of this application, but this application may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some embodiments of this application, and not all embodiments.

[0105] See Figure 1 , Figure 1 This is a flowchart of an image segmentation model training method in an embodiment of this application, which may include the following steps:

[0106] Step S102: Obtain multiple sets of sample images and the sample segmentation image corresponding to each set of sample images.

[0107] Each set of sample images includes: infrared and visible light images of the same scene at the same scale. That is, the content of the infrared and visible light images in each set is the same, but the image types are different. Each set of sample images serves as input, and the corresponding segmented images serve as label data to train and generate an image segmentation model. The segmented images corresponding to each set of sample images include segmentation results for different objects; for example, the image may contain multiple objects, each with different pixel values.

[0108] Step S104: For the sample infrared light image and sample visible light image in each group of sample images, extract M infrared light basic features and M visible light basic features at different scales from the sample infrared light image and sample visible light image, respectively, where M is a positive integer.

[0109] During model training, the processing procedure for each set of sample images is the same; therefore, we will use one set of sample images as an example for illustration. To better fuse the infrared and visible light images in each set of sample images, M features at different scales can be extracted from the infrared and visible light images respectively, allowing the fusion of these M features at the same scale. The value of M can be 1, or it can be 2, 3, 4, 5, 6, etc. To improve the fusion effect without making the computation process too complex, the value of M can be, for example, 6. This application does not limit the feature extraction method; for example, a trained neural network model can be used.

[0110] Optionally, the VGG model (e.g., VGG-16) can be integrated with ResNet to obtain the final feature extraction module. Specifically, the first convolutional layer of the VGG model is used to extract features from the sample infrared and visible light images, respectively, to obtain the first-scale infrared and visible light basic features at the same scale as the original infrared and visible light images. Then, a residual network is used to process the sample infrared and visible light images, respectively, to obtain M-1 scales of successively decreasing infrared and visible light basic features, where the largest scale among the M-1 scales is smaller than the original scale of the sample infrared and visible light images. Integrating the two independent feature extraction modules (VGG model and ResNet) can mitigate the resolution degradation.

[0111] Since ResNet processes three-channel images, while infrared images are single-channel images, the infrared image can be copied into a three-channel format. Assuming the value of M is 6, in ResNet, max pooling layers and four residual layers can be used to gradually reduce the resolution, outputting five different scale features.

[0112] Assuming the original scale of the sample infrared and visible light images is 1024×1024, with the first scale being 1024×1024 and the remaining five scales being 512×512, 256×256, 128×128, 64×64, and 32×32 respectively, then six scales of basic infrared features can be extracted from the sample infrared image, and six scales of basic visible light features can be extracted from the sample visible light image.

[0113] Step S106: The infrared light basic features and visible light basic features of the same scale are fused to obtain fused features of M scales.

[0114] In this embodiment, the infrared light basic features and visible light basic features of the same scale are fused to complement the heterogeneous features of the two images and enhance the common features between the two images, thus preventing the neural network from being biased during supervision.

[0115] In each of the M scales, the basic infrared and visible light features are feature vectors of the same dimension. The basic infrared feature is the infrared feature vector, and the basic visible light feature is the visible light feature vector. In some embodiments, the elements at the same position in the infrared and visible light feature vectors can be multiplied to obtain a common feature vector. The sum of the infrared feature vector, the visible light feature vector, and the common feature vector is determined as the fusion feature of that scale, expressed as formula (1):

[0116] F({X 可见光 X 红外光})=X 可见光 +X 红外光 +(X 可见光 ×X 红外光 (1)

[0117] Among them, X 红外光 X represents the infrared light feature vector. 可见光 X represents the visible light feature vector. 可见光 ×X 红外光 Represents common feature vectors.

[0118] F({X 可见光 X 红外光}) represents the fusion feature, F({X) 可见光 ×X 红外光}) can be considered as a combination of X 可见光 (X 可见光 +X 红外光 ) and X 红外光 (X 红外光 +X 可见光 The interactive residual attention mechanisms are fused with another feature type in a residual attention manner.

[0119] See Figure 2A , Figure 2A This is a schematic diagram of residual attention based on infrared light basic features in an embodiment of this application. Based on the infrared light basic features, the final output features are augmented with X. 红外光 ×X 可见光 These are common characteristics. See also Figure 2B , Figure 2B This is a schematic diagram of residual attention based on visible light basic features in an embodiment of this application. Based on visible light basic features, the final output features are augmented with X. 可见光 ×X 红外光 That is, common characteristics.

[0120] It can be seen that the fused features include basic infrared light features and basic visible light features, i.e. heterogeneous features, as well as common feature vectors, i.e. common features.

[0121] Optionally, during feature fusion, residual dense modules can also be used to transfer features of similar or dissimilar nature. Residual dense modules combine residual blocks and dense blocks; see [link to relevant documentation]. Figure 3A and Figure 3B , Figure 3A This is a schematic diagram of a dense block. Figure 3B This is a schematic diagram of a residual block. Combining the two yields a residual dense block. See also... Figure 3C , Figure 3C This diagram illustrates the fusion and transfer of features of the same and different types. Residual blocks prevent gradient vanishing in deep networks, while dense blocks improve the reusability of model features.

[0122] Step S108: Extract boundary features from the fused features at M scales to obtain predicted boundary features at the same scale as the original infrared and visible light images of the sample.

[0123] Boundaries belong to low-level semantic information, while segmentation contains both low-level and high-level semantic information; that is, boundaries and segmentation are not semantic information at the same level. Simply using a single network to simultaneously supervise multiple tasks at different levels may lead to training difficulties. Therefore, in this embodiment, two networks are used to supervise boundaries and segmentation separately to reduce training difficulty. Based on the characteristics of boundary information, a boundary extraction module can be specifically designed to better utilize the extracted image features.

[0124] After obtaining the fused features at M scales, boundary features can be extracted from the fused features to obtain the predicted boundary features. See also Figure 4A , Figure 4A This is a flowchart of a boundary feature extraction embodiment of the present application, which may include the following steps:

[0125] Step S402: Sort the fused features of M scales from largest to smallest scale, and perform residual processing on the fused features of the first scale to obtain the boundary features.

[0126] The first scale of the fused features contains rich details and can be adjusted using residual blocks to obtain boundary features.

[0127] Step S404: For the fusion features of each remaining scale, the fusion features of that scale are upsampled to the original scale of the sample infrared light image and the sample visible light image, and feature extraction is performed to obtain the first channel features.

[0128] For the fusion features at each remaining scale, after upsampling the fusion features at that scale, a small amount of detailed information can be extracted from the deep network through 1×1 kernel convolution.

[0129] Step S406: Perform residual processing on the boundary features output corresponding to the fusion features of the previous scale to obtain the second channel features.

[0130] In this embodiment, the boundary features output by each layer are used as input to the next layer, and residual blocks are used to enhance the feature representation.

[0131] Step S408: Based on the first channel features and the second channel features, obtain the boundary features corresponding to the fusion features at this scale.

[0132] In this embodiment, the first channel feature and the second channel feature can be concatenated to form a dual-channel feature in the channel dimension. Convolution processing of the dual-channel feature can output the corresponding single-channel feature, i.e., the boundary feature.

[0133] Optionally, the dual-channel features can also be input into a first spatial attention module to extract boundary features. See also Figure 4B , Figure 4B This is a schematic diagram of boundary feature extraction in an embodiment of this application. The area inside the dashed box represents the processing of the first spatial attention module, which can also be expressed as the following formula:

[0134] X out =Conv{[1+Sigmoid(Conv(R(Conv(N(X in )))))]·X in}

[0135] Among them, X in X represents the two-channel feature of the input. outThe boundary features of the output are represented by Conv, which represents a 1×1 kernel convolution, N represents batch normalization, R represents the ReLU activation function, and Sigmoid represents the Sigmoid function.

[0136] Step S410: The boundary features corresponding to the fusion features of M scales are spliced ​​together to obtain spliced ​​boundary features, and the spliced ​​boundary features are convolved to obtain predicted boundary features with the same scale as the original infrared light image and the visible light image of the sample.

[0137] Each of the M-scale fused features corresponds to a boundary feature, and the size of the boundary feature is the same as the original scale of the sample infrared light image and the sample visible light image. The boundary features corresponding to the M-scale fused features are concatenated and input into a 1x1 kernel convolution to obtain the final single-channel predicted boundary feature.

[0138] Step S110: Aggregate the fusion features and predicted boundary features of M scales to obtain predicted segmentation features with the same scale as the original scale of the sample infrared light image and the sample visible light image.

[0139] In this embodiment, the method for aggregating fused features and predicted boundary features can be found in [reference needed]. Figure 5 This may include the following steps:

[0140] Step S502: Downsample the predicted boundary features to obtain downsampled boundary features at M scales.

[0141] Since the scale of the predicted boundary features is the same as the original scale of the sample infrared and visible light images, while the fused features contain multiple scales, the predicted boundary features can be downsampled to obtain M downsampled boundary features at different scales. This allows for the aggregation of downsampled boundary features and fused features at the same scale.

[0142] In this embodiment, kernel convolutions with sizes of 1×1, 3×3, and 5×5 can be set for adaptive selection by the network. That is, different kernel convolutions can be used when aggregating downsampled boundary features and fused features at different scales.

[0143] Step S504: Sort the fusion features of M scales in ascending order of scale, concatenate the fusion feature of the first scale with the downsampled boundary feature of the same scale as the fusion feature, and determine the integrated feature corresponding to the fusion feature of the first scale based on the obtained basic boundary concatenation feature.

[0144] In this embodiment, an improved Inception module can be obtained by adding a 1×1 convolution before the 3×3 and 5×5 convolutions. See also Figure 6, Figure 6 This is a schematic diagram of the improved Inception module in this application embodiment. After processing the fused features through the improved Inception module, the downsampled boundary features with the same scale as the fused features are then concatenated. This can reduce the dimensionality of the features, reduce the amount of computation, and improve the nonlinear expression capability.

[0145] In this embodiment, the basic boundary splicing features can be processed by a second spatial attention module to obtain the corresponding integrated features. For example, it can be done using the following formula:

[0146]

[0147] The basic boundary splicing features are processed to obtain the corresponding integrated features.

[0148] Among them, X in X represents the basic boundary splicing feature of all input channels. out The output features are represented by Conv, where Conv represents a 1×1 kernel convolution, N represents batch normalization, Avg represents channel-based mean pooling, Max represents channel-based max pooling, MLP represents a fully connected layer, and exp represents a normalized exponential function. Let j represent the basic boundary splicing feature of the i-th channel, and j represent the total number of channels.

[0149] When aggregating the fused features and predicted boundary features of M scales through the second spatial attention module, information from the basic feature channels can be suppressed in the boundary region and information from the boundary feature channels can be emphasized. In the segmentation of the internal region, information from the boundary feature channels can be suppressed and information from the basic feature channels can be emphasized, thereby improving the accuracy of the finally determined predicted segmentation features.

[0150] Step S506: For the remaining fusion features at each scale, the fusion features at that scale, the downsampled boundary features at the same scale as the fusion features at that scale, and the integrated features corresponding to the fusion features before that scale are concatenated. Based on the obtained basic boundary concatenation features, the integrated features corresponding to the fusion features at that scale are determined.

[0151] In this embodiment of the application, the integrated features output by each layer are used as the input of each remaining layer. That is, when each layer aggregates the fused features and the downsampled boundary features, it takes the integrated features output by each layer before it, the fused features corresponding to this layer, and the downsampled boundary features as input, and outputs the corresponding integrated features.

[0152] Step S508: The integrated feature corresponding to the fusion feature of the last scale is determined as the prediction segmentation feature with the same original scale as the sample infrared light image and the sample visible light image.

[0153] It is understandable that the fusion feature corresponding to the last scale fusion feature is the predicted segmentation feature, and the scale of the predicted segmentation feature is the same as the original scale of the sample infrared light image and the sample visible light image.

[0154] Step S112: Extract the actual boundary features from the sample segmentation image, and obtain the boundary loss value based on the actual boundary features, predicted boundary features, and preset boundary loss function.

[0155] After obtaining the predicted boundary features and predicted segmentation features, the actual boundary features and actual segmentation features can be determined based on the label data (i.e., the sample segmentation image) to obtain the boundary loss value and segmentation loss value, respectively.

[0156] Optionally, the single-channel sample segmentation image is converted into a one-hot encoding with the number of channels equal to the number of segmentation categories according to formula (1):

[0157] OnehotMask[SegLabel[i,j],i,j]=1 (1)

[0158] Where OnehotMask is the one-hot encoding of shape (N,H,W), where N represents the number of segmentation categories, H and W represent the height and width of the sample segmentation image, respectively, and the initial value can be set to 0; SegLabel[i,j] represents the pixel category of the i-th row and j-th column in the sample segmentation image; SegLabel is the label data corresponding to the sample segmentation image of shape (H,W);

[0159] According to formula (2), the one-hot encoding is converted into the label data corresponding to the boundary image:

[0160] Distance n =R(OnehotMask[n,:,:])+R(1-OnehotMask[n,:,:])

[0161]

[0162]

[0163] Where R(·) represents the distance operator from a non-zero point in the segmented image to the nearest pixel with a value of 0, n represents the nth channel in OnehotMask encoding, and Distance n[i,j] represents the distance from the pixel at row i and column j in the segmented image of the nth category to the nearest pixel with a value of 0. If this distance is greater than the radius, then the pixel is not an edge point. n If the distance [i,j] is set to 0, then the pixel is an edge point if the distance is less than or equal to the radius. n [i,j] is assigned the value 1.

[0164] In this embodiment, since two decoding networks are used to supervise segmentation and boundary prediction respectively, an extreme imbalance will occur between the foreground and background in the boundary prediction during training. Therefore, the cross-entropy function can be adaptively weighted according to the number of pixels in each category and their training difficulty. The cross-entropy loss function is defined as follows:

[0165]

[0166] In the above, y∈{±1} represents the positive and negative labels, and a∈[0,1] is the network model's prediction probability for the class with label y=1.

[0167] Assumption a t This represents the predicted boundary features of position t in the sample infrared image and the sample visible light image, that is, whether position t is a boundary point.

[0168] The cross-entropy function is weighted based on the number of pixels belonging to each category. That is, for the current batch of training samples, the fewer pixels belonging to a particular category, the higher the weight of that category in calculating the cross-entropy loss, and vice versa. The weight coefficients can be defined as follows:

[0169]

[0170] Where n1 represents the two categories of boundary foreground and background, f n1 The frequency of a certain category n1 appearing in all category pixels can be obtained by calculating a histogram.

[0171] The cross-entropy function is weighted based on the training difficulty; that is, the lower the prediction reliability of a pixel at a certain position t in the image, the larger the weight coefficient is added to that position when calculating the loss. The boundary loss value can finally be calculated using the following formula:

[0172] L 边界 =-ω m1 ∑B t (1-α t ) μ log(a t )

[0173] Among them, L边界 This represents the boundary loss value, where μ is the weighting exponent coefficient. The lower the pixel prediction reliability, the more μ is used to amplify the proportion of that pixel category loss in the loss function; for example, it can be set to 2. B t This represents the actual boundary features of position t in the sample infrared image and the sample visible light image, that is, whether position t is a boundary point.

[0174] Step S114: Extract the actual segmentation features from the sample segmentation image, and obtain the segmentation loss value based on the actual segmentation features, the predicted segmentation features, and the preset boundary loss function.

[0175] To strengthen the tight coupling between fusion features (i.e., basic features) and boundary features, embodiments of this application disclose a local segmentation cross-entropy loss, that is, the cross-entropy loss is calculated only in the region near the boundary, which can be based on the following formula (3):

[0176] L 分割全局 =-W n2 ∑B t (1-A t ) μ log(A t (3)

[0177] Determine the global loss value L for segmentation 分割全局 ;

[0178] According to the following public notice (4):

[0179]

[0180] Determine the local loss value L for segmentation 分割局部 Among them, A t B represents the class prediction probability of the pixel value at position t in the sample infrared image and the sample visible light image. t W represents the actual probability of the class of the pixel value at position t in the segmented image, μ is the weighting exponent coefficient, and W n2 Represents the cross-entropy weighting coefficient for category n2; EdgeArea t This represents the pixel value at position t in the boundary image. A pixel value of 1 indicates that the position belongs to the region near the boundary, while a pixel value of 0 indicates that the position does not belong to the region near the boundary. EdgeArea can be the actual boundary region, and the region near the boundary can be obtained by widening the boundary range based on the actual boundary region by adjusting the radius.

[0181] Next, the segmentation loss value is determined based on the global segmentation loss value and the local segmentation loss value. For example, the sum of the global segmentation loss value and the local segmentation loss value can be directly used as the segmentation loss value, or the weighted average of the global segmentation loss value and the local segmentation loss value can be used as the segmentation loss value.

[0182] Step S116: The boundary loss value and the segmentation loss value are weighted and averaged to obtain the overall loss value, and the image segmentation model is trained based on the overall loss value.

[0183] Assuming the segmentation loss is the sum of the global segmentation loss and the local segmentation loss, it can be expressed by the following formula:

[0184] L 总 =λ1L 边界 +λ2(L 分割全局 +L 分割局部 )

[0185] Determine the overall loss value L 总 λ1 and λ2 are hyperparameters that control the boundary and balance the segmentation loss.

[0186] The image segmentation model training method in this application extracts infrared and visible light basic features at multiple scales and fuses infrared and visible light basic features at the same scale. This complements the dissimilar features of the two images and enhances the common features between them, preventing bias in the neural network during supervision and improving the accuracy of the trained image segmentation model. Furthermore, since boundaries and segmentation are not semantic information at the same level, two decoding networks are used to calculate predicted boundary features and predicted segmentation features separately, and boundary loss and segmentation loss values ​​are calculated separately. These are then weighted and averaged to obtain the overall loss value, which is used to train the image segmentation model. By separately supervising boundaries and segmentation, processing information at different levels together is avoided. This prevents training difficulties, reduces the training cycle, and improves the accuracy of the trained image segmentation model. Thus, image segmentation using the image segmentation model improves the accuracy of image segmentation.

[0187] This application also provides an image segmentation method, see [link to relevant documentation] Figure 8 This may include the following steps:

[0188] Step S802: Obtain the infrared light image and the visible light image to be processed in the same scene.

[0189] Step S804: Input the infrared light image to be processed and the visible light image to be processed into a pre-trained image segmentation model to obtain segmented images corresponding to the infrared light image to be processed and the visible light image to be processed. The image segmentation model is based on... Figure 1 The model training method in the embodiment is used for training and generation.

[0190] because Figure 1The model training method of the embodiment can generate a more accurate image segmentation model. Therefore, in the model application stage, image segmentation can be performed using this image segmentation model to improve the accuracy of image segmentation.

[0191] Corresponding to the above method embodiments, this application also provides an image segmentation model training device, see [link to relevant documentation]. Figure 9 The image segmentation model training device 900 includes:

[0192] The sample data acquisition module 902 is used to acquire multiple sets of sample images and sample segmentation images corresponding to each set of sample images. Each set of sample images includes: sample infrared light images and sample visible light images with the same scale in the same scene.

[0193] The basic feature extraction module 904 is used to extract M infrared light basic features and M visible light basic features at different scales from the sample infrared light image and sample visible light image in each group of sample images, respectively, where M is a positive integer;

[0194] The feature fusion module 906 is used to fuse infrared light basic features and visible light basic features of the same scale to obtain M fused features.

[0195] The boundary feature extraction module 908 is used to extract boundary features from the fused features at M scales to obtain predicted boundary features at the same scale as the original infrared light image and the visible light image of the sample.

[0196] The feature aggregation module 910 is used to aggregate the fused features and predicted boundary features of M scales to obtain the predicted segmentation features with the same scale as the original infrared light image and the visible light image of the sample.

[0197] The boundary loss value determination module 912 is used to extract actual boundary features from the sample segmentation image and obtain the boundary loss value based on the actual boundary features, predicted boundary features and preset boundary loss function;

[0198] The segmentation loss value determination module 914 is used to extract the actual segmentation features from the sample segmentation image and obtain the segmentation loss value based on the actual segmentation features, the predicted segmentation features and the preset boundary loss function.

[0199] The model training module 916 is used to perform a weighted average of the boundary loss value and the segmentation loss value to obtain the overall loss value, and to train and generate an image segmentation model based on the overall loss value.

[0200] Optionally, the basic feature extraction module 904 is specifically used to extract features from the sample infrared light image and sample visible light image in each group of sample images using the first convolutional layer of the VGG model, respectively, to obtain the basic infrared light features and basic visible light features at the same scale as the original scale of the sample infrared light image and sample visible light image; and to process the sample infrared light image and sample visible light image respectively through a residual network to obtain the basic infrared light features and basic visible light features at M-1 scales that decrease sequentially, wherein the largest scale among the M-1 scales is smaller than the original scale of the sample infrared light image and sample visible light image.

[0201] Optionally, the infrared light basic features and visible light basic features of each of the M scales are feature vectors of the same dimension, with the infrared light basic features being infrared light feature vectors and the visible light basic features being visible light feature vectors.

[0202] The feature fusion module 906 is specifically used to multiply the elements at the same position in the infrared light feature vector and the visible light feature vector to obtain the common feature vector; and to determine the sum of the infrared light feature vector, the visible light feature vector and the common feature vector as the scale fusion feature.

[0203] Optionally, the boundary feature extraction module 908 is specifically used to sort the fusion features of M scales in descending order of scale, perform residual processing on the fusion features of the first scale to obtain boundary features; for the remaining fusion features of each scale, upsample the fusion features of the scale to the original scale of the sample infrared light image and the sample visible light image, and perform feature extraction to obtain the first channel feature; perform residual processing on the boundary features output corresponding to the fusion features of the previous scale to obtain the second channel feature, and obtain the boundary features corresponding to the fusion features of that scale based on the first channel feature and the second channel feature; concatenate the boundary features corresponding to the fusion features of the M scales to obtain the concatenated boundary features, and perform convolution processing on the concatenated boundary features to obtain the predicted boundary features with the same original scale as the sample infrared light image and the sample visible light image.

[0204] Optionally, the feature aggregation module 910 is specifically used to downsample the predicted boundary features to obtain downsampled boundary features at M scales; sort the fusion features at the M scales in ascending order of scale; concatenate the fusion feature at the first scale with the downsampled boundary features at the same scale as the fusion feature; determine the integrated feature corresponding to the fusion feature at the first scale based on the obtained basic boundary concatenation feature; for each remaining scale of fusion feature, concatenate the fusion feature at that scale, the downsampled boundary features at the same scale as the fusion feature at that scale, and the integrated feature corresponding to the fusion features before that scale; determine the integrated feature corresponding to the fusion feature at that scale based on the obtained basic boundary concatenation feature; and determine the integrated feature corresponding to the fusion feature at the last scale as the predicted segmentation feature at the same scale as the original scale of the sample infrared light image and the sample visible light image.

[0205] Optionally, the boundary loss value determination module 912 is specifically used to extract the actual boundary features from the sample segmentation image through the following steps:

[0206] According to formula (1), the single-channel sample segmentation image is converted into a one-hot encoding with the number of channels equal to the number of segmentation categories:

[0207] OnehotMask[SegLabel[i,j],i,j]=1 (1)

[0208] Where OnehotMask is the one-hot encoding of shape (N,H,W), where N represents the number of segmentation categories, H and W represent the height and width of the sample segmentation image, respectively, and the initial value can be set to 0; SegLabel[i,j] represents the pixel category of the i-th row and j-th column in the sample segmentation image; SegLabel is the label data corresponding to the sample segmentation image of shape (H,W);

[0209] According to formula (2), the one-hot encoding is converted into the label data corresponding to the boundary image:

[0210] Distance n =R(OnehotMask[n,:,:])+R(1-OnehotMask[n,:,:])

[0211]

[0212]

[0213] Where R(·) represents the distance operator from a non-zero point in the segmented image to the nearest pixel with a value of 0, n represents the nth channel in OnehotMask encoding, and Distance n[i,j] represents the distance from the pixel at row i and column j in the segmented image of the nth category to the nearest pixel with a value of 0. If this distance is greater than the radius, then the pixel is not an edge point. n If the distance [i,j] is set to 0, then the pixel is an edge point if the distance is less than or equal to the radius. n [i,j] is assigned the value 1.

[0214] Optionally, the segmentation loss determination module 914 is used to extract the actual segmentation features from the sample segmentation image, according to the following formula (3):

[0215] L 分割全局 =-W n2 ∑B t (1-A t ) μ log(A t (3)

[0216] Determine the global loss value L for segmentation 分割全局 ;

[0217] According to the following public notice (4):

[0218]

[0219] Determine the local loss value L for segmentation 分割局部 Among them, A t B represents the class prediction probability of the pixel value at position t in the sample infrared image and the sample visible light image. t W represents the actual probability of the class of the pixel value at position t in the segmented image, μ is the weighting exponent coefficient, and W n2 Represents the cross-entropy weighting coefficient for category n2; EdgeArea t The pixel value at position t in the boundary image represents the region near the boundary. A pixel value of 1 indicates that the position belongs to the region near the boundary, and a pixel value of 0 indicates that the position does not belong to the region near the boundary. The segmentation loss value is determined based on the global segmentation loss value and the local segmentation loss value.

[0220] This application also provides an image segmentation apparatus, see [link to relevant documentation] Figure 10 The image segmentation device 1000 includes:

[0221] The image acquisition module 1002 is used to acquire an infrared light image to be processed and a visible light image to be processed in the same scene;

[0222] Image segmentation module 1004 is used to input the infrared light image to be processed and the visible light image to be processed into a pre-trained image segmentation model to obtain segmented images corresponding to the infrared light image to be processed and the visible light image to be processed. The image segmentation model is based on... Figure 1 The model training method shown in the embodiment is used for training and generation.

[0223] The specific details of each module or unit in the above-mentioned device have been described in detail in the corresponding methods, so they will not be repeated here.

[0224] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0225] This application also provides an electronic device, including: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute the image segmentation method described in this example embodiment.

[0226] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the above-described image segmentation method.

[0227] It should be noted that a computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on a computer-readable storage medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, radio frequency, etc., or any suitable combination thereof.

[0228] This application also provides a computer program product that, when run on a computer, causes the computer to execute the above-described image segmentation method.

[0229] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0230] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for training an image segmentation model, characterized in that, include: Acquire multiple sets of sample images and the corresponding sample segmentation images for each set of sample images. Each set of sample images includes: an infrared light image and a visible light image of a sample with the same scale in the same scene. For each set of sample images, extract M infrared light basic features and M visible light basic features at different scales from the sample infrared light image and the sample visible light image, respectively, where M is a positive integer; The infrared light basic features and the visible light basic features of the same scale are fused to obtain M fused features; Boundary features are extracted from the fused features at the M scales to obtain predicted boundary features at the same scale as the original infrared and visible light images of the sample. The fusion features of the M scales and the predicted boundary features are aggregated to obtain predicted segmentation features with the same original scale as the sample infrared light image and the sample visible light image; The actual boundary features are extracted from the sample segmentation image, and the boundary loss value is obtained based on the actual boundary features, the predicted boundary features, and the preset boundary loss function; The actual segmentation features are extracted from the sample segmentation image, and the segmentation loss value is obtained based on the actual segmentation features, the predicted segmentation features, and the preset boundary loss function; The boundary loss value and the segmentation loss value are weighted and averaged to obtain the overall loss value, and an image segmentation model is trained based on the overall loss value.

2. The method according to claim 1, characterized in that, The extraction of M-scale infrared fundamental features and M-scale visible fundamental features from the sample infrared light image and the sample visible light image, respectively, includes: The first convolutional layer of the VGG model is used to extract features from the sample infrared light image and the sample visible light image respectively, to obtain the basic infrared light features and basic visible light features at the same scale as the original scale of the sample infrared light image and the sample visible light image; The sample infrared image and the sample visible image are processed by a residual network to obtain M-1 infrared and visible basic features with successively decreasing scales. The largest scale among the M-1 scales is smaller than the original scale of the sample infrared image and the sample visible image.

3. The method according to claim 1, characterized in that, The infrared light fundamental feature and the visible light fundamental feature at each of the M scales are feature vectors of the same dimension, wherein the infrared light fundamental feature is an infrared light feature vector and the visible light fundamental feature is a visible light feature vector; The process of fusing the infrared and visible light fundamental features of the same scale to obtain M fused features includes: The common feature vector is obtained by multiplying the elements at the same position in the infrared light feature vector and the visible light feature vector. The sum of the infrared light feature vector, the visible light feature vector, and the common feature vector is determined as the fusion feature of the scale.

4. The method according to claim 1, characterized in that, The step of extracting boundary features from the fused features at the M scales to obtain predicted boundary features at the same scale as the original infrared and visible light images of the samples includes: The fused features of M scales are sorted from largest to smallest scale. The residual of the fused feature of the first scale is processed to obtain the boundary features. For the fusion features of each remaining scale, the fusion features of the scale are upsampled to the original scale of the sample infrared light image and the sample visible light image, and feature extraction is performed to obtain the first channel feature; the boundary features corresponding to the output of the fusion features of the previous scale are subjected to residual processing to obtain the second channel feature; based on the first channel feature and the second channel feature, the boundary features corresponding to the fusion features of the scale are obtained. The boundary features corresponding to the fusion features of M scales are concatenated to obtain concatenated boundary features, and the concatenated boundary features are then convolved to obtain predicted boundary features with the same scale as the original infrared light image and the visible light image of the sample.

5. The method according to claim 1, characterized in that, The step of aggregating the fused features of the M scales and the predicted boundary features to obtain predicted segmentation features with the same original scale as the sample infrared light image and the sample visible light image includes: The predicted boundary features are downsampled to obtain M scaled downsampled boundary features; The fusion features of M scales are sorted in ascending order of scale. The fusion feature of the first scale is concatenated with the downsampled boundary feature of the same scale as the fusion feature. Based on the obtained basic boundary concatenation feature, the integrated feature corresponding to the fusion feature of the first scale is determined. For the remaining fusion features at each scale, the fusion features at that scale, the downsampled boundary features at the same scale as the fusion features at that scale, and the integrated features corresponding to the fusion features before that scale are concatenated. Based on the obtained basic boundary concatenation features, the integrated features corresponding to the fusion features at that scale are determined. The integrated feature corresponding to the fusion feature of the last scale is determined as the prediction segmentation feature with the same original scale as the sample infrared light image and the sample visible light image.

6. The method according to claim 1, characterized in that, Extracting actual boundary features from the sample segmentation image includes: According to formula (1), the single-channel sample segmentation image is converted into a one-hot encoding with the number of channels equal to the number of segmentation categories: OnehotMask[SegLabel[i,j],i,j]=1 (1) Where OnehotMask is the one-hot encoding of shape (N,H,W), where N represents the number of segmentation categories, H and W represent the height and width of the sample segmentation image, respectively, and the initial value can be set to 0; SegLabel[i,j] represents the pixel category of the i-th row and j-th column in the sample segmentation image; SegLabel is the label data corresponding to the sample segmentation image of shape (H,W); According to formula (2), the one-hot encoding is converted into the label data corresponding to the boundary image: Distance n =R(OnehotMask[n,:,:])+R(1-OmehotMask[n,:,:]) Where R(·) represents the distance operator from a non-zero point in the segmented image to the nearest pixel with a value of 0, n represents the nth channel in OnehotMask encoding, and Distance n [i,j] represents the distance from the pixel at row i and column j in the segmented image of the nth category to the nearest pixel with a value of 0. If this distance is greater than the radius, then the pixel is not an edge point. n If the distance [i,j] is set to 0, then the pixel is an edge point if the distance is less than or equal to the radius. n [i,j] is assigned the value 1.

7. The method according to claim 1, characterized in that, Based on the actual segmentation features, the predicted segmentation features, and the preset boundary loss function, a segmentation loss value is obtained, including: According to the following public notice (3): L 分割全局 =-W n2 ΣB t (1-A t ) μ log(A t ) (3) Determine the global loss value L for segmentation 分割全局 ; According to the following public notice (4): Determine the local loss value L for segmentation 分割局部 Among them, A t B represents the class prediction probability of the pixel value at position t in the sample infrared image and the sample visible light image. t W represents the actual probability of the class of the pixel value at position t in the segmented image, μ is the weighting exponent coefficient, and W n2 Represents the cross-entropy weighting coefficient for category n2; EdgeArea t This represents the pixel value at position t in the boundary image. A pixel value of 1 indicates that the position belongs to the region near the boundary, and a pixel value of 0 indicates that the position does not belong to the region near the boundary. The segmentation loss value is determined based on the global segmentation loss value and the local segmentation loss value.

8. An image segmentation method, characterized in that, include: Acquire the infrared light image and the visible light image to be processed in the same scene; The infrared light image to be processed and the visible light image to be processed are input into a pre-trained image segmentation model to obtain segmented images corresponding to the infrared light image to be processed and the visible light image to be processed, wherein the image segmentation model is trained and generated based on the method described in any one of claims 1 to 7.

9. An image segmentation model training device, characterized in that, include: The sample data acquisition module is used to acquire multiple sets of sample images and the sample segmentation images corresponding to each set of sample images. Each set of sample images includes: an infrared light image and a visible light image of the sample with the same scale in the same scene. The basic feature extraction module is used to extract M infrared light basic features and M visible light basic features at the scale from the sample infrared light image and the sample visible light image in each group of sample images, respectively, where M is a positive integer; The feature fusion module is used to fuse the infrared light basic features and the visible light basic features of the same scale to obtain M fused features; The boundary feature extraction module is used to extract boundary features from the fused features at the M scales to obtain predicted boundary features with the same scale as the original scale of the sample infrared light image and the sample visible light image. The feature aggregation module is used to aggregate the fused features of the M scales and the predicted boundary features to obtain predicted segmentation features with the same original scale as the sample infrared light image and the sample visible light image; The boundary loss value determination module is used to extract actual boundary features from the sample segmentation image and obtain the boundary loss value based on the actual boundary features, the predicted boundary features and the preset boundary loss function. The segmentation loss value determination module is used to extract actual segmentation features from the sample segmentation image, and obtain the segmentation loss value based on the actual segmentation features, the predicted segmentation features, and the preset boundary loss function; The model training module is used to perform a weighted average of the boundary loss value and the segmentation loss value to obtain the overall loss value, and to train and generate an image segmentation model based on the overall loss value.

10. An image segmentation apparatus, characterized in that, include: The image acquisition module is used to acquire infrared light images and visible light images to be processed in the same scene. An image segmentation module is used to input the infrared light image to be processed and the visible light image to be processed into a pre-trained image segmentation model to obtain segmented images corresponding to the infrared light image to be processed and the visible light image to be processed, wherein the image segmentation model is trained and generated based on the method described in any one of claims 1 to 7.

Images