A training method for a semantic segmentation model and a semantic segmentation method

Abstract

This specification relates to the field of image processing technology, and relates to a training method for a semantic segmentation model and a semantic segmentation method. The training method includes obtaining training set data, the training set data includes RGB images and thermal value images; performing complementary masking on the RGB images and thermal value images; inputting the masked RGB images and thermal value images into a first semantic segmentation model, respectively, and outputting a semantic segmentation prediction map; obtaining the semantic segmentation prediction map of the unmasked RGB image and thermal value image through a second semantic segmentation model, and updating the parameters of the first semantic segmentation model with the unmasked semantic segmentation prediction map as the target until the number of training iterations is reached. The semantic segmentation method includes obtaining the RGB image and thermal value image of the current environment and inputting them into a semantic segmentation model, and outputting a semantic segmentation map. The semantic segmentation model is trained based on the above method. The semantic segmentation accuracy and stability of the model in the embodiment of this specification are high, not easily affected by external environmental interference, and can adapt to more complex application scenarios.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of image processing technology, and in particular to a training method for a semantic segmentation model and a semantic segmentation method. Background Art

[0002] During the production and operation of elevators, the images captured by elevator monitoring cameras are easily blurred due to environmental factors such as light changes, reduced visibility, and lens contamination, requiring further processing of the captured images. The different shooting angles and sizes of the objects captured by the monitoring cameras can also easily affect the semantic segmentation, target detection, and data enhancement of the captured images, leading to dependence on single-angle monitoring cameras. Therefore, it is necessary to improve the accuracy and stability of image processing processes such as semantic segmentation and target detection to meet the needs of elevator monitoring in complex environments. The semantic segmentation methods in the existing technology overly rely on the information captured by the RGB lens and ignore the thermal map information. Moreover, when the RGB lens is blocked by mosquitoes or stains, it will lead to a significant decrease in semantic segmentation accuracy and a deterioration in scene understanding capabilities. Therefore, it is necessary to perform masking operations on the training data of the semantic segmentation model so that the semantic segmentation model can predict the missing pixel values ​​of the image through learning. Summary of the Invention

[0003] The embodiments of this specification provide a semantic segmentation model training method and a semantic segmentation method, which aim to solve one or more of the above-mentioned problems and other potential problems.

[0004] To achieve the above objectives, the following technical solutions are provided:

[0005] According to a first aspect of this specification, a method for training a semantic segmentation model is provided, the method comprising:

[0006] Acquire training set data, wherein the training set data includes an RGB image and a corresponding thermal value image;

[0007] Masking the RGB image and the corresponding thermal value image; wherein the area with the lowest similarity between adjacent image blocks in the RGB image and the thermal value image is not masked, and the masked areas of the RGB image and the thermal value image are complementary;

[0008] Inputting the complementary masked RGB image and the thermal value image into a first semantic segmentation model and outputting a semantic segmentation prediction map; wherein the first semantic segmentation model includes a feature extraction module, a global context fusion module and a feature decoding module, the feature extraction module extracts feature maps of the RGB image and the thermal value image respectively, the feature fusion module fuses the feature maps of the RGB image and the thermal value image, the feature decoding module performs semantic segmentation on the RGB image, the thermal value image and the fused feature map, and outputs the semantic segmentation prediction map based on the semantic segmentation result;

[0009] Inputting the unmasked RGB image and the corresponding thermal value image into a second semantic segmentation model, and outputting an unmasked semantic segmentation prediction map;

[0010] The semantic segmentation prediction map and the unmasked semantic segmentation prediction map are input into a knowledge distillation module. The knowledge distillation module is used to guide the first semantic segmentation model to update model parameters with the unmasked semantic segmentation prediction map as a target until the first semantic segmentation model reaches the number of training iterations and the training of the first semantic segmentation model is completed.

[0011] The training method of the semantic segmentation model of the embodiment of this specification combines RGB images and thermal value images for training, so that the trained semantic segmentation model does not rely too much on the image information of a single modality, can adapt to more application scenarios with more interference factors, and realize an accurate and efficient semantic segmentation process; and performs complementary masking on the RGB image and the thermal value image to extract the complementary semantic information and geometric information in the image scene, so that the semantic model can make full use of information from different images and different image areas, reduce the influence of single modality dependence and image occlusion on the model prediction results, and have better semantic segmentation effect and stability; finally, the unmasked semantic segmentation prediction map is used as the learning target to update the model parameters, thereby further improving the prediction performance of the semantic segmentation model.

[0012] In some embodiments, masking the RGB image and the corresponding thermal value image includes:

[0013] Segmenting the RGB image and the corresponding thermal value image into a plurality of image blocks;

[0014] Calculating the similarity between each pair of image blocks of the thermal value image, recording the position of the image block with the lowest similarity, and the thermal value image and the corresponding RGB image at this position are not masked;

[0015] The unrecorded image blocks are randomly masked. After the random masking, the mask areas of the remaining image blocks of the RGB image and the corresponding thermal value image are complementary except for the recorded image block positions.

[0016] In some embodiments, the knowledge distillation module uses the unmasked semantic segmentation prediction map as a target to guide the first semantic segmentation model to update model parameters, including: calculating a distillation loss function of the semantic segmentation prediction map and the unmasked semantic segmentation prediction map, and updating the first semantic segmentation model parameters based on the distillation loss function. The distillation loss function L is formulated as follows:

[0017] L=L MCM +L MWS +L MCS

[0018] L MCM represents the loss function between the category prediction embedding of the masked semantic segmentation map and the category prediction embedding of the unmasked semantic segmentation map, L MWS represents the self-supervised learning loss function, L MCS Represents the sum of the loss functions between the category prediction embedding of the masked semantic segmentation map of the RGB image, the category prediction embedding of the masked semantic segmentation map of the thermal value image, and the category prediction embedding of the masked semantic segmentation map.

[0019] In some embodiments, each loss function in the distillation loss function is a loss function in the form of JS divergence.

[0020] In some embodiments, updating the first semantic segmentation model parameters includes:

[0021] The mask areas of the RGB image and the thermal value image are replaced with a learnable vector, and the learnable vector is initialized; wherein the formula of the learnable vector is as follows:

[0022]

[0023]

[0024] and Represent the masked RGB image and thermal image respectively, M rgb and M thr are the mask matrices of RGB image and thermal image respectively, T rgb and T thr Learnable vectors representing RGB images and thermal images respectively;

[0025] The learnable vector is updated based on the distillation loss function.

[0026] In some embodiments, the model structures and initialization parameters of the first semantic segmentation model and the second semantic segmentation model are consistent.

[0027] In some embodiments, the feature fusion module performs global context feature fusion, local context fusion, and consistency fusion on the feature maps of the RGB image and the thermal value image.

[0028] In some embodiments, the feature fusion module performs global context feature fusion on the feature maps of the RGB image and the thermal value image, including:

[0029] Updating the RGB image: calculating the similarity between the RGB image feature map and the thermal value image feature map, converting the similarity into a probability distribution to obtain an RGB image normalization weight, and using the RGB image normalization weight as a coefficient of the RGB image feature map to update the RGB image;

[0030] Updating the thermal value image: calculating the similarity between the thermal value image feature map and the RGB image feature map, converting the similarity into a probability distribution to obtain a thermal value map normalization weight, and using the thermal value map normalization weight as a coefficient of the thermal value image feature map to update the thermal value image;

[0031] The updated RGB image and the thermal image are fused, and the formula is as follows:

[0032]

[0033] in, Represents the feature map after feature fusion, represents the updated RGB image feature map, Represents the updated thermal image feature map.

[0034] In some embodiments, the feature extraction module uses a swin transformer to extract feature maps of the RGB image and the thermal value image respectively.

[0035] According to the second aspect of this specification, a semantic segmentation method is provided, which includes: obtaining an RGB image and a thermal value image of the current environment through an RGB-T camera, inputting the RGB image and the thermal value image into a first semantic segmentation model, and outputting a semantic segmentation map; wherein, the first semantic segmentation model is trained based on the above-mentioned semantic segmentation model training method. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The above and other objects, features and advantages of the embodiments of the present specification will become readily understood by reading the following detailed description with reference to the accompanying drawings, in which several embodiments of the present specification are shown by way of example and not limitation.

[0037] Figure 1A flowchart of a method for training a semantic segmentation model according to an embodiment of this specification is shown. DETAILED DESCRIPTION

[0038] The preferred embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although preferred embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to make the present invention more thorough and complete and to fully convey the scope of the present invention to those skilled in the art.

[0039] The term "including" and its variations used in this document indicate open inclusion, that is, "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "based at least in part on". The terms "an example embodiment" and "an embodiment" mean "at least one example embodiment". The term "another embodiment" means "at least one other embodiment". Terms such as "upper", "lower", "front", and "rear" indicating placement or positional relationships are based on the orientation or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the principles of this specification, and do not indicate or imply that the referred elements must have a specific orientation, be constructed or operate in a specific orientation, and therefore should not be understood as limiting this specification.

[0040] The following is a flowchart of a method for training a semantic segmentation model according to an embodiment of the present disclosure, described in detail with reference to the accompanying drawings. The method includes:

[0041] S1. Obtain training set data, wherein the training set data includes an RGB image and a corresponding thermal value image;

[0042] S3, masking the RGB image and the corresponding thermal value image;

[0043] S5, inputting the complementary masked RGB image and the thermal value image into a first semantic segmentation model, and outputting a semantic segmentation prediction map;

[0044] S7, inputting the unmasked RGB image and the corresponding thermal value image into a second semantic segmentation model, and outputting an unmasked semantic segmentation prediction map;

[0045] S9. Input the semantic segmentation prediction map and the unmasked semantic segmentation prediction map into a knowledge distillation module. The knowledge distillation module is used to guide the first semantic segmentation model to update model parameters with the unmasked semantic segmentation prediction map as a target until the first semantic segmentation model reaches the number of training iterations and the training of the first semantic segmentation model is completed.

[0046] In order to solve the problem of elevator dependence on single-angle lenses or images when visibility is reduced due to changes in light, contamination of camera lenses, etc., and to increase the accuracy of semantic segmentation results, in step S1, the semantic segmentation model in the embodiment of this specification uses RGB images and corresponding thermal images as training set data. RGB images contain rich information such as color and texture. Thermal images rely on surface temperature imaging of objects to capture object contours and temperature information. The two types of image information are combined for analysis and reasoning, so that the semantic segmentation model can make full use of image information from different modalities, avoiding dependence on single modal information and the influence of image occlusion.

[0047] In step S3, the area with the lowest similarity between adjacent image blocks in the RGB image and the thermal value image is not masked, and the masked areas of the RGB image and the thermal value image are complementary, which specifically includes the following steps:

[0048] S31, dividing the RGB image and the corresponding thermal value image into a plurality of image blocks;

[0049] S33, calculating the similarity between each pair of image blocks of the thermal value image, recording the position of the image block with the lowest similarity, and the thermal value image and the corresponding RGB image at this position are not masked;

[0050] S35 , randomly masking the unrecorded image blocks. After the random masking, the masked areas of the remaining image blocks of the RGB image and the corresponding thermal value image are complementary except for the recorded image block positions.

[0051] In object detection and semantic segmentation tasks, masking is often used to improve the model's attention, focusing it more on the target area. However, when the similarity between adjacent image blocks in an image is too low, the images at that location may not contain the same target or have significantly different features. Masking at these locations cannot effectively remove noise and interference and improve recognition accuracy. Therefore, in step S33, the image blocks with the lowest similarity are not masked to reduce computational effort and focus on key areas in the image. Specifically, a 2×2 sliding window is set, with each window selecting four image blocks. The similarity between each pair of image blocks within the window is calculated. The window slides sequentially across the image in columns or rows and performs the calculations to calculate the similarity between all adjacent image blocks in the image. In step S35, the training image is randomly masked to reduce the model's over-reliance on specific noise or local features, lowering the risk of overfitting and increasing model robustness. Furthermore, the masked regions of the RGB image and the thermal image complement each other, enabling the model to fully utilize information from different modalities and different local regions, ensuring stable performance across different datasets and scenarios.

[0052] In step S5, the first semantic segmentation model includes a feature extraction module, a global context fusion module and a feature decoding module. The feature extraction module extracts the feature maps of the RGB image and the thermal value image respectively, the feature fusion module fuses the feature maps of the RGB image and the thermal value image, and the feature decoding module performs semantic segmentation on the RGB image, the thermal value image and the fused feature map, and outputs the semantic segmentation prediction map based on the semantic segmentation result.

[0053] Among them, the feature extraction module uses the Swin transformer to extract the feature maps of the RGB image and the thermal value image respectively. The Swin transformer adopts a hierarchical structure while performing multi-scale feature extraction, dividing the input image into multiple small blocks and performing Transformer calculations on each small block. The computational complexity is low, effectively improving the processing speed. While maintaining high precision, it reduces unnecessary global calculations, further improving model performance, and ensuring the efficiency and accuracy of the feature extraction process. The feature decoding module uses a transformer decoder to perform semantic segmentation, converting the fused feature map into a semantic segmentation map with the same resolution as the input image, and assigning appropriate categories to the pixels in the image. In order to fully combine the feature information between each modality, the feature fusion module performs global context feature fusion, local context fusion, and consistency fusion on the feature maps of the RGB image and the thermal value image. The feature representation of the above multi-scale fusion method is more comprehensive, and can capture the details and features of the image at different scales, so that the performance and generalization ability of the model are higher. Specifically, the global context feature fusion includes the following steps:

[0054] S51. Update the RGB image: Use the cross-attention mechanism to calculate the similarity of the corresponding rows of the RGB image feature map and the thermal value image feature map, convert the similarity into a probability distribution to obtain the RGB image normalization weight, and use the RGB image normalization weight as the coefficient of the RGB image feature map to update the RGB image. The formula is as follows:

[0055]

[0056] Among them, x rgb Represents the feature map of the RGB image, x thr Represents the feature map of the thermal image, d represents the number of channels of the RGB image feature map, Represents the feature map of the updated RGB image;

[0057] S53, updating the thermal value image: using a cross-attention mechanism to calculate the similarity of the corresponding rows of the thermal value image feature map and the RGB image feature map, converting the similarity into a probability distribution to obtain a thermal value map normalization weight, and using the thermal value map normalization weight as a coefficient of the thermal value image feature map to update the thermal value image, the formula is as follows:

[0058]

[0059] Among them, x rg b Represents the feature map of the RGB image, x thr represents the feature map of the thermal value image, d represents the number of channels of the feature map of the thermal value image, A feature map representing the updated thermal image;

[0060] S55: Fusing the updated RGB image and the thermal value image, the formula is as follows:

[0061]

[0062] in, Represents the feature map after feature fusion, represents the updated RGB image feature map, Represents the updated thermal image feature map.

[0063] In the global context fusion process, the RGB image and the thermal image are updated before feature fusion, and the similarity with the other image is introduced, so that the fused feature map has the characteristics of both modalities, reducing the holes and discontinuities in the segmentation mask and improving the edge quality of the semantic segmentation result.

[0064] In step S9, the unmasked semantic segmentation prediction map is used as a target to guide the first semantic segmentation model to update its model parameters, thereby improving the performance of the first semantic segmentation model through knowledge distillation. Therefore, the model structure and initialization parameters of the second semantic segmentation model that outputs the unmasked semantic segmentation prediction map in step S7 must be consistent with those of the first semantic segmentation model. This minimizes the differences between the first and second semantic segmentation models beyond the masking operation, allowing the first semantic segmentation model to more specifically learn from the output of the second semantic segmentation model.

[0065] Specifically, the distillation loss function L of the semantic segmentation prediction map and the unmasked semantic segmentation prediction map is as follows:

[0066] L=L MCM +L MWS +L MCS

[0067] L MCM represents the loss function between the category prediction embedding of the masked semantic segmentation map and the category prediction embedding of the unmasked semantic segmentation map, L MWS represents the self-supervised learning loss function, L MCS Represents the sum of the loss functions between the category prediction embedding of the masked semantic segmentation map of the RGB image, the category prediction embedding of the masked semantic segmentation map of the thermal value image, and the category prediction embedding of the masked semantic segmentation map.

[0068]

[0069] L MWS =L MCL (E gt ,E(M rgb ,M thr ))+L MCL (E gt ,E(M thr ))+L MCL (E gt ,E(M rgb ))

[0070]

[0071] Among them, M rgb Represents the RGB image classification result predicted by the second semantic segmentation model, M thr Represents the thermal image classification result predicted by the second semantic segmentation model, Represents the RGB image classification result predicted by the first semantic segmentation model, Represents the thermal image classification result predicted by the first semantic segmentation model, E C represents the category prediction embedding, E gt Indicates the label value, L JS represents JS divergence, L MCL Represents the loss function for self-supervised learning.

[0072] In addition, the model objects updated in the first semantic segmentation include the multi-layer perceptron in swintransformer, the learnable vector formed by replacing the mask area, etc. Specifically, the mask areas of the RGB image and the thermal value image are replaced with the learnable vector, and the learnable vector is initialized; the formula of the learnable vector is as follows:

[0073]

[0074]

[0075] in, and Represent the masked RGB image and thermal image respectively, M rgb and M thr are the mask matrices of RGB image and thermal image respectively, T rgb and T thr Represent the learnable vectors of RGB image and thermal value image respectively.

[0076] The embodiments of this specification also disclose a semantic segmentation method, which includes:

[0077] The RGB image and thermal image of the current environment are acquired through an RGB-T camera, and the RGB image and the thermal image are input into the first semantic segmentation model to output a semantic segmentation map; wherein, the first semantic segmentation model is trained based on the above-mentioned semantic segmentation model training method. When specifically applied to the field of elevator monitoring, an RGB image and a thermal image are acquired by installing an RGB-T camera in the elevator. The RGB-T camera transmits the acquired two images to a processor deployed with a semantic segmentation model to obtain a semantic segmentation result. Then, based on the semantic segmentation result, elevator intrusion detection, fault diagnosis and prevention, passenger behavior analysis and other processes are performed, effectively ensuring the safety of the elevator production and operation process.

[0078] Although several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this specification. Certain features described in the context of separate embodiments may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented in multiple implementations individually or in any suitable subcombination.

[0079] Although the subject matter has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

[0080] While the embodiments of the present disclosure have been described above, the above descriptions are illustrative, non-exhaustive, and not intended to be limiting of the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the disclosed embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technological improvements in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for training a semantic segmentation model, characterized in that: Methods include: Acquire training set data, wherein the training set data includes an RGB image and a corresponding thermal value image; Masking the RGB image and the corresponding thermal value image; wherein the area with the lowest similarity between adjacent image blocks in the RGB image and the thermal value image is not masked, and the masked areas of the RGB image and the thermal value image are complementary; Inputting the complementary masked RGB image and the thermal value image into a first semantic segmentation model and outputting a semantic segmentation prediction map; wherein the first semantic segmentation model includes a feature extraction module, a global context fusion module and a feature decoding module, the feature extraction module extracts feature maps of the RGB image and the thermal value image respectively, the fusion module fuses the feature maps of the RGB image and the thermal value image, the feature decoding module performs semantic segmentation on the RGB image, the thermal value image and the fused feature map, and outputs the semantic segmentation prediction map based on the semantic segmentation result; Inputting the unmasked RGB image and the corresponding thermal value image into a second semantic segmentation model, and outputting an unmasked semantic segmentation prediction map; The semantic segmentation prediction map and the unmasked semantic segmentation prediction map are input into a knowledge distillation module. The knowledge distillation module is used to guide the first semantic segmentation model to update model parameters with the unmasked semantic segmentation prediction map as a target until the first semantic segmentation model reaches the number of training iterations and the training of the first semantic segmentation model is completed.

2. The method according to claim 1, characterized in that Masking the RGB image and the corresponding thermal value image includes: Segmenting the RGB image and the corresponding thermal value image into a plurality of image blocks; Calculating the similarity between each pair of image blocks of the thermal value image, recording the position of the image block with the lowest similarity, and the thermal value image and the corresponding RGB image at this position are not masked; The unrecorded image blocks are randomly masked. After the random masking, the mask areas of the remaining image blocks of the RGB image and the corresponding thermal value image are complementary except for the recorded image block positions.

3. The method according to claim 2, characterized in that The knowledge distillation module uses the unmasked semantic segmentation prediction map as a target to guide the first semantic segmentation model to update model parameters, including: calculating a distillation loss function of the semantic segmentation prediction map and the unmasked semantic segmentation prediction map, and updating the first semantic segmentation model parameters based on the distillation loss function. The formula of the distillation loss function L is as follows: L=L MCM +L MWS +L MCS L MCM represents the loss function between the category prediction embedding of the masked semantic segmentation prediction map and the category prediction embedding of the unmasked semantic segmentation prediction map, L MWS represents the self-supervised learning loss function, L MCS Represents the sum of the loss functions between the category prediction embedding of the masked semantic segmentation prediction map of the RGB image, the category prediction embedding of the masked semantic segmentation prediction map of the thermal value image, and the category prediction embedding of the masked semantic segmentation prediction map.

4. The method according to claim 3, characterized in that Each loss function in the distillation loss function is a loss function in the form of JS divergence.

5. The method according to claim 3, characterized in that Updating the first semantic segmentation model parameters includes: The mask areas of the RGB image and the thermal value image are replaced with a learnable vector, and the learnable vector is initialized; wherein the formula of the learnable vector is as follows: and Represent the masked RGB image and thermal image respectively, M rgb and M thr are the mask matrices of RGB image and thermal image respectively, T rgb and T thr Learnable vectors representing RGB images and thermal images respectively; The learnable vector is updated based on the distillation loss function.

6. The method according to claim 1, characterized in that The model structure and initialization parameters of the first semantic segmentation model and the second semantic segmentation model are consistent.

7. The method according to claim 1, characterized in that The fusion module performs global context feature fusion, local context fusion and consistency fusion on the feature maps of the RGB image and the thermal value image.

8. The method according to claim 7, characterized in that The fusion module performs global context feature fusion on the feature maps of the RGB image and the thermal value image, including: Updating the RGB image: calculating the similarity between the RGB image feature map and the thermal value image feature map, converting the similarity into a probability distribution to obtain an RGB image normalization weight, and using the RGB image normalization weight as a coefficient of the RGB image feature map to update the RGB image; Updating the thermal value image: calculating the similarity between the thermal value image feature map and the RGB image feature map, converting the similarity into a probability distribution to obtain a thermal value map normalization weight, and using the thermal value map normalization weight as a coefficient of the thermal value image feature map to update the thermal value image; The updated RGB image and the thermal image are fused, and the formula is as follows: in, Represents the feature map after feature fusion, represents the updated RGB image feature map, Represents the updated thermal image feature map.

9. The method according to claim 7, characterized in that The feature extraction module uses a swin transformer to extract the feature maps of the RGB image and the thermal value image respectively.

10. A semantic segmentation method, characterized in that: include: An RGB image and a thermal image of the current environment are acquired through an RGB-T camera, the RGB image and the thermal image are input into the first semantic segmentation model, and a semantic segmentation map is output; wherein the first semantic segmentation model is trained based on the method described in claim 1.

Images