Semantic segmentation model training method and semantic segmentation method

By masking and knowledge distillation of RGB images and calorific value images, combined with multimodal information, the problem of insufficient dependence and environmental adaptability of semantic segmentation methods in the prior art is solved, and the problem of insufficient adaptability of single modes and environmentally in complex environments is realized. efficient semantic segmentation effect.

CN119068196AActive Publication Date: 2024-12-03ZHEJIANG UNIV OF TECH
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202411191979.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-28
Publication Date
2024-12-03
Estimated Expiration
2044-08-28

AI Technical Summary

Technical Problem

现有语义分割方法过度依赖RGB图像信息,忽略热值图信息,导致在光线变化、能见度下降和镜头污染等环境下语义分割精度下降,且对单一角度监控镜头依赖严重,难以适应复杂环境的电梯 监控需求。

Method used

By acquiring RGB images and calorific value images, masking, the image after complementary masking is input into the semantic segmentation model, combining the knowledge distillation module to update the model parameters, using multimodal information to reduce single mode dependence, and improving the stability of semantic segmentation and accuracy.

Benefits of technology

It realizes efficient accuracy of semantic segmentation in complex environments, reduces the impact of image occlusion and interference on model prediction results, and improves the performance and stability of semantic segmentation models.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068196A_ABST
    Figure CN119068196A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of image processing, in particular to a training method of a semantic segmentation model and a semantic segmentation method. The training method comprises the following steps: acquiring training set data, wherein the training set data comprises an RGB image and a calorific value image; performing complementary masking on the RGB image and the calorific value image; respectively inputting the masked RGB image and calorific value image into a first semantic segmentation model, and outputting a semantic segmentation prediction map; and obtaining a semantic segmentation prediction map of the unmasked RGB image and the calorific value image through the second semantic segmentation model, and updating the parameters of the first semantic segmentation model by taking the unmasked semantic segmentation prediction map as a target until the number of training iterations is reached. The semantic segmentation method comprises the steps of obtaining an RGB image and a calorific value image of a current environment, inputting the RGB image and the calorific value image into a semantic segmentation model, and outputting a semantic segmentation image which is obtained by training based on the method. The semantic segmentation precision and stability of the model in the embodiment of the invention are high, the model is not easily interfered by the external environment, and the model can adapt to a complex application scene.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of this specification relate to the field of image processing technology, and in particular to a training method for a semantic segmentation model and a semantic segmentation method. Background Art

[0002] During the production and operation of elevators, the elevator monitoring lens is prone to unclear images due to environmental factors such as light changes, reduced visibility, and lens contamination, and the captured images need to be further processed; and the different shooting angles and sizes of the monitoring lens are also likely to have a certain impact on the semantic segmentation, target detection, and data enhancement of the captured images, making them dependent on the monitoring lens of a single angle; therefore, it is necessary to improve the accuracy and stability of image processing processes such as semantic segmentation and target detection in order to meet the needs of elevator monitoring in complex environments. The semantic segmentation method in the prior art overly relies on the information captured by the RGB lens and ignores the thermal map information. Moreover, when the RGB lens is blocked by mosquitoes or stains, it will cause a significant decrease in the accuracy of semantic segmentation and a deterioration in the ability to understand the scene. Therefore, it is necessary to perform masking operations on the training data of the semantic segmentation model so that the semantic segmentation model can predict the missing pixel values ​​of the image through learning. Summary of the invention

[0003] The embodiments of this specification provide a semantic segmentation model training method and a semantic segmentation method, aiming to solve one or more of the above-mentioned problems and other potential problems.

[0004] In order to achieve the above objectives, the following technical solutions are provided:

[0005] According to a first aspect of the present specification, a method for training a semantic segmentation model is provided, the method comprising:

[0006] Acquire training set data, wherein the training set data includes an RGB image and a corresponding thermal value image;

[0007] Masking the RGB image and the corresponding thermal value image; wherein the area with the lowest similarity between adjacent image blocks in the RGB image and the thermal value image is not masked, and the masked areas of the RGB image and the thermal value image are complementary;

[0008] Input the complementary masked RGB image and the thermal value image into a first semantic segmentation model, and output a semantic segmentation prediction map; wherein the first semantic segmentation model includes a feature extraction module, a global context fusion module and a feature decoding module, the feature extraction module extracts feature maps of the RGB image and the thermal value image respectively, the feature fusion module fuses the feature maps of the RGB image and the thermal value image, the feature decoding module performs semantic segmentation on the RGB image, the thermal value image and the fused feature map, and outputs the semantic segmentation prediction map based on the semantic segmentation result;

[0009] Inputting the unmasked RGB image and the corresponding thermal value image into a second semantic segmentation model, and outputting an unmasked semantic segmentation prediction map;

[0010] The semantic segmentation prediction map and the unmasked semantic segmentation prediction map are input into a knowledge distillation module, and the knowledge distillation module is used to guide the first semantic segmentation model to update model parameters with the unmasked semantic segmentation prediction map as a target until the first semantic segmentation model reaches the number of training iterations and the training of the first semantic segmentation model is completed.

[0011] The training method of the semantic segmentation model of the embodiment of the present specification combines RGB images and thermal value images for training, so that the trained semantic segmentation model does not rely too much on the image information of a single modality, can adapt to more application scenarios with more interference factors, and realize an accurate and efficient semantic segmentation process; and performs complementary masking on the RGB images and thermal value images to extract complementary semantic information and geometric information in the image scene, so that the semantic model can make full use of information from different images and different image areas, reduce the influence of single modality dependence and image occlusion on the model prediction results, and have better semantic segmentation effect and stability; finally, the unmasked semantic segmentation prediction map is used as the learning target to update the model parameters, and further improve the prediction performance of the semantic segmentation model.

[0012] In some embodiments, masking the RGB image and the corresponding thermal value image includes:

[0013] Segment the RGB image and the corresponding thermal value image into a plurality of image blocks;

[0014] Calculate the similarity between each pair of image blocks of the thermal value image, record the position of the image block with the lowest similarity, and neither the thermal value image nor the corresponding RGB image at this position is masked;

[0015] The unrecorded image blocks are randomly masked. After the random masking, the masked areas of the remaining image blocks of the RGB image and the corresponding thermal value image are complementary except for the recorded image block positions.

[0016] In some embodiments, the knowledge distillation module uses the unmasked semantic segmentation prediction map as a target to guide the first semantic segmentation model to update model parameters, including: calculating a distillation loss function of the semantic segmentation prediction map and the unmasked semantic segmentation prediction map, and updating the first semantic segmentation model parameters based on the distillation loss function, wherein the distillation loss function L formula is as follows:

[0017] L=L MCM +L MWS +L MCS

[0018] L MCM represents the loss function between the category prediction embedding of the masked semantic segmentation map and the category prediction embedding of the unmasked semantic segmentation map, L MWS represents the self-supervised learning loss function, L MCS Represents the sum of the loss functions between the category prediction embedding of the masked semantic segmentation map of the RGB image, the category prediction embedding of the masked semantic segmentation map of the thermal value image, and the category prediction embedding of the masked semantic segmentation map.

[0019] In some embodiments, each loss function in the distillation loss function is a loss function in the form of JS divergence.

[0020] In some embodiments, updating the first semantic segmentation model parameters includes:

[0021] The mask areas of the RGB image and the thermal value image are replaced with a learnable vector, and the learnable vector is initialized; wherein the formula of the learnable vector is as follows:

[0022]

[0023]

[0024] and Represent the masked RGB image and thermal image, M rgb and M thr are the mask matrices of RGB image and thermal image respectively, T rgb and T thr Represent the learnable vectors of RGB image and thermal value image respectively;

[0025] The learnable vector is updated based on the distillation loss function.

[0026] In some embodiments, the model structures and initialization parameters of the first semantic segmentation model and the second semantic segmentation model are consistent.

[0027] In some embodiments, the feature fusion module performs global context feature fusion, local context fusion and consistency fusion on the feature maps of the RGB image and the thermal value image.

[0028] In some embodiments, the feature fusion module performs global context feature fusion on the feature maps of the RGB image and the thermal value image, including:

[0029] Updating the RGB image: calculating the similarity between the RGB image feature map and the thermal value image feature map, converting the similarity into a probability distribution to obtain a normalized weight of the RGB image, and using the normalized weight of the RGB image as a coefficient of the RGB image feature map to update the RGB image;

[0030] Updating the thermal value image: calculating the similarity between the thermal value image feature map and the RGB image feature map, converting the similarity into a probability distribution to obtain a normalized weight of the thermal value map, and using the normalized weight of the thermal value map as a coefficient of the thermal value image feature map to update the thermal value image;

[0031] The updated RGB image and the thermal value image are fused, and the formula is as follows:

[0032]

[0033] in, Represents the feature map after feature fusion, represents the updated RGB image feature map, Represents the updated thermal image feature map.

[0034] In some embodiments, the feature extraction module uses a swin transformer to extract feature maps of the RGB image and the thermal value image respectively.

[0035] According to the second aspect of the present specification, a semantic segmentation method is provided, the method comprising: obtaining an RGB image and a thermal value image of the current environment through an RGB-T camera, inputting the RGB image and the thermal value image into the first semantic segmentation model, and outputting a semantic segmentation map; wherein the first semantic segmentation model is trained based on the above-mentioned semantic segmentation model training method. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] By reading the following detailed description with reference to the accompanying drawings, the above and other objects, features and advantages of the embodiments of the present specification will become readily understood. In the accompanying drawings, several embodiments of the present specification are shown by way of example and not limitation.

[0037] Figure 1A flowchart of a method for training a semantic segmentation model according to an embodiment of the present specification is shown. DETAILED DESCRIPTION

[0038] The preferred embodiments of the present specification will be described in more detail below with reference to the accompanying drawings. Although the preferred embodiments of the present specification are shown in the accompanying drawings, it should be understood that the present specification can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to make the present specification more thorough and complete, and to fully convey the scope of the present specification to those skilled in the art.

[0039] The term "including" and its variations used in this document represent open inclusion, that is, "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "based at least in part on". The terms "an example embodiment" and "an embodiment" mean "at least one example embodiment". The term "another embodiment" means "at least one other embodiment". Terms such as "upper", "lower", "front", and "rear" indicating placement or positional relationships are based on the orientation or positional relationships shown in the drawings, and are only for the convenience of describing the principles of this specification, and do not indicate or imply that the referred elements must have a specific orientation, be constructed or operate in a specific orientation, and therefore should not be understood as limitations on this specification.

[0040] The following is a flowchart of a method for training a semantic segmentation model according to an embodiment of the present specification described in detail in conjunction with the accompanying drawings. The method includes:

[0041] S1. Acquire training set data, wherein the training set data includes an RGB image and a corresponding thermal value image;

[0042] S3, masking the RGB image and the corresponding thermal value image;

[0043] S5, inputting the complementary masked RGB image and the thermal value image into a first semantic segmentation model, and outputting a semantic segmentation prediction map;

[0044] S7, inputting the unmasked RGB image and the corresponding thermal value image into a second semantic segmentation model, and outputting an unmasked semantic segmentation prediction map;

[0045] S9. Input the semantic segmentation prediction map and the unmasked semantic segmentation prediction map into a knowledge distillation module, wherein the knowledge distillation module is used to guide the first semantic segmentation model to update model parameters with the unmasked semantic segmentation prediction map as a target until the first semantic segmentation model reaches the number of training iterations and the training of the first semantic segmentation model is completed.

[0046] In order to solve the problem of elevator dependence on single-angle lenses or images when visibility is reduced due to changes in light, camera lens pollution, etc., and to increase the accuracy of semantic segmentation results, in step S1, the semantic segmentation model in the embodiment of this specification uses RGB images and corresponding thermal images as training set data. RGB images contain rich information such as color and texture. Thermal images rely on object surface temperature imaging to capture object contours and temperature information. The two types of image information are combined for analysis and reasoning, so that the semantic segmentation model can make full use of image information from different modalities and avoid dependence on single modal information and the influence of image occlusion.

[0047] In step S3, the area with the lowest similarity between adjacent image blocks in the RGB image and the thermal value image is not masked, and the masked areas of the RGB image and the thermal value image are complementary, which specifically includes the following steps:

[0048] S31, dividing the RGB image and the corresponding thermal value image into a plurality of image blocks;

[0049] S33, calculating the similarity between each pair of image blocks of the thermal value image, recording the position of the image block with the lowest similarity, and the thermal value image and the corresponding RGB image at this position are not masked;

[0050] S35, randomly masking the unrecorded image blocks. After the random masking, the masked areas of the remaining image blocks of the RGB image and the corresponding thermal value image are complementary except for the recorded image block positions.

[0051] In target detection and semantic segmentation tasks, masking operations are usually used to improve the model's attention and make it more focused on the target area. However, when the similarity between adjacent image blocks in the image is too low, the image at this position may not contain the same target or the feature difference is significant. At these positions, the mask cannot effectively remove noise and interference and improve recognition accuracy. Therefore, in step S33, the image block position with the lowest similarity is not masked to reduce the amount of calculation and focus on the key area in the image. Specifically, a 2×2 sliding pane is set, each sliding pane selects 4 image blocks, and the similarity between the two image blocks in the pane is calculated. The sliding pane slides and calculates in sequence on the image by column or row, and the similarity between all adjacent image blocks in the image can be calculated. In step S35, the training image is randomly masked to reduce the model's excessive dependence on specific noise or local features, reduce the risk of overfitting, and increase the robustness of the model. At the same time, the mask areas of the RGB image and the thermal value image are complementary, which enables the model to make full use of information from different modalities and different local areas, and ensure the stable performance of the model in different data sets and different scenarios.

[0052] In step S5, the first semantic segmentation model includes a feature extraction module, a global context fusion module and a feature decoding module. The feature extraction module extracts the feature maps of the RGB image and the thermal value image respectively, the feature fusion module fuses the feature maps of the RGB image and the thermal value image, and the feature decoding module performs semantic segmentation on the RGB image, the thermal value image and the fused feature map, and outputs the semantic segmentation prediction map based on the semantic segmentation result.

[0053] Among them, the feature extraction module uses Swin transformer to extract the feature maps of the RGB image and the thermal value image respectively. Swin transformer adopts a hierarchical structure while performing multi-scale feature extraction, divides the input image into multiple small blocks, and performs Transformer calculation on each small block. The computational complexity is low, effectively improves the processing speed, reduces unnecessary global calculations while maintaining high precision, further improves the model performance, and ensures the efficiency and accuracy of the feature extraction process. The feature decoding module uses a transformer decoder for semantic segmentation, converts the fused feature map into a semantic segmentation map with the same resolution as the input image, and assigns appropriate categories to the pixels in the image. In order to fully combine the feature information between each modality, the feature fusion module performs global context feature fusion, local context fusion and consistency fusion on the feature maps of the RGB image and the thermal value image. The feature representation of the above multi-scale fusion method is more comprehensive, and can capture the details and features of the image at different scales, so that the performance and generalization ability of the model are higher. Specifically, the global context feature fusion includes the following steps:

[0054] S51, updating the RGB image: using the cross attention mechanism to calculate the similarity of the corresponding rows of the RGB image feature map and the thermal value image feature map, converting the similarity into a probability distribution to obtain the RGB image normalization weight, and using the RGB image normalization weight as the coefficient of the RGB image feature map to update the RGB image, the formula is as follows:

[0055]

[0056] Among them, x rgb Represents the feature map of the RGB image, x thr represents the feature map of the thermal image, d represents the number of channels of the RGB image feature map, Represents the feature map of the updated RGB image;

[0057] S53, updating the thermal value image: using the cross attention mechanism to calculate the similarity of the corresponding rows of the thermal value image feature map and the RGB image feature map, converting the similarity into a probability distribution to obtain the normalized weight of the thermal value map, and using the normalized weight of the thermal value map as the coefficient of the thermal value image feature map to update the thermal value image, the formula is as follows:

[0058]

[0059] Among them, x rg b Represents the feature map of the RGB image, x thr represents the feature map of the thermal value image, d represents the number of channels of the feature map of the thermal value image, A feature map representing an updated thermal value image;

[0060] S55, fusing the updated RGB image and the thermal value image, the formula is as follows:

[0061]

[0062] in, Represents the feature map after feature fusion, represents the updated RGB image feature map, Represents the updated thermal image feature map.

[0063] In the global context fusion process, before feature fusion, the RGB image and the thermal value image are updated first, and the similarity with another image is introduced, so that the fused feature map has the characteristics of both modalities, reducing the holes and discontinuities in the segmentation mask and improving the edge quality of the semantic segmentation result.

[0064] In step S9, the unmasked semantic segmentation prediction map is used as a target to guide the first semantic segmentation model to update model parameters, and the performance of the first semantic model is improved by the knowledge distillation method. Therefore, the model structure and initialization parameters of the second semantic segmentation model that outputs the unmasked semantic segmentation prediction map in step S7 and the first semantic segmentation model must be consistent, so as to reduce the difference between the first semantic segmentation model and the second semantic segmentation model except for the mask operation, so that the first semantic segmentation model can learn from the output result of the second semantic segmentation model in a more targeted manner.

[0065] Specifically, the distillation loss function L formula of the semantic segmentation prediction map and the unmasked semantic segmentation prediction map is as follows:

[0066] L=L MCM +L MWS +L MCS

[0067] L MCM represents the loss function between the category prediction embedding of the masked semantic segmentation map and the category prediction embedding of the unmasked semantic segmentation map, L MWS represents the self-supervised learning loss function, L MCS Represents the sum of the loss functions between the category prediction embedding of the masked semantic segmentation map of the RGB image, the category prediction embedding of the masked semantic segmentation map of the thermal value image, and the category prediction embedding of the masked semantic segmentation map.

[0068]

[0069] L MWS =L MCL (E gt ,E(M rgb ,M thr ))+L MCL (E gt ,E(M thr ))+L MCL (E gt ,E(M rgb ))

[0070]

[0071] Among them, M rgb represents the RGB image classification result predicted by the second semantic segmentation model, M thr represents the thermal image classification result predicted by the second semantic segmentation model, Represents the RGB image classification result predicted by the first semantic segmentation model, represents the thermal image classification result predicted by the first semantic segmentation model, E C represents the category prediction embedding, E gt Indicates the label value, L JS represents the JS divergence, L MCL Represents the loss function for self-supervised learning.

[0072] In addition, the model object updated by the first semantic segmentation includes a multilayer perceptron in swintransformer, a learnable vector replaced by a mask region, etc. Specifically, the mask regions of the RGB image and the thermal value image are replaced by a learnable vector, and the learnable vector is initialized; the formula of the learnable vector is as follows:

[0073]

[0074]

[0075] in, and Represent the masked RGB image and thermal image, M rgb and M thr are the mask matrices of RGB image and thermal image respectively, T rgb and T thr Represent the learnable vectors of RGB image and thermal value image respectively.

[0076] The embodiment of this specification also discloses a semantic segmentation method, which includes:

[0077] The RGB image and thermal value image of the current environment are obtained through the RGB-T camera, and the RGB image and the thermal value image are input into the first semantic segmentation model to output a semantic segmentation map; wherein, the first semantic segmentation model is trained based on the training method of the above-mentioned semantic segmentation model. When specifically applied to the field of elevator monitoring, the RGB image and the thermal value image are obtained by setting an RGB-T camera in the elevator. After the RGB-T camera transmits the two obtained images to the processor deployed with the semantic segmentation model, the semantic segmentation result is obtained, and then the elevator intrusion detection, fault diagnosis and prevention, passenger behavior analysis and other processes are carried out based on the semantic segmentation results, effectively ensuring the safety of the elevator production and operation process.

[0078] Although several specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of this specification. Certain features described in the context of separate embodiments may also be implemented in a single implementation in combination. Conversely, various features described in the context of a single implementation may also be implemented in multiple implementations individually or in any suitable sub-combination.

[0079] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

[0080] The embodiments of the present specification have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for training a semantic segmentation model, characterized in that: Methods include: Acquire training set data, wherein the training set data includes an RGB image and a corresponding thermal value image; Masking the RGB image and the corresponding thermal value image; wherein the area with the lowest similarity between adjacent image blocks in the RGB image and the thermal value image is not masked, and the masked areas of the RGB image and the thermal value image are complementary; Input the complementary masked RGB image and the thermal value image into a first semantic segmentation model, and output a semantic segmentation prediction map; wherein the first semantic segmentation model includes a feature extraction module, a global context fusion module and a feature decoding module, the feature extraction module extracts feature maps of the RGB image and the thermal value image respectively, the feature fusion module fuses the feature maps of the RGB image and the thermal value image, the feature decoding module performs semantic segmentation on the RGB image, the thermal value image and the fused feature map, and outputs the semantic segmentation prediction map based on the semantic segmentation result; Inputting the unmasked RGB image and the corresponding thermal value image into a second semantic segmentation model, and outputting an unmasked semantic segmentation prediction map; The semantic segmentation prediction map and the unmasked semantic segmentation prediction map are input into a knowledge distillation module, and the knowledge distillation module is used to guide the first semantic segmentation model to update model parameters with the unmasked semantic segmentation prediction map as a target until the first semantic segmentation model reaches the number of training iterations and the training of the first semantic segmentation model is completed.

2. The method according to claim 1, characterized in that Masking the RGB image and the corresponding thermal value image includes: Segment the RGB image and the corresponding thermal value image into a plurality of image blocks; Calculate the similarity between each pair of image blocks of the thermal value image, record the position of the image block with the lowest similarity, and neither the thermal value image nor the corresponding RGB image at this position is masked; The unrecorded image blocks are randomly masked. After the random masking, the masked areas of the remaining image blocks of the RGB image and the corresponding thermal value image are complementary except for the recorded image block positions.

3. The method according to claim 2, characterized in that The knowledge distillation module uses the unmasked semantic segmentation prediction map as a target to guide the first semantic segmentation model to update model parameters, including: calculating a distillation loss function of the semantic segmentation prediction map and the unmasked semantic segmentation prediction map, and updating the first semantic segmentation model parameters based on the distillation loss function. The distillation loss function L formula is as follows: L=L MCM +L MWS +L MCS L MCM represents the loss function between the category prediction embedding of the masked semantic segmentation map and the category prediction embedding of the unmasked semantic segmentation map, L MWS represents the self-supervised learning loss function, L MCS Represents the sum of the loss functions between the category prediction embedding of the masked semantic segmentation map of the RGB image, the category prediction embedding of the masked semantic segmentation map of the thermal value image, and the category prediction embedding of the masked semantic segmentation map.

4. The method according to claim 3, characterized in that: Each loss function in the distillation loss function is a loss function in the form of JS divergence.

5. The method according to claim 3, characterized in that: Updating the first semantic segmentation model parameters includes: The mask areas of the RGB image and the thermal value image are replaced with a learnable vector, and the learnable vector is initialized; wherein the formula of the learnable vector is as follows: and Represent the masked RGB image and thermal image, M rgb and M thr are the mask matrices of RGB image and thermal image respectively, T rgb and T thr Represent the learnable vectors of RGB image and thermal value image respectively; The learnable vector is updated based on the distillation loss function.

6. The method according to claim 1, characterized in that The model structure and initialization parameters of the first semantic segmentation model and the second semantic segmentation model are consistent.

7. The method according to claim 1, characterized in that The feature fusion module performs global context feature fusion, local context fusion and consistency fusion on the feature maps of the RGB image and the thermal value image.

8. The method according to claim 7, characterized in that The feature fusion module performs global context feature fusion on the feature maps of the RGB image and the thermal value image, including: Updating the RGB image: calculating the similarity between the RGB image feature map and the thermal value image feature map, converting the similarity into a probability distribution to obtain a normalized weight of the RGB image, and using the normalized weight of the RGB image as a coefficient of the RGB image feature map to update the RGB image; Updating the thermal value image: calculating the similarity between the thermal value image feature map and the RGB image feature map, converting the similarity into a probability distribution to obtain a normalized weight of the thermal value map, and using the normalized weight of the thermal value map as a coefficient of the thermal value image feature map to update the thermal value image; The updated RGB image and the thermal value image are fused, and the formula is as follows: in, Represents the feature map after feature fusion, represents the updated RGB image feature map, Represents the updated thermal image feature map.

9. The method according to claim 7, characterized in that: The feature extraction module uses a swin transformer to extract the feature maps of the RGB image and the thermal value image respectively.

10. A semantic segmentation method, characterized in that: include: An RGB image and a thermal image of the current environment are acquired through an RGB-T camera, the RGB image and the thermal image are input into the first semantic segmentation model, and a semantic segmentation map is output; wherein the first semantic segmentation model is trained based on the method described in claim 1.

Citation Information

Patent Citations

  • RGB-T image semantic segmentation method and device

    CN116091765A

  • Semantic segmentation method, system and equipment based on knowledge distillation and medium

    CN117079276A

  • Limited data continuous semantic segmentation method and system based on decoupling knowledge distillation

    CN117218347A

  • Dynamic feature elimination method of visual SLAM (Simultaneous Localization and Mapping)

    CN118447091A