Semantic segmentation model training method and device, equipment and storage medium
By extracting and utilizing event information and content information of day and night images, the semantic segmentation model is trained, which solves the problem of poor performance of traditional models in night image segmentation, and significantly improves the accuracy of night image segmentation.
Patent Information
- Application Number
- CN202411696568.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-25
- Publication Date
- 2025-03-21
AI Technical Summary
Traditional semantic segmentation models perform poorly at night or under low light conditions, mainly due to insufficient light, shadows, reflections and image quality, resulting in limited semantic understanding and boundary segmentation performance of night images.
By obtaining training data containing night and daytime images, extracting event information and content information of daytime and nighttime images, and using this information as target training data, the semantic segmentation model is trained to improve its performance in nighttime image segmentation.
By calculating daytime event information and content information, the gap between image mode and event mode is bridged, and by calculating daytime and nighttime content information, the style difference between daytime and nighttime is bridged, which significantly improves the segmentation performance of the semantic segmentation model for nighttime images.
Smart Images
Figure CN119672332A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of semantic segmentation, and in particular to a training method and device for a semantic segmentation model, an electronic device and a storage medium. Background Art
[0002] Nighttime segmentation is the application of semantic segmentation tasks at night or in low-light conditions, aiming to achieve pixel-level semantic understanding of images in dark environments. This task is of great significance for promoting the application of nighttime scenes in many fields such as autonomous driving, robotics, and surveillance systems. However, the presence of factors such as insufficient nighttime lighting, shadows, reflections, and the low dynamic range of traditional frame cameras has led to a significant decrease in the quality of nighttime images acquired by traditional frame cameras. The loss of color contrast and details further reduces image clarity and hinders the effective distinction of object boundaries. In addition, the lack of high-quality annotations for nighttime images also increases the difficulty of training semantic segmentation models.
[0003] Therefore, how to train a semantic segmentation model that can effectively perform semantic segmentation on images taken at night has become an urgent problem to be solved in this field. Summary of the invention
[0004] An embodiment of the present application provides a method for training a semantic segmentation model to solve the problem of how to train a semantic segmentation model that can effectively perform semantic segmentation on images taken at night.
[0005] Correspondingly, an embodiment of the present application also provides a training device for a semantic segmentation model, an electronic device and a storage medium to ensure the implementation and application of the above method.
[0006] In order to solve the above problems, the present application discloses a method for training a semantic segmentation model, the method comprising:
[0007] Acquire training data; the training data includes a night image set and a day image set, the night image set includes night image information and night event information, and the day image set includes day image information with corresponding semantic categories;
[0008] Extracting daytime event information corresponding to the daytime image information, daytime content information corresponding to the daytime image information, and nighttime content information corresponding to the nighttime image information;
[0009] using the daytime event information, the daytime content information, the nighttime content information, and the training data as target training data;
[0010] The target training data is used to train a semantic segmentation model to be trained to obtain a target semantic segmentation model, and the target semantic segmentation model is used to perform semantic segmentation on the image to be segmented taken at night.
[0011] Optionally, the extracting the daytime event information corresponding to the daytime image information, the daytime content information corresponding to the daytime image information, and the nighttime content information corresponding to the nighttime image information includes:
[0012] Acquiring target daytime image information from the daytime image information;
[0013] Calculating target daytime event information corresponding to the target daytime image information according to the target daytime image information and adjacent daytime image information that is adjacent to the target daytime image information in time;
[0014] The target daytime event information corresponding to the target daytime image information is sorted to obtain the daytime event information corresponding to the daytime image information.
[0015] Optionally, the extracting the daytime event information corresponding to the daytime image information, the daytime content information corresponding to the daytime image information, and the nighttime content information corresponding to the nighttime image information includes:
[0016] Acquiring target daytime image information from the daytime image information;
[0017] Randomly shifting the target daytime image information to obtain shifted target daytime image information;
[0018] Calculating target daytime content information corresponding to the target daytime image information according to the translated target daytime image information and the target daytime image information;
[0019] Arranging the target daytime content information corresponding to the target daytime image information to obtain the daytime content information corresponding to the daytime image information;
[0020] Acquiring target nighttime image information from the nighttime image information;
[0021] Randomly translate the target nighttime image information to obtain translated target nighttime image information;
[0022] Calculating target nighttime content information corresponding to the target nighttime image information according to the translated target nighttime image information and the target nighttime image information;
[0023] The target nighttime content information corresponding to the target nighttime image information is sorted to obtain the nighttime content information corresponding to the nighttime image information.
[0024] Optionally, the semantic segmentation model to be trained is a teacher model, the teacher model has a corresponding student model, the student model includes an image encoder, an event encoder and a decoder, and the using the target training data to train the semantic segmentation model to be trained includes:
[0025] Inputting the daytime image information into the image encoder to obtain a first intermediate feature;
[0026] Inputting the daytime event information and the daytime content information into the event encoder to obtain a second intermediate feature;
[0027] fusing the first intermediate feature and the second intermediate feature to obtain a first cross-modal feature;
[0028] Inputting the first cross-modal feature into the decoder to obtain predicted student segmentation data;
[0029] Obtaining a student loss function according to the semantic category corresponding to the daytime image information and the predicted student segmentation data;
[0030] The teacher model is trained according to the student loss function, the night image information, the night event information and the night content information.
[0031] Optionally, the training of the teacher model according to the student loss function, the night image information, the night event information and the night content information includes:
[0032] Inputting the night image information, the night event information and the night content information into the teacher model to obtain a pseudo label;
[0033] Obtaining a second cross-modal feature according to the nighttime image information, the nighttime event information, and the nighttime content information;
[0034] Inputting the second cross-modal feature into the student model to obtain predicted teacher segmentation data;
[0035] Obtaining a teacher loss function according to the pseudo-label and the predicted teacher segmentation data;
[0036] Using the teacher loss function to update the student model, to obtain an updated student model;
[0037] Update the teacher model according to the model parameters of the updated student model and the student loss function.
[0038] Optionally, the fusing the first intermediate feature with the second intermediate feature to obtain a first cross-modal feature includes:
[0039] splicing intermediate features of the same scale in the first intermediate feature and the second intermediate feature to obtain a spliced intermediate feature;
[0040] Perform dimensionality restoration on the concatenated intermediate features to obtain the first cross-modal features.
[0041] The present application also discloses a semantic segmentation method for nighttime images, the method comprising:
[0042] Obtain an image to be segmented taken at night;
[0043] The target semantic segmentation model is used to perform semantic segmentation on the image to be segmented to obtain a target image after semantic segmentation; the target image after semantic segmentation has a corresponding semantic category; the target semantic segmentation model is the target semantic segmentation model described in the embodiment of the present application.
[0044] The present application also discloses a training device for a semantic segmentation model, the device comprising:
[0045] A training data acquisition module, used to acquire training data; the training data includes a night image set and a day image set, the night image set includes night image information and night event information, and the day image set includes day image information with corresponding semantic categories;
[0046] A training data extraction module, used to extract daytime event information corresponding to the daytime image information, daytime content information corresponding to the daytime image information, and nighttime content information corresponding to the nighttime image information;
[0047] a training data merging module, configured to use the daytime event information, the daytime content information, the nighttime content information and the training data as target training data;
[0048] The model training module is used to train the semantic segmentation model to be trained using the target training data to obtain a target semantic segmentation model.
[0049] An embodiment of the present application also discloses an electronic device, including: a processor; and a memory, on which executable code is stored. When the executable code is executed, the processor executes the training method of the semantic segmentation model as described in one or more of the embodiments of the present application.
[0050] The embodiments of the present application also disclose one or more machine-readable media on which executable codes are stored. When the executable codes are executed, the processor executes the training method of the semantic segmentation model as described in one or more of the embodiments of the present application.
[0051] Compared with the prior art, the embodiments of the present application have the following advantages:
[0052] In an embodiment of the present application, training data is obtained; the training data includes a night image set and a day image set, the night image set includes night image information and night event information, and the day image set includes day image information with corresponding semantic categories; the day event information corresponding to the day image information, the day content information corresponding to the day image information, and the night content information corresponding to the night image information are extracted; the day event information, the day content information, the night content information, and the training data are used as target training data; the target training data is used to train the semantic segmentation model to be trained to obtain the target semantic segmentation model, and the target semantic segmentation model is used to perform semantic segmentation on the images to be segmented taken at night. The embodiment of the present application bridges the gap between the image modality and the event modality in the training data by calculating the day event information, and at the same time bridges the style difference between day and night in the training data by calculating the day content information and the night content information, so that the trained target semantic segmentation model can better understand the relationship between the image information and the event information, thereby significantly improving the performance of semantic segmentation of images taken at night. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] Figure 1 It is a flowchart of the steps of an embodiment of a method for training a semantic segmentation model of the present application;
[0054] Figure 2 This is a training data processing diagram of an embodiment of a training method for a semantic segmentation model of the present application;
[0055] Figure 3 This is a model training schematic diagram of an embodiment of a training method for a semantic segmentation model of the present application;
[0056] Figure 4 It is a flowchart of an embodiment of a method for training a semantic segmentation model of the present application;
[0057] Figure 5 It is a structural block diagram of an embodiment of a training device for a semantic segmentation model of the present application;
[0058] Figure 6 It is a schematic diagram of the structure of a device provided in one embodiment of the present application. DETAILED DESCRIPTION
[0059] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.
[0060] Night segmentation is to assign a semantic label to each pixel in the night image, thereby dividing the night image into semantically meaningful areas. Unlike target detection, it is not only required to identify the objects in the image, but also to associate each pixel with a specific category.
[0061] As an application of semantic segmentation tasks under low-light conditions, night segmentation is of great significance for promoting night scene applications in fields such as autonomous driving, robotics, and monitoring systems. However, night images are affected by factors such as insufficient lighting, shadows, and reflections, and the image quality is greatly reduced. In addition, high-quality annotated data is scarce, which limits the performance of traditional semantic segmentation models in night scenes. To solve this problem, the embodiment of the present application proposes to use high-quality labeled data during the day to train the semantic segmentation model to improve its semantic segmentation performance on night images.
[0062] Furthermore, the embodiment of the present application further introduces an event camera to make up for the shortcomings of traditional frame cameras in low dynamic range. The event camera has a higher dynamic range and contrast by generating events when the pixel light intensity changes, and can better preserve details, which is particularly suitable for low-light scenes. However, event data is asynchronous and spatially sparse, lacking a comprehensive representation of the scene, so relying solely on event data to train the semantic segmentation model is usually not effective.
[0063] To overcome this difficulty, the embodiment of the present application also designs an image-event cross-modal fusion framework. This framework reduces the domain offset between daytime and nighttime image representations through image motion extractors and image content extractors, and uses the features of both image and event modalities for cross-modal fusion. Through this fusion method, the trained semantic segmentation model can process nighttime images more effectively, improve the semantic segmentation accuracy of nighttime scenes, and enhance the ability to understand and perceive nighttime environments, thereby promoting the widespread application of computer vision technology in nighttime environments.
[0064] Reference Figure 1 , is a flowchart of a method for training a semantic segmentation model according to an embodiment of the present application, comprising the following steps:
[0065] Step 101, obtaining training data; the training data includes a night image set and a day image set, the night image set includes night image information and night event information, and the day image set includes day image information with corresponding semantic categories.
[0066] In the embodiment of the present application, the training data includes a set of unlabeled nighttime images (I t ,E t ) and the daytime image set with semantic category labels (I s ,Y s). Among them, the night image set includes night image information I acquired by traditional frame cameras t and nighttime event information E acquired by the event camera t , the daytime image set includes images with corresponding semantic categories Y s Daytime image information obtained by traditional frame cameras s .
[0067] Data with different data distributions are called different domains. The data distribution of the training set data is a known domain, called the source domain; the data distribution of the test set data, that is, the domain to be learned, is called the target domain. The source domain data of the embodiment of the present application is a daytime image set, and the target domain data is a nighttime image set.
[0068] When the training data and test data come from different distributions or domains, due to the difference in distribution, the model learned from one domain may not be generalized enough to another domain. The goal of domain adaptation is to use samples from the source domain (labeled data) and the target domain (unlabeled data) to enable the model to achieve better performance in the target domain. Therefore, in the embodiment of the present application, it is necessary to use samples from the daytime image set and the nighttime image set to enable the trained semantic segmentation model to achieve domain adaptation and achieve better performance in data processing in the target domain.
[0069] Step 102: extracting daytime event information corresponding to the daytime image information, daytime content information corresponding to the daytime image information, and nighttime content information corresponding to the nighttime image information.
[0070] In the embodiment of the present application, an image motion extractor can be used to obtain the daytime event information E corresponding to the daytime image information in the extraction source domain. s , use the image content extractor to obtain the daytime content information C corresponding to the daytime image information in the source domain s The nighttime content information C corresponding to the nighttime image information of the target domain t , to bridge the gap between different modalities (images vs. events) and domains (daytime vs. nighttime).
[0071] Step 103: Use the daytime event information, the daytime content information, the nighttime content information and the training data as target training data.
[0072] In step 103, the training data obtained in step 101, and the daytime event information, daytime content information, and nighttime content information obtained in step 102 are integrated into target training data.
[0073] Step 104: Use the target training data to train the semantic segmentation model to be trained to obtain a target semantic segmentation model, where the target semantic segmentation model is used to perform semantic segmentation on the image to be segmented taken at night.
[0074] In an embodiment of the present application, after obtaining the target training data, the target training data is used to train the semantic segmentation model to be trained to obtain a target semantic segmentation model. The target semantic segmentation model is used to perform semantic segmentation on the image to be segmented taken at night to obtain a target image after semantic segmentation with a corresponding semantic category.
[0075] In one embodiment, since the event camera generates events when the pixel light intensity changes, it has a higher dynamic range and contrast, can better preserve details, and is particularly suitable for low-light scenes. Therefore, the image to be segmented taken at night can be an image of event information obtained by the event camera.
[0076] In an embodiment of the present application, training data is obtained; the training data includes a night image set and a day image set, the night image set includes night image information and night event information, and the day image set includes day image information with corresponding semantic categories; the day event information corresponding to the day image information, the day content information corresponding to the day image information, and the night content information corresponding to the night image information are extracted; the day event information, the day content information, the night content information, and the training data are used as target training data; the target training data is used to train the semantic segmentation model to be trained to obtain the target semantic segmentation model, and the target semantic segmentation model is used to perform semantic segmentation on the images to be segmented taken at night. The embodiment of the present application bridges the gap between the image modality and the event modality in the training data by calculating the day event information, and at the same time bridges the style difference between day and night in the training data by calculating the day content information and the night content information, so that the trained target semantic segmentation model can better understand the relationship between the image information and the event information, thereby significantly improving the performance of semantic segmentation of images taken at night.
[0077] Optionally, in step 102, extracting daytime event information corresponding to the daytime image information, daytime content information corresponding to the daytime image information, and nighttime content information corresponding to the nighttime image information includes:
[0078] Acquiring target daytime image information from the daytime image information;
[0079] Calculating target daytime event information corresponding to the target daytime image information according to the target daytime image information and adjacent daytime image information that is adjacent to the target daytime image information in time;
[0080] The target daytime event information corresponding to the target daytime image information is sorted to obtain the daytime event information corresponding to the daytime image information.
[0081] In this embodiment, the source domain (daytime image information) lacks corresponding event information, which prevents the model from effectively associating the image modality with the event modality. Considering that event information is essentially a reflection of the relative motion between the camera and the scene, and a single image lacks motion information, the embodiment of the present application designs an image motion extractor to extract relative motion information from two temporally adjacent daytime images as daytime event information E. s In addition, considering that the event camera records the logarithmic intensity changes of pixels, this embodiment simulates the generation process of event information by comparing the same pixel of two adjacent daytime images in the logarithmic domain.
[0082] In this embodiment, the daytime image information is traversed, and the currently traversed daytime image information is used as the target daytime image information I s . Calculate the target daytime event information E corresponding to the target daytime image information according to the target daytime image information and the adjacent daytime image information that is adjacent to the target daytime image information in time. s .
[0083] In one embodiment, for two time-adjacent daytime image information H and W are the image sizes of the daytime image information, and the target daytime image information I can be calculated by the following formula (1): k Corresponding target daytime event information E s .
[0084] E s =F Filter left(I k-1 ,I k ):
[0085] F Filter (I1,I2)=F Norm (F ClipIgn (F LogDiff (I1,I2)))
[0086] F LogDiff (I1,I2)=ln(I1+∈)-ln(I2+∈)
[0087]
[0088] Among them, E s I is the target daytime event information; k is the target daytime image information; I k-1is the adjacent daytime image information; left() represents the daytime image information used for calculation; F LogDiff (I1, I2) represents the difference between I1 and I2 in the logarithmic domain; ln() represents taking the logarithm; ∈ is a small scalar constant to prevent the logarithm from being zero; F ClipIgn The purpose of (x) is to clip larger values and ignore smaller values through two hyperparameters α and β; is the indicator function; sgn(·) is the sign function; F Norm (x) is the minimum-maximum normalization, which scales the value to the range of -1 to 1; min() takes the minimum value; max() takes the maximum value.
[0089] Arrange the target daytime event information corresponding to all target daytime image information to obtain the daytime event information E corresponding to the daytime image information s .
[0090] The embodiment of the present application bridges the gap between image modality and event modality by designing an image motion extractor, so that the trained target semantic segmentation model can better understand the relationship between images and events, thereby improving the performance of the target semantic segmentation model for night image segmentation.
[0091] Optionally, in step 102, extracting daytime event information corresponding to the daytime image information, daytime content information corresponding to the daytime image information, and nighttime content information corresponding to the nighttime image information includes:
[0092] Acquiring target daytime image information from the daytime image information;
[0093] Randomly shifting the target daytime image information to obtain shifted target daytime image information;
[0094] Calculating target daytime content information corresponding to the target daytime image information according to the translated target daytime image information and the target daytime image information;
[0095] Arranging the target daytime content information corresponding to the target daytime image information to obtain the daytime content information corresponding to the daytime image information;
[0096] Acquiring target nighttime image information from the nighttime image information;
[0097] Randomly translate the target nighttime image information to obtain translated target nighttime image information;
[0098] Calculating target nighttime content information corresponding to the target nighttime image information according to the translated target nighttime image information and the target nighttime image information;
[0099] The target nighttime content information corresponding to the target nighttime image information is sorted to obtain the nighttime content information corresponding to the nighttime image information.
[0100] Since night scenes are complex and varied, it is very challenging to construct a realistic night style, which means that it is very difficult to transfer the style of daytime images to nighttime styles. Therefore, the embodiment of the present application designs an image content extractor specifically for obtaining the content information of an image. By eliminating the style differences between daytime and nighttime images, both the source domain and the target domain are transferred to the intermediate domain, thereby avoiding the generation and use of nighttime styles, and thus eliminating the need to rely on a style transfer network.
[0101] In this embodiment, the daytime image information is traversed, and the currently traversed daytime image information is used as the target daytime image information I s .
[0102] The target daytime image information is randomly translated to obtain the translated target daytime image information, and the target daytime content information corresponding to the target daytime image information is calculated based on the translated target daytime image information and the target daytime image information. The target daytime content information corresponding to all target daytime image information is sorted to obtain the daytime content information C corresponding to the daytime image information. s .
[0103] Similarly, the night image information is traversed, and the currently traversed night image information is used as the target night image information I t .
[0104] The target night image information is randomly translated to obtain the translated target night image information, and the target night content information corresponding to the target night image information is calculated based on the translated target night image information and the target night image information. The target night content information corresponding to all target night image information is sorted to obtain the night content information C corresponding to the night image information t .
[0105] In an optional embodiment, given a grayscale image I, it can be randomly translated to the left / right and up / down by pixels γ to obtain I x±γ and I y±γ Then, the intermediate shared content domain can be generated as the content information of the image by the following formula (2).
[0106]
[0107] F Filter (I1,I2)=F Norm (F ClipIgn (F LogDiff (I1,I2)))
[0108] FLogDiff (I1,I2)=ln(I1+∈)-ln(I2+∈)
[0109]
[0110] Among them, C is the content information of the image; I is the grayscale image; I x±γ and I y±γ is the image after random translation; F LogDiff (I1, I2) represents the difference between I1 and I2 in the logarithmic domain; ln() represents taking the logarithm; ∈ is a small scalar constant to prevent the logarithm from being zero; F ClipIgn The purpose of (x) is to clip larger values and ignore smaller values through two hyperparameters α and β; is the indicator function; sgn(·) is the sign function; F Norm (x) is the minimum-maximum normalization, which scales the value to the range of -1 to 1; min() takes the minimum value; max() takes the maximum value.
[0111] By subtracting the translated image from the grayscale image itself, the pixels of the same color are erased, leaving only the pixels at the edge of the image, that is, the content information. s and night image information I t To obtain the daytime content information C s and night content information C t After converting the image information I into content information C, the domain-specific style information is largely eliminated, and only the domain-invariant content information is retained.
[0112] The embodiment of the present application bridges the gap between daytime style and nighttime style by designing an image content extractor, thereby avoiding the generation and use of nighttime style during the training process, and further improving the performance of the target semantic segmentation model for nighttime image segmentation without relying on a style transfer network.
[0113] Reference Figure 2 , is a training data processing diagram of an embodiment of a training method for a semantic segmentation model of the present application.
[0114] like Figure 2 As shown, the daytime image information I s Can be passed through the image motion extractor F FILTER , for two adjacent daytime image information I k and I k-1 Calculate the daytime event information E s , to narrow down the nighttime event information E of the image modality and target domain t The gap between event modes.
[0115] Source domain daytime image information I s and the randomly translated image I x±γ ,I y±γ Can be passed through the image content extractor F FILTER Calculate the daytime content information C s , night image information E of the target domain s The same method can be used to extract the image content through the image content extractor F FILTER Calculate night content information C s , to bridge the gap between daytime style and nighttime style.
[0116] Optionally, the semantic segmentation model to be trained is a teacher model, the teacher model has a corresponding student model, the student model includes an image encoder, an event encoder and a decoder, and in step 104, the use of the target training data to train the semantic segmentation model to be trained includes:
[0117] Inputting the daytime image information into the image encoder to obtain a first intermediate feature;
[0118] Inputting the daytime event information and the daytime content information into the event encoder to obtain a second intermediate feature;
[0119] fusing the first intermediate feature and the second intermediate feature to obtain a first cross-modal feature;
[0120] Inputting the first cross-modal feature into the decoder to obtain predicted student segmentation data;
[0121] Obtaining a student loss function according to the semantic category corresponding to the daytime image information and the predicted student segmentation data;
[0122] The teacher model is trained according to the student loss function, the night image information, the night event information and the night content information.
[0123] In this embodiment, the specific process of training the semantic segmentation model to be trained may include the following steps.
[0124] The daytime image information I s The image encoder of the student model is input to obtain the first intermediate feature, and the daytime event information E s and daytime content information C s The second intermediate features are input into the event encoder of the student model. In one embodiment, after the data is input into the image encoder and the event encoder of the student model, the first intermediate features and the second intermediate features of four different scales are output.
[0125] The first intermediate feature and the second intermediate feature are input into the cross-modal fusion module, and the intermediate features of the same scale in the first intermediate feature and the second intermediate feature are fused to obtain the first cross-modal feature X fs The embodiment of the present application fully integrates high-resolution coarse-grained features and low-resolution fine-grained features through a cross-modal fusion module, thereby improving the nighttime segmentation performance of the trained target semantic segmentation model.
[0126] The first cross-modal feature X fs Input decoder to get predicted student segmentation data P s , and according to the semantic category Y corresponding to the daytime image information s and predict student segmentation data P s , we get the student loss function Among them, the predicted student segmentation data is the predicted nighttime semantic segmentation output by the student model.
[0127] In an optional embodiment, the student loss function can be calculated by the following formula (3):
[0128]
[0129] in, is the student loss function; Fusion is a hyperparameter; P s is the predicted student segmentation data; Y is the semantic category corresponding to the daytime image information; H and W are the sizes of the daytime image information; C is the number of segmentation categories; δ(P) represents the softmax output of the prediction result P.
[0130] Afterwards, according to the student loss function Night Image Information I t , Night event information E t and night content information C t , train the teacher model. The trained teacher model is the target semantic segmentation model.
[0131] The embodiment of the present application uses a cross-modal fusion module to train the model, fully integrating high-resolution coarse-grained features and low-resolution fine-grained features, giving full play to the advantages of both, improving the performance of nighttime segmentation of the trained target semantic segmentation model, and making the model more robust.
[0132] Optionally, when training the semantic segmentation model to be trained, training the teacher model according to the student loss function, the night image information, the night event information and the night content information includes:
[0133] Inputting the night image information, the night event information and the night content information into the teacher model to obtain a pseudo label;
[0134] Obtaining a second cross-modal feature according to the nighttime image information, the nighttime event information, and the nighttime content information;
[0135] Inputting the second cross-modal feature into the student model to obtain predicted teacher segmentation data;
[0136] Obtaining a teacher loss function according to the pseudo-label and the predicted teacher segmentation data;
[0137] Using the teacher loss function to update the student model, to obtain an updated student model;
[0138] Update the teacher model according to the model parameters of the updated student model and the student loss function.
[0139] In this embodiment, the specific process of performing unsupervised training on the teacher-student network may include the following steps.
[0140] Night image information t , Night event information E t and night content information C t Input the teacher model with the same structure as the student model but different parameters, and get the pseudo label Y output by the teacher model t .
[0141] Based on the night image information t , Night event information E t and night content information C t , the image encoder, event encoder and cross-modal fusion module are used to process and obtain the second cross-modal feature X of the target domain ft , and the second cross-modal feature X ft Input the student model and get the predicted teacher segmentation data P t . Further, according to the pseudo label Y t and predict teacher segmentation data P t , we get the teacher loss function
[0142] Using the teacher loss function Update the student model to obtain an updated student model. Then, according to the model parameters and student loss function of the updated student model Updating the teacher model: In one embodiment, the teacher model may be updated by an exponential moving average (EMA) of relevant parameters of the student model.
[0143] The embodiment of the present application uses a teacher-student network for unsupervised training, and uses the parameters of the updated student model to update the teacher model, thereby improving the performance of nighttime segmentation of the trained target semantic segmentation model.
[0144] Optionally, when the cross-modal fusion module is used, fusing the first intermediate feature with the second intermediate feature to obtain a first cross-modal feature includes:
[0145] splicing intermediate features of the same scale in the first intermediate feature and the second intermediate feature to obtain a spliced intermediate feature;
[0146] Perform dimensionality restoration on the concatenated intermediate features to obtain the first cross-modal features.
[0147] In this embodiment, the first intermediate feature and the second intermediate feature are input into the cross-modal fusion module, and the intermediate features of the same scale in the first intermediate feature and the second intermediate feature are fused to obtain the first cross-modal feature X fs The specific process may be as follows.
[0148] In the channel dimension, the intermediate features of the same scale in the first intermediate feature and the second intermediate feature are concatenated to obtain the concatenated intermediate feature. Then, the concatenated intermediate feature is restored to its dimension through a 3*3 convolution layer to obtain the first cross-modal feature X fs .
[0149] The embodiment of the present application fully integrates high-resolution coarse-grained features and low-resolution fine-grained features through a cross-modal fusion module, thereby improving the nighttime segmentation performance of the trained target semantic segmentation model.
[0150] Reference Figure 3 , is a model training schematic diagram of an embodiment of a training method for a semantic segmentation model of the present application.
[0151] like Figure 3 As shown, the daytime image information I of the source domain data s The image encoder of the student model is input to obtain the first intermediate feature, and the daytime event information E s and daytime content information C s Input the event encoder of the student model to obtain the second intermediate feature
[0152] The first intermediate feature and the second intermediate feature are input into the cross-modal fusion module. In the channel dimension, the intermediate features of the same scale in the first intermediate feature and the second intermediate feature are spliced to obtain the spliced intermediate feature, and then the dimensionality of the spliced intermediate feature is restored through a 3*3 convolution layer to obtain the first cross-modal feature X fs .
[0153] The first cross-modal feature X fs Input decoder to get predicted student segmentation data P s , and according to the semantic category Y corresponding to the daytime image information s and predict student segmentation data P s , we get the student loss function
[0154] The night image information of the target domain data I t , Night event information E t and night content information C t Input the teacher model and get the pseudo label Y output by the teacher model t . The pseudo label Y t 、Night image information I t , Night event information E t and night content information C t , input the student model and get the teacher loss function
[0155] Using the teacher loss function Update the student model to obtain an updated student model. Then, according to the model parameters and student loss function of the updated student model The exponential moving average EMA updates the teacher model.
[0156] Reference Figure 4 , is a flowchart of an embodiment of a training method for a semantic segmentation model of the present application.
[0157] like Figure 4 As shown, the complete process of a training method for a semantic segmentation model of the present application can be expressed as follows.
[0158] Step 401: Input data is divided into labeled daytime images (source domain) and unlabeled images-event nighttime data (target domain).
[0159] Step 402: Design an image motion extractor to obtain event information of the source domain, and design an image content extractor to obtain content information of the source domain and the target domain.
[0160] Step 403: Input the source domain information into the image encoder and the event encoder to generate features of different scales.
[0161] Step 404: input to the cross-modal fusion module to obtain fused multi-modal features.
[0162] Step 405: Input the features into the decoder to obtain a prediction value, and use the source domain label for training to obtain a loss function.
[0163] Step 406: The target domain information is input into the teacher-student network for unsupervised training to obtain the teacher network as the final model.
[0164] In an embodiment of the present application, training data is obtained; the training data includes a night image set and a day image set, the night image set includes night image information and night event information, and the day image set includes day image information with corresponding semantic categories; the day event information corresponding to the day image information, the day content information corresponding to the day image information, and the night content information corresponding to the night image information are extracted; the day event information, the day content information, the night content information, and the training data are used as target training data; the target training data is used to train the semantic segmentation model to be trained to obtain the target semantic segmentation model, and the target semantic segmentation model is used to perform semantic segmentation on the images to be segmented taken at night. The embodiment of the present application bridges the gap between the image modality and the event modality in the training data by calculating the day event information, and at the same time bridges the style difference between day and night in the training data by calculating the day content information and the night content information, so that the trained target semantic segmentation model can better understand the relationship between the image information and the event information, thereby significantly improving the performance of semantic segmentation of images taken at night.
[0165] Based on the above embodiment, this embodiment further provides a semantic segmentation method for nighttime images, the method comprising:
[0166] Obtain an image to be segmented taken at night;
[0167] The target semantic segmentation model is used to perform semantic segmentation on the image to be segmented to obtain a target image after semantic segmentation; the target image after semantic segmentation has a corresponding semantic category; the target semantic segmentation model is the target semantic segmentation model described in the embodiment of the present application.
[0168] In this embodiment, the target semantic segmentation model trained by the above-mentioned semantic segmentation model training method embodiment can be used to perform semantic segmentation on night images.
[0169] Acquire the image to be segmented taken at night. In one embodiment, the image to be segmented taken at night may be night event information acquired by an event camera.
[0170] The target semantic segmentation model is used to perform semantic segmentation on the image to be segmented to obtain a target image after semantic segmentation; the target image after semantic segmentation has a corresponding semantic category. In one embodiment, the target semantic segmentation model is used to identify and segment different semantic regions of the image to be segmented, and these semantic regions may include roads, pedestrians, vehicles, etc.
[0171] In an embodiment of the present application, an image to be segmented taken at night is obtained; a target semantic segmentation model is used to perform semantic segmentation on the image to be segmented, and a target image after semantic segmentation is obtained; the target image after semantic segmentation has a corresponding semantic category; the target semantic segmentation model is the target semantic segmentation model described in the embodiment of the present application. The embodiment of the present application uses the trained target semantic segmentation model to perform semantic segmentation on night images, thereby improving the accuracy of semantic segmentation of night images, significantly improving the target semantic segmentation model's ability to understand and perceive night scenes, and further promoting the application of computer vision technology in night environments.
[0172] It should be noted that, for the method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited by the described order of actions, because according to the embodiments of the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present application.
[0173] On the basis of the above embodiments, this embodiment further provides a training device for a semantic segmentation model, which is applied to electronic devices such as terminal devices and servers.
[0174] Reference Figure 5 , shows a structural block diagram of an embodiment of a training device for a semantic segmentation model of the present application, which may specifically include the following modules:
[0175] The training data acquisition module 501 is used to acquire training data; the training data includes a night image set and a day image set, the night image set includes night image information and night event information, and the day image set includes day image information with corresponding semantic categories;
[0176] The training data extraction module 502 is used to extract the daytime event information corresponding to the daytime image information, the daytime content information corresponding to the daytime image information, and the nighttime content information corresponding to the nighttime image information;
[0177] A training data merging module 503, configured to use the daytime event information, the daytime content information, the nighttime content information and the training data as target training data;
[0178] The model training module 504 is used to train the semantic segmentation model to be trained using the target training data to obtain a target semantic segmentation model.
[0179] Optionally, the training data extraction module 502 includes:
[0180] A first target daytime image information acquisition submodule, used to acquire target daytime image information from the daytime image information;
[0181] a target daytime event information calculation submodule, configured to calculate the target daytime event information corresponding to the target daytime image information based on the target daytime image information and adjacent daytime image information that is temporally adjacent to the target daytime image information;
[0182] The daytime event information sorting submodule is used to sort the target daytime event information corresponding to the target daytime image information to obtain the daytime event information corresponding to the daytime image information.
[0183] Optionally, the training data extraction module 502 includes:
[0184] A second target daytime image information acquisition submodule, used to acquire target daytime image information from the daytime image information;
[0185] A first translation submodule is used to randomly translate the target daytime image information to obtain translated target daytime image information;
[0186] a target daytime content information calculation submodule, configured to calculate the target daytime content information corresponding to the target daytime image information according to the translated target daytime image information and the target daytime image information;
[0187] A daytime content information sorting submodule, used for sorting the target daytime content information corresponding to the target daytime image information, to obtain the daytime content information corresponding to the daytime image information;
[0188] A target nighttime image information acquisition submodule, used to acquire target nighttime image information from the nighttime image information;
[0189] A second translation submodule is used to randomly translate the target nighttime image information to obtain translated target nighttime image information;
[0190] A target nighttime content information calculation submodule, configured to calculate target nighttime content information corresponding to the target nighttime image information according to the translated target nighttime image information and the target nighttime image information;
[0191] The nighttime content information sorting submodule is used to sort the target nighttime content information corresponding to the target nighttime image information to obtain the nighttime content information corresponding to the nighttime image information.
[0192] Optionally, the semantic segmentation model to be trained is a teacher model, the teacher model has a corresponding student model, the student model includes an image encoder, an event encoder and a decoder, and the model training module 504 includes:
[0193] A first intermediate feature calculation submodule, used for inputting the daytime image information into the image encoder to obtain a first intermediate feature;
[0194] An event encoder submodule, configured to input the daytime event information and the daytime content information into the event encoder to obtain a second intermediate feature;
[0195] a first cross-modal feature calculation submodule, configured to fuse the first intermediate feature with the second intermediate feature to obtain a first cross-modal feature;
[0196] A decoder submodule, configured to input the first cross-modal feature into the decoder to obtain predicted student segmentation data;
[0197] A student loss function calculation submodule, used to obtain a student loss function according to the semantic category corresponding to the daytime image information and the predicted student segmentation data;
[0198] The teacher model training submodule is used to train the teacher model according to the student loss function, the night image information, the night event information and the night content information.
[0199] Optionally, the teacher model training submodule is further used to:
[0200] Inputting the night image information, the night event information and the night content information into the teacher model to obtain a pseudo label;
[0201] Obtaining a second cross-modal feature according to the nighttime image information, the nighttime event information, and the nighttime content information;
[0202] Inputting the second cross-modal feature into the student model to obtain predicted teacher segmentation data;
[0203] Obtaining a teacher loss function according to the pseudo-label and the predicted teacher segmentation data;
[0204] Using the teacher loss function to update the student model, to obtain an updated student model;
[0205] Update the teacher model according to the model parameters of the updated student model and the student loss function.
[0206] Optionally, the first cross-modal feature calculation submodule is further used to:
[0207] splicing intermediate features of the same scale in the first intermediate feature and the second intermediate feature to obtain a spliced intermediate feature;
[0208] Perform dimensionality restoration on the concatenated intermediate features to obtain the first cross-modal features.
[0209] In an embodiment of the present application, training data is obtained; the training data includes a night image set and a day image set, the night image set includes night image information and night event information, and the day image set includes day image information with corresponding semantic categories; the day event information corresponding to the day image information, the day content information corresponding to the day image information, and the night content information corresponding to the night image information are extracted; the day event information, the day content information, the night content information, and the training data are used as target training data; the target training data is used to train the semantic segmentation model to be trained to obtain the target semantic segmentation model, and the target semantic segmentation model is used to perform semantic segmentation on the images to be segmented taken at night. The embodiment of the present application bridges the gap between the image modality and the event modality in the training data by calculating the day event information, and at the same time bridges the style difference between day and night in the training data by calculating the day content information and the night content information, so that the trained target semantic segmentation model can better understand the relationship between the image information and the event information, thereby significantly improving the performance of semantic segmentation of images taken at night.
[0210] Based on the above embodiment, this embodiment further provides a semantic segmentation device for nighttime images, the device comprising:
[0211] The module for acquiring the image to be segmented is used to acquire the image to be segmented taken at night;
[0212] A semantic segmentation module is used to perform semantic segmentation on the image to be segmented using a target semantic segmentation model to obtain a target image after semantic segmentation; the target image after semantic segmentation has a corresponding semantic category; the target semantic segmentation model is the target semantic segmentation model described in the embodiment of the present application.
[0213] In an embodiment of the present application, an image to be segmented taken at night is obtained; a target semantic segmentation model is used to perform semantic segmentation on the image to be segmented, and a target image after semantic segmentation is obtained; the target image after semantic segmentation has a corresponding semantic category; the target semantic segmentation model is the target semantic segmentation model described in the embodiment of the present application. The embodiment of the present application uses the trained target semantic segmentation model to perform semantic segmentation on night images, thereby improving the accuracy of semantic segmentation of night images, significantly improving the target semantic segmentation model's ability to understand and perceive night scenes, and further promoting the application of computer vision technology in night environments.
[0214] The embodiment of the present application also provides a non-volatile readable storage medium, which stores one or more modules (programs). When the one or more modules are applied to a device, the device can execute instructions (instructions) of each method step in the embodiment of the present application.
[0215] The present application embodiment provides one or more machine-readable media on which instructions are stored, and when executed by one or more processors, an electronic device executes one or more of the methods described in the above embodiments. In the present application embodiment, the electronic device includes various types of devices such as terminal devices and servers (clusters).
[0216] The embodiments of the present disclosure may be implemented as a device configured as desired using any appropriate hardware, firmware, software, or any combination thereof, and the device may include electronic devices such as terminal devices and servers (clusters). Figure 6 An exemplary apparatus 600 that may be used to implement various embodiments described in this application is schematically illustrated.
[0217] For one embodiment, Figure 6 An exemplary apparatus 600 is shown having one or more processors 602, a control module (chip set) 604 coupled to at least one of the processor(s) 602, a memory 606 coupled to the control module 604, a non-volatile memory (NVM) / storage device 608 coupled to the control module 604, one or more input / output devices 610 coupled to the control module 604, and a network interface 612 coupled to the control module 604.
[0218] The processor 602 may include one or more single-core or multi-core processors, and the processor 602 may include any combination of general-purpose processors or special-purpose processors (such as graphics processors, application processors, baseband processors, etc.). In some embodiments, the device 600 can be used as a terminal device, server (cluster), etc. described in the embodiments of the present application.
[0219] In some embodiments, the apparatus 600 may include one or more computer-readable media (e.g., memory 606 or NVM / storage device 608) having instructions 614 and one or more processors 602 combined with the one or more computer-readable media and configured to execute the instructions 614 to implement a module to perform the actions described in the present disclosure.
[0220] For one embodiment, control module 604 may include any suitable interface controller to provide any suitable interface to at least one of processor(s) 602 and / or any suitable device or component in communication with control module 604 .
[0221] The control module 604 may include a memory controller module to provide an interface to the memory 606. The memory controller module may be a hardware module, a software module, and / or a firmware module.
[0222] The memory 606 may be used, for example, to load and store data and / or instructions 614 for the device 600. For one embodiment, the memory 606 may include any suitable volatile memory, such as a suitable DRAM. In some embodiments, the memory 606 may include double data rate type four synchronous dynamic random access memory (DDR4 SDRAM).
[0223] For one embodiment, control module 604 may include one or more input / output controllers to provide an interface to NVM / storage device 608 and input / output device(s) 610 .
[0224] For example, NVM / storage 608 may be used to store data and / or instructions 614. NVM / storage 608 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable non-volatile storage device(s) (e.g., one or more hard disk drives (HDDs), one or more compact disk (CD) drives, and / or one or more digital versatile disk (DVD) drives).
[0225] NVM / storage device 608 may include storage resources that are physically part of the device on which apparatus 600 is installed, or it may be accessible to the device without being part of the device. For example, NVM / storage device 608 may be accessed via input / output device(s) 610 over a network.
[0226] (One or more) input / output devices 610 may provide an interface for the apparatus 600 to communicate with any other appropriate device, and the input / output device 610 may include a communication component, an audio component, a sensor component, etc. The network interface 612 may provide an interface for the apparatus 600 to communicate through one or more networks, and the apparatus 600 may wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols, for example, access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G, 5G, etc., or a combination thereof for wireless communication.
[0227] For one embodiment, at least one of the processor(s) 602 may be packaged together with the logic of one or more controllers (e.g., a memory controller module) of the control module 604. For one embodiment, at least one of the processor(s) 602 may be packaged together with the logic of one or more controllers of the control module 604 to form a system-in-package (SiP). For one embodiment, at least one of the processor(s) 602 may be integrated on the same die with the logic of one or more controllers of the control module 604. For one embodiment, at least one of the processor(s) 602 may be integrated on the same die with the logic of one or more controllers of the control module 604 to form a system-on-chip (SoC).
[0228] In various embodiments, the device 600 may be, but is not limited to, a terminal device such as a server, a desktop computing device, or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a tablet computer, a netbook, etc.). In various embodiments, the device 600 may have more or fewer components and / or different architectures. For example, in some embodiments, the device 600 includes one or more cameras, a keyboard, a liquid crystal display (LCD) screen (including a touch screen display), a non-volatile memory port, multiple antennas, a graphics chip, an application-specific integrated circuit (ASIC), and a speaker.
[0229] Among them, the main control chip can be used as a processor or control module in the detection device, sensor data, location information, etc. are stored in a memory or NVM / storage device, the sensor group can be used as an input / output device, and the communication interface may include a network interface.
[0230] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0231] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0232] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable semantic segmentation model training terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable semantic segmentation model training terminal device generate instructions for implementing the process Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0233] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable semantic segmentation model training terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which is implemented in the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0234] These computer program instructions can also be loaded onto a computer or other programmable semantic segmentation model training terminal device, so that a series of operation steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable terminal device provide for implementing the process Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0235] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.
[0236] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.
[0237] The above is a detailed introduction to a training method and device for a semantic segmentation model, an electronic device and a storage medium provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.
Claims
1. A method for training a semantic segmentation model, characterized in that: The method comprises: Acquire training data; the training data includes a night image set and a day image set, the night image set includes night image information and night event information, and the day image set includes day image information with corresponding semantic categories; Extracting daytime event information corresponding to the daytime image information, daytime content information corresponding to the daytime image information, and nighttime content information corresponding to the nighttime image information; using the daytime event information, the daytime content information, the nighttime content information, and the training data as target training data; The target training data is used to train a semantic segmentation model to be trained to obtain a target semantic segmentation model, and the target semantic segmentation model is used to perform semantic segmentation on the image to be segmented taken at night.
2. The method according to claim 1, characterized in that The extracting the daytime event information corresponding to the daytime image information, the daytime content information corresponding to the daytime image information, and the nighttime content information corresponding to the nighttime image information includes: Acquiring target daytime image information from the daytime image information; Calculating target daytime event information corresponding to the target daytime image information according to the target daytime image information and adjacent daytime image information that is adjacent to the target daytime image information in time; The target daytime event information corresponding to the target daytime image information is sorted to obtain the daytime event information corresponding to the daytime image information.
3. The method according to claim 1, characterized in that The extracting the daytime event information corresponding to the daytime image information, the daytime content information corresponding to the daytime image information, and the nighttime content information corresponding to the nighttime image information includes: Acquiring target daytime image information from the daytime image information; Randomly shifting the target daytime image information to obtain shifted target daytime image information; Calculating target daytime content information corresponding to the target daytime image information according to the translated target daytime image information and the target daytime image information; Arranging the target daytime content information corresponding to the target daytime image information to obtain the daytime content information corresponding to the daytime image information; Acquiring target nighttime image information from the nighttime image information; Randomly translate the target nighttime image information to obtain translated target nighttime image information; Calculating target nighttime content information corresponding to the target nighttime image information according to the translated target nighttime image information and the target nighttime image information; The target nighttime content information corresponding to the target nighttime image information is sorted to obtain the nighttime content information corresponding to the nighttime image information.
4. The method according to claim 1, characterized in that: The semantic segmentation model to be trained is a teacher model, the teacher model has a corresponding student model, the student model includes an image encoder, an event encoder and a decoder, and the semantic segmentation model to be trained using the target training data includes: Inputting the daytime image information into the image encoder to obtain a first intermediate feature; Inputting the daytime event information and the daytime content information into the event encoder to obtain a second intermediate feature; fusing the first intermediate feature and the second intermediate feature to obtain a first cross-modal feature; Inputting the first cross-modal feature into the decoder to obtain predicted student segmentation data; Obtaining a student loss function according to the semantic category corresponding to the daytime image information and the predicted student segmentation data; The teacher model is trained according to the student loss function, the night image information, the night event information and the night content information.
5. The method according to claim 4, characterized in that The training of the teacher model according to the student loss function, the night image information, the night event information and the night content information comprises: Inputting the night image information, the night event information and the night content information into the teacher model to obtain a pseudo label; Obtaining a second cross-modal feature according to the nighttime image information, the nighttime event information, and the nighttime content information; Inputting the second cross-modal feature into the student model to obtain predicted teacher segmentation data; Obtaining a teacher loss function according to the pseudo-label and the predicted teacher segmentation data; Using the teacher loss function to update the student model, to obtain an updated student model; Update the teacher model according to the model parameters of the updated student model and the student loss function.
6. The method according to claim 4, characterized in that The fusing the first intermediate feature and the second intermediate feature to obtain a first cross-modal feature includes: splicing intermediate features of the same scale in the first intermediate feature and the second intermediate feature to obtain a spliced intermediate feature; Perform dimensionality restoration on the concatenated intermediate features to obtain the first cross-modal features.
7. A semantic segmentation method for nighttime images, characterized in that: The method comprises: Obtain an image to be segmented taken at night; The target semantic segmentation model is used to perform semantic segmentation on the image to be segmented to obtain a target image after semantic segmentation; the target image after semantic segmentation has a corresponding semantic category; the target semantic segmentation model is the target semantic segmentation model described in any one of claims 1-6.
8. A training device for a semantic segmentation model, characterized in that: The device comprises: A training data acquisition module, used to acquire training data; the training data includes a night image set and a day image set, the night image set includes night image information and night event information, and the day image set includes day image information with corresponding semantic categories; A training data extraction module, used to extract daytime event information corresponding to the daytime image information, daytime content information corresponding to the daytime image information, and nighttime content information corresponding to the nighttime image information; a training data merging module, configured to use the daytime event information, the daytime content information, the nighttime content information and the training data as target training data; The model training module is used to train the semantic segmentation model to be trained using the target training data to obtain a target semantic segmentation model.
9. An electronic device, characterized in that: include: processor; and A memory having executable code stored thereon, which, when executed, enables the processor to execute the training method of the semantic segmentation model as described in one or more of claims 1-7.
10. One or more machine-readable media having executable codes stored thereon, which, when executed, cause a processor to execute a method for training a semantic segmentation model as described in one or more of claims 1-7.