Nighttime semantic segmentation method, nighttime semantic segmentation model training method and device

Abstract

This invention belongs to the field of nighttime semantic segmentation, and specifically relates to a nighttime semantic segmentation method, a nighttime semantic segmentation model training method, and an apparatus. The method includes constructing and training a nighttime semantic segmentation model, inputting an image of a nighttime scene to be segmented into the trained model to obtain a pixel-level segmentation map. The nighttime semantic segmentation model includes a feature encoder, a spatial domain branch, a frequency domain branch, a multi-frequency fully connected layer module, a multiplier, a reshaping module, and a hybrid domain fusion module. The nighttime semantic segmentation method and apparatus of this invention can effectively improve the segmentation accuracy in nighttime scenes by fusing frequency information into spatial features, thereby enhancing the generalization and robustness of the segmentation network. The training method and apparatus of the nighttime semantic segmentation model used in this invention can effectively improve the accuracy of the nighttime semantic segmentation model.

Description

Technical Field

[0001] This invention belongs to the field of semantic segmentation in nighttime scenes, and specifically relates to a nighttime semantic segmentation method, a nighttime semantic segmentation model training method, and an apparatus. Background Technology

[0002] In recent years, with the rapid development of mobile robot technology and autonomous vehicles, autonomous vehicles have gradually become the first generation of digital robots trusted by people. Thanks to the increased computing speed of computers, enhanced hardware computing power, and continuous advancements in deep learning technology, cars can achieve autonomous driving capabilities similar to humans in various outdoor scenarios, ensuring driving safety and stability. In this process, semantic segmentation technology plays an irreplaceable role. It can accurately identify and segment the position, quantity, and type of target objects such as roads, pedestrians, and vehicles, thereby providing intelligent decision support for vehicles, effectively avoiding obstacles, and improving the safety and robustness of autonomous driving systems.

[0003] However, low-light conditions in nighttime scenes present new challenges for semantic segmentation in autonomous driving. Nighttime images are typically characterized by low contrast, blurriness, and high noise, leading to blurred target edges and loss of detail when extracting spatial domain features using traditional methods, thus reducing segmentation accuracy. While some methods improve image quality through image enhancement techniques (such as contrast enhancement and histogram equalization) or expand training data through deep learning model design and generative adversarial networks (GANs), these methods primarily focus on spatial domain feature extraction and optimization, with insufficient utilization of frequency information. In fact, frequency domain information in images, especially the accurate description of target boundaries by high-frequency features and the structural supplementation of the global background by low-frequency features, can provide stronger robustness and auxiliary information for semantic segmentation. However, existing segmentation methods fail to fully integrate frequency domain information, limiting the model's performance in nighttime scenes. Summary of the Invention

[0004] To address the aforementioned issues, this invention provides a nighttime semantic segmentation method, a nighttime semantic segmentation model training method, and an apparatus. First, the frequency domain branch transforms spatial features into frequency information using a two-dimensional discrete cosine transform (DCT), enabling the network to extract high-frequency details and low-frequency structures from the image, helping the segmentation model better capture target boundaries and background features. Next, the Multi-frequency Fully Connected Layer (MFC) module dynamically weights and adjusts the frequency information. By calculating weights and multiplying them pixel-by-pixel with the input frequency information, the expression of frequency features is further enhanced, allowing important spectral features to be better highlighted and improving the accuracy of image segmentation. Finally, the Hybrid Domain Fusion (HDF) module fuses spatial and frequency features, optimizing the model's comprehensive utilization of both. By effectively combining spatial and frequency information, the comprehensiveness and robustness of image segmentation are improved.

[0005] In a first aspect of the present invention, a nighttime semantic segmentation method is provided, the method comprising:

[0006] Obtain the image to be segmented;

[0007] The image to be segmented is input into a nighttime semantic segmentation model; the nighttime semantic segmentation model includes a feature encoder, a spatial branch, a frequency branch, a multi-frequency fully connected layer module, a multiplier, a reshaping module, and a hybrid domain fusion module;

[0008] The original feature map of the image to be segmented is obtained by extracting features from the image to be segmented using a feature encoder.

[0009] The original feature map of the image to be segmented is enhanced by spatial domain branching to obtain the spatial feature map of the image to be segmented.

[0010] Frequency information of the image to be segmented is obtained by extracting the frequency from the original feature map of the image to be segmented through frequency domain branching.

[0011] The frequency information of the image to be segmented is integrated by a multi-frequency fully connected layer module to obtain the weights of the frequency information of the image to be segmented.

[0012] The frequency information and weights of the image to be segmented are multiplied element by element by a multiplier to obtain the adjusted frequency information of the image to be segmented.

[0013] The frequency information of the image to be segmented is adjusted by the reshaping module to obtain the frequency feature map of the image to be segmented.

[0014] The spatial feature map and frequency feature map of the image to be segmented are fused by the hybrid domain fusion module to obtain the semantic segmentation result of the image to be segmented.

[0015] In a second aspect, the present invention also provides a method for training a nighttime semantic segmentation model, the method comprising:

[0016] Obtain a training segmentation image, wherein the training segmentation image is a training image of a night scene, and the training image includes pixel-by-pixel segmentation map labels;

[0017] The trained segmented image is input into the nighttime semantic segmentation model; the nighttime semantic segmentation model includes a feature encoder, a spatial branch, a frequency branch, a multi-frequency fully connected layer module, a multiplier, a reshaping module, and a hybrid domain fusion module;

[0018] The training segmentation image is subjected to feature extraction by a feature encoder to obtain the original feature map of the training segmentation image;

[0019] The original segmentation map of the training segmentation image is enhanced by spatial domain branching to obtain the spatial feature map of the training segmentation image;

[0020] Frequency information of the training segmentation image is obtained by extracting the frequency of the original segmentation map of the training segmentation image through frequency domain branching.

[0021] The frequency information of the training segmentation image is integrated by a multi-frequency fully connected layer module to obtain the weights of the frequency information of the training segmentation image.

[0022] The frequency information and weights of the training segmentation image are multiplied element-wise by a multiplier to obtain the adjusted frequency information of the training segmentation image.

[0023] The frequency information of the training segmentation image is adjusted by the reshaping module to obtain the frequency feature map of the training segmentation image;

[0024] The spatial feature map and frequency feature map of the training segmentation image are fused by the hybrid domain fusion module to obtain the semantic segmentation result of the training segmentation image;

[0025] The nighttime semantic segmentation model is trained based on the semantic segmentation results of the training segmented image to obtain the trained nighttime semantic segmentation model.

[0026] In a third aspect, the present invention also provides a nighttime semantic segmentation apparatus, the apparatus comprising:

[0027] The first acquisition unit is used to acquire the image to be segmented;

[0028] The first extraction unit is used to extract features from the image to be segmented through a feature encoder to obtain the original feature map of the image to be segmented.

[0029] The first spatial unit is used to enhance the original feature map of the image to be segmented through spatial domain branching to obtain the spatial feature map of the image to be segmented.

[0030] The first frequency unit is used to extract the frequency of the original feature map of the image to be segmented through frequency domain branching to obtain the frequency information of the image to be segmented.

[0031] The first integration unit is used to integrate the frequency information of the image to be segmented through the multi-frequency fully connected layer module to obtain the weight of the frequency information of the image to be segmented.

[0032] The first multiplication unit is used to multiply the frequency information and weights of the image to be segmented element by element through a multiplier to obtain the adjusted frequency information of the image to be segmented.

[0033] The first reshaping unit is used to adjust the adjustment frequency information of the image to be segmented through the reshaping module to obtain the frequency feature map of the image to be segmented.

[0034] The first fusion unit is used to fuse the spatial feature map and frequency feature map of the image to be segmented through the hybrid domain fusion module to obtain the semantic segmentation result of the image to be segmented.

[0035] The image to be segmented is input into a nighttime semantic segmentation model, which includes a feature encoder, a spatial branch, a frequency branch, a multi-frequency fully connected layer module, a multiplier, a reshaping module, and a hybrid domain fusion module.

[0036] In a fourth aspect, the present invention also provides a nighttime semantic segmentation model training apparatus, the apparatus comprising:

[0037] The second acquisition unit is used to acquire the training segmentation image;

[0038] The second extraction unit is used to extract features from the training segmentation image through a feature encoder to obtain the original feature map of the training segmentation image;

[0039] The second spatial unit is used to enhance the original segmentation map of the training segmentation image through spatial domain branching to obtain the spatial feature map of the training segmentation image;

[0040] The second frequency unit is used to extract the frequency of the original segmentation map of the training segmentation image through frequency domain branching to obtain the frequency information of the training segmentation image;

[0041] The second integration unit is used to integrate the frequency information of the training segmentation image through the multi-frequency fully connected layer module to obtain the weights of the frequency information of the training segmentation image.

[0042] The second multiplication unit is used to multiply the frequency information and weights of the training segmentation image element by element through a multiplier to obtain the adjusted frequency information of the training segmentation image.

[0043] The second reshaping unit is used to adjust the adjustment frequency information of the training segmentation image through the reshaping module to obtain the frequency feature map of the training segmentation image.

[0044] The second fusion unit is used to fuse the spatial feature map and frequency feature map of the training segmentation image through the hybrid domain fusion module to obtain the semantic segmentation result of the training segmentation image;

[0045] The first training unit is used to train the nighttime semantic segmentation model based on the semantic segmentation results of the training segmented image, so as to obtain the trained nighttime semantic segmentation model.

[0046] The trained segmented image is input into a nighttime semantic segmentation model, which includes a feature encoder, a spatial branch, a frequency branch, a multi-frequency fully connected layer module, a multiplier, a reshaping module, and a hybrid domain fusion module.

[0047] The beneficial effects of this invention are:

[0048] This invention significantly improves the accuracy and robustness of semantic segmentation in nighttime scenes through a nighttime semantic segmentation method, a nighttime semantic segmentation model training method, and a device. First, the frequency domain branch transforms spatial features into frequency information using a two-dimensional discrete cosine transform (DCT), enabling the network to extract high-frequency details and low-frequency structures in the image, helping the segmentation model better capture target boundaries and background features. Next, the MFC (Multi-frequency Fully Connected Layer) module dynamically weights and adjusts the frequency information, further enhancing the expression of frequency features by calculating weights and multiplying them pixel-by-pixel with the input frequency information. This results in better highlighting of important spectral features and improving the accuracy of image segmentation. Finally, the HDF (Hybrid Domain Fusion) module fuses spatial and frequency features, optimizing the model's comprehensive utilization of both. By effectively combining spatial and frequency information, the comprehensiveness and robustness of image segmentation are improved. This method not only effectively addresses the problems of low contrast, blur, and noise in nighttime scenes but also optimizes computational efficiency and training results, exhibiting strong versatility and scalability, and can be widely applied to image segmentation tasks in various low-light and complex environments. Attached Figure Description

[0049] Figure 1 This is a flowchart of a nighttime semantic segmentation method according to an embodiment of the present invention;

[0050] Figure 2 This is a schematic diagram of the nighttime semantic segmentation model according to an embodiment of the present invention;

[0051] Figure 3 This is a schematic diagram of the structure of the multi-frequency fully connected layer MFC unit according to an embodiment of the present invention;

[0052] Figure 4 This is a flowchart of a nighttime semantic segmentation model training method according to an embodiment of the present invention;

[0053] Figure 5 This is a schematic diagram of a nighttime semantic segmentation device according to an embodiment of the present invention;

[0054] Figure 6 This is a schematic diagram of a nighttime semantic segmentation model training device according to an embodiment of the present invention;

[0055] Figure 7 This is a schematic diagram of the structure of the first training unit in an embodiment of the present invention. Detailed Implementation

[0056] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] It should be understood that, unless otherwise stated, the descriptions of "first," "second," etc., appearing in the embodiments of this application are only for illustration and to distinguish the objects being described, and have no order, nor do they indicate any special limitation on the number of devices in the embodiments of this application, and cannot constitute any limitation on the embodiments of this application.

[0058] It should also be understood that specific features, structures, or characteristics relating to embodiments in the specification are included in at least one embodiment of this application. Furthermore, these specific features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0059] Furthermore, the terms “comprising” and “having”, and any variations thereof, are intended to cover non-exclusive inclusion, such that a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such processes, methods, products, or devices.

[0060] This application's embodiments may relate to Computer Vision (CV) technology within artificial intelligence. Computer vision is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in recognizing and measuring targets, and further performs image processing to create images more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision researches related theories and technologies, attempting to establish artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and other technologies, as well as common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0061] This application provides a nighttime semantic segmentation method, a nighttime semantic segmentation model training method, and an apparatus, which can help improve the accuracy of nighttime scene segmentation results.

[0062] This application's embodiments can be used in any scenario requiring nighttime semantic segmentation, including autonomous driving, security monitoring, drone inspection, and nighttime navigation. For example, in autonomous driving scenarios, nighttime semantic segmentation can help vehicles identify key elements such as road signs, pedestrians, other vehicles, and obstacles, enabling accurate judgments and reactions even in low-light conditions. This application's embodiments can accurately segment each key element, thus improving the accuracy of nighttime scene segmentation results.

[0063] Figure 1 This is a flowchart of a nighttime semantic segmentation method according to an embodiment of the present invention, as follows: Figure 1 As shown, the method can be performed by any electronic device with data processing capabilities; for example, the electronic device can be implemented as a server, and this application does not limit it. The method includes:

[0064] 101. Obtain the image to be segmented;

[0065] In this embodiment of the invention, the image to be segmented may be a road environment photo collected in an autonomous driving scenario, or a photo of industrial parts in an industrial quality inspection scenario, etc., and this embodiment does not limit it.

[0066] 102. Input the image to be segmented into the nighttime semantic segmentation model; the nighttime semantic segmentation model includes a feature encoder, a spatial branch, a frequency branch, a multi-frequency fully connected layer module, a multiplier, a reshaping module, and a hybrid domain fusion module;

[0067] Please see Figure 2 After the nighttime image to be segmented is input into the nighttime semantic segmentation model, each pixel in the input image is classified, so that pixels belonging to the same object or having the same semantic meaning are labeled as the same category, and different colors are used to distinguish pixels of different categories to obtain the semantic segmentation result of the image to be segmented; for example, the dark purple area occupies the lower half of the image, representing the ground, the light purple area occupies the right part of the image, representing the sidewalk, the green area is in the middle left of the image, representing trees or vegetation, the red area is in the lower right corner of the image, representing pedestrians, and so on. The nighttime semantic segmentation module is mainly divided into spatial domain branch and frequency domain branch. After the image is input into the nighttime semantic segmentation module, the original feature map of the input image is first extracted by the feature encoder; then, the original feature map is processed by the spatial domain branch and the frequency branch respectively. The frequency feature map of the input image is learned by the multi-frequency fully connected layer module of the frequency domain branch, and the spatial feature map of the input image is extracted by the spatial domain branch; then, the spatial feature map and the frequency feature map of the input image are integrated by the multiplier, the reshaping module and the hybrid domain fusion module to obtain the segmentation feature map of the input image.

[0068] 103. Extract features from the image to be segmented using a feature encoder to obtain the original feature map of the image to be segmented;

[0069] In this embodiment of the invention, the feature encoder processes the input image to be segmented through a series of mathematical transformations and calculations. During this process, it extracts key features from the image to be segmented and transforms these features into vector representations of a specific dimension, which are the original feature maps of the image to be segmented. These vector representations can capture the inherent information and patterns of the image to be segmented, enabling subsequent models to learn and make decisions more effectively.

[0070] It should be noted that the feature encoder can be a common neural network. For example, convolutional neural networks (CNNs) are commonly used for feature encoding of image data. They extract local and global features of an image step by step through operations such as convolutional layers and pooling layers, transforming the original image into a high-dimensional feature vector. This invention does not impose specific limitations on this.

[0071] 104. Enhance the original feature map of the image to be segmented by spatial domain branching to obtain the spatial feature map of the image to be segmented;

[0072] In some embodiments of the present invention, the spatial branch enhances the original feature map of the image to be segmented by a pyramid pooling module. The pyramid pooling module pools the original feature map F of the input image to be segmented at four scales: 1×1, 2×2, 3×3, and 6×6, and upsamples the pooling results. After upsampling, the multi-scale feature maps after pooling are stitched together by channel to form a feature map F0 containing multi-scale information. Finally, the feature map F0 is fused with the input feature map F to generate a spatial feature map F1 containing rich contextual information.

[0073] 105. Frequency extraction is performed on the original feature map of the image to be segmented through frequency domain branching to obtain the frequency information of the image to be segmented;

[0074] In this embodiment of the invention, a two-dimensional discrete cosine transform is used to convert the spatial feature map of the image to be segmented in the spatial domain into frequency information in the frequency domain. This allows for the analysis of low-frequency and high-frequency features of the image through different frequency components. The low-frequency component mainly describes the global shape and coarse variations of the image to be segmented, while the high-frequency component contains detailed and texture information. Frequency information can help improve the effectiveness of image segmentation, compression, denoising, and other operations.

[0075] 106. The frequency information of the image to be segmented is integrated through a multi-frequency fully connected layer module to obtain the weights of the frequency information of the image to be segmented;

[0076] In this embodiment of the invention, the multi-frequency fully connected layer module is as follows: Figure 3 As shown, after the frequency information is transformed by DCT, the frequency information of each channel is obtained. The frequency information of each channel is then concatenated and processed by a fully connected layer to obtain the adjusted frequency information of the image to be segmented.

[0077] In this embodiment of the invention, step 106 includes:

[0078] The frequency information S of the image to be segmented is divided along the channel dimension to obtain the frequency information S0, S1, S2...S of the image in each channel. n-1 , where n represents the number of channels;

[0079] The frequency information S0, S1, S2...S of the image to be segmented in each channel is defined as follows: n-1 By inputting each input into a fully connected layer, the weights W0, W1…W of the image to be segmented in each channel are obtained. n-1 Then, the weights of these n channels are concatenated to obtain the weight matrix W.

[0080] Specifically, fully connected layers can adaptively adjust the importance of each channel, helping the network focus on the more informative channels, thereby improving its ability to express frequency features.

[0081] 107. The frequency information and weights of the image to be segmented are multiplied element by element by a multiplier to obtain the adjusted frequency information of the image to be segmented.

[0082] In this embodiment of the invention, considering that the frequency information of the image to be segmented contains components of different spatial frequencies, the high-frequency components typically correspond to details and edge information in the image, while the low-frequency components are related to the overall structure and background of the image. Therefore, by performing element-wise multiplication with weights, for example, multiplying the weight W of the frequency information of the image to be segmented by the dot product of the frequency information S of the image to be segmented to obtain the adjusted weight W', certain frequency components can be selectively enhanced or weakened, thereby highlighting or suppressing specific features in the image. For example, if it is desired to highlight details in the image, the weight of the high-frequency components can be appropriately increased, so that the high-frequency information is enhanced in the adjusted frequency information. By adjusting the frequency information, this embodiment of the invention can enhance or suppress different features in the image to be segmented, thereby improving the effect of subsequent processing tasks. This adjustment can make features related to the segmentation target more prominent, making it easier to identify and distinguish target objects from the background in nighttime scenes.

[0083] 108. The frequency information of the image to be segmented is adjusted by the reshaping module to obtain the frequency feature map of the image to be segmented;

[0084] In this embodiment of the invention, considering that the dimension and structure of the adjusted frequency information after multiplication may not be directly applicable to subsequent processing tasks, one of the functions of the reshaping module is to adjust the dimension and structure of this information so that it can better match the input requirements of subsequent models. For example, the weights W' are reshaped to the same shape as the spatial feature map F1 of the image to be segmented, resulting in the frequency feature map F2 of the image to be segmented. This provides a more effective feature representation for the subsequent nighttime semantic segmentation model, thereby improving the model's performance. The frequency feature map obtained through reshaping can better retain and highlight information related to the segmentation target, making it easier for the model to learn features that distinguish different objects or regions, thereby improving the accuracy and precision of segmentation.

[0085] 109. The spatial feature map and frequency feature map of the image to be segmented are fused by the hybrid domain fusion module to obtain the semantic segmentation result of the image to be segmented.

[0086] In this embodiment of the invention, step 109 includes:

[0087] The spatial feature map and the frequency feature map of the image to be segmented are fused along the spatial plane to obtain the spatial-frequency hybrid feature map of the image to be segmented.

[0088] In this embodiment of the invention, the extraction method of the space-frequency mixing feature map includes:

[0089] The spatial feature map and the frequency feature map of the image to be segmented are convolved by 1×1 respectively to obtain the first intermediate feature and the second intermediate feature of the image to be segmented.

[0090] After pixel-by-pixel summing of the first and second intermediate features of the image to be segmented, the spatial-frequency hybrid feature map of the image to be segmented is obtained by applying the Sigmoid activation function.

[0091] This embodiment integrates the spatial and frequency information of the image to be segmented at the spatial level, fully combining the features of the object's position, shape, and texture in the spatial domain with the image's frequency components and periodicity information reflected in the frequency domain. This generates a spatial-frequency hybrid feature map containing richer features, preserving both the details and spatial structure information of the objects in the image to be segmented, while also incorporating the overall image features implied by the frequency information. This provides a more comprehensive feature representation for subsequent processing and helps improve the understanding of the image content.

[0092] The spatial feature map and the frequency feature map of the image to be segmented are fused along the channel level to obtain the frequency-space hybrid feature map of the image to be segmented.

[0093] In this embodiment of the invention, the extraction method of the frequency-space hybrid feature map includes:

[0094] The spatial feature map of the image to be segmented is concatenated with the frequency feature map of the image to be segmented to obtain the third intermediate feature of the image to be segmented.

[0095] The third intermediate feature of the image to be segmented is processed by cascaded average pooling, max pooling, 1×1 convolution, and sigmoid activation function to obtain the fourth intermediate feature of the image to be segmented.

[0096] The third intermediate feature of the image to be segmented is multiplied by the fourth intermediate feature of the image to be segmented, and then a 1×1 convolution is performed to obtain the spatial frequency hybrid feature map of the image to be segmented.

[0097] This embodiment provides an alternative fusion method for different features by fusing spatial feature maps and frequency feature maps at the channel level, further mining and combining the information of the two types of features. This fusion method can enrich feature expression at the channel dimension, enabling the frequency-space hybrid feature map to integrate spatial and frequency features from different perspectives, enhancing the diversity and complementarity of features, and providing more dimensional information support for subsequent analysis and processing.

[0098] The semantic segmentation result of the image to be segmented is obtained by multiplying the space-frequency mixture feature map of the image to be segmented with the frequency-space mixture feature map of the image to be segmented.

[0099] The dot product operation used in this embodiment can perform deep fusion of the feature maps obtained from the two different fusion methods, highlighting the common important information in both feature maps and suppressing irrelevant or redundant information. Through this operation, the fused features are further transformed into semantic segmentation results, comprehensively utilizing the advantages of the spatial and frequency features fused in the previous steps, thereby more accurately dividing different semantic regions in the image and improving the precision and accuracy of semantic segmentation.

[0100] Figure 4 This is a flowchart of a nighttime semantic segmentation model training method according to an embodiment of the present invention, as follows: Figure 4 As shown, the method includes:

[0101] 201. Obtain a training segmentation image, wherein the training segmentation image is a training image of a night scene, and the training image includes pixel-by-pixel segmentation map labels;

[0102] In this embodiment of the invention, the training segmentation images can use the NightCity dataset as the training dataset for nighttime scenes. The NightCity dataset contains various types of nighttime urban street scenes, including city roads, intersections, parking lots, etc. The scenes include different lighting conditions, such as streetlights, car lights, neon lights, etc. This provides rich test cases for training and evaluating the performance of deep learning models in complex nighttime environments. It contains 4297 finely annotated images, of which 2998 are used for training and 1299 for validation. The dataset labels contain 19 categories, and the image resolution is 512×1024. In this implementation, the size of all sample images and ground truth mask images can be adjusted to 384×768 during network training.

[0103] 202. Input the trained segmented image into the nighttime semantic segmentation model; the nighttime semantic segmentation model includes a feature encoder, a spatial branch, a frequency branch, a multi-frequency fully connected layer module, a multiplier, a reshaping module, and a hybrid domain fusion module;

[0104] 203. Extract features from the training segmentation image using a feature encoder to obtain the original feature map of the training segmentation image;

[0105] In this embodiment of the invention, the feature encoder extracts image features layer by layer through multiple convolutional layers, pooling layers, and residual blocks. During training, residual connections help optimize gradient propagation in the network, solving the gradient vanishing problem in deep network training. This enables the feature encoder to efficiently extract multi-level, multi-scale features from low to high levels, ultimately generating a feature map F with rich representations.

[0106] 204. Enhance the original segmentation map of the training segmentation image through spatial domain branching to obtain the spatial feature map of the training segmentation image;

[0107] 205. Extract the frequency of the original segmentation map of the training segmentation image through frequency domain branching to obtain the frequency information of the training segmentation image;

[0108] 206. The frequency information of the training segmentation image is integrated through a multi-frequency fully connected layer module to obtain the weights of the frequency information of the training segmentation image;

[0109] 207. The frequency information and weights of the training segmentation image are multiplied element by element by a multiplier to obtain the adjusted frequency information of the training segmentation image;

[0110] 208. The frequency information of the training segmentation image is adjusted by the reshaping module to obtain the frequency feature map of the training segmentation image;

[0111] 209. The spatial feature map and frequency feature map of the training segmentation image are fused by the hybrid domain fusion module to obtain the semantic segmentation result of the training segmentation image;

[0112] 210. The nighttime semantic segmentation model is trained based on the semantic segmentation results of the trained segmented image to obtain the trained nighttime semantic segmentation model.

[0113] In this embodiment of the invention, training the nighttime semantic segmentation model based on the semantic segmentation results of the trained segmented image to obtain the trained nighttime semantic segmentation model includes:

[0114] Based on the pixel-by-pixel segmentation map labels of the training segmentation image, the basic semantic edge information of each pixel of the training segmentation image and the probability of each pixel belonging to a specific category are obtained;

[0115] Based on the semantic segmentation results of the training segmented image, the predicted edge information of each pixel in the training segmented image and the predicted probability of each pixel belonging to a specific category are obtained;

[0116] The segmentation loss of the training segmentation image is calculated based on the probability that each pixel of the training segmentation image belongs to a specific category and the predicted probability that each pixel of the training segmentation image belongs to a specific category.

[0117] The edge loss of the training segmentation image is calculated based on the true semantic edge information of each pixel of the training segmentation image and the predicted edge information of each pixel of the training segmentation image.

[0118] The loss of the training segmentation image is calculated based on the segmentation loss and the edge loss of the training segmentation image.

[0119] The nighttime semantic segmentation model is trained based on the loss of the image to be segmented;

[0120] When the loss of the image to be segmented converges, the trained nighttime semantic segmentation model is obtained.

[0121] To better illustrate, a loss function for the nighttime semantic segmentation model is constructed based on the semantic segmentation result P of the trained segmented image and the pixel-by-pixel segmentation map labels corresponding to the original nighttime scene image. The parameters of the nighttime semantic segmentation model are updated with the minimum loss function as the optimization objective, thus completing the training of the nighttime semantic segmentation model. The formula for calculating the loss function L is:

[0122] L = L edge +L seg

[0123]

[0124] Among them, L edge L represents the edge loss. seg G represents the segmentation loss; i,c P represents the ground truth value of the i-th pixel in the c-th class, which is also the segmentation map label of the i-th pixel. i,c Let b represent the semantic segmentation result of the c-th class for the i-th pixel. i It is the basic semantic edge value of the i-th pixel. It is an indicator function that indicates whether the i-th pixel belongs to a semantic edge region. For example, it can be determined whether the pixel is an edge pixel based on its position; N is the total number of pixels in the image, and C is the number of categories.

[0125] The nighttime semantic segmentation model was trained using the above method. During the training process, the loss function of the nighttime semantic segmentation model was constructed using the semantic segmentation results of the training segmentation image and the pixel-by-pixel segmentation map label of the training image as constraints, which resulted in higher output accuracy of the nighttime semantic segmentation model.

[0126] Figure 5 This is a structural diagram of a nighttime semantic segmentation device according to an embodiment of the present invention, as shown below. Figure 5 As shown, the device includes:

[0127] The first acquisition unit 111 is used to acquire the image to be segmented;

[0128] The first extraction unit 112 is used to extract features from the image to be segmented by a feature encoder to obtain the original feature map of the image to be segmented.

[0129] The first spatial unit 113 is used to enhance the original feature map of the image to be segmented through spatial domain branching to obtain the spatial feature map of the image to be segmented.

[0130] The first frequency unit 114 is used to extract the frequency of the original feature map of the image to be segmented through frequency domain branching to obtain the frequency information of the image to be segmented.

[0131] The first integration unit 115 is used to integrate the frequency information of the image to be segmented through the multi-frequency fully connected layer module to obtain the weight of the frequency information of the image to be segmented.

[0132] The first multiplication unit 116 is used to multiply the frequency information and weights of the image to be segmented element by element through a multiplier to obtain the adjusted frequency information of the image to be segmented.

[0133] The first reshaping unit 117 is used to adjust the adjustment frequency information of the image to be segmented through the reshaping module to obtain the frequency feature map of the image to be segmented.

[0134] The first fusion unit 118 is used to fuse the spatial feature map and frequency feature map of the image to be segmented through the hybrid domain fusion module to obtain the semantic segmentation result of the image to be segmented.

[0135] The image to be segmented is input into a nighttime semantic segmentation model, which includes a feature encoder, a spatial branch, a frequency branch, a multi-frequency fully connected layer module, a multiplier, a reshaping module, and a hybrid domain fusion module.

[0136] Figure 6 This is a structural diagram of a nighttime semantic segmentation model training device according to an embodiment of the present invention, as shown below. Figure 6 As shown, the device includes:

[0137] The second acquisition unit 211 is used to acquire the training segmentation image;

[0138] The second extraction unit 212 is used to extract features from the training segmentation image through a feature encoder to obtain the original feature map of the training segmentation image;

[0139] The second spatial unit 213 is used to enhance the original segmentation map of the training segmentation image through spatial domain branching to obtain the spatial feature map of the training segmentation image.

[0140] The second frequency unit 214 is used to extract the frequency of the original segmentation map of the training segmentation image through frequency domain branching to obtain the frequency information of the training segmentation image.

[0141] The second integration unit 215 is used to integrate the frequency information of the training segmentation image through the multi-frequency fully connected layer module to obtain the weights of the frequency information of the training segmentation image.

[0142] The second multiplication unit 216 is used to multiply the frequency information and weights of the training segmentation image element by element through a multiplier to obtain the adjusted frequency information of the training segmentation image.

[0143] The second reshaping unit 217 is used to adjust the adjustment frequency information of the training segmentation image through the reshaping module to obtain the frequency feature map of the training segmentation image.

[0144] The second fusion unit 218 is used to fuse the spatial feature map and frequency feature map of the training segmentation image through the hybrid domain fusion module to obtain the semantic segmentation result of the training segmentation image;

[0145] The first training unit 219 is used to train the night semantic segmentation model based on the semantic segmentation results of the training segmented image to obtain the trained night semantic segmentation model.

[0146] The trained segmented image is input into a nighttime semantic segmentation model, which includes a feature encoder, a spatial branch, a frequency branch, a multi-frequency fully connected layer module, a multiplier, a reshaping module, and a hybrid domain fusion module.

[0147] In a preferred embodiment of the present invention, such as Figure 7 As shown, the first training unit includes:

[0148] The first determining unit 291 is used to obtain the basic semantic edge information of each pixel of the training segmentation image and the probability that each pixel belongs to a specific category based on the pixel-by-pixel segmentation map label of the training segmentation image.

[0149] The second determining unit 292 is used to obtain the predicted edge information of each pixel of the training segmented image and the predicted probability of each pixel belonging to a specific category based on the semantic segmentation result of the training segmented image.

[0150] The first calculation unit 293 is used to calculate the segmentation loss of the training segmentation image based on the probability that each pixel of the training segmentation image belongs to a specific category and the predicted probability that each pixel of the training segmentation image belongs to a specific category.

[0151] The second calculation unit 294 is used to calculate the edge loss of the training segmentation image based on the true semantic edge information of each pixel of the training segmentation image and the predicted edge information of each pixel of the training segmentation image.

[0152] The third calculation unit 295 is used to calculate the loss of the training segmentation image based on the segmentation loss of the training segmentation image and the edge loss of the training segmentation image.

[0153] The first adjustment unit 296 is used to train the nighttime semantic segmentation model based on the loss of the image to be segmented; when the loss of the image to be segmented converges, the trained nighttime semantic segmentation model is obtained.

[0154] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include ROM, RAM, disk, or optical disk, etc.

[0155] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A nighttime semantic segmentation method, characterized in that, The method includes: Obtain the image to be segmented; The image to be segmented is input into a nighttime semantic segmentation model; the nighttime semantic segmentation model includes a feature encoder, a spatial branch, a frequency branch, a multi-frequency fully connected layer module, a multiplier, a reshaping module, and a hybrid domain fusion module; The original feature map of the image to be segmented is obtained by extracting features from the image to be segmented using a feature encoder. The original feature map of the image to be segmented is enhanced by spatial domain branching to obtain the spatial feature map of the image to be segmented. Frequency information of the image to be segmented is obtained by extracting the frequency from the original feature map of the image to be segmented through frequency domain branching. The frequency information of the image to be segmented is integrated by a multi-frequency fully connected layer module to obtain the weights of the frequency information of the image to be segmented, including: The frequency information of the image to be segmented is divided along the channel dimension to obtain the frequency information of the image to be segmented in each channel; The frequency information of the image to be segmented in each channel is passed through a fully connected layer one by one to obtain the weight of the image to be segmented in each channel; The frequency information and weights of the image to be segmented are multiplied element by element by a multiplier to obtain the adjusted frequency information of the image to be segmented. The frequency information of the image to be segmented is adjusted by the reshaping module to obtain the frequency feature map of the image to be segmented. The spatial feature map and frequency feature map of the image to be segmented are fused by the hybrid domain fusion module to obtain the semantic segmentation result of the image to be segmented, including: The spatial feature map and the frequency feature map of the image to be segmented are fused along the spatial plane to obtain the spatial-frequency hybrid feature map of the image to be segmented. The spatial feature map and the frequency feature map of the image to be segmented are fused along the channel level to obtain the frequency-space hybrid feature map of the image to be segmented. The semantic segmentation result of the image to be segmented is obtained by multiplying the space-frequency mixture feature map of the image to be segmented with the frequency-space mixture feature map of the image to be segmented.

2. The nighttime semantic segmentation method according to claim 1, characterized in that, The step of fusing the spatial feature map and the frequency feature map of the image to be segmented along a spatial plane to obtain the spatial-frequency hybrid feature map of the image to be segmented includes: The spatial feature map and the frequency feature map of the image to be segmented are convolved by 1×1 respectively to obtain the first intermediate feature and the second intermediate feature of the image to be segmented. After pixel-by-pixel summing of the first and second intermediate features of the image to be segmented, the spatial-frequency hybrid feature map of the image to be segmented is obtained by applying the Sigmoid activation function.

3. A nighttime semantic segmentation method according to claim 1 or 2, characterized in that, The step of fusing the spatial feature map and the frequency feature map of the image to be segmented along a spatial plane to obtain the spatial-frequency hybrid feature map of the image to be segmented includes: The spatial feature map of the image to be segmented is concatenated with the frequency feature map of the image to be segmented to obtain the third intermediate feature of the image to be segmented. The third intermediate feature of the image to be segmented is processed by cascaded average pooling, max pooling, 1×1 convolution, and sigmoid activation function to obtain the fourth intermediate feature of the image to be segmented. The third intermediate feature of the image to be segmented is multiplied by the fourth intermediate feature of the image to be segmented, and then a 1×1 convolution is performed to obtain the spatial frequency hybrid feature map of the image to be segmented.

4. A method for training a nighttime semantic segmentation model, characterized in that, The method includes: Obtain a training segmentation image, wherein the training segmentation image is a training image of a night scene, and the training image includes pixel-by-pixel segmentation map labels; The trained segmented image is input into the nighttime semantic segmentation model; the nighttime semantic segmentation model includes a feature encoder, a spatial branch, a frequency branch, a multi-frequency fully connected layer module, a multiplier, a reshaping module, and a hybrid domain fusion module; The training segmentation image is subjected to feature extraction by a feature encoder to obtain the original feature map of the training segmentation image; The original segmentation map of the training segmentation image is enhanced by spatial domain branching to obtain the spatial feature map of the training segmentation image; Frequency information of the training segmentation image is obtained by extracting the frequency of the original segmentation map of the training segmentation image through frequency domain branching. The frequency information of the training segmentation image is integrated through a multi-frequency fully connected layer module to obtain the weights of the frequency information of the training segmentation image, including: The frequency information of the image to be segmented is divided along the channel dimension to obtain the frequency information of the image to be segmented in each channel; The frequency information of the image to be segmented in each channel is passed through a fully connected layer one by one to obtain the weight of the image to be segmented in each channel; The frequency information and weights of the training segmentation image are multiplied element-wise by a multiplier to obtain the adjusted frequency information of the training segmentation image. The frequency information of the training segmentation image is adjusted by the reshaping module to obtain the frequency feature map of the training segmentation image; The spatial feature map and frequency feature map of the training segmentation image are fused by the hybrid domain fusion module to obtain the semantic segmentation result of the training segmentation image, including: The spatial feature map and the frequency feature map of the image to be segmented are fused along the spatial plane to obtain the spatial-frequency hybrid feature map of the image to be segmented. The spatial feature map and the frequency feature map of the image to be segmented are fused along the channel level to obtain the frequency-space hybrid feature map of the image to be segmented. The spatial-frequency mixture feature map of the image to be segmented is multiplied by the frequency-spatial mixture feature map of the image to be segmented to obtain the semantic segmentation result of the image to be segmented. The nighttime semantic segmentation model is trained based on the semantic segmentation results of the training segmented image to obtain the trained nighttime semantic segmentation model.

5. The nighttime semantic segmentation model training method according to claim 4, characterized in that, The step of training the nighttime semantic segmentation model based on the semantic segmentation results of the trained segmented image to obtain the trained nighttime semantic segmentation model includes: Based on the pixel-by-pixel segmentation map labels of the training segmentation image, the basic semantic edge information of each pixel of the training segmentation image and the probability of each pixel belonging to a specific category are obtained; Based on the semantic segmentation results of the training segmented image, the predicted edge information of each pixel in the training segmented image and the predicted probability of each pixel belonging to a specific category are obtained; The segmentation loss of the training segmentation image is calculated based on the probability that each pixel of the training segmentation image belongs to a specific category and the predicted probability that each pixel of the training segmentation image belongs to a specific category. The edge loss of the training segmentation image is calculated based on the true semantic edge information of each pixel of the training segmentation image and the predicted edge information of each pixel of the training segmentation image. The loss of the training segmentation image is calculated based on the segmentation loss and the edge loss of the training segmentation image. The nighttime semantic segmentation model is trained based on the loss of the trained segmented images; When the loss of the training segmented image converges, the trained nighttime semantic segmentation model is obtained.

6. A nighttime semantic segmentation device, characterized in that, The device includes: The first acquisition unit is used to acquire the image to be segmented; The first extraction unit is used to extract features from the image to be segmented through a feature encoder to obtain the original feature map of the image to be segmented. The first spatial unit is used to enhance the original feature map of the image to be segmented through spatial domain branching to obtain the spatial feature map of the image to be segmented. The first frequency unit is used to extract the frequency of the original feature map of the image to be segmented through frequency domain branching to obtain the frequency information of the image to be segmented. The first integration unit is used to integrate the frequency information of the image to be segmented through a multi-frequency fully connected layer module, and to obtain the weights of the frequency information of the image to be segmented, including: The frequency information of the image to be segmented is divided along the channel dimension to obtain the frequency information of the image to be segmented in each channel; The frequency information of the image to be segmented in each channel is passed through a fully connected layer one by one to obtain the weight of the image to be segmented in each channel; The first multiplication unit is used to multiply the frequency information and weights of the image to be segmented element by element through a multiplier to obtain the adjusted frequency information of the image to be segmented. The first reshaping unit is used to adjust the adjustment frequency information of the image to be segmented through the reshaping module to obtain the frequency feature map of the image to be segmented. The first fusion unit is used to fuse the spatial feature map and frequency feature map of the image to be segmented through a hybrid domain fusion module to obtain the semantic segmentation result of the image to be segmented, including: The spatial feature map and the frequency feature map of the image to be segmented are fused along the spatial plane to obtain the spatial-frequency hybrid feature map of the image to be segmented. The spatial feature map and the frequency feature map of the image to be segmented are fused along the channel level to obtain the frequency-space hybrid feature map of the image to be segmented. The spatial-frequency mixture feature map of the image to be segmented is multiplied by the frequency-spatial mixture feature map of the image to be segmented to obtain the semantic segmentation result of the image to be segmented. The image to be segmented is input into a nighttime semantic segmentation model, which includes a feature encoder, a spatial branch, a frequency branch, a multi-frequency fully connected layer module, a multiplier, a reshaping module, and a hybrid domain fusion module.

7. A nighttime semantic segmentation model training device, characterized in that, The device includes: The second acquisition unit is used to acquire the training segmentation image; The second extraction unit is used to extract features from the training segmentation image through a feature encoder to obtain the original feature map of the training segmentation image; The second spatial unit is used to enhance the original segmentation map of the training segmentation image through spatial domain branching to obtain the spatial feature map of the training segmentation image; The second frequency unit is used to extract the frequency of the original segmentation map of the training segmentation image through frequency domain branching to obtain the frequency information of the training segmentation image; The second integration unit is used to integrate the frequency information of the training segmentation image through a multi-frequency fully connected layer module to obtain the weights of the frequency information of the training segmentation image, including: The frequency information of the image to be segmented is divided along the channel dimension to obtain the frequency information of the image to be segmented in each channel; The frequency information of the image to be segmented in each channel is passed through a fully connected layer one by one to obtain the weight of the image to be segmented in each channel; The second multiplication unit is used to multiply the frequency information and weights of the training segmentation image element by element through a multiplier to obtain the adjusted frequency information of the training segmentation image. The second reshaping unit is used to adjust the adjustment frequency information of the training segmentation image through the reshaping module to obtain the frequency feature map of the training segmentation image. The second fusion unit is used to fuse the spatial feature map and frequency feature map of the training segmentation image through the hybrid domain fusion module to obtain the semantic segmentation result of the training segmentation image, including: The spatial feature map and the frequency feature map of the image to be segmented are fused along the spatial plane to obtain the spatial-frequency hybrid feature map of the image to be segmented. The spatial feature map and the frequency feature map of the image to be segmented are fused along the channel level to obtain the frequency-space hybrid feature map of the image to be segmented. The spatial-frequency mixture feature map of the image to be segmented is multiplied by the frequency-spatial mixture feature map of the image to be segmented to obtain the semantic segmentation result of the image to be segmented. The first training unit is used to train the nighttime semantic segmentation model based on the semantic segmentation results of the training segmented image, so as to obtain the trained nighttime semantic segmentation model. The trained segmented image is input into a nighttime semantic segmentation model, which includes a feature encoder, a spatial branch, a frequency branch, a multi-frequency fully connected layer module, a multiplier, a reshaping module, and a hybrid domain fusion module.

8. The nighttime semantic segmentation model training device according to claim 7, characterized in that, The first training unit includes: The first determining unit is used to obtain the basic semantic edge information of each pixel of the training segmentation image and the probability that each pixel belongs to a specific category based on the pixel-by-pixel segmentation map label of the training segmentation image; The second determining unit is used to obtain the predicted edge information of each pixel of the training segmented image and the predicted probability of each pixel belonging to a specific category based on the semantic segmentation result of the training segmented image. The first calculation unit is used to calculate the segmentation loss of the training segmentation image based on the probability that each pixel of the training segmentation image belongs to a specific category and the predicted probability that each pixel of the training segmentation image belongs to a specific category. The second calculation unit is used to calculate the edge loss of the training segmentation image based on the true semantic edge information of each pixel of the training segmentation image and the predicted edge information of each pixel of the training segmentation image. The third calculation unit is used to calculate the loss of the training segmentation image based on the segmentation loss of the training segmentation image and the edge loss of the training segmentation image. The first adjustment unit is used to train the nighttime semantic segmentation model based on the loss of the training segmentation image; when the loss of the training segmentation image converges, the trained nighttime semantic segmentation model is obtained.

Images