Night semantic segmentation method and night semantic segmentation model training method and device

Abstract

The invention belongs to the field of night semantic segmentation, and particularly relates to a night semantic segmentation method and a night semantic segmentation model training method and device.The method comprises the steps that a night semantic segmentation model is constructed and trained, and an image of a night scene to be segmented is input into the trained night semantic segmentation model to obtain a pixel-level segmentation map; the night semantic segmentation model comprises a feature encoder, a spatial domain branch, a frequency domain branch, a multi-frequency full-connection layer module, a multiplier, a remodeling module and a mixed domain fusion module; according to the nighttime semantic segmentation method and device, the segmentation accuracy in a nighttime scene can be effectively improved through the nighttime semantic segmentation model, frequency information is fused into spatial features, and generalization and robustness of a segmentation network are enhanced. According to the training method and device of the night semantic segmentation model, the accuracy of the night semantic segmentation model can be effectively improved.

Description

Technical Field

[0001] The present invention belongs to the field of semantic segmentation in night scenes, and in particular relates to a night semantic segmentation method, a night semantic segmentation model training method and a device. Background Art

[0002] In recent years, with the rapid development of mobile robot technology and driverless cars, driverless cars have gradually become the first generation of digital robots that people trust. With the improvement of computer computing speed, the enhancement of hardware computing power and the continuous advancement of deep learning technology, cars can achieve autonomous driving capabilities similar to those of humans in a variety of outdoor scenarios, ensuring the safety and stability of driving. In this process, semantic segmentation technology plays an irreplaceable role. It can accurately identify and segment the location, quantity and type of target objects such as driving roads, pedestrians, and vehicles, thereby providing intelligent decision support for vehicles, effectively avoiding obstacles, and improving the safety and robustness of autonomous driving systems.

[0003] However, the low-light conditions of night scenes bring new challenges to the semantic segmentation task of unmanned driving. Night images usually show low contrast, blur and high noise, which leads to blurred target edges and loss of detail information when traditional methods extract spatial domain features, thereby reducing segmentation accuracy. Although some methods improve image quality through image enhancement techniques (such as contrast enhancement and histogram equalization), or expand training data through deep learning model design and generative adversarial networks (GANs), these methods mainly focus on feature extraction and optimization in the spatial domain, and there is a clear lack of utilization of frequency information. In fact, the frequency domain information in the image, especially the accurate description of the target boundary by high-frequency features and the structural supplement of the global background by low-frequency features, can provide stronger robustness and auxiliary information for semantic segmentation. However, existing segmentation methods fail to fully incorporate frequency domain information, which limits the performance of the model in night scenes. Summary of the invention

[0004] To solve the above problems, the present invention provides a nighttime semantic segmentation method, a nighttime semantic segmentation model training method and a device. First, the frequency domain branch converts spatial features into frequency information through two-dimensional discrete cosine transform (DCT), so that the network can extract high-frequency details and low-frequency structures in the image, and help the segmentation model better capture the target boundary and background features. Then, the multi-frequency fully connected layer module MFC (Multi-frequency Fully Connected Layer) module dynamically weights and adjusts the frequency information, and further enhances the expression of frequency features by calculating the weights and multiplying them pixel by pixel with the input frequency information, so that important spectral features are better highlighted, and the accuracy of image segmentation is improved. Finally, the hybrid domain fusion module HDF (Hybird Domain Fusion) fuses spatial features and frequency features together, optimizes the model's comprehensive utilization of the two, and improves the comprehensiveness and robustness of image segmentation by effectively combining spatial and frequency information.

[0005] In a first aspect of the present invention, a nighttime semantic segmentation method of the present invention comprises:

[0006] Obtain the image to be segmented;

[0007] Inputting the image to be segmented into a nighttime semantic segmentation model; the nighttime semantic segmentation model comprises a feature encoder, a spatial domain branch, a frequency domain branch, a multi-frequency fully connected layer module, a multiplier, a reshaping module, and a mixed domain fusion module;

[0008] Extracting features of the image to be segmented by a feature encoder to obtain an original feature map of the image to be segmented;

[0009] The original feature map of the image to be segmented is enhanced by a spatial domain branch to obtain a spatial feature map of the image to be segmented;

[0010] Performing frequency extraction on the original feature map of the image to be segmented through a frequency domain branch to obtain frequency information of the image to be segmented;

[0011] Integrating the frequency information of the image to be segmented through a multi-frequency fully connected layer module to obtain the weight of the frequency information of the image to be segmented;

[0012] The frequency information of the image to be segmented and its weight are multiplied element by element by a multiplier to obtain the adjusted frequency information of the image to be segmented;

[0013] The frequency information of the image to be segmented is adjusted by a reshaping module to obtain a frequency characteristic diagram of the image to be segmented;

[0014] The spatial feature map and the frequency feature map of the image to be segmented are fused through a hybrid domain fusion module to obtain a semantic segmentation result of the image to be segmented.

[0015] In a second aspect of the present invention, the present invention further provides a nighttime semantic segmentation model training method, the method comprising:

[0016] Acquire a training segmentation image, where the training segmentation image is a training image of a night scene, and the training image includes a pixel-by-pixel segmentation map label;

[0017] Inputting the training segmented image into a nighttime semantic segmentation model; the nighttime semantic segmentation model comprises a feature encoder, a spatial domain branch, a frequency domain branch, a multi-frequency fully connected layer module, a multiplier, a reshaping module, and a mixed domain fusion module;

[0018] Extracting features from the training segmented image using a feature encoder to obtain an original feature map of the training segmented image;

[0019] The original segmentation map of the training segmentation image is enhanced by a spatial domain branch to obtain a spatial feature map of the training segmentation image;

[0020] Performing frequency extraction on the original segmentation map of the training segmentation image through a frequency domain branch to obtain frequency information of the training segmentation image;

[0021] Integrate the frequency information of the training segmented image through a multi-frequency fully connected layer module to obtain the weight of the frequency information of the training segmented image;

[0022] The frequency information of the training segmented image and its weight are multiplied element by element by a multiplier to obtain the adjusted frequency information of the training segmented image;

[0023] The adjustment frequency information of the training segmented image is adjusted by a reshaping module to obtain a frequency characteristic graph of the training segmented image;

[0024] The spatial feature map and the frequency feature map of the training segmented image are fused by a hybrid domain fusion module to obtain a semantic segmentation result of the training segmented image;

[0025] The nighttime semantic segmentation model is trained according to the semantic segmentation result of the training segmented image to obtain a trained nighttime semantic segmentation model.

[0026] In a third aspect of the present invention, the present invention further provides a nighttime semantic segmentation device, the device comprising:

[0027] A first acquisition unit, used for acquiring an image to be segmented;

[0028] A first extraction unit, configured to extract features of the image to be segmented by using a feature encoder to obtain an original feature map of the image to be segmented;

[0029] A first spatial unit is used to enhance the original feature map of the image to be segmented through a spatial domain branch to obtain a spatial feature map of the image to be segmented;

[0030] A first frequency unit, used for performing frequency extraction on an original feature map of the image to be segmented through a frequency domain branch to obtain frequency information of the image to be segmented;

[0031] A first integration unit is used to integrate the frequency information of the image to be segmented through a multi-frequency fully connected layer module to obtain the weight of the frequency information of the image to be segmented;

[0032] A first multiplication unit, configured to multiply the frequency information of the image to be segmented and its weight element by element through a multiplier to obtain the adjusted frequency information of the image to be segmented;

[0033] A first reshaping unit, configured to adjust the adjustment frequency information of the image to be segmented through a reshaping module to obtain a frequency characteristic graph of the image to be segmented;

[0034] A first fusion unit is used to fuse the spatial feature map and the frequency feature map of the image to be segmented through a hybrid domain fusion module to obtain a semantic segmentation result of the image to be segmented;

[0035] Among them, the image to be segmented is input into a nighttime semantic segmentation model; the nighttime semantic segmentation model includes a feature encoder, a spatial domain branch, a frequency domain branch, a multi-frequency fully connected layer module, a multiplier, a reshaping module and a mixed domain fusion module.

[0036] In a fourth aspect of the present invention, the present invention further provides a nighttime semantic segmentation model training device, the device comprising:

[0037] A second acquisition unit, used for acquiring a training segmentation image;

[0038] A second extraction unit, configured to perform feature extraction on the training segmented image through a feature encoder to obtain an original feature map of the training segmented image;

[0039] A second spatial unit is used to enhance the original segmentation map of the training segmentation image through a spatial domain branch to obtain a spatial feature map of the training segmentation image;

[0040] A second frequency unit is used to perform frequency extraction on the original segmentation map of the training segmentation image through a frequency domain branch to obtain frequency information of the training segmentation image;

[0041] A second integration unit is used to integrate the frequency information of the training segmented image through a multi-frequency fully connected layer module to obtain the weight of the frequency information of the training segmented image;

[0042] A second multiplication unit is used to multiply the frequency information of the training segmented image and its weight element by element through a multiplier to obtain the adjusted frequency information of the training segmented image;

[0043] A second reshaping unit, configured to adjust the adjustment frequency information of the training segmented image through a reshaping module to obtain a frequency characteristic graph of the training segmented image;

[0044] A second fusion unit is used to fuse the spatial feature map and the frequency feature map of the training segmented image through a hybrid domain fusion module to obtain a semantic segmentation result of the training segmented image;

[0045] A first training unit is used to train the nighttime semantic segmentation model according to the semantic segmentation result of the training segmented image to obtain a trained nighttime semantic segmentation model;

[0046] The training segmented image is input into a nighttime semantic segmentation model; the nighttime semantic segmentation model includes a feature encoder, a spatial domain branch, a frequency domain branch, a multi-frequency fully connected layer module, a multiplier, a reshaping module, and a mixed domain fusion module.

[0047] Beneficial effects of the present invention:

[0048] The present invention significantly improves the accuracy and robustness of semantic segmentation of night scenes through a night semantic segmentation method, a night semantic segmentation model training method and a device. First, the frequency domain branch converts spatial features into frequency information through a two-dimensional discrete cosine transform (DCT), so that the network can extract high-frequency details and low-frequency structures in the image, and help the segmentation model better capture the target boundary and background features. Then, the MFC (Multi-frequency Fully Connected Layer) module dynamically weights and adjusts the frequency information, and further enhances the expression of frequency features by calculating the weights and multiplying them pixel by pixel with the input frequency information, so that important spectral features are better highlighted, and the accuracy of image segmentation is improved. Finally, the HDF (Hybird Domain Fusion) hybrid domain fusion module fuses spatial features and frequency features together, optimizes the comprehensive utilization of the two by the model, and improves the comprehensiveness and robustness of image segmentation by effectively combining spatial and frequency information. This method can not only effectively deal with low contrast, blur and noise problems in night scenes, but also optimizes computational efficiency and training effects. It has strong versatility and extensibility and can be widely used in various image segmentation tasks under low light and complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A flowchart of a nighttime semantic segmentation method according to an embodiment of the present invention;

[0050] Figure 2 It is a structural schematic diagram of a nighttime semantic segmentation model according to an embodiment of the present invention;

[0051] Figure 3 It is a structural schematic diagram of a multi-frequency fully connected layer MFC unit according to an embodiment of the present invention;

[0052] Figure 4 This is a flow chart of a nighttime semantic segmentation model training method according to an embodiment of the present invention;

[0053] Figure 5 This is a schematic diagram of the structure of a nighttime semantic segmentation device according to an embodiment of the present invention;

[0054] Figure 6 This is a schematic diagram of the structure of a nighttime semantic segmentation model training device according to an embodiment of the present invention;

[0055] Figure 7 Schematic diagram of the structure of the first training unit according to an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] It should be understood that in the description of the present application, unless otherwise specified, the first, second, etc. descriptions appearing in the embodiments of the present application are only for illustration and distinction of the description objects, without any distinction of order, nor do they indicate any special limitation on the number of devices in the embodiments of the present application, and cannot constitute any limitation on the embodiments of the present application.

[0058] It should also be understood that the specific features, structures or characteristics related to the embodiments in the specification are included in at least one embodiment of the present application. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner.

[0059] In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such process, method, product, or device.

[0060] The embodiments of the present application may involve computer vision (CV) technology in artificial intelligence technology. Computer vision is a science that studies how to make machines "see". To put it more concretely, it refers to machine vision such as using cameras and computers to replace human eyes to identify and measure targets, and further perform graphic processing to make computer processing into images that are more suitable for human eye observation or transmission to instrument detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multidimensional data. Computer vision technology generally includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0061] The embodiments of the present application provide a nighttime semantic segmentation method, a nighttime semantic segmentation model training method and a device, which can help improve the accuracy of nighttime scene segmentation results.

[0062] The embodiments of the present application can be used in any scenario that requires nighttime semantic segmentation, including scenarios such as autonomous driving, security monitoring, drone inspections, and nighttime navigation. For example, in an autonomous driving scenario, nighttime semantic segmentation can help vehicles identify key elements such as road signs, pedestrians, other vehicles, and obstacles, and make accurate judgments and responses even in low light conditions. The embodiments of the present application can accurately segment out each key element, which is conducive to improving the accuracy of the nighttime scene segmentation results.

[0063] Figure 1 FIG. 1 is a flow chart of a nighttime semantic segmentation method according to an embodiment of the present invention. Figure 1 As shown, the method can be executed by any electronic device with data processing capability, for example, the electronic device can be implemented as a server, which is not limited in this application. The method includes:

[0064] 101. Obtain an image to be segmented;

[0065] In an embodiment of the present invention, the image to be segmented may be a road environment photograph collected in an autonomous driving scenario, or a photograph of an industrial part in an industrial quality inspection scenario, etc., which is not limited in this embodiment.

[0066] 102. Input the image to be segmented into a nighttime semantic segmentation model; the nighttime semantic segmentation model includes a feature encoder, a spatial domain branch, a frequency domain branch, a multi-frequency fully connected layer module, a multiplier, a reshaping module, and a mixed domain fusion module;

[0067] See also Figure 2 After the night image to be segmented is input into the night semantic segmentation model, each pixel in the input image is classified so that pixels belonging to the same object or having the same semantics are marked as the same category, and pixels of different categories are distinguished by different colors to obtain the semantic segmentation result of the image to be segmented; for example, the dark purple area occupies the lower half of the image, representing the ground, the light purple area occupies the right part of the image, representing the sidewalk, the green area is in the middle left of the image, representing trees or vegetation, and the red area is in the lower right corner of the image, representing pedestrians, etc. The night semantic segmentation module is mainly divided into a spatial domain branch and a frequency domain branch. After the image is input into the night semantic segmentation module, the original feature map of the input image is first extracted through the feature encoder; secondly, the original feature map passes through the spatial domain branch and the frequency branch respectively, and the frequency feature map of the input image is learned through the multi-frequency fully connected layer module of the frequency domain branch, and the spatial feature map of the input image is extracted through the spatial domain branch; then the spatial feature map and the frequency feature map of the input image are integrated through the multiplier, the reshaping module and the mixed domain fusion module to obtain the segmentation feature map of the input image.

[0068] 103. Perform feature extraction on the image to be segmented by a feature encoder to obtain an original feature map of the image to be segmented;

[0069] In the embodiment of the present invention, the feature encoder processes the input image to be segmented through a series of mathematical transformations and calculations. In this process, it extracts key features in the image to be segmented and converts these features into vector representations of specific dimensions, which are the original feature maps of the image to be segmented. These vector representations can capture the intrinsic information and patterns of the image to be segmented, so that the subsequent model can learn and make decisions more effectively.

[0070] It should be noted that the feature encoder can be a common neural network. For example, a convolutional neural network (CNN) is commonly used for feature encoding of image data. It gradually extracts local and global features of the image through operations such as convolutional layers and pooling layers, and converts the original image into a high-dimensional feature vector. The present invention does not make specific limitations on this.

[0071] 104. Enhance the original feature map of the image to be segmented by using a spatial domain branch to obtain a spatial feature map of the image to be segmented;

[0072] In some embodiments of the present invention, the spatial domain branch enhances the original feature map of the image to be segmented through a pyramid pooling module. The pyramid pooling module pools the original feature map F of the input image to be segmented at four scales of 1×1, 2×2, 3×3, and 6×6, and upsamples the pooling result. After upsampling, the pooled multi-scale feature map is spliced ​​together by channel to form a feature map F0 containing multi-scale information. The feature map F0 is finally fused with the input feature map F to generate a spatial feature map F1 containing rich contextual information.

[0073] 105. Perform frequency extraction on the original feature map of the image to be segmented through a frequency domain branch to obtain frequency information of the image to be segmented;

[0074] In an embodiment of the present invention, the embodiment of the present invention uses a two-dimensional discrete cosine transform to convert the spatial feature map of the image to be segmented in the spatial domain into frequency information in the frequency domain, so that the low-frequency and high-frequency features of the image can be analyzed through components of different frequencies. The low-frequency part mainly describes the global shape and rough changes of the image to be segmented, while the high-frequency part contains details and texture information. Frequency information can help improve the effects of operations such as image segmentation, compression, and denoising.

[0075] 106. Integrate the frequency information of the image to be segmented through a multi-frequency fully connected layer module to obtain the weight of the frequency information of the image to be segmented;

[0076] In the embodiment of the present invention, the multi-frequency fully connected layer module is as follows: Figure 3 As shown, after the frequency information is transformed by DCT, the frequency information of each channel is obtained. After the frequency information of each channel is spliced ​​and processed through a fully connected layer, the adjusted frequency information of the image to be segmented is obtained.

[0077] In the embodiment of the present invention, step 106 includes:

[0078] The frequency information S of the image to be segmented is segmented along the channel dimension to obtain the frequency information S0, S1, S2, ... S of the image to be segmented in each channel n-1 , where n represents the number of channels;

[0079] The frequency information S0, S1, S2...S of the image to be segmented in each channel n-1 Input the fully connected layer one by one to obtain the weights W0, W1…W of each channel of the image to be segmented n-1 , and then concat the weights of these n channels to get the weight matrix W.

[0080] Specifically, the fully connected layer can adaptively adjust the importance of each channel, helping the network focus on more informative channels, thereby improving the ability to express frequency features.

[0081] 107. Multiply the frequency information of the image to be segmented and its weight element by element by a multiplier to obtain the adjusted frequency information of the image to be segmented;

[0082] In an embodiment of the present invention, considering that the frequency information of the image to be segmented contains components of different spatial frequencies, the high-frequency part usually corresponds to the details and edge information in the image, and the low-frequency part is related to the overall structure and background of the image. Therefore, by multiplying the weight element by element, for example, dot-multiplying the weight W of the frequency information of the image to be segmented and the frequency information S of the image to be segmented to obtain the adjusted weight W'; certain frequency components can be targeted to enhance or weaken, thereby highlighting or suppressing specific features in the image. For example, if you want to highlight the details in the image, you can appropriately increase the weight of the high-frequency part so that the high-frequency information is enhanced in the adjusted frequency information. By adjusting the frequency information, the embodiment of the present invention can enhance or suppress different features in the image to be segmented, thereby improving the effect of subsequent processing tasks. This adjustment can make the features related to the segmentation target more prominent, and can make it easier to identify and distinguish the target object from the background in the night scene.

[0083] 108. Adjust the adjustment frequency information of the image to be segmented by a reshaping module to obtain a frequency characteristic graph of the image to be segmented;

[0084] In an embodiment of the present invention, it is considered that the dimension and structure of the adjusted frequency information after processing by the multiplier may not be directly applicable to subsequent processing tasks. One of the functions of the reshaping module is to adjust the dimension and structure of this information so that it can better match the input requirements of the subsequent model. For example, the weight W' is reshaped into the same shape as the spatial feature map F1 of the image to be segmented, and the frequency feature map F2 of the image to be segmented is obtained. This can provide a more effective feature representation for the subsequent night semantic segmentation model, thereby improving the performance of the model. The frequency feature map obtained by reshaping can better retain and highlight the information related to the segmentation target, making it easier for the model to learn the features that distinguish different objects or regions, thereby improving the accuracy and precision of the segmentation.

[0085] 109. Fusing the spatial feature map and the frequency feature map of the image to be segmented through a hybrid domain fusion module to obtain a semantic segmentation result of the image to be segmented.

[0086] In the embodiment of the present invention, step 109 includes:

[0087] The spatial feature map of the image to be segmented and the frequency feature map of the image to be segmented are merged along the spatial level to obtain a spatial-frequency mixed feature map of the image to be segmented;

[0088] In an embodiment of the present invention, the method for extracting the space-frequency mixing feature map includes:

[0089] Performing 1×1 convolution on the spatial feature map of the image to be segmented and the frequency feature map of the image to be segmented to obtain a first intermediate feature and a second intermediate feature of the image to be segmented;

[0090] After adding the first intermediate feature and the second intermediate feature of the image to be segmented pixel by pixel, a space-frequency mixed feature map of the image to be segmented is obtained through a Sigmoid activation function.

[0091] This embodiment integrates the spatial information and frequency information of the image to be segmented at the spatial level, fully combining the position, shape, texture and other features of the object in the spatial domain and the image frequency components and periodic information reflected in the frequency domain. It can generate a space-frequency hybrid feature map containing richer features, which not only retains the details and spatial structure information of the object in the image to be segmented, but also integrates the overall image features contained in the frequency information, providing a more comprehensive feature representation for subsequent processing, which helps to improve the understanding of the image content.

[0092] The spatial feature map of the image to be segmented is merged with the frequency feature map of the image to be segmented along the channel level to obtain a frequency-space mixed feature map of the image to be segmented;

[0093] In an embodiment of the present invention, the method for extracting the frequency-space mixed feature map includes:

[0094] splicing the spatial feature map of the image to be segmented and the frequency feature map of the image to be segmented to obtain a third intermediate feature of the image to be segmented;

[0095] The third intermediate feature of the image to be segmented is processed by cascade average pooling, maximum pooling, 1×1 convolution, and Sigmoid activation function to obtain the fourth intermediate feature of the image to be segmented;

[0096] The third intermediate feature of the image to be segmented is multiplied by the fourth intermediate feature of the image to be segmented, and then a 1×1 convolution is performed to obtain a space-frequency mixed feature map of the image to be segmented.

[0097] This embodiment provides another fusion method for different features by fusing the spatial feature map and the frequency feature map at the channel level, further mining and combining the information of the two features. This fusion method can enrich the feature expression in the channel dimension, so that the frequency-space hybrid feature map can integrate the spatial and frequency features from different angles, enhance the diversity and complementarity of the features, and provide more dimensional information support for subsequent analysis and processing.

[0098] The space-frequency mixed feature map of the image to be segmented is point-multiplied with the frequency-space mixed feature map of the image to be segmented to obtain a semantic segmentation result of the image to be segmented.

[0099] The dot multiplication operation used in this embodiment can deeply fuse the feature maps of the two different fusion methods obtained above, highlight the common important information in the two feature maps, and suppress irrelevant or redundant information. Through this operation, the fused features are further converted into semantic segmentation results, and the advantages of the spatial and frequency features fused in the previous steps are comprehensively utilized, so as to more accurately divide different semantic areas in the image and improve the precision and accuracy of semantic segmentation.

[0100] Figure 4 is a flow chart of a nighttime semantic segmentation model training method according to an embodiment of the present invention. Figure 4 As shown, the method includes:

[0101] 201. Acquire a training segmentation image, where the training segmentation image is a training image of a night scene, and the training image includes a pixel-by-pixel segmentation map label;

[0102] In an embodiment of the present invention, the training segmented image can use the NightCity dataset as an image training dataset for night scenes. The NightCity dataset is a dataset that contains various types of night city street scenes, including urban roads, intersections, parking lots, etc. The scene includes different lighting conditions, such as city street lights, car lights, neon lights, etc. This provides a rich test case for training and evaluating the performance of deep learning models in complex night environments. It contains 4297 finely annotated images, of which 2998 are used for training and 1299 are used for verification. The dataset labels contain 19 categories and the image resolution is 512×1024. When the present invention is implemented, the size of all sample images and true value label mask images can be resized to 384×768 during network training.

[0103] 202. Input the training segmented image into a nighttime semantic segmentation model; the nighttime semantic segmentation model includes a feature encoder, a spatial domain branch, a frequency domain branch, a multi-frequency fully connected layer module, a multiplier, a reshaping module, and a mixed domain fusion module;

[0104] 203. Perform feature extraction on the training segmented image by using a feature encoder to obtain an original feature map of the training segmented image;

[0105] In the embodiment of the present invention, the feature encoder extracts the features of the image layer by layer through multiple convolutional layers, pooling layers and residual blocks. During the training process, the residual connection helps optimize the gradient propagation of the network, solves the gradient vanishing problem in deep network training, and enables the feature encoder to efficiently extract multi-level and multi-scale features from low-level to high-level, and finally generates a feature map F with rich representation.

[0106] 204. Enhance the original segmentation map of the training segmentation image through a spatial domain branch to obtain a spatial feature map of the training segmentation image;

[0107] 205. Perform frequency extraction on the original segmentation map of the training segmented image through a frequency domain branch to obtain frequency information of the training segmented image;

[0108] 206. Integrate the frequency information of the training segmented image through a multi-frequency fully connected layer module to obtain the weight of the frequency information of the training segmented image;

[0109] 207. Multiply the frequency information of the training segmented image and its weight element by element through a multiplier to obtain the adjusted frequency information of the training segmented image;

[0110] 208. Adjust the adjustment frequency information of the training segmented image by a reshaping module to obtain a frequency characteristic graph of the training segmented image;

[0111] 209. Fusing the spatial feature map and the frequency feature map of the training segmented image through a hybrid domain fusion module to obtain a semantic segmentation result of the training segmented image;

[0112] 210. Train the nighttime semantic segmentation model according to the semantic segmentation result of the training segmented image to obtain a trained nighttime semantic segmentation model.

[0113] In an embodiment of the present invention, the step of training the nighttime semantic segmentation model according to the semantic segmentation result of the training segmented image to obtain the trained nighttime semantic segmentation model comprises:

[0114] According to the pixel-by-pixel segmentation map label of the training segmented image, obtaining the ground truth semantic edge information of each pixel of the training segmented image and the probability that each pixel belongs to a specific category;

[0115] According to the semantic segmentation result of the training segmented image, the predicted edge information of each pixel of the training segmented image and the predicted probability that each pixel belongs to a specific category are obtained;

[0116] Calculating the segmentation loss of the training segmented image according to the probability that each pixel of the training segmented image belongs to a specific category and the predicted probability that each pixel of the training segmented image belongs to the specific category;

[0117] Calculating the edge loss of the training segmented image according to the ground truth semantic edge information of each pixel of the training segmented image and the predicted edge information of each pixel of the training segmented image;

[0118] Calculating the loss of the training segmented image according to the segmentation loss of the training segmented image and the edge loss of the training segmented image;

[0119] Training the nighttime semantic segmentation model according to the loss of the image to be segmented;

[0120] When the loss of the image to be segmented converges, the trained nighttime semantic segmentation model is obtained.

[0121] For better explanation, the loss function of the night semantic segmentation model is constructed according to the semantic segmentation result P of the training segmented image and the pixel-by-pixel segmentation map label corresponding to the image of the original night scene, and the parameters of the night semantic segmentation model are updated with the minimum loss function as the optimization goal to complete the training of the night semantic segmentation model. Among them, the loss function L calculation formula is:

[0122] L=L edge +L seg

[0123]

[0124] Among them, L edge represents the marginal loss, L seg represents the segmentation loss; G i,c represents the c-th ground truth value of the ith pixel, that is, the segmentation map label of the ith pixel, P i,c represents the c-th semantic segmentation result of the ith pixel, b i is the ground truth semantic edge value of the ith pixel, is an indicator function indicating whether the i-th pixel belongs to the semantic edge region. For example, whether the pixel is an edge pixel can be determined based on the pixel position. N is the total number of pixels in the image, and C is the number of categories.

[0125] The training of the night semantic segmentation model is completed in the above manner. During the training process of the night semantic segmentation model, the construction of the loss function of the night semantic segmentation model uses the semantic segmentation results of the training segmented image and the pixel-by-pixel segmentation map labels of the training image as constraints, so that the output accuracy of the night semantic segmentation model is higher.

[0126] Figure 5 is a structural diagram of a nighttime semantic segmentation device according to an embodiment of the present invention. Figure 5 As shown, the device comprises:

[0127] A first acquisition unit 111 is used to acquire an image to be segmented;

[0128] A first extraction unit 112 is used to extract features of the image to be segmented through a feature encoder to obtain an original feature map of the image to be segmented;

[0129] A first spatial unit 113 is used to enhance the original feature map of the image to be segmented through a spatial domain branch to obtain a spatial feature map of the image to be segmented;

[0130] A first frequency unit 114 is used to extract the frequency of the original feature map of the image to be segmented through a frequency domain branch to obtain the frequency information of the image to be segmented;

[0131] A first integration unit 115 is used to integrate the frequency information of the image to be segmented through a multi-frequency fully connected layer module to obtain the weight of the frequency information of the image to be segmented;

[0132] A first multiplication unit 116 is used to multiply the frequency information of the image to be segmented and its weight element by element through a multiplier to obtain the adjusted frequency information of the image to be segmented;

[0133] A first reshaping unit 117, configured to adjust the adjustment frequency information of the image to be segmented through a reshaping module to obtain a frequency characteristic graph of the image to be segmented;

[0134] A first fusion unit 118 is used to fuse the spatial feature map and the frequency feature map of the image to be segmented through a hybrid domain fusion module to obtain a semantic segmentation result of the image to be segmented;

[0135] Among them, the image to be segmented is input into a nighttime semantic segmentation model; the nighttime semantic segmentation model includes a feature encoder, a spatial domain branch, a frequency domain branch, a multi-frequency fully connected layer module, a multiplier, a reshaping module and a mixed domain fusion module.

[0136] Figure 6 is a structural diagram of a night semantic segmentation model training device according to an embodiment of the present invention, such as Figure 6 As shown, the device comprises:

[0137] A second acquisition unit 211 is used to acquire a training segmentation image;

[0138] A second extraction unit 212 is used to extract features from the training segmented image through a feature encoder to obtain an original feature map of the training segmented image;

[0139] The second spatial unit 213 is used to enhance the original segmentation map of the training segmentation image through a spatial domain branch to obtain a spatial feature map of the training segmentation image;

[0140] A second frequency unit 214 is used to perform frequency extraction on the original segmentation map of the training segmentation image through a frequency domain branch to obtain frequency information of the training segmentation image;

[0141] A second integration unit 215 is used to integrate the frequency information of the training segmented image through a multi-frequency fully connected layer module to obtain the weight of the frequency information of the training segmented image;

[0142] A second multiplication unit 216 is used to multiply the frequency information of the training segmented image and its weight element by element through a multiplier to obtain the adjusted frequency information of the training segmented image;

[0143] A second reshaping unit 217 is used to adjust the adjustment frequency information of the training segmented image through a reshaping module to obtain a frequency characteristic map of the training segmented image;

[0144] A second fusion unit 218 is used to fuse the spatial feature map and the frequency feature map of the training segmented image through a hybrid domain fusion module to obtain a semantic segmentation result of the training segmented image;

[0145] A first training unit 219 is used to train the night semantic segmentation model according to the semantic segmentation result of the training segmented image to obtain a trained night semantic segmentation model;

[0146] The training segmented image is input into a nighttime semantic segmentation model; the nighttime semantic segmentation model includes a feature encoder, a spatial domain branch, a frequency domain branch, a multi-frequency fully connected layer module, a multiplier, a reshaping module, and a mixed domain fusion module.

[0147] In a preferred embodiment of the present invention, Figure 7 As shown, the first training unit includes:

[0148] A first determining unit 291 is used to obtain the ground truth semantic edge information of each pixel of the training segmented image and the probability that each pixel belongs to a specific category according to the pixel-by-pixel segmentation map label of the training segmented image;

[0149] A second determining unit 292 is used to obtain predicted edge information of each pixel of the training segmented image and a predicted probability that each pixel belongs to a specific category according to the semantic segmentation result of the training segmented image;

[0150] A first calculation unit 293, configured to calculate the segmentation loss of the training segmented image according to the probability that each pixel of the training segmented image belongs to a specific category and the predicted probability that each pixel of the training segmented image belongs to the specific category;

[0151] A second calculation unit 294 is used to calculate the edge loss of the training segmented image according to the ground truth semantic edge information of each pixel of the training segmented image and the predicted edge information of each pixel of the training segmented image;

[0152] A third calculation unit 295 is used to calculate the loss of the training segmented image according to the segmentation loss of the training segmented image and the edge loss of the training segmented image;

[0153] The first adjustment unit 296 is used to train the night semantic segmentation model according to the loss of the image to be segmented; when the loss of the image to be segmented converges, the trained night semantic segmentation model is obtained.

[0154] A person skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium, which can include: ROM, RAM, disk or CD, etc.

[0155] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A nighttime semantic segmentation method, characterized in that: The method comprises: Obtain the image to be segmented; Inputting the image to be segmented into a nighttime semantic segmentation model; the nighttime semantic segmentation model comprises a feature encoder, a spatial domain branch, a frequency domain branch, a multi-frequency fully connected layer module, a multiplier, a reshaping module, and a mixed domain fusion module; Extracting features of the image to be segmented by a feature encoder to obtain an original feature map of the image to be segmented; The original feature map of the image to be segmented is enhanced by a spatial domain branch to obtain a spatial feature map of the image to be segmented; Performing frequency extraction on the original feature map of the image to be segmented through a frequency domain branch to obtain frequency information of the image to be segmented; Integrating the frequency information of the image to be segmented through a multi-frequency fully connected layer module to obtain the weight of the frequency information of the image to be segmented; The frequency information of the image to be segmented and its weight are multiplied element by element by a multiplier to obtain the adjusted frequency information of the image to be segmented; The frequency information of the image to be segmented is adjusted by a reshaping module to obtain a frequency characteristic diagram of the image to be segmented; The spatial feature map and the frequency feature map of the image to be segmented are fused through a hybrid domain fusion module to obtain a semantic segmentation result of the image to be segmented.

2. A nighttime semantic segmentation method according to claim 1, characterized in that: The integrating the frequency information of the image to be segmented by the multi-frequency fully connected layer module to obtain the weight of the frequency information of the image to be segmented comprises: Segmenting the frequency information of the image to be segmented along the channel dimension to obtain the frequency information of the image to be segmented in each channel; The frequency information of the image to be segmented in each channel is passed through the fully connected layer one by one to obtain the weight of the image to be segmented in each channel.

3. The nighttime semantic segmentation method according to claim 1, characterized in that: The step of fusing the spatial feature map and the frequency feature map of the image to be segmented by a hybrid domain fusion module to obtain a semantic segmentation result of the image to be segmented includes: The spatial feature map of the image to be segmented is merged with the frequency feature map of the image to be segmented along the spatial level to obtain a spatial-frequency mixed feature map of the image to be segmented; The spatial feature map of the image to be segmented is merged with the frequency feature map of the image to be segmented along the channel level to obtain a frequency-space mixed feature map of the image to be segmented; The space-frequency mixed feature map of the image to be segmented is point-multiplied with the frequency-space mixed feature map of the image to be segmented to obtain a semantic segmentation result of the image to be segmented.

4. A nighttime semantic segmentation method according to claim 3, characterized in that: The step of fusing the spatial feature map of the image to be segmented with the frequency feature map of the image to be segmented along the spatial level to obtain the spatial-frequency mixed feature map of the image to be segmented comprises: Performing 1×1 convolution on the spatial feature map of the image to be segmented and the frequency feature map of the image to be segmented to obtain a first intermediate feature and a second intermediate feature of the image to be segmented; After adding the first intermediate feature and the second intermediate feature of the image to be segmented pixel by pixel, a space-frequency mixed feature map of the image to be segmented is obtained through a Sigmoid activation function.

5. A nighttime semantic segmentation method according to claim 3 or 4, characterized in that: The step of fusing the spatial feature map of the image to be segmented with the frequency feature map of the image to be segmented along the spatial level to obtain the spatial-frequency mixed feature map of the image to be segmented comprises: splicing the spatial feature map of the image to be segmented and the frequency feature map of the image to be segmented to obtain a third intermediate feature of the image to be segmented; The third intermediate feature of the image to be segmented is processed by cascade average pooling, maximum pooling, 1×1 convolution, and Sigmoid activation function to obtain the fourth intermediate feature of the image to be segmented; The third intermediate feature of the image to be segmented is multiplied by the fourth intermediate feature of the image to be segmented, and then a 1×1 convolution is performed to obtain a space-frequency mixed feature map of the image to be segmented.

6. A nighttime semantic segmentation model training method, characterized in that: The method comprises: Acquire a training segmentation image, where the training segmentation image is a training image of a night scene, and the training image includes a pixel-by-pixel segmentation map label; Inputting the training segmented image into a nighttime semantic segmentation model; the nighttime semantic segmentation model comprises a feature encoder, a spatial domain branch, a frequency domain branch, a multi-frequency fully connected layer module, a multiplier, a reshaping module, and a mixed domain fusion module; Extracting features from the training segmented image using a feature encoder to obtain an original feature map of the training segmented image; The original segmentation map of the training segmentation image is enhanced by a spatial domain branch to obtain a spatial feature map of the training segmentation image; Performing frequency extraction on the original segmentation map of the training segmentation image through a frequency domain branch to obtain frequency information of the training segmentation image; Integrate the frequency information of the training segmented image through a multi-frequency fully connected layer module to obtain the weight of the frequency information of the training segmented image; The frequency information of the training segmented image and its weight are multiplied element by element by a multiplier to obtain the adjusted frequency information of the training segmented image; The adjustment frequency information of the training segmented image is adjusted by a reshaping module to obtain a frequency characteristic graph of the training segmented image; The spatial feature map and the frequency feature map of the training segmented image are fused by a hybrid domain fusion module to obtain a semantic segmentation result of the training segmented image; The nighttime semantic segmentation model is trained according to the semantic segmentation result of the training segmented image to obtain a trained nighttime semantic segmentation model.

7. A nighttime semantic segmentation model training method according to claim 6, characterized in that: The step of training the nighttime semantic segmentation model according to the semantic segmentation result of the training segmented image to obtain the trained nighttime semantic segmentation model comprises: According to the pixel-by-pixel segmentation map label of the training segmented image, obtaining the ground truth semantic edge information of each pixel of the training segmented image and the probability that each pixel belongs to a specific category; According to the semantic segmentation result of the training segmented image, the predicted edge information of each pixel of the training segmented image and the predicted probability that each pixel belongs to a specific category are obtained; Calculating the segmentation loss of the training segmented image according to the probability that each pixel of the training segmented image belongs to a specific category and the predicted probability that each pixel of the training segmented image belongs to the specific category; Calculating the edge loss of the training segmented image according to the ground truth semantic edge information of each pixel of the training segmented image and the predicted edge information of each pixel of the training segmented image; Calculating the loss of the training segmented image according to the segmentation loss of the training segmented image and the edge loss of the training segmented image; Training the nighttime semantic segmentation model according to the loss of the image to be segmented; When the loss of the image to be segmented converges, the trained nighttime semantic segmentation model is obtained.

8. A nighttime semantic segmentation device, characterized in that: The device comprises: A first acquisition unit, used for acquiring an image to be segmented; A first extraction unit, configured to extract features of the image to be segmented by using a feature encoder to obtain an original feature map of the image to be segmented; A first spatial unit is used to enhance the original feature map of the image to be segmented through a spatial domain branch to obtain a spatial feature map of the image to be segmented; A first frequency unit, used for performing frequency extraction on an original feature map of the image to be segmented through a frequency domain branch to obtain frequency information of the image to be segmented; A first integration unit is used to integrate the frequency information of the image to be segmented through a multi-frequency fully connected layer module to obtain the weight of the frequency information of the image to be segmented; A first multiplication unit, configured to multiply the frequency information of the image to be segmented and its weight element by element through a multiplier to obtain the adjusted frequency information of the image to be segmented; A first reshaping unit, configured to adjust the adjustment frequency information of the image to be segmented through a reshaping module to obtain a frequency characteristic graph of the image to be segmented; A first fusion unit is used to fuse the spatial feature map and the frequency feature map of the image to be segmented through a hybrid domain fusion module to obtain a semantic segmentation result of the image to be segmented; Among them, the image to be segmented is input into a nighttime semantic segmentation model; the nighttime semantic segmentation model includes a feature encoder, a spatial domain branch, a frequency domain branch, a multi-frequency fully connected layer module, a multiplier, a reshaping module and a mixed domain fusion module.

9. A nighttime semantic segmentation model training device, characterized in that: The device comprises: A second acquisition unit, used for acquiring a training segmentation image; A second extraction unit, configured to perform feature extraction on the training segmented image through a feature encoder to obtain an original feature map of the training segmented image; A second spatial unit is used to enhance the original segmentation map of the training segmentation image through a spatial domain branch to obtain a spatial feature map of the training segmentation image; A second frequency unit is used to perform frequency extraction on the original segmentation map of the training segmentation image through a frequency domain branch to obtain frequency information of the training segmentation image; A second integration unit is used to integrate the frequency information of the training segmented image through a multi-frequency fully connected layer module to obtain the weight of the frequency information of the training segmented image; A second multiplication unit is used to multiply the frequency information of the training segmented image and its weight element by element through a multiplier to obtain the adjusted frequency information of the training segmented image; A second reshaping unit, configured to adjust the adjustment frequency information of the training segmented image through a reshaping module to obtain a frequency characteristic graph of the training segmented image; A second fusion unit is used to fuse the spatial feature map and the frequency feature map of the training segmented image through a hybrid domain fusion module to obtain a semantic segmentation result of the training segmented image; A first training unit is used to train the nighttime semantic segmentation model according to the semantic segmentation result of the training segmented image to obtain a trained nighttime semantic segmentation model; The training segmented image is input into a nighttime semantic segmentation model; the nighttime semantic segmentation model includes a feature encoder, a spatial domain branch, a frequency domain branch, a multi-frequency fully connected layer module, a multiplier, a reshaping module, and a mixed domain fusion module.

10. The nighttime semantic segmentation model training device according to claim 9, characterized in that: The first training unit comprises: A first determining unit is used to obtain the ground truth semantic edge information of each pixel of the training segmented image and the probability that each pixel belongs to a specific category according to the pixel-by-pixel segmentation map label of the training segmented image; A second determination unit is used to obtain predicted edge information of each pixel of the training segmented image and a predicted probability that each pixel belongs to a specific category according to the semantic segmentation result of the training segmented image; A first calculation unit, configured to calculate a segmentation loss of the training segmented image according to a probability that each pixel of the training segmented image belongs to a specific category and a predicted probability that each pixel of the training segmented image belongs to the specific category; A second calculation unit is used to calculate the edge loss of the training segmented image according to the ground truth semantic edge information of each pixel of the training segmented image and the predicted edge information of each pixel of the training segmented image; A third calculation unit, configured to calculate the loss of the training segmented image according to the segmentation loss of the training segmented image and the edge loss of the training segmented image; The first adjustment unit is used to train the night semantic segmentation model according to the loss of the image to be segmented; when the loss of the image to be segmented converges, the trained night semantic segmentation model is obtained.

Images