Image segmentation method, device, equipment and storage medium

Abstract

Embodiments of the present application disclose a kind of image segmentation method, device, equipment and storage medium, belong to image processing technical field.The method comprises: by image segmentation model, the first image spatial domain feature of night scene image is extracted;The first image spatial domain feature is transformed, and the first image frequency domain feature is obtained, and the first image frequency domain feature includes the sub-frequency domain feature of at least two frequency bands;Based on the illumination sensitivity difference between the sub-frequency domain feature of at least two frequency bands, at least two different feature fine-tuning ways are used to fine-tune the sub-frequency domain feature of at least two frequency bands, and the second image frequency domain feature is obtained;The second image frequency domain feature is inversely transformed, and the second image spatial domain feature is obtained;The second image spatial domain feature is decoded by the image segmentation model, and the image segmentation result corresponding to the night scene image is obtained.The present scheme can improve the image segmentation accuracy of night scene image.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image segmentation method, apparatus, device, and storage medium. Background Technology

[0002] Image segmentation is a fundamental technique in digital image processing. Its goal is to divide an image into several non-overlapping regions, where pixels within each region share similarities in certain features (such as grayscale, color, and texture). Scene segmentation, an important branch of image segmentation, achieves a detailed understanding of the entire scene by assigning each pixel in an image to a specific semantic category. Image segmentation has wide applications in fields such as autonomous driving, intelligent recognition, robot navigation, and medical image analysis.

[0003] In related technologies, a scene segmentation algorithm based on deep learning is adopted. Based on a deep image recognition network, the semantic features of the scene image are extracted and processed through multiple convolutional layers to determine the category confidence of each pixel in the scene image, thereby obtaining the image segmentation result corresponding to the scene image.

[0004] However, the pixel features of each pixel in a scene image are usually affected by illumination. There are significant differences between the image features of daytime images and nighttime images in the same scene. Using the scene segmentation method for daytime images to segment nighttime images will reduce the accuracy of image segmentation. Summary of the Invention

[0005] This application provides an image segmentation method, apparatus, device, and storage medium, the technical solution of which is as follows:

[0006] On one hand, embodiments of this application provide an image segmentation method, the method comprising:

[0007] The first spatial features of the night scene image are extracted using an image segmentation model;

[0008] The first image spatial domain features are transformed to obtain the first image frequency domain features, which include sub-frequency domain features of at least two frequency bands.

[0009] Based on the difference in light sensitivity between the sub-frequency domain features of the at least two frequency bands, the sub-frequency domain features of the at least two frequency bands are fine-tuned using at least two different feature fine-tuning methods to obtain the second image frequency domain features;

[0010] The second image frequency domain features are inversely transformed to obtain the second image spatial domain features;

[0011] The image segmentation result corresponding to the night scene image is obtained by decoding the spatial features of the second image using the image segmentation model.

[0012] On the other hand, embodiments of this application provide an image segmentation apparatus, the apparatus comprising:

[0013] The feature extraction module is used to extract the first image spatial features of the night scene image through an image segmentation model;

[0014] The feature transformation module is used to transform the spatial domain features of the first image to obtain the frequency domain features of the first image, wherein the frequency domain features of the first image include sub-frequency domain features of at least two frequency bands.

[0015] The feature fine-tuning module is used to fine-tune the sub-frequency domain features of the at least two frequency bands based on the difference in light sensitivity between the sub-frequency domain features of the at least two frequency bands using at least two different feature fine-tuning methods to obtain the second image frequency domain features.

[0016] The feature inverse transformation module is used to perform an inverse transformation on the frequency domain features of the second image to obtain the spatial domain features of the second image.

[0017] The feature decoding module is used to decode the spatial features of the second image through the image segmentation model to obtain the image segmentation result corresponding to the night scene image.

[0018] On the other hand, embodiments of this application provide a computer device including a processor and a memory, wherein the memory stores at least one instruction, which is loaded and executed by the processor to implement the image segmentation method as described above.

[0019] On the other hand, embodiments of this application provide a computer-readable storage medium storing at least one instruction, which is loaded and executed by a processor to implement the image segmentation method as described above.

[0020] On the other hand, embodiments of this application provide a computer program product including at least one instruction stored in a computer-readable storage medium. A processor of a computer device reads the at least one instruction from the computer-readable storage medium and executes the at least one instruction, causing the computer device to perform the image segmentation method described above.

[0021] In this embodiment, during the image segmentation of a night scene image using an image segmentation model, after extracting the first image spatial domain features of the night scene image, the first image spatial domain features are transformed to obtain a first image frequency domain feature including sub-frequency domain features of at least two frequency bands. Furthermore, based on the differences in light sensitivity between the sub-frequency domain features of the at least two frequency bands, at least two different feature fine-tuning methods are used to fine-tune the sub-frequency domain features of the at least two frequency bands, thereby obtaining a second image frequency domain feature. Then, the second image frequency domain feature is inversely transformed into a second image spatial domain feature, and the second image spatial domain feature is decoded to obtain the image segmentation result corresponding to the night scene image.

[0022] The solution provided in this application, based on the different sensitivities of sub-frequency domain features to illumination across different frequency bands, fine-tunes image features in the frequency domain using different feature fine-tuning methods. This allows the image segmentation model to focus more on the relatively stable sub-frequency domain features with lower illumination sensitivity, while simultaneously increasing the robustness of sub-frequency domain features with higher illumination sensitivity. Furthermore, the image segmentation model can also learn the fine-tuned sub-frequency domain features with higher illumination sensitivity, thereby improving the accuracy of image segmentation for night scene images. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This application shows a structural block diagram of a computer system provided in an exemplary embodiment;

[0025] Figure 2 A flowchart illustrating an exemplary embodiment of the image segmentation method provided in this application is shown;

[0026] Figure 3 This illustration shows a flowchart of generating image segmentation results according to an exemplary embodiment of this application;

[0027] Figure 4 A flowchart of an image segmentation method provided by another exemplary embodiment of this application is shown;

[0028] Figure 5 The figure shows experimental results of frequency domain analysis provided by an exemplary embodiment of this application;

[0029] Figure 6A schematic diagram illustrating the channel response values ​​of different feature channels provided in an exemplary embodiment of this application is shown;

[0030] Figure 7 A flowchart illustrating an exemplary embodiment of the image segmentation method provided in this application is shown.

[0031] Figure 8 This application illustrates a flowchart of training an image segmentation model according to an exemplary embodiment.

[0032] Figure 9 This illustration shows a flowchart of training an image segmentation model according to an exemplary embodiment of this application;

[0033] Figure 10 This illustration shows a schematic diagram of the entire training and inference process of an image segmentation method provided in an exemplary embodiment of this application;

[0034] Figure 11 A flowchart illustrating an exemplary embodiment of this application of an image segmentation method applied to an autonomous driving scenario is shown.

[0035] Figure 12 A flowchart illustrating an exemplary embodiment of this application of an image segmentation method applied to a road recognition scenario is shown.

[0036] Figure 13 This invention provides a structural block diagram of an image segmentation apparatus according to an exemplary embodiment of the present application.

[0037] Figure 14 A schematic diagram of the structure of a computer device provided in an exemplary embodiment of this application is shown. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0039] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0040] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0041] It should be understood that although the terms first, second, etc., may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, a first parameter may also be referred to as a second parameter, and similarly, a second parameter may also be referred to as a first parameter. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0042] First, a brief introduction to the terms used in the embodiments of this application:

[0043] Image segmentation is a fundamental technique in digital image processing. Its goal is to divide an image into several non-overlapping regions, where pixels within each region share similarities in certain features (such as grayscale, color, texture, etc.). Optionally, image segmentation can include semantic segmentation, instance segmentation, panoptic segmentation, and other types.

[0044] Scene segmentation, an important branch of image segmentation, aims to classify pixels or regions in an image into different categories or objects, thereby understanding the structure and composition of the scene. Scene segmentation not only focuses on objects in an image but also on the semantic information of the entire scene, including elements such as background, ground, and sky.

[0045] Image spatial features refer to features extracted directly from the pixel values ​​of an image, reflecting information about the image's spatial location. Optionally, image spatial features mainly focus on the distribution, positional relationships, and local structure of pixels in the image. Image spatial features can include grayscale features, color features, texture features, edge features, and shape features, etc.

[0046] Image frequency domain features: Features extracted by transforming an image from the spatial domain to the frequency domain, or features extracted by transforming spatial domain features from the spatial domain to the frequency domain. Frequency domain features are used to describe the distribution and intensity of different frequency components in an image, and they can separate high-frequency and low-frequency components in an image. High-frequency components usually correspond to details and edge information, while low-frequency components correspond to overall structure and background information.

[0047] Please refer to Figure 1This illustration shows a structural block diagram of a computer system provided in an exemplary embodiment of this application. The computer system may include a terminal 110 and a server 120. The terminal 110 and the server 120 communicate via a communication network. Optionally, the communication network may be a wired network or a wireless network, and the communication network may be at least one of a local area network (LAN), a metropolitan area network (MAN), and a wide area network (WAN).

[0048] Terminal 110 is an electronic device with an application that has image segmentation functionality installed. This image segmentation functionality can be a native application in terminal 110 or a third-party application; terminal 110 can be a smartphone, tablet, laptop, desktop computer, smart TV, wearable device, or in-vehicle terminal, etc. Figure 1 The example of terminal 110 being a desktop computer is used for illustration only, but it is not a limitation.

[0049] Server 120 includes at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. In this embodiment, server 120 can be a backend server for an application with image segmentation capabilities.

[0050] In some embodiments, data interaction exists between server 120 and terminal 110. (Illustrative example, such as...) Figure 1 As shown, after receiving the image upload operation, terminal 110 sends the received night scene image to server 120. Server 120 then inputs the night scene image into an image segmentation model, extracts image features from the night scene image using the image encoder in the image segmentation model, and obtains the first image spatial domain features of the night scene image. Further, server 120 transforms the first image spatial domain features from the spatial domain to the frequency domain to obtain the first image frequency domain features. Based on the difference in light sensitivity between the sub-frequency domain features of at least two frequency bands in the first image frequency domain features, at least two different feature fine-tuning methods are used to fine-tune the first image frequency domain features, thereby obtaining the second image frequency domain features. Finally, by inversely transforming the second image frequency domain features from the frequency domain to the spatial domain, the second image spatial domain features are obtained. The second image spatial domain features are then decoded by the decoder in the image segmentation model, thus obtaining the image segmentation result corresponding to the night scene image. Server 120 then returns the image segmentation result to terminal 110, allowing terminal 110 to display the image segmentation result to the user.

[0051] Based on the above introduction, the image segmentation method provided in this application will be described. This method can be executed by a server or a terminal, or by both a server and a terminal.

[0052] Please refer to Figure 2This document illustrates a flowchart of an image segmentation method provided in an exemplary embodiment of this application. This embodiment uses the method applied to a computer device (including a terminal and / or a server) as an example for illustration. The method includes the following steps:

[0053] Step 210: Extract the first image spatial features of the night scene image using an image segmentation model.

[0054] Optionally, night scene images refer to images taken under low-light conditions, typically with low light intensity and a dark background. Optionally, compared to daytime images, night scene images have a narrower pixel value distribution range, with pixel values ​​concentrated in low-brightness areas, usually between 0 and 100, while highlight areas are typically concentrated in areas with artificial lights. Furthermore, night scene images tend to have a more uniform color, often leaning towards blue or black.

[0055] Optionally, the night scene image can be an image taken indoors under low light conditions or an image taken outdoors at night. That is, when taking the night scene image, the ambient light intensity is relatively weak. For example, the light intensity in indoor low light conditions is usually between 5 and 50 lux, while the light intensity in outdoor nighttime conditions is usually between 0.001 and 0.3 lux.

[0056] Optionally, the night scene image can be an image that has been artificially marked; it can also be an image obtained by taking pictures when the light intensity meets the conditions, such as a street night scene image taken when the street light intensity is between 10 and 50 lux; or it can be an image taken within a specific time period, such as an image taken between sunset and sunrise.

[0057] Optionally, to improve the image processing efficiency of the image segmentation model, the image size of night scene images can be standardized. For example, the image size of night scene images can be uniformly set to 512*512 pixels.

[0058] Optionally, the image segmentation model can be a model based on Vision Foundation Models (VFM). Optionally, the image segmentation model includes an image encoder (for extracting image features), an adapter (for feature fine-tuning), and a decoder (for feature decoding).

[0059] Optionally, the image encoder may include a patch embedding layer and a Transformer encoder. For example, the image encoder may be a DINO (Distillation with No Labels) encoder or other VFM image encoders, which are not limited in this application embodiment.

[0060] In some embodiments, in order to achieve image segmentation based on image features, the computer device inputs the night scene image into the image segmentation model. First, the image features of the night scene image are extracted by the image encoder in the image segmentation model, thereby obtaining the first image spatial domain features of the night scene image.

[0061] The first image spatial domain feature refers to the image features of the night scene image in the spatial domain, which can reflect the information of the night scene image in spatial location, such as the distribution, positional relationship and local structure of pixels in the night scene image. Optionally, the first image spatial domain feature can be represented as f, and the feature dimension of the first image spatial domain feature is C×N, where C is the number of feature channels and N is the number of patches.

[0062] Indicative, such as Figure 3 As shown, the computer device inputs the night scene image 301 into the image segmentation model 310, thereby extracting the image features of the night scene image 301 through the image encoder 302 in the image segmentation model 310, and obtaining the first image spatial domain feature 303 of the night scene image 301.

[0063] Step 220: Transform the spatial domain features of the first image to obtain the frequency domain features of the first image. The frequency domain features of the first image include sub-frequency domain features of at least two frequency bands.

[0064] Optionally, transforming the spatial domain features of the first image means transforming the spatial domain features of the first image from the spatial domain to the frequency domain. The spatial domain, also known as the image space, refers to the space composed of image pixels. Optionally, the frequency domain describes the features of the image through frequency. Optionally, spatial domain features and frequency domain features can be transformed into each other.

[0065] In this embodiment of the application, in order to fine-tune the image features based on the difference in light sensitivity between different frequency domain features, after extracting the first image spatial domain features, it is also necessary to transform the first image spatial domain features from the spatial domain to the frequency domain, so as to obtain the first image frequency domain features corresponding to the first image spatial domain features.

[0066] Optionally, the spatial domain features can be transformed into frequency domain features using the Discrete Sine Transform (DST), the Fourier Transform, or other feature transformation methods. This application does not limit the specific methods used in this embodiment.

[0067] Optionally, the first image frequency domain feature can be represented as V, and the feature dimension of the first image frequency domain feature is also C×N, where C is the number of feature channels and N is the number of patches.

[0068] Optionally, the first image frequency domain features may include high-frequency features and low-frequency features. High-frequency features are used to characterize the details and edge information of the image and are usually more affected by illumination; while low-frequency features are used to characterize the overall structure and background information of the image and are usually less affected by illumination.

[0069] In some embodiments, to fine-tune image features based on differences in light sensitivity between different frequency domain features, a computer device can divide the first image frequency domain features into sub-frequency domain features of at least two frequency bands according to the frequency values ​​corresponding to different frequency domain features in the first image frequency domain features. Here, a frequency band refers to a frequency range, that is, the range from the lowest frequency to the highest frequency within the frequency range.

[0070] Indicative, such as Figure 3 As shown, in order to achieve feature fine-tuning based on frequency domain features, the computer device transforms the first image spatial domain feature 303 from the spatial domain to the frequency domain, thereby obtaining the first image frequency domain feature 304.

[0071] Step 230: Based on the difference in light sensitivity between the sub-frequency domain features of at least two frequency bands, at least two different feature fine-tuning methods are used to fine-tune the sub-frequency domain features of at least two frequency bands to obtain the second image frequency domain features.

[0072] In some embodiments, after obtaining sub-frequency domain features of at least two frequency bands through feature division, the computer device can fine-tune the frequency domain features of at least two frequency bands based on the difference in light sensitivity between the sub-frequency domain features of at least two frequency bands using at least two different feature fine-tuning methods, so that the sub-frequency domain features of at least two frequency bands after fine-tuning constitute the second image frequency domain features.

[0073] Optionally, feature fine-tuning can also be understood as feature optimization. Optionally, after feature fine-tuning, the second image frequency domain features are more robust to changes in illumination. For illumination-insensitive sub-frequency domain features, feature fine-tuning can make the model pay more attention to these types of frequency domain features; for illumination-sensitive sub-frequency domain features, feature fine-tuning can increase the robustness of these types of frequency domain features.

[0074] Among these, at least two different feature fine-tuning methods result in different degrees of feature fine-tuning for the sub-frequency domain features. Optionally, the difference between the at least two different feature fine-tuning methods could be whether preprocessing is used to reduce the inter-domain gap of the sub-frequency domain features in different scenarios to improve the robustness of the sub-frequency domain features; or it could be that different degrees of preprocessing are used to reduce the inter-domain gap of the sub-frequency domain features in different scenarios to different degrees. That is, different degrees of preprocessing will result in different inter-domain gaps of the preprocessed sub-frequency domain features in different scenarios, and the higher the degree of preprocessing, the smaller the inter-domain gap of the preprocessed sub-frequency domain features in different scenarios.

[0075] Optionally, the relationship between sub-frequency domain features and feature fine-tuning methods is many-to-many, meaning that different sub-frequency domain features can adopt their own feature fine-tuning methods. Optionally, different sub-frequency domain features correspond to different feature fine-tuning methods, or at least two sub-frequency domain features adopt the same feature fine-tuning method.

[0076] In one possible approach, at least two feature fine-tuning methods include at least one feature fine-tuning method that does not perform preprocessing and at least one feature fine-tuning method that performs preprocessing, wherein, in the case of including at least two feature fine-tuning methods that perform preprocessing, the at least two feature fine-tuning methods that perform preprocessing may perform the same degree of preprocessing or may perform different degrees of preprocessing.

[0077] In another possible approach, at least two feature fine-tuning methods perform preprocessing, but at least two feature fine-tuning methods perform preprocessing to different degrees.

[0078] For example, when there are low-frequency, mid-frequency, and high-frequency features, since the light sensitivity varies among these features, different feature fine-tuning methods can be used to fine-tune each feature separately. However, for mid-frequency and high-frequency features, since the difference in light sensitivity is relatively small, the same feature fine-tuning method can be used for both.

[0079] Optionally, the image segmentation model includes at least two adapters, with different adapters used to fine-tune the features of sub-frequency domain features in different frequency bands, and each adapter corresponding to its own feature fine-tuning method.

[0080] Optionally, the second image frequency domain features can be represented as The feature dimension of the second image frequency domain feature is also C×N, where C is the number of feature channels and N is the number of patches.

[0081] Indicative, such as Figure 3 As shown, after obtaining the first image frequency domain feature 304, the computer device can fine-tune the feature based on the differences in light sensitivity between the sub-frequency domain features of different frequency bands in the first image frequency domain feature 304, using different feature fine-tuning methods, thereby obtaining the second image frequency domain feature 305. The feature fine-tuning process can be executed through the adapter 311, with different feature fine-tuning methods corresponding to their respective adapters. Figure 3 The example is based on adapter 311.

[0082] Step 240: Perform an inverse transform on the frequency domain features of the second image to obtain the spatial domain features of the second image.

[0083] In some embodiments, after obtaining the second image frequency domain features, the computer device can inversely transform the second image frequency domain features from the frequency domain to the spatial domain by means of inverse transformation, thereby obtaining the second image spatial domain features corresponding to the night scene image.

[0084] Optionally, the inverse transformation of frequency domain features into spatial domain features can be performed using the inverse discrete sine transform (IDST), the inverse fourier transform (Inverse Fourier Transform), or other feature transformation methods. This application does not limit these methods.

[0085] Optionally, the second image spatial domain feature can be represented as f′, and the feature dimension of the second image spatial domain feature is also C×N, where C is the number of feature channels and N is the number of patches.

[0086] For example, such as Figure 3 As shown, in order to facilitate subsequent feature decoding, the computer device also needs to perform an inverse transformation to convert the second image frequency domain feature 305 from the frequency domain to the spatial domain, thereby obtaining the second image spatial domain feature 306 corresponding to the night scene image 301.

[0087] Step 250: Decode the spatial features of the second image using an image segmentation model to obtain the image segmentation result corresponding to the night scene image.

[0088] The image segmentation result refers to the division of a night scene image into at least two non-overlapping regions. Optionally, the image segmentation result is a semantic segmentation result, where the value of each pixel in the semantic segmentation result is used to characterize the semantic category of that pixel in the night scene image. Optionally, the image segmentation result can be understood as a pixel-level labeled image of the night scene image, that is, each pixel in the image segmentation result corresponds to its own label, which is used to characterize the semantic category to which the pixel belongs in the night scene image.

[0089] For example, a night scene image includes roads, cars, buildings, and the sky. In the semantic annotation process, the category number corresponding to the road is represented by the number 1, the category number corresponding to the car is represented by the number 2, the category number corresponding to the building is represented by the number 3, and the category number corresponding to the sky is represented by the number 4. Then, in the image segmentation result corresponding to the night scene image, the pixel value representing the road is 1, the pixel value representing the car is 2, the pixel value representing the building is 3, and the pixel value representing the sky is 4.

[0090] In some embodiments, after obtaining the second image spatial features, the computer device can input the second image spatial features into the decoder, and by decoding the second image spatial features, the image segmentation result corresponding to the night scene image can be obtained.

[0091] Optionally, the feature decoding process can be understood as the process of restoring the spatial domain features of the second image to an output of the same size as the input image through the decoder. The decoder can gradually recover detailed image semantic information based on the spatial domain features of the second image, thereby generating image segmentation results based on the image semantic information.

[0092] Optionally, the decoder can be the decoder part of Mask2Former, or other decoders that can predict the semantic category of pixels. This application embodiment does not limit this.

[0093] Indicative, such as Figure 3 As shown, the computer device decodes the second image spatial domain features 306 through decoder 307, recovers the image semantic information of each pixel in the night scene image 301, thereby completing the image segmentation of the night scene image 301 and outputting its corresponding image segmentation result 308.

[0094] In summary, in the embodiments of this application, during the image segmentation of a night scene image using an image segmentation model, after extracting the first image spatial domain features of the night scene image, the first image spatial domain features are transformed to obtain first image frequency domain features including sub-frequency domain features of at least two frequency bands. Furthermore, based on the differences in light sensitivity between the sub-frequency domain features of the at least two frequency bands, at least two different feature fine-tuning methods are used to fine-tune the sub-frequency domain features of the at least two frequency bands, thereby obtaining second image frequency domain features. Then, the second image frequency domain features are inversely transformed into second image spatial domain features, and the second image spatial domain features are decoded to obtain the image segmentation result corresponding to the night scene image.

[0095] The solution provided in this application, based on the different sensitivities of sub-frequency domain features to illumination across different frequency bands, fine-tunes image features in the frequency domain using different feature fine-tuning methods. This allows the image segmentation model to focus more on the relatively stable sub-frequency domain features with lower illumination sensitivity, while simultaneously increasing the robustness of sub-frequency domain features with higher illumination sensitivity. Furthermore, the image segmentation model can also learn the fine-tuned sub-frequency domain features with higher illumination sensitivity, thereby improving the accuracy of image segmentation for night scene images.

[0096] In some embodiments, based on the sensitivity of sub-frequency domain features of different frequency bands to illumination, the first image frequency domain features can be divided into a first sub-frequency domain feature and a second sub-frequency domain feature. The first sub-frequency domain feature can also be called an illumination-insensitive frequency domain feature, and the second sub-frequency domain feature can also be called an illumination-sensitive frequency domain feature. Therefore, different feature fine-tuning methods are used for the illumination-sensitive and illumination-insensitive frequency domain features. This process will be described below through specific embodiments.

[0097] Please refer to Figure 4 This document illustrates a flowchart of an image segmentation method provided in an exemplary embodiment of this application. This embodiment uses the method applied to a computer device (including a terminal and / or a server) as an example for illustration. The method includes the following steps:

[0098] Step 210: Extract the first image spatial features of the night scene image using an image segmentation model.

[0099] In some embodiments, in order to fully learn the semantic information of night scene images, the image segmentation model may include at least two image encoders and connect them in series, thereby using at least two image encoders to extract spatial features of the image layer by layer, realizing the construction of complex high-level image features from simple low-level image features, and increasing the model's understanding of night scene images.

[0100] Furthermore, to improve the accuracy of image feature understanding, an adapter can be placed between every two image encoders. This adapter fine-tunes the spatial features output by the previous image encoder, and the fine-tuned spatial features are then used as input to the next image encoder. In other words, the input to the next image encoder is the finely-tuned spatial features output by the previous image encoder.

[0101] For example, the first image spatial feature output by the i-th image encoder is f iThis feature is not directly input into the (i+1)th image encoder, but needs to be fine-tuned to obtain the second image spatial domain feature f. i ′, thereby further processing the spatial domain features f of the second image i ′ is used as the input to the (i+1)th image encoder.

[0102] Step 221: Transform the spatial domain features of the first image using discrete sine transform to obtain the frequency domain features of the first image.

[0103] Considering that the discrete sine transform has good energy compression characteristics and can concentrate most of the image information in a few frequency bands, in this embodiment, in order to improve the efficiency of subsequent fine-tuning of the image frequency domain features, the discrete sine transform is used to transform the first image spatial domain features from the spatial domain to the frequency domain, thereby obtaining the first image frequency domain features.

[0104] Optionally, the Discrete Sine Transform (DST) can project spatial features onto a frequency domain space represented by a pure real matrix. The process of projecting the spatial features of the first image into the frequency features of the first image via DST can be represented as follows:

[0105]

[0106] Where C is the number of feature channels, N is the number of patches, u = 1, ..., C, n = 1, ..., N.

[0107] Step 231: Based on at least two frequency bands, divide the frequency domain features of the first image into a first sub-frequency domain feature and a second sub-frequency domain feature. The frequency of the frequency band corresponding to the first sub-frequency domain feature is lower than the frequency of the frequency band corresponding to the second sub-frequency domain feature. The sensitivity of the first sub-frequency domain feature to illumination is lower than the sensitivity of the second sub-frequency domain feature to illumination.

[0108] Optionally, in order to determine the sensitivity of different frequency domain features in the first image frequency domain features to illumination, in this embodiment of the application, before dividing the frequency band, frequency domain analysis is first performed through the sample image set and the verification image set, so as to determine that the low frequency features are less sensitive to illumination and the mid-to-high frequency features are more sensitive to illumination based on the frequency domain analysis results.

[0109] Frequency domain analysis

[0110] Regarding the frequency domain analysis process, in one possible implementation, for the sample image used for frequency domain analysis, a computer device can extract features from the sample image using an image encoder and perform a discrete sine transform to obtain the corresponding image frequency domain features. Then, the computer device sorts the image frequency domain features in descending order of sine coefficients; that is, as the number of feature channels gradually increases, the frequency of the frequency domain features gradually decreases, and divides the image frequency domain features into 8 frequency bands. For example, taking a feature channel number C of 1024 as an example, the frequency domain feature located in the 1st feature channel corresponds to the highest frequency, and the frequency domain feature located in the 1024th feature channel corresponds to the lowest frequency. The frequency domain features of the 8 frequency bands obtained after frequency band division are V... [0 ,128) V [128,256) V [256,384) V [384,512) V [512,640) V [640,768) V [768,896) and V [896,1024) .

[0111] Furthermore, to determine the impact of frequency domain features from different frequency bands in the unseen nighttime region, the computer device utilizes a band rejection filter to perform classification experiments on daytime and nighttime scenes by removing the frequency domain features of each band. Optionally, the process of removing frequency bands using a band rejection filter can be represented as:

[0112]

[0113] Among them, B r This indicates a band rejection filter. The flag is the number of band removal markers. Setting the flag corresponding to a band to 0 means removing that band.

[0114] For the frequency domain features of an image after removing one frequency band, the computer device processes them using a two-layer multilayer perceptron (MLP). The first MLP layer maps the frequency domain features to feature embeddings, and the second MLP layer performs binary classification. For example, the computer device maps the frequency domain features V... [0,128) By removing these features, the image's frequency domain features are reduced to only the remaining seven frequency bands. After feature mapping and binary classification using an MLP (Multi-Level Processing) to the removed frequency domain features, the computer can obtain the binary classification result output by the MLP. Furthermore, the computer combines this with the ground truth binary values ​​corresponding to the sample images to determine the accuracy of the binary classification task after removing these frequency domain features.

[0115] Indicative, such as Figure 5The diagram illustrates experimental results of frequency domain analysis provided in an exemplary embodiment of this application. In this embodiment, the CityScapes training set is used as the source domain to train a binary classification MLP, which is then validated using the CityScapes validation set and the ACDC-night dataset. Source accuracy and target accuracy line graphs are obtained by calculating the accuracy of the binary classification task without removing frequency domain features and the accuracy of the binary classification task with the removal of frequency domain features from each frequency band. Simultaneously, the NightCity training set is used as the source domain to train a binary classification MLP, which is then validated using the NightCity validation set and the ACDC-night dataset. Source accuracy and target accuracy line graphs are obtained by calculating the accuracy of the binary classification task without removing frequency domain features and the accuracy of the binary classification task with the removal of frequency domain features from each frequency band.

[0116] from Figure 5 It can be seen that after removing the frequency domain feature V [768,896) or V [896,1024) That is, when low-frequency features are removed, the accuracy of binary classification tasks is significantly lower; while when frequency domain features V are removed... [0,128) V [128,256) V [256,384) V [384,512) V [512,640) or V [640,768) In other words, when using mid-to-high frequency features, the accuracy of binary classification tasks changes relatively little.

[0117] Based on the frequency domain analysis process described above, it can be seen that removing high-frequency features has little impact on the model's understanding of image features and can significantly improve the generalization ability from the daytime or nighttime source domain to the unseen nighttime target domain. Removing mid-frequency features can also slightly enhance the generalization ability, while removing low-frequency features has a greater impact on the model's understanding of image features and will have a negative impact on the generalization ability of the source and target domains.

[0118] Indicative, such as Figure 6 The diagram illustrates the channel response values ​​of different feature channels provided in an exemplary embodiment of this application. The sample images are from the Cityscapes dataset, Night City dataset, Night Driving dataset, BDD-Night dataset, ACDC-Night dataset, and All-day City dataset. Frequency domain analysis of the sample images reveals that between different sample images (between daytime images and different nighttime images), the low-frequency feature V... [768,896) or V [896,1024) The channel response value of the characteristic channel is relatively stable, while the mid-to-high frequency characteristic V [0,128) V [128,256) V[256,384) V [384,512) V [512,640) or V [640,768) The channel response value of the characteristic channel is relatively volatile.

[0119] Combination Figure 5 and Figure 6 It can be seen that the inter-domain difference between daytime and unseen nighttime scenes is mainly distributed in high frequency and part of mid frequency, that is, mid-to-high frequency features are more sensitive to light, while low frequency features are less sensitive to light.

[0120] Therefore, after obtaining the first image frequency domain features, the computer device divides the first image frequency domain features into light-sensitive frequency domain features (second sub-frequency domain features) and light-insensitive frequency domain features (first sub-frequency domain features) according to the difference in light sensitivity between the sub-frequency domain features of different frequency bands.

[0121] Among them, the frequency of the frequency band corresponding to the second sub-frequency domain feature is higher than the frequency of the frequency band corresponding to the first sub-frequency domain feature, and the sensitivity of the second sub-frequency domain feature to illumination is higher than that of the first sub-frequency domain feature to illumination.

[0122] In one possible implementation, in order to ensure the accuracy of frequency band division, the computer device may further divide the first image frequency domain features into frequency domain features of at least two frequency bands through at least two bandpass filters, thereby determining the frequency domain features corresponding to the frequency bands in the first frequency range as the first sub-frequency domain features, and determining the frequency domain features corresponding to the frequency bands in the second frequency range as the second sub-frequency domain features, wherein the frequency of the first frequency range is lower than the frequency of the second frequency range.

[0123] The number of bandpass filters is the same as the number of frequency bands. For example, if there are 8 frequency bands, then 8 bandpass filters can be set, with each bandpass filter corresponding to one frequency band.

[0124] For example, let's take dividing the first image frequency domain features into 8 frequency bands. After obtaining the first image frequency domain features V, the computer device inputs the first image frequency domain features into 8 bandpass filters, and each bandpass filter corresponds to one frequency band, thereby obtaining the sub-frequency domain features of the 8 frequency bands output by the 8 bandpass filters.

[0125] Optionally, to facilitate the division of the frequency domain features of the first image according to frequency, before frequency band division, the computer device can first sort the frequency domain features of each feature channel in the frequency domain features of the first image in descending order according to the sine coefficient, so that as the number of feature channels gradually increases, the frequency of the frequency domain features gradually decreases. For example, if the number of feature channels in the first image frequency domain features is 1024, after descending sorting and frequency band division, 8 frequency domain features can be obtained, namely V... [0,128)V [128,256) V [256,384) V [384,512) V [512,640) V [640,768) V [768,896) and V [896,1024) Optionally, the process of uniformly dividing the frequency domain features of the first image into 8 frequency bands can be represented as:

[0126] V b =B p (V)={V [0,128) ,…,V [896,1024)}

[0127] Among them, the frequency domain characteristics V of each frequency band b The number of feature channels is 128 for each.

[0128] Furthermore, since the frequency of the frequency domain features gradually decreases as the number of feature channels increases, the second frequency range can be set to the range between the frequency of the first feature channel and the frequency of the 768th feature channel, and the first frequency range can be set to the range between the frequency of the 768th feature channel and the frequency of the 1024th feature channel. That is, V [0,128) V [128,256) V [256,384) V [384,512) V [512,640) V [640,768) For light-sensitive frequency domain features (second sub-frequency domain features), V [768,896) V [896 ,1024) This represents the illumination-insensitive frequency domain characteristics (first sub-frequency domain characteristics).

[0129] Fine-tuning the first sub-frequency domain features

[0130] Step 232: Fine-tune the first sub-frequency domain feature using the first feature fine-tuning method to obtain the fine-tuned first sub-frequency domain feature.

[0131] In some embodiments, for light-insensitive frequency domain features, the computer device may fine-tune them using a first feature fine-tuning method to obtain fine-tuned light-insensitive frequency domain features.

[0132] Optionally, based on the above frequency domain analysis, it can be seen that light-insensitive frequency domain features, i.e., low-frequency features, have a significant impact on accuracy in binary classification tasks. In other words, light-insensitive frequency domain features are necessary features. Therefore, in this embodiment, fine-tuning the light-insensitive frequency domain features is to make the model pay more attention to them.

[0133] Optionally, considering that the first sub-frequency domain features, i.e., the illumination-insensitive frequency domain features, have small inter-domain differences in different scenes, and that the channel response values ​​of the feature channels containing the illumination-insensitive frequency domain features in daytime and nighttime images have small differences, a first feature fine-tuning method without preprocessing can be used for the first sub-frequency domain features. This first feature fine-tuning method can include three parts: determining the similarity map, generating the first frequency domain fine-tuned features, and feature fine-tuning.

[0134] Optionally, the first sub-frequency domain features correspond to a first token sequence. The first token sequence can be used to characterize the features of each image instance in the image, and the image instances can include scene instances such as roads, cars, buildings, and the sky. Optionally, the first token sequence is learned during the training of the image segmentation model. In one possible implementation, the computer device can obtain the first token sequence through random initialization and update the first token sequence during the training of the image segmentation model, so that the final first token sequence can be obtained after training is completed.

[0135] Regarding the feature fine-tuning process, in one possible implementation, the computer device performs a dot product operation on the first sub-frequency domain features and the first label sequence to obtain a first similarity map, which is used to characterize the feature similarity between the first sub-frequency domain features and the first label sequence. Simultaneously, the computer device performs a first linear transformation on the first label sequence and performs a dot product operation on the transformed first label sequence and the first similarity map to obtain the first frequency domain fine-tuned features. Finally, the computer device fine-tunes the first sub-frequency domain features based on the first frequency domain fine-tuned features, thus obtaining the fine-tuned first sub-frequency domain features.

[0136] Optionally, the first marker sequence can be represented as T L The feature dimension of the first labeled sequence is M×C, where M is the sequence length and C is the number of feature channels. Therefore, the computer device performs a dot product operation on the first sub-frequency domain features and the transpose of the first labeled sequence, and after normalization, obtains the first similarity map S. L The first similarity map S L The feature dimension is N×M.

[0137] Optionally, the calculation process of the first similarity map can be represented as follows:

[0138]

[0139] Where softmax is the activation function used for normalization, (V [768,896) V [896,1024) The first sub-frequency domain feature V represents the low-frequency characteristics of the two spliced ​​frequency bands. L , C is the transpose of the first labeled sequence, and C is the number of feature channels.

[0140] Optionally, performing a first linear transformation on the first labeled sequence is to project the first labeled sequence onto a feature space. This process can be implemented using a linear layer. Optionally, this linear layer is a first MLP, and the first MLP is composed of weights W. 1 and bias b 1 Parameterization. Optionally, the process of generating the first frequency domain fine-tuned feature can be represented as:

[0141]

[0142] Regarding the process of fine-tuning the first sub-frequency domain feature through the first frequency domain fine-tuning feature, in one possible implementation, the computer device can first fuse the first frequency domain fine-tuning feature and the first sub-frequency domain feature to obtain the first frequency domain fused feature, and then fuse the first frequency domain fused feature after the second linear transformation and the first sub-frequency domain feature to obtain the fine-tuned first sub-frequency domain feature.

[0143] Optionally, the process of performing a second linear transformation on the first frequency domain fused features can be implemented using another linear layer. Optionally, this linear layer is a second MLP, and the second MLP is composed of weights W. 2 and bias b 2 Parameterization. Optionally, the fine-tuning process can be represented as:

[0144]

[0145] in, That is, to integrate the first frequency domain fine-tuning features With the first sub-frequency domain feature V L , This involves performing a second linear transformation on the first frequency domain fusion feature.

[0146] Fine-tuning the second sub-frequency domain features

[0147] Step 233: Fine-tune the second sub-frequency domain feature using the second feature fine-tuning method to obtain the fine-tuned second sub-frequency domain feature. The second feature fine-tuning method is different from the first feature fine-tuning method.

[0148] In some embodiments, for light-sensitive frequency domain features, the computer device can fine-tune them using a second feature fine-tuning method to obtain fine-tuned light-insensitive frequency domain features. The second feature fine-tuning method differs from the first feature fine-tuning method.

[0149] Optionally, based on the above frequency domain analysis, it is known that light-sensitive frequency domain features, i.e., mid-to-high frequency features, have a relatively small impact on accuracy in binary classification tasks. Although removing mid-to-high frequency features can improve the model's generalization ability, they still need to be retained because they can characterize edge details in the image. Therefore, in this embodiment, fine-tuning the light-sensitive frequency domain features aims to increase their robustness while retaining them.

[0150] Optionally, considering the significant inter-domain differences in the second sub-frequency domain features, namely the illumination-sensitive frequency domain features, between different scenes, and the large differences in the channel response values ​​of the feature channels containing the illumination-sensitive frequency domain features between daytime and nighttime images, a second feature fine-tuning method involving preprocessing can be adopted for the second sub-frequency domain features. This second feature fine-tuning method can include four parts: randomization, determining a similarity map, generating the second frequency domain fine-tuned features, and feature fine-tuning.

[0151] Optionally, the second sub-frequency domain features correspond to a second label sequence, which can be used to characterize the features of each image instance in the image. Image instances can include scene instances such as roads, cars, buildings, and the sky. Optionally, the second label sequence is learned during the training of the image segmentation model. In one possible implementation, the computer device can obtain the second label sequence through random initialization and update it during the training of the image segmentation model, thus obtaining the final second label sequence after training is complete.

[0152] Optionally, to maintain consistency with the process of fine-tuning the first sub-frequency domain features, the mid-to-high frequency features in the first image frequency domain features can be divided into multiple second sub-frequency domain features, such that each second sub-frequency domain feature maintains the same feature dimension as the first sub-frequency domain feature. That is, the first image frequency domain features include at least two second sub-frequency domain features. Based on this, in order to fine-tune the features of each second sub-frequency domain feature, it is necessary to set a corresponding second label sequence for each of the at least two second sub-frequency domain features.

[0153] For example, when the first image frequency domain features are divided into 8 frequency bands, the first sub-frequency domain feature is [V [768,896) V [896,1024) To ensure that each second sub-frequency domain feature maintains the same feature dimension as the first sub-frequency domain feature, three second sub-frequency domain features can be determined, namely [V]. [0,128) V [128,256) ]、[V [256,384) V [384 ,512) ]、[V [512,640) V [640,768)Thus, the computer equipment performs feature fine-tuning on the three second sub-frequency domain features respectively.

[0154] Among them, [V [0,128) V [128,256) ] can be defined as high-frequency feature V H [V] [256,384) V [384,512) This can be defined as the first intermediate frequency feature. [V] [512,640) V [640,768) This can be defined as the second intermediate frequency feature.

[0155] Regarding the feature fine-tuning process, in one possible implementation, the computer device performs randomization processing on the second sub-frequency domain features to obtain randomized second sub-frequency domain features. Then, a dot product operation is performed on the randomized second sub-frequency domain features and the second labeled sequence to obtain a second similarity map, which is used to characterize the feature similarity between the second sub-frequency domain features and the second labeled sequence. Simultaneously, the computer device performs a third linear transformation on the second labeled sequence and performs a dot product operation on the transformed second labeled sequence and the second similarity map to obtain the second frequency domain fine-tuned features. Finally, the computer device fine-tunes the randomized second sub-frequency domain features based on the second frequency domain fine-tuned features to obtain the fine-tuned second sub-frequency domain features.

[0156] Combination Figure 6 Frequency domain analysis results of images from different scenes: Among different sample images, the channel response values ​​of the feature channels containing low-frequency features, i.e., light-insensitive frequency domain features, are relatively stable, while the channel response values ​​of the feature channels containing mid-to-high-frequency features, i.e., light-sensitive frequency domain features, are more volatile. Therefore, in this embodiment, before feature fine-tuning, the second sub-frequency domain features are randomized to reduce the inter-domain differences between different scenes.

[0157] Optionally, since there are also differences in light sensitivity among the second sub-frequency domain features, different degrees of randomization can be applied when using the second feature fine-tuning method for different second sub-frequency domain features. For example, compared to mid-frequency features, high-frequency features are more sensitive to light, so high-frequency light-sensitive frequency domain features can be randomized to a greater extent to minimize the inter-domain differences of high-frequency features in different scenarios.

[0158] Regarding the randomization process, in one possible implementation, the computer device can randomly assign values ​​to the frequency domain features in each feature channel of the second sub-frequency domain feature based on the channel response threshold, thereby obtaining the randomized second sub-frequency domain feature.

[0159] The channel response values ​​of the feature channels after randomization are not greater than the channel response threshold. Optionally, the channel response values ​​are used to characterize the response characteristics of the frequency domain features within the feature channels.

[0160] Optionally, during the randomization process, when randomly assigning values ​​to the frequency domain features in each feature channel, it is necessary to calculate the channel response value corresponding to the randomized feature channel in real time to ensure that the channel response value of the feature channel after randomization is not greater than the channel response threshold.

[0161] Optionally, for different second sub-frequency domain characteristics, different channel response thresholds can be set when applying different degrees of randomization processing. That is, different degrees of randomization processing can be achieved by setting different channel response thresholds.

[0162] For example, a first channel response threshold can be set for mid-frequency domain features, and a second channel response threshold can be set for high-frequency domain features, with the first channel response threshold being slightly smaller than the second channel response threshold.

[0163] Optionally, the channel response threshold can be a pre-set empirical value or a value determined based on the model training process; this embodiment of the application does not limit this. For example, based on experience, the channel response threshold can be set to 0.3.

[0164] Optionally, the random value assigned to each frequency domain feature within the feature channel can be represented as V. j,k Where j = 0, 1, ..., 767, k = 0, 1, ..., 1023. That is, the randomization process requires randomly assigning values ​​to each mid-to-high frequency feature within the 0th to 767th feature channels. To ensure that the channel response values ​​of the randomized feature channels do not exceed the channel response threshold, V... j,k It needs to follow a uniform distribution on the interval [0,T], i.e., V j,k ~[0,T], where T is the channel response threshold.

[0165] By randomizing the second sub-frequency domain features based on the channel response threshold, the second sub-frequency domain features can be randomized within a moderate range, thereby reducing the inter-domain gap between different scenes while preserving the frequency domain information required for image segmentation.

[0166] Optionally, for the second sub-frequency domain feature, the high-frequency feature V H The corresponding second marker sequence can be represented as T H First intermediate frequency characteristics The corresponding second marker sequence can be represented as Second intermediate frequency characteristics The corresponding second marker sequence can be represented as Optionally, consistent with the first marker sequence, the feature dimension of the second marker sequence is M×C, where M is the sequence length and C is the number of feature channels.

[0167] Taking feature fine-tuning of high-frequency features as an example, the computer device performs a dot product operation on the randomized second sub-frequency domain features and the transpose of the second label sequence, and after normalization, the second similarity map S can be obtained. H The second similarity map S H The feature dimension is N×M.

[0168] Optionally, the calculation process for the second similarity map can be represented as follows:

[0169]

[0170] Where softmax is the activation function used for normalization, V H These are the high-frequency features after randomization. C is the transpose of the second tokens, and C is the number of feature channels.

[0171] Optionally, a third linear transformation is performed on the second tokens to project them into the feature space. This process can be implemented using a linear layer. Optionally, this linear layer is a third MLP, which is composed of weights W. 3 and bias b 3 Parameterization. Optionally, the process of generating the second frequency domain fine-tuned feature can be represented as:

[0172]

[0173] Regarding the process of fine-tuning the second sub-frequency domain features through the second frequency domain fine-tuning features, in one possible implementation, the computer device can first fuse the second frequency domain features with the randomized second sub-frequency domain features to obtain the second frequency domain fused features, and then fuse the second frequency domain fused features after the fourth linear transformation with the randomized second sub-frequency domain features to obtain the fine-tuned second sub-frequency domain features.

[0174] Taking feature fine-tuning of high-frequency features as an example, optionally, the process of performing a fourth linear transformation on the second frequency domain fused features can be implemented through another linear layer. Optionally, this linear layer is a fourth MLP, and the fourth MLP is composed of weights W. 4 and bias b 4 Parameterization. Optionally, the fine-tuning process can be represented as:

[0175]

[0176] in, This refers to the fusion of second frequency domain fine-tuning features. With the second sub-frequency domain feature V after randomization H , This involves performing a fourth linear transformation on the second frequency domain fusion feature.

[0177] Step 234: Fuse the finely tuned first sub-frequency domain features and the finely tuned second sub-frequency domain features to obtain the second image frequency domain features.

[0178] After fine-tuning the first and second sub-frequency domain features respectively, the computer device can fuse the fine-tuned first and second sub-frequency domain features to obtain the second image frequency domain features.

[0179] Optionally, the fine-tuned first sub-frequency domain feature is The fine-tuned second sub-frequency domain features are as well as Optionally, the feature fusion method can be feature concatenation, that is, the frequency domain features of the second image are...

[0180] Step 241: The frequency domain features of the second image are inversely transformed by the inverse discrete sine transform to obtain the spatial domain features of the second image.

[0181] Corresponding to the method of transforming from the spatial domain to the frequency domain, in the inverse transformation process, the computer device can inversely transform the frequency domain features of the second image from the frequency domain to the spatial domain through the inverse discrete sine transform, thereby obtaining the spatial domain features of the second image.

[0182] Optionally, the second image spatial domain features can be represented as

[0183] Step 250: Decode the spatial features of the second image using an image segmentation model to obtain the image segmentation result corresponding to the night scene image.

[0184] Optionally, the image segmentation model includes N-1 image encoders, where N-1 is a positive integer. In one possible implementation, the computer device extracts features from the night scene image using a first image encoder to obtain a first image spatial domain feature f1 output by the first image encoder. By transforming the first image spatial domain feature f1 from the spatial domain to the frequency domain, performing feature fine-tuning, and then performing an inverse transform, a second image spatial domain feature f1′ can be obtained. Furthermore, the computer device inputs the second image spatial domain feature f1′ into a second image encoder, extracts features from f1′ using the second image encoder, and obtains a first image spatial domain feature f2 output by the second image encoder. By transforming the first image spatial domain feature f2 from the spatial domain to the frequency domain, performing feature fine-tuning, and then performing an inverse transform, a second image spatial domain feature f2′ can be obtained.

[0185] Furthermore, the computer device will use the second image spatial domain features f i Continue inputting the (i+1)th image encoder, and process f through the (i+1)th image encoder. i Feature extraction is performed to obtain the first image spatial domain feature f output by the (i+1)th image encoder. i+1 And by using the spatial features f of the first image i+1 After spatial domain transformation to frequency domain, feature fine-tuning, and inverse transformation, the spatial domain feature f′ of the second image can be obtained. i+1 .

[0186] It should be noted that between any two image encoders, the first label sequence used for fine-tuning the first sub-frequency domain features is different, as is the MLP; similarly, the second label sequence used for fine-tuning the second sub-frequency domain features is different, as is the MLP. That is, in the case of N-1 image encoders, the first sub-frequency domain features correspond to N-1 first label sequences and 2×(N-1) MLPs, and each second sub-frequency domain feature corresponds to N-1 second label sequences and 2×(N-1) MLPs.

[0187] This process continues until the second image spatial domain feature f′ is obtained. N-1 Finally, the computer device uses the decoder in the image segmentation model to process the spatial features f′ of the second image. N-1 Decoding yields the image segmentation results corresponding to the night scene image.

[0188] In this embodiment, by dividing the first image frequency domain features into a first sub-frequency domain feature and a second sub-frequency domain feature, different feature fine-tuning methods are used to fine-tune the two types of frequency domain features, which can optimize the fine-tuning effect of the features and thus improve the accuracy of image segmentation.

[0189] Furthermore, for the first sub-frequency domain feature, namely the light-insensitive frequency domain feature, feature fine-tuning can make the model pay more attention to this type of frequency domain feature; for the second sub-frequency domain feature, namely the light-sensitive frequency domain feature, feature fine-tuning can increase the robustness of this type of frequency domain feature.

[0190] Furthermore, during the feature partitioning process, keeping the feature dimensions of multiple second sub-frequency domain features consistent with those of the first sub-frequency domain features facilitates subsequent feature fusion and processing, thereby improving the efficiency of feature partitioning and fusion.

[0191] Please refer to Figure 7 This illustrates a flowchart of an image segmentation method provided in an exemplary embodiment of this application. Figure 7 As shown, the image segmentation model includes N-1 image encoders. This embodiment takes the feature fine-tuning process between the i-th image encoder and the i+1-th image encoder as an example.

[0192] like Figure 7 As shown, the computer device inputs the night scene image 701 into the image segmentation model. After feature extraction by the i-th image encoder, the first image spatial feature f output by the i-th image encoder can be obtained. i To achieve fine-tuning of sub-frequency domain features in different frequency bands, the computer equipment employs a discrete sine transform to transform the first image spatial domain features f i The first image frequency domain feature V is obtained by mapping from the spatial domain to the frequency domain. i .

[0193] Furthermore, the computer device will use the first image frequency domain features V i The frequency domain features of each feature channel are sorted in descending order of sinusoidal coefficients, and the frequency domain features V of the first image are analyzed based on different frequency bands. i This allows us to obtain the high-frequency features of the image. Image mid-frequency features as well as Low-frequency features of images and image high frequency features Image mid-frequency features as well as Determined as the second sub-frequency domain feature, low-frequency feature of the image. It was determined to be the first sub-frequency domain feature.

[0194] For low-frequency features of images The computer device employs a first feature fine-tuning method to fine-tune the image's features. Firstly, the computer device fine-tunes the low-frequency features of the image. and low-frequency features of images Perform a dot product operation on the corresponding first tokens to obtain a similarity graph. And the first tokens after linear transformation and the similarity graph Performing the dot product operation yields the frequency domain fine-tuning features. Thus, frequency domain fine-tuning features are integrated. Low-frequency features of images After undergoing linear transformation, it is then compared with the low-frequency features of the image. Fusion allows us to obtain the finely tuned low-frequency features of the image.

[0195] The above process can be represented as:

[0196] Where C is the number of feature channels. Low-frequency features of the image between the i-th image encoder and the (i+1)-th image encoder The corresponding first tokens, and The weights and biases of the MLP used to implement the first linear transformation between the i-th image encoder and the (i+1)-th image encoder are given. and The weights and biases of the MLP used to implement the second linear transformation between the i-th image encoder and the (i+1)-th image encoder.

[0197] For image high frequency features The computer device employs a second feature fine-tuning method to fine-tune the image's features. First, the computer device performs feature fine-tuning on the high-frequency features of the image based on channel response thresholds. Randomization is performed to obtain the high-frequency features of the image after randomization. Furthermore, the high-frequency features of the image and image high-frequency features Perform a dot product operation on the corresponding second tokens to obtain a similarity graph. And the second tokens after linear transformation and the similarity graph Performing the dot product operation yields the frequency domain fine-tuning features. Thus, the frequency domain fine-tuning features are integrated. High-frequency features of images After undergoing linear transformation, it is then compared with the high-frequency features of the image. Fusion, in this way, yields the finely tuned high-frequency features of the image.

[0198] The above process can be represented as:

[0199] Where C is the number of feature channels. High-frequency features of the image between the i-th image encoder and the (i+1)-th image encoder The corresponding second tokens, and The weights and biases of the MLP used to implement the third linear transformation between the i-th image encoder and the (i+1)-th image encoder are given. and The weights and biases are used between the i-th image encoder and the (i+1)-th image encoder to implement the fourth linear transformation in the MLP.

[0200] For image mid-frequency features The computer device employs a second feature fine-tuning method to fine-tune the image's features. First, the computer device adjusts the mid-frequency features of the image based on channel response thresholds. Randomization is performed to obtain the mid-frequency features of the image after randomization. Furthermore, the mid-frequency features of the image and image mid-frequency features Perform a dot product operation on the corresponding second tokens to obtain a similarity graph. And the second tokens after linear transformation and the similarity graph Performing the dot product operation yields the frequency domain fine-tuning features. Thus, the frequency domain fine-tuning features are integrated. Image mid-frequency features After undergoing linear transformation, it is then compared with the mid-frequency features of the image. Fusion, in this way, yields the finely tuned mid-frequency features of the image.

[0201] The above process can be represented as:

[0202] Where C is the number of feature channels. The image intermediate frequency features between the i-th image encoder and the (i+1)-th image encoder The corresponding second tokens, and The weights and biases of the MLP used to implement the third linear transformation between the i-th image encoder and the (i+1)-th image encoder are given. and The weights and biases are used between the i-th image encoder and the (i+1)-th image encoder to implement the fourth linear transformation in the MLP.

[0203] For image mid-frequency features The computer device employs a second feature fine-tuning method to fine-tune the image's features. First, the computer device adjusts the mid-frequency features of the image based on channel response thresholds. Randomization is performed to obtain the mid-frequency features of the image after randomization. Furthermore, the mid-frequency features of the image and image mid-frequency features Perform a dot product operation on the corresponding second tokens to obtain a similarity graph. And the second tokens after linear transformation and the similarity graph Performing the dot product operation yields the frequency domain fine-tuning features. Thus, the frequency domain fine-tuning features are integrated. Image mid-frequency features After undergoing linear transformation, it is then compared with the mid-frequency features of the image. Fusion, in this way, yields the finely tuned mid-frequency features of the image.

[0204] The above process can be represented as:

[0205] Where C is the number of feature channels. The image intermediate frequency features between the i-th image encoder and the (i+1)-th image encoder The corresponding second tokens, and The weights and biases of the MLP used to implement the third linear transformation between the i-th image encoder and the (i+1)-th image encoder are given. and The weights and biases are used between the i-th image encoder and the (i+1)-th image encoder to implement the fourth linear transformation in the MLP.

[0206] The low-frequency features of the fine-tuned image were obtained. Image high frequency features Image mid-frequency features as well as Then, the computer equipment can obtain the second image frequency domain features through feature fusion. Furthermore, by using the inverse discrete sine transform, the frequency domain features of the second image are transformed from the frequency domain to the spatial domain, thus obtaining the spatial domain features f of the second image. i ′.

[0207] Furthermore, the computer device will use the second image spatial domain features f i 'Continue to input the (i+1)th image encoder, and extract features from it. After feature extraction and fine-tuning by the (N-1)th image encoder, the computer device can obtain the second image spatial domain feature f' corresponding to the night scene image 701. N-1 Finally, the computer device will use the second image spatial domain feature f′ N-1 The input is to the decoder, which processes the spatial features f′ of the second image. N-1 Decoding yields the image segmentation result 702 corresponding to the night scene image 701.

[0208] Please refer to Figure 8 This document illustrates a flowchart of training an image segmentation model according to an exemplary embodiment of this application. This embodiment uses the method applied to a computer device (including a terminal and / or server) as an example for illustration. The method includes the following steps:

[0209] Step 810: Extract the spatial features of the first sample image from the sample image using the image segmentation model.

[0210] Optionally, the sample image can be any image in the sample training set. In order for the image segmentation model to learn image features from different domains, the sample image can be a daytime image or a nighttime image.

[0211] In this embodiment, the training set mainly includes the Cityscapes training set, the NightCity training set, and the NightCity-fine training set. The Cityscapes dataset is a commonly used driving scene segmentation dataset, captured under well-lit daytime conditions. It contains 2975 training images and 500 validation images, covering 19 common semantic categories in driving scenes. The NightCity dataset is the largest semantic segmentation dataset for nighttime driving scenes to date, containing 2998 training images and 1299 validation images. The NightCity-fine dataset corrects the ground truth labels of 2554 images from the original NightCity dataset, fixing missing and incorrect annotations, and is used for nighttime-to-night generalization experiments.

[0212] In some embodiments, the computer device inputs a sample image into an image segmentation model, and first extracts the image features of the sample image through the image encoder in the image segmentation model, thereby obtaining the first sample image spatial features of the sample image.

[0213] Among them, the spatial features of the first sample image refer to the image features of the sample image in the spatial domain, which can reflect the information of the sample image in spatial location, such as the distribution of pixels, positional relationships and local structure in the sample image.

[0214] Step 820: Transform the spatial domain features of the first sample image to obtain the frequency domain features of the first sample image. The frequency domain features of the first sample image include sample sub-frequency domain features of at least two frequency bands.

[0215] Optionally, transforming the spatial features of the first sample image means transforming the spatial features of the first sample image from the spatial domain to the frequency domain. The spatial domain, also known as the image space, refers to the space composed of image pixels. Optionally, the frequency domain describes the features of the image through frequency. Optionally, spatial domain features and frequency domain features can be transformed into each other.

[0216] In this embodiment of the application, in order to fine-tune the image features based on the difference in light sensitivity between different sample frequency domain features, after extracting the spatial domain features of the first sample image, it is also necessary to transform the spatial domain features of the first sample image from the spatial domain to the frequency domain, so as to obtain the frequency domain features of the first sample image corresponding to the spatial domain features of the first sample image.

[0217] Optionally, the spatial domain features can be transformed into frequency domain features using discrete sine transform, Fourier transform, or other feature transformation methods. This application does not limit the specific methods used in this embodiment.

[0218] Optionally, the frequency domain features of the first sample image may include high-frequency sample features and low-frequency sample features. High-frequency sample features are used to characterize the details and edge information of the sample image and are typically more affected by illumination; while low-frequency sample features are used to characterize the overall structure and background information of the sample image and are typically less affected by illumination.

[0219] In some embodiments, in order to fine-tune image features based on the differences in light sensitivity between different sample frequency domain features, the computer device can divide the frequency domain features of the first sample image into sample frequency domain features of at least two frequency bands according to the frequency values ​​corresponding to different sample frequency domain features in the frequency domain features of the first sample image.

[0220] Step 830: Based on the difference in light sensitivity between the sample sub-frequency domain features of at least two frequency bands, fine-tune the sample sub-frequency domain features of at least two frequency bands using at least two different feature fine-tuning methods to obtain the frequency domain features of the second sample image.

[0221] In some embodiments, after obtaining sample frequency domain features of at least two frequency bands through feature division, the computer device can fine-tune the sample frequency domain features of at least two frequency bands based on the difference in light sensitivity between the sample sub-frequency domain features of at least two frequency bands using at least two different feature fine-tuning methods, so that the sub-sample frequency domain features of at least two frequency bands after fine-tuning constitute the second sample image frequency domain features.

[0222] Optionally, after feature fine-tuning, the frequency domain features of the second sample image are more robust to changes in illumination. For sample sub-frequency domain features that are not sensitive to illumination, feature fine-tuning can make the model pay more attention to these types of frequency domain features; for sample sub-frequency domain features that are sensitive to illumination, feature fine-tuning can increase the robustness of these types of frequency domain features.

[0223] Among them, at least two different feature fine-tuning methods have different degrees of feature fine-tuning for sub-frequency domain features. The main difference between the at least two different feature fine-tuning methods lies in whether preprocessing is used to reduce the inter-domain gap of sub-frequency domain features between different scenarios in order to improve the robustness of sub-frequency domain features.

[0224] Optionally, there is a many-to-many relationship between sub-frequency domain features and feature fine-tuning methods, meaning that different sub-frequency domain features can use their own feature fine-tuning methods or the same feature fine-tuning method.

[0225] For example, when there are low-frequency, mid-frequency, and high-frequency features, since the light sensitivity varies among these features, different feature fine-tuning methods can be used to fine-tune each feature separately. However, for mid-frequency and high-frequency features, since the difference in light sensitivity is relatively small, the same feature fine-tuning method can be used for both.

[0226] Optionally, the image segmentation model includes at least two adapters, with different adapters used to fine-tune the sub-frequency domain features of samples in different frequency bands, and each adapter corresponding to its own feature fine-tuning method.

[0227] Step 840: Perform an inverse transform on the frequency domain features of the second sample image to obtain the spatial domain features of the second sample image.

[0228] In some embodiments, after obtaining the frequency domain features of the second sample image, the computer device can inversely transform the frequency domain features of the second sample image from the frequency domain to the spatial domain by means of inverse transformation, thereby obtaining the spatial domain features of the second sample image corresponding to the sample image.

[0229] Optionally, the inverse transformation of frequency domain features into spatial domain features can be performed using inverse discrete sine transform, inverse Fourier transform, or other feature transformation methods. This application does not limit the specific methods used in this embodiment.

[0230] Step 850: Decode the spatial features of the second sample image using an image segmentation model to obtain the sample image segmentation result corresponding to the sample image.

[0231] The sample image segmentation result refers to the result of dividing the sample image into at least two non-overlapping regions. Optionally, the sample image segmentation result is a sample semantic segmentation result, where the value of each pixel in the sample semantic segmentation result is used to characterize the semantic category of that pixel in the sample image. Optionally, the sample image segmentation result can be understood as a pixel-level labeled image of the sample image, that is, each pixel in the sample image segmentation result corresponds to its own label, which is used to characterize the semantic category to which the pixel belongs in the sample image.

[0232] For example, the sample image includes a road surface, a car, a building, and the sky. During semantic annotation, the category number corresponding to the road surface is represented by the number 1, the category number corresponding to the car is represented by the number 2, the category number corresponding to the building is represented by the number 3, and the category number corresponding to the sky is represented by the number 4. Then, in the sample image segmentation result corresponding to the sample image, the pixel value representing the road surface is 1, the pixel value representing the car is 2, the pixel value representing the building is 3, and the pixel value representing the sky is 4.

[0233] In some embodiments, after obtaining the spatial features of the second sample image, the computer device can input the spatial features of the second sample image into the decoder, and by decoding the spatial features of the second sample image, the sample image segmentation result corresponding to the sample image can be obtained.

[0234] Optionally, the feature decoding process can be understood as the process of restoring the spatial features of the second sample image to an output of the same size as the sample image through the decoder. The decoder can gradually recover detailed image semantic information based on the spatial features of the second sample image, thereby generating the sample image segmentation result based on the image semantic information.

[0235] Step 860: Train the image segmentation model based on the ground truth of the image segmentation result corresponding to the sample image and the sample image segmentation result.

[0236] In some embodiments, after obtaining the sample image segmentation result, the computer device can determine the image segmentation loss based on the ground truth value of the image segmentation result corresponding to the sample image and the sample image segmentation result, thereby training the image segmentation model with the image segmentation loss.

[0237] Optionally, the image segmentation model includes, in addition to the image encoder and decoder, an adapter for fine-tuning frequency domain features. Optionally, this adapter includes a first label sequence corresponding to the first sample sub-frequency domain features, a second label sequence corresponding to the second sample sub-frequency domain features, and multiple MLPs for implementing linear transformation processing. Based on this, the training process of the image segmentation model can be understood as the training process of the image encoder, adapter, and decoder. The image encoder can be a pre-trained encoder, and therefore remains frozen during the training process of the image segmentation model.

[0238] In one possible implementation, the computer device determines an image segmentation loss based on the ground truth of the image segmentation result corresponding to the sample image and the sample image segmentation result. Based on this loss, it updates the first and second label sequences, and updates the parameters of the decoder and each MLP. Then, when the training completion condition is met, training of the image segmentation model can be stopped.

[0239] Optionally, in this embodiment, AdamW can be used as the optimizer, and model training can be stopped when the number of training iterations reaches a threshold. The initial learning rates for the image encoder, adapter, and decoder are each set to 1×10⁻⁶. -4 1×10 -5 and 1×10 -5 .

[0240] In this embodiment, during the training process of the image segmentation model, the parameters of the label sequence used for feature fine-tuning and each MLP of the linear transformation are updated to ensure the accuracy of feature fine-tuning; and the parameters of the decoder are updated to ensure the accuracy of image segmentation.

[0241] It should be noted that the above embodiments only provide a general description of the training process of the image segmentation model. During the training process of the image segmentation model, the mutual transformation process between the spatial domain features and the frequency domain features of the sample image can refer to the mutual transformation process between the spatial domain features and the frequency domain features of the image in the above application-side embodiments. The feature fine-tuning process of the sample sub-frequency domain features of different frequency bands can also refer to the feature fine-tuning process of the sub-frequency domain features of different frequency bands in the above application-side embodiments. This embodiment will not elaborate on these details.

[0242] Please refer to Figure 9 This illustrates a flowchart of the image segmentation model training process provided in an exemplary embodiment of this application. Figure 9 As shown, the image segmentation model includes N-1 image encoders. In this embodiment, N-1 image encoders are frozen during training as an example.

[0243] like Figure 9 As shown, the computer device inputs sample image 901 into the image segmentation model, extracts features from it sequentially through N-1 image encoders, and fine-tunes the spatial features of the sample image output by each pair of image encoders between each pair. Then, after obtaining the spatial features of the second sample image, the computer device decodes these features using a decoder and outputs the sample image segmentation result 902. The specific process of fine-tuning the spatial features of the sample image can be found in [reference needed]. Figure 7 The process of fine-tuning the spatial features of the image is not described in detail in this embodiment.

[0244] After obtaining the sample image segmentation results corresponding to the sample images, the computer device can determine the image segmentation loss based on the ground truth value of the image segmentation results corresponding to the sample images and the sample image segmentation results. Based on the image segmentation loss, the parameters of the N-1 first tokens and 2×(N-1) MLPs used for feature fine-tuning of the first sample sub-frequency domain features, the N-1 second tokens and 2×(N-1) MLPs used for feature fine-tuning of each second sample sub-frequency domain feature, and the decoder are updated. Then, after reaching a preset number of training iterations, training of the image segmentation model is stopped.

[0245] Please refer to Figure 10 This illustrates a schematic diagram of the entire training and inference process of an image segmentation method provided in an exemplary embodiment of this application. Figure 10 As shown in the embodiments of this application, an image segmentation model is trained using daytime image training and nighttime image training sets, enabling the trained image segmentation model to have relatively accurate image segmentation capabilities. Furthermore, by applying the image segmentation model to segment nighttime images, the image segmentation results corresponding to each nighttime image can be obtained.

[0246] The beneficial effects of the image segmentation method proposed in this application will be explained in conjunction with the above embodiments.

[0247] First, the image segmentation method proposed in this application has a good generalization ability from daytime to nighttime scenes.

[0248] 1) Comparison with DA (Domain Adaptation) and UDA (Unsupervised Domain Adaptation) methods.

[0249] Among them, DA-based methods include GCMA, MCGDA, DANNet, and CCDistill, while UDA-based methods include AdaptSeg, TENT, DAFormer, MALL-D, SepiCo, HRDA, STA, MIC, PASS, CoDA, and DGInStyle.

[0250] The CityScapes dataset was used as the source domain, while the AN (Adverse Conditions Dataset with Correspondences-Night) dataset, the DZ (Dark Zurich) dataset, and the ND (Nighttime Driving) dataset were used as unseen target domains.

[0251] As illustrated in Table 1, the performance comparison results of the image segmentation method proposed in this application with the DA and UDA methods are shown. Table 1 shows that compared with the Transformer-based DA and UDA methods, the proposed method improves the mIoU (mean Intersection over Union, a key metric for segmentation accuracy) by approximately 25% and 15% on AN and DZ datasets, respectively. Compared with the REIN baseline, the proposed method improves the mIoU by 13.4%, 9.8%, and 10.0% on the AN, DZ, and ND datasets, respectively, demonstrating its effectiveness in unseen nighttime domains.

[0252] Table 1

[0253] Method Backbone AN DZ ND DA Models: GCMA Res-101 - 42.0 45.6 MCGDA Res-101 - 42.5 49.4 DANNet Res-101 - 44.3 47.7 CCDistil Res-101 37.3 47.5 - UDA Models: AdaptSeg Res-101 - 30.4 34.5 TENT Res-101 25.2 21.6 39.3 DAFormer MiT-B5 44.7 48.5 51.8 MALL-D Res-101 26.9 22.8 42.7 SePiCo Res-101 50.5 54.2 56.9 HRDA MiT-B5 53.1 55.9 - STA MiT-B5 49.7 - - MIC MiT-B5 57.2 60.2 58.6 PASS MiT-B5 60.3 60.2 57.0 CoDA MiT-B5 66.4 61.2 59.2 DGInStyle MiT-B5 59.2 32.6 - REIN ViT-L 70.6 64.3 50.3 This plan ViT-L 84.0 74.1 60.3 ↑13.4 ↑9.8 ↑1.1

[0254] 2) Comparison with PEFT (Parameter-Efficient Fine-Tuning) method.

[0255] Among them, the PEFT-based methods include LoRA, AdaptFormer, VPT, EVP, WHT, FourierFT, and SET. In addition, three traditional style illusion methods (GTR-LTR, AdvStyle, and PASTA) and the REIN baseline are added as references.

[0256] The CityScapes dataset was used as the source domain, while the AN (Adverse Conditions Dataset with Correspondences-Night) dataset, the DZ (Dark Zurich) dataset, and the ND (Nighttime Driving) dataset were used as unseen target domains.

[0257] As illustrated in Table 2, the performance comparison results of the image segmentation method proposed in this application and the PEFT method are shown. Table 2 shows that on the AN dataset, DZ dataset, and ND dataset, the proposed method outperforms the second-best method by 13.4%, 8.3%, and 9.6% in mIoU, respectively. Notably, it significantly outperforms frequency-based PEFT methods such as EVP, WHT, FourierFT, and SET.

[0258] Table 2

[0259] Method #Para. AN DZ ND Full 304.2M 65.8 59.4 45.2 GTR-LTR 304.2M 66.7 60.3 46.4 AdvStyle 304.2M 67.0 60.6 46.7 PASTA 304.2M 67.1 61.3 46.9 Freeze 0.0M 67.6 60.8 47.1 LoRA 0.8M 68.7 62.0 48.3 AdaptFormer 3.2M 68.5 61.8 47.8 VPT 3.7M 68.0 61.6 47.7 EVP 3.7M 67.4 58.3 45.6 WHT 3.5M 68.5 61.7 46.4 FourierFT 0.7M 69.8 63.0 49.5 SET 6.1M 70.5 65.8 50.7 REIN 3.0M 70.6 64.3 50.3 This plan 11.5M 84.0 74.1 60.3 ↑13.4 ↑8.3 ↑9.6

[0260] Secondly, the image segmentation method proposed in this application has a good ability to segment images of nighttime scenes.

[0261] Optionally, compared fully supervised nighttime image segmentation methods include NightCity, NightLab, FDLNet, DTP, and NightFormer. Optionally, this scheme is trained and tested on the training and test sets of NightCity (NC), NightCity-fine (NC-f), and BDD100K-Night (BN), respectively.

[0262] As illustrated in Table 3, the performance comparison results of the image segmentation method proposed in this application with other methods in night scene image segmentation are shown. Table 3 shows that the proposed method significantly outperforms all compared methods, achieving mIoU of 65.1%, 66.2%, and 57.8% on NC, NC-f, and BN, respectively. Notably, the proposed method outperforms the REIN baseline by 2.7%, 1.6%, and 3.9% on NC, NC-f, and BN, respectively. Furthermore, Table 3 shows that CNN-based and ViT-based methods typically have approximately 50M and 100M parameters, respectively. However, the proposed method has only 11.5M parameters, effectively reducing computational costs.

[0263] Table 3

[0264] Method #Para. NC NC-f BN ResNet-101 Encoder: DeepLabV3 60.1M 54.7 58.8 43.4 NightCity 84.6M 51.8 55.9 39.7 NightLab 98.5M 55.9 62.3 50.4 DTP 63.9M 57.6 60.4 - NightFormer 58.4M 59.8 62.8 47.5 Swin-Base Encoder: UPer-Swin 119.9M 58.4 61.1 31.7 NightLab 242.4M 59.8 62.3 45.1 FDLNet - 55.5 - 45.9 DTP 122.5M 61.2 64.2 53.1 NightFormer 111.9M 63.5 65.9 55.4 VFM Encoder: REIN 3.0M 62.4 64.6 53.9 This plan 11.5M 65.1 66.2 57.8 1.6 0.3 2.4

[0265] Finally, the image segmentation method proposed in this application has a good generalization ability from night scenes to night scenes.

[0266] Optional, comparable Domain Generalized Semantic Segmentation (DGSS) methods include RobustNet, SHADE, SAW, WildNet, SPC, HGFormer, CMFormer, BlindNet, SET, DGInStyle, tqdm, and REIN.

[0267] In this study, NC and NC-f were used as source domains, while AN, DZ, ND, BN, and All-day CityScapes (AC) datasets were used as unseen target domains in both experiments.

[0268] As illustrated in Table 4, the performance comparison results of the image segmentation method proposed in this application and the DGSS method are shown. Table 4 shows that compared with the ResNet-50-based DGSS method, the proposed method achieves approximately 25%, 25%, 25%, 15%, and 20% higher mIoU on AN, DZ, ND, BN, and AD, respectively. Compared with Transformer-based methods, the proposed method achieves approximately 15%, 15%, 15%, 7%, and 12% higher mIoU on AN, DZ, ND, BN, and AD, respectively. Compared with VFM methods such as SET, tqdm, and REIN, when using NC as the source domain, the mIoU improvements on AN, DZ, ND, BN, and AD are 5.2%, 3.0%, 1.1%, 5.7%, and 4.0%, respectively. Similarly, when using NC-f as the source domain, the mIoU improvements on AN, DZ, ND, BN, and AD are 4.1%, 3.0%, 1.5%, 3.4%, and 3.1%, respectively. This demonstrates that our proposed scheme exhibits better performance than other methods when generalizing to different nighttime domains.

[0269] Table 4

[0270]

[0271]

[0272] Optionally, the image segmentation method provided in this application embodiment can be applied to various scenarios, such as autonomous driving, road recognition, robot navigation, etc.

[0273] For autonomous driving scenarios:

[0274] Optionally, in the process of achieving autonomous driving, in order to ensure driving safety, it is necessary to acquire driving images in real time and identify image instances in the driving images, such as pedestrians, vehicles, traffic lights, etc. Based on this, the image segmentation method proposed in this application can be applied to image segmentation of nighttime driving images to improve the recognition accuracy of nighttime driving images.

[0275] Please refer to Figure 11 This document illustrates a flowchart of an image segmentation method for autonomous driving scenarios provided by an exemplary embodiment of this application. This embodiment uses the method applied to a computer device (including a terminal and / or a server) as an example for illustration. The method includes the following steps:

[0276] Step 1101: Extract the first image spatial features of the night driving image using an image segmentation model.

[0277] Optionally, nighttime driving images refer to images captured by a camera or dashcam while the vehicle is driving at night. In one possible implementation, to achieve image segmentation of nighttime driving images, the computer device inputs the nighttime driving images into an image segmentation model. First, the image encoder in the image segmentation model extracts the image features of the nighttime driving images, thereby obtaining the first spatial domain features of the nighttime driving images.

[0278] Step 1102: Transform the spatial domain features of the first image to obtain the frequency domain features of the first image. The frequency domain features of the first image include sub-frequency domain features of at least two frequency bands.

[0279] In order to fine-tune the image features based on the differences in light sensitivity between different frequency domain features, after extracting the first image spatial domain features, it is also necessary to transform the first image spatial domain features from the spatial domain to the frequency domain to obtain the first image frequency domain features corresponding to the first image spatial domain features.

[0280] In one possible implementation, the computer device can use discrete sine transform to transform the spatial domain features of the first image from the spatial domain to the frequency domain, thereby obtaining the frequency domain features of the first image corresponding to the spatial domain features of the first image.

[0281] Optionally, in order to fine-tune the image features based on the differences in light sensitivity between different frequency domain features, the computer device can divide the first image frequency domain features into sub-frequency domain features of at least two frequency bands according to the frequency values ​​corresponding to different frequency domain features in the first image frequency domain features.

[0282] Step 1103: Based on the difference in light sensitivity between the sub-frequency domain features of at least two frequency bands, at least two different feature fine-tuning methods are used to fine-tune the sub-frequency domain features of at least two frequency bands to obtain the second image frequency domain features.

[0283] After obtaining sub-frequency domain features of at least two frequency bands through feature division, the computer device can fine-tune the sub-frequency domain features of at least two frequency bands based on the difference in light sensitivity between the sub-frequency domain features of at least two frequency bands using at least two different feature fine-tuning methods. Thus, the sub-frequency domain features of at least two frequency bands after fine-tuning constitute the second image frequency domain features.

[0284] Optionally, after feature fine-tuning, the second image frequency domain features become more robust to changes in illumination. For the first sub-frequency domain features that are insensitive to illumination, feature fine-tuning can make the model pay more attention to this type of frequency domain feature; for the second sub-frequency domain features that are sensitive to illumination, feature fine-tuning can increase the robustness of this type of frequency domain feature.

[0285] In one possible implementation, the computer device divides the first image frequency domain features into light-sensitive frequency domain features and light-insensitive frequency domain features based on at least two frequency bands. The frequency of the frequency band corresponding to the light-sensitive frequency domain features is higher than the frequency of the frequency band corresponding to the light-insensitive frequency domain features, and the light-sensitive frequency domain features are more sensitive to light than the light-insensitive frequency domain features. The light-insensitive frequency domain features are fine-tuned using a first feature fine-tuning method to obtain fine-tuned light-insensitive frequency domain features. The light-sensitive frequency domain features are fine-tuned using a second feature fine-tuning method to obtain fine-tuned light-sensitive frequency domain features, where the second feature fine-tuning method is different from the first feature fine-tuning method. The fine-tuned light-insensitive frequency domain features and the fine-tuned light-sensitive frequency domain features are fused to obtain the second image frequency domain features.

[0286] Regarding the process of fine-tuning the illumination-insensitive frequency domain features, optionally, the computer device performs a dot product operation on the illumination-insensitive frequency domain features and the first label sequence corresponding to the illumination-insensitive frequency domain features to obtain a first similarity map. The first label sequence is learned during the image segmentation model training process, and the first similarity map is used to characterize the feature similarity between the illumination-insensitive frequency domain features and the first label sequence. A dot product operation is performed on the first label sequence after the first linear transformation and the first similarity map to obtain the first frequency domain fine-tuned features. The first frequency domain fine-tuned features are fused with the illumination-insensitive frequency domain features to obtain the first frequency domain fused features. The first frequency domain fused features after the second linear transformation are fused with the illumination-insensitive frequency domain features to obtain the fine-tuned illumination-insensitive frequency domain features.

[0287] Regarding the process of fine-tuning the light-sensitive frequency domain features, optionally, the computer device performs randomization processing on the light-sensitive frequency domain features to obtain randomized light-sensitive frequency domain features; a dot product operation is performed on the randomized light-sensitive frequency domain features and the corresponding second label sequence to obtain a second similarity map. The second label sequence is learned during the image segmentation model training process, and the second similarity map is used to characterize the feature similarity between the randomized light-sensitive frequency domain features and the second label sequence; a dot product operation is performed on the second label sequence after the third linear transformation and the second similarity map to obtain the second frequency domain fine-tuned features; the second frequency domain fine-tuned features are fused with the randomized light-sensitive frequency domain features to obtain the second frequency domain fused features; the second frequency domain fused features after the fourth linear transformation are fused with the randomized light-sensitive frequency domain features to obtain the fine-tuned light-insensitive frequency domain features.

[0288] Step 1104: Perform an inverse transform on the frequency domain features of the second image to obtain the spatial domain features of the second image.

[0289] After obtaining the frequency domain features of the second image, the computer device can inversely transform the frequency domain features of the second image from the frequency domain to the spatial domain through an inverse transformation, thereby obtaining the spatial domain features of the second image corresponding to the night driving image.

[0290] In one possible implementation, the computer device can use inverse discrete sine transform to inversely transform the frequency domain features of the second image from the frequency domain to the spatial domain, thereby obtaining the spatial domain features of the second image corresponding to the night driving image.

[0291] Step 1105: Decode the spatial features of the second image using an image segmentation model to obtain the image segmentation result corresponding to the nighttime driving image.

[0292] After obtaining the second image spatial features, the computer device can input the second image spatial features into the decoder. By decoding the second image spatial features, the image segmentation result corresponding to the night driving image can be obtained.

[0293] For road recognition scenarios:

[0294] Optionally, in the process of road recognition, poor lighting conditions at night increase the difficulty of recognizing nighttime road images. Therefore, the image segmentation method proposed in this application can be applied to segment nighttime road images to improve the accuracy of nighttime road image recognition.

[0295] Please refer to Figure 12 This document illustrates a flowchart of an image segmentation method for a road recognition scenario provided by an exemplary embodiment of this application. This embodiment uses the method applied to a computer device (including a terminal and / or a server) as an example for illustration. The method includes the following steps:

[0296] Step 1201: Extract the first image spatial features of the nighttime road image using an image segmentation model.

[0297] Optionally, a nighttime road image refers to an image of a road taken at night. In one possible implementation, to achieve image segmentation of a nighttime road image, the computer device inputs the nighttime road image into an image segmentation model. First, the image encoder in the image segmentation model extracts the image features of the nighttime road image, thereby obtaining the first spatial domain features of the nighttime road image.

[0298] Step 1202: Transform the spatial domain features of the first image to obtain the frequency domain features of the first image. The frequency domain features of the first image include sub-frequency domain features of at least two frequency bands.

[0299] In order to fine-tune the image features based on the differences in light sensitivity between different frequency domain features, after extracting the first image spatial domain features, it is also necessary to transform the first image spatial domain features from the spatial domain to the frequency domain to obtain the first image frequency domain features corresponding to the first image spatial domain features.

[0300] In one possible implementation, the computer device can use discrete sine transform to transform the spatial domain features of the first image from the spatial domain to the frequency domain, thereby obtaining the frequency domain features of the first image corresponding to the spatial domain features of the first image.

[0301] Optionally, in order to fine-tune the image features based on the differences in light sensitivity between different frequency domain features, the computer device can divide the first image frequency domain features into sub-frequency domain features of at least two frequency bands according to the frequency values ​​corresponding to different frequency domain features in the first image frequency domain features.

[0302] Step 1203: Based on the difference in light sensitivity between the sub-frequency domain features of at least two frequency bands, at least two different feature fine-tuning methods are used to fine-tune the sub-frequency domain features of at least two frequency bands to obtain the second image frequency domain features.

[0303] After obtaining sub-frequency domain features of at least two frequency bands through feature division, the computer device can fine-tune the sub-frequency domain features of at least two frequency bands based on the difference in light sensitivity between the sub-frequency domain features of at least two frequency bands using at least two different feature fine-tuning methods. Thus, the frequency domain features of at least two frequency bands after fine-tuning constitute the second image frequency domain features.

[0304] Optionally, after feature fine-tuning, the second image frequency domain features become more robust to changes in illumination. For the first sub-frequency domain features that are insensitive to illumination, feature fine-tuning can make the model pay more attention to this type of frequency domain feature; for the second sub-frequency domain features that are sensitive to illumination, feature fine-tuning can increase the robustness of this type of frequency domain feature.

[0305] In one possible implementation, the computer device divides the first image frequency domain features into light-sensitive frequency domain features and light-insensitive frequency domain features based on at least two frequency bands. The frequency of the frequency band corresponding to the light-sensitive frequency domain features is higher than the frequency of the frequency band corresponding to the light-insensitive frequency domain features, and the light-sensitive frequency domain features are more sensitive to light than the light-insensitive frequency domain features. The light-insensitive frequency domain features are fine-tuned using a first feature fine-tuning method to obtain fine-tuned light-insensitive frequency domain features. The light-sensitive frequency domain features are fine-tuned using a second feature fine-tuning method to obtain fine-tuned light-sensitive frequency domain features, where the second feature fine-tuning method is different from the first feature fine-tuning method. The fine-tuned light-insensitive frequency domain features and the fine-tuned light-sensitive frequency domain features are fused to obtain the second image frequency domain features.

[0306] Regarding the process of fine-tuning the illumination-insensitive frequency domain features, optionally, the computer device performs a dot product operation on the illumination-insensitive frequency domain features and the first label sequence corresponding to the illumination-insensitive frequency domain features to obtain a first similarity map. The first label sequence is learned during the image segmentation model training process, and the first similarity map is used to characterize the feature similarity between the illumination-insensitive frequency domain features and the first label sequence. A dot product operation is performed on the first label sequence after the first linear transformation and the first similarity map to obtain the first frequency domain fine-tuned features. The first frequency domain fine-tuned features are fused with the illumination-insensitive frequency domain features to obtain the first frequency domain fused features. The first frequency domain fused features after the second linear transformation are fused with the illumination-insensitive frequency domain features to obtain the fine-tuned illumination-insensitive frequency domain features.

[0307] Regarding the process of fine-tuning the light-sensitive frequency domain features, optionally, the computer device performs randomization processing on the light-sensitive frequency domain features to obtain randomized light-sensitive frequency domain features; a dot product operation is performed on the randomized light-sensitive frequency domain features and the corresponding second label sequence to obtain a second similarity map. The second label sequence is learned during the image segmentation model training process, and the second similarity map is used to characterize the feature similarity between the randomized light-sensitive frequency domain features and the second label sequence; a dot product operation is performed on the second label sequence after the third linear transformation and the second similarity map to obtain the second frequency domain fine-tuned features; the second frequency domain fine-tuned features are fused with the randomized light-sensitive frequency domain features to obtain the second frequency domain fused features; the second frequency domain fused features after the fourth linear transformation are fused with the randomized light-sensitive frequency domain features to obtain the fine-tuned light-insensitive frequency domain features.

[0308] Step 1204: Perform an inverse transform on the frequency domain features of the second image to obtain the spatial domain features of the second image.

[0309] After obtaining the frequency domain features of the second image, the computer device can inversely transform the frequency domain features of the second image from the frequency domain to the spatial domain through an inverse transformation, thereby obtaining the spatial domain features of the second image corresponding to the nighttime road image.

[0310] In one possible implementation, the computer device can use inverse discrete sine transform to inversely transform the frequency domain features of the second image from the frequency domain to the spatial domain, thereby obtaining the spatial domain features of the second image corresponding to the nighttime road image.

[0311] Step 1205: Decode the spatial features of the second image using an image segmentation model to obtain the image segmentation result corresponding to the nighttime road image.

[0312] After obtaining the second image spatial features, the computer device can input the second image spatial features into the decoder. By decoding the second image spatial features, the image segmentation result corresponding to the nighttime road image can be obtained.

[0313] Please refer to Figure 13 This illustrates a structural block diagram of an image segmentation apparatus provided in an exemplary embodiment of this application, the apparatus comprising:

[0314] Feature extraction module 1301 is used to extract the first image spatial features of the night scene image through an image segmentation model;

[0315] The feature transformation module 1302 is used to transform the spatial domain features of the first image to obtain the frequency domain features of the first image, wherein the frequency domain features of the first image include sub-frequency domain features of at least two frequency bands.

[0316] The feature fine-tuning module 1303 is used to fine-tune the sub-frequency domain features of the at least two frequency bands based on the difference in light sensitivity between the sub-frequency domain features of the at least two frequency bands using at least two different feature fine-tuning methods to obtain the second image frequency domain features.

[0317] The feature inverse transformation module 1304 is used to perform an inverse transformation on the frequency domain features of the second image to obtain the spatial domain features of the second image.

[0318] The feature decoding module 1305 is used to decode the spatial features of the second image through the image segmentation model to obtain the image segmentation result corresponding to the night scene image.

[0319] Optionally, the feature fine-tuning module 1303 includes:

[0320] The feature segmentation unit is used to divide the first image frequency domain features into a first sub-frequency domain feature and a second sub-frequency domain feature based on the at least two frequency bands. The frequency of the frequency band corresponding to the first sub-frequency domain feature is lower than the frequency of the frequency band corresponding to the second sub-frequency domain feature, and the sensitivity of the first sub-frequency domain feature to illumination is lower than the sensitivity of the second sub-frequency domain feature to illumination.

[0321] The first feature fine-tuning unit is used to fine-tune the first sub-frequency domain feature using the first feature fine-tuning method to obtain the fine-tuned first sub-frequency domain feature.

[0322] The second feature fine-tuning unit is used to fine-tune the second sub-frequency domain feature using a second feature fine-tuning method to obtain the fine-tuned second sub-frequency domain feature. The second feature fine-tuning method is different from the first feature fine-tuning method.

[0323] The feature fusion unit is used to fuse the finely tuned first sub-frequency domain features and the finely tuned second sub-frequency domain features to obtain the second image frequency domain features.

[0324] Optionally, the first feature fine-tuning unit is used for:

[0325] A dot product operation is performed on the first sub-frequency domain feature and the first label sequence corresponding to the first sub-frequency domain feature to obtain a first similarity map. The first label sequence is learned during the training of the image segmentation model. The first similarity map is used to characterize the feature similarity between the first sub-frequency domain feature of light and the first label sequence.

[0326] Perform a dot product operation on the first label sequence after the first linear transformation and the first similarity map to obtain the first frequency domain fine-tuning feature;

[0327] The first sub-frequency domain feature is finely adjusted based on the first frequency domain fine-tuning feature to obtain the finely adjusted first sub-frequency domain feature.

[0328] Optionally, the first feature fine-tuning unit is used for:

[0329] The first frequency domain fine-tuning feature and the first sub-frequency domain feature are fused to obtain the first frequency domain fusion feature;

[0330] The first frequency domain fusion feature after the second linear transformation is combined with the first sub-frequency domain feature to obtain the fine-tuned first sub-frequency domain feature.

[0331] Optionally, the second feature fine-tuning unit is used for:

[0332] Randomization is performed on the second sub-frequency domain features to obtain the randomized second sub-frequency domain features;

[0333] A dot product operation is performed on the randomized second sub-frequency domain features and the second label sequence corresponding to the second sub-frequency domain features to obtain a second similarity map. The second label sequence is learned during the training of the image segmentation model. The second similarity map is used to characterize the feature similarity between the randomized second sub-frequency domain features and the second label sequence.

[0334] Perform a dot product operation on the second label sequence after the third linear transformation and the second similarity map to obtain the second frequency domain fine-tuning feature;

[0335] The second sub-frequency domain feature after randomization is finely adjusted based on the second frequency domain fine-tuning feature to obtain the finely adjusted second sub-frequency domain feature.

[0336] Optionally, the second feature fine-tuning unit is used for:

[0337] Based on the channel response threshold, the frequency domain features in each feature channel of the second sub-frequency domain feature are randomly assigned values ​​to obtain the randomized second sub-frequency domain feature.

[0338] Wherein, the channel response value of the feature channel after randomization is not greater than the channel response threshold, and the channel response value is used to characterize the response characteristics of the frequency domain features within the feature channel.

[0339] Optionally, the first image frequency domain feature includes at least two second sub-frequency domain features, and each of the at least two second sub-frequency domain features corresponds to its respective second label sequence.

[0340] Optionally, the second feature fine-tuning unit is further configured to:

[0341] By fusing the second frequency domain fine-tuning feature with the randomized second sub-frequency domain feature, a second frequency domain fusion feature is obtained;

[0342] The second frequency domain fusion feature after the fourth linear transformation is combined with the second sub-frequency domain feature after randomization to obtain the fine-tuned second sub-frequency domain feature.

[0343] Optionally, the feature partitioning unit is used for:

[0344] The first image frequency domain features are divided into frequency domain features of at least two frequency bands by using at least two bandpass filters;

[0345] The frequency domain features corresponding to the frequency bands within the first frequency range are determined as the first sub-frequency domain features, and the frequency domain features corresponding to the frequency bands within the second frequency range are determined as the second sub-frequency domain features, wherein the frequencies of the first frequency range are lower than the frequencies of the second frequency range.

[0346] Optionally, the feature transformation module 1302 is used for:

[0347] The spatial domain features of the first image are transformed by discrete sine transform to obtain the frequency domain features of the first image;

[0348] The feature inverse transformation module 1304 is used for:

[0349] The second image's frequency domain features are inversely transformed by the inverse discrete sine transform to obtain the second image's spatial domain features.

[0350] Optionally, the first image spatial features are extracted by the image encoder in the image segmentation model;

[0351] The image segmentation model includes at least two image encoders connected in series; wherein the input of the subsequent image encoder is the image spatial features output by the previous image encoder and finely processed.

[0352] Optionally, the device further includes:

[0353] The sample feature extraction module is used to extract the first sample image spatial features of the sample image through the image segmentation model;

[0354] The sample feature transformation module is used to transform the spatial domain features of the first sample image to obtain the frequency domain features of the first sample image, wherein the frequency domain features of the first sample image include sample sub-frequency domain features of at least two frequency bands.

[0355] The sample feature fine-tuning module is used to fine-tune the sample sub-frequency domain features of the at least two frequency bands based on the difference in light sensitivity between the sample sub-frequency domain features of the at least two frequency bands using at least two different feature fine-tuning methods to obtain the second sample image frequency domain features.

[0356] The sample feature inverse transformation module is used to perform an inverse transformation on the frequency domain features of the second sample image to obtain the spatial domain features of the second sample image.

[0357] The sample feature decoding module is used to decode the spatial features of the second sample image through the image segmentation model to obtain the sample image segmentation result corresponding to the sample image;

[0358] The model training module is used to train the image segmentation model based on the ground truth value of the image segmentation result corresponding to the sample image and the sample image segmentation result.

[0359] Optionally, the image segmentation model includes a first label sequence corresponding to the first sample sub-frequency domain features and a second label sequence corresponding to the second sample sub-frequency domain features;

[0360] The model training module is used for:

[0361] Based on the ground truth value of the image segmentation result corresponding to the sample image and the sample image segmentation result, determine the image segmentation loss;

[0362] Based on the image segmentation loss, update the first label sequence and the second label sequence.

[0363] In summary, in the embodiments of this application, during the image segmentation of a night scene image using an image segmentation model, after extracting the first image spatial domain features of the night scene image, the first image spatial domain features are transformed to obtain a first image frequency domain feature including sub-frequency domain features of at least two frequency bands. Based on the difference in light sensitivity between the sub-frequency domain features of the at least two frequency bands, at least two different feature fine-tuning methods are used to fine-tune the sub-frequency domain features of the at least two frequency bands, thereby obtaining the second image frequency domain feature. Then, the second image frequency domain feature is inversely transformed into the second image spatial domain feature, and the second image spatial domain feature is decoded to obtain the image segmentation result corresponding to the night scene image. The solution provided in this application starts from the perspective that the sub-frequency domain features of different frequency bands have different sensitivities to illumination. By fine-tuning the image features in the frequency domain through different feature fine-tuning methods, it is possible to make the image segmentation model pay more attention to the sub-frequency domain features with lower illumination sensitivity and more stable features, while increasing the robustness of the sub-frequency domain features with higher illumination sensitivity. This allows the image segmentation model to also learn the fine-tuned sub-frequency domain features with higher illumination sensitivity, thereby improving the image segmentation accuracy of the image segmentation model for night scene images.

[0364] It should be noted that the apparatus provided in the above embodiments is only an example of the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the apparatus can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and their implementation process can be found in the method embodiments, which will not be repeated here.

[0365] Please refer to Figure 14 This illustration shows a schematic diagram of a computer device provided in an exemplary embodiment of this application. Specifically, the computer device 1400 includes a Central Processing Unit (CPU) 1401, a system memory 1404 including a random access memory 1402 and a read-only memory 1403, and a system bus 1405 connecting the system memory 1404 and the CPU 1401. The computer device 1400 may also include a basic input / output system (I / O system) 1406 to facilitate the transfer of information between various devices within the computer, and a mass storage device 1407 for storing the operating system 1413, application programs 1414, and other program modules 1415.

[0366] In some embodiments, the basic input / output system 1406 includes a display 1408 for displaying information and an input device 1409 for user input, such as a mouse or keyboard. Both the display 1408 and the input device 1409 are connected to the central processing unit 1401 via an input / output controller 1410 connected to a system bus 1405. The basic input / output system 1406 may also include the input / output controller 1410 for receiving and processing input from multiple other devices such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 1410 also provides output to a display screen, printer, or other types of output devices.

[0367] The mass storage device 1407 is connected to the central processing unit 1401 via a mass storage controller (not shown) connected to the system bus 1405. The mass storage device 1407 and its associated computer-readable media provide non-volatile storage for the computer device 1400. That is, the mass storage device 1407 may include computer-readable media (not shown) such as a hard disk or drive.

[0368] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include random access memory (RAM), read-only memory (ROM), flash memory or other solid-state storage technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that the computer storage media are not limited to the above-mentioned types. The system memory 1404 and mass storage device 1407 described above can be collectively referred to as memory.

[0369] The memory stores one or more programs, which are configured to be executed by one or more central processing units 1401. The one or more programs contain instructions for implementing the methods described above. The central processing unit 1401 executes the one or more programs to implement the image segmentation methods provided in the various method embodiments described above.

[0370] According to various embodiments of this application, the computer device 1400 can also be connected to a remote computer on a network, such as the Internet. That is, the computer device 1400 can be connected to the network 1411 via the network interface unit 1412 connected to the system bus 1405, or the network interface unit 1412 can be used to connect to other types of networks or remote computer systems (not shown).

[0371] This application also provides a computer-readable storage medium storing at least one instruction, which is loaded and executed by a processor to implement the image segmentation method described in the above embodiments.

[0372] Optionally, the computer-readable storage medium may include ROM, RAM, solid-state drives (SSDs), or optical discs, etc. The RAM may include resistive random access memory (ReRAM) and dynamic random access memory (DRAM).

[0373] This application provides a computer program product including at least one instruction stored in a computer-readable storage medium. A processor of a computer device reads the at least one instruction from the computer-readable storage medium and executes the at least one instruction, causing the computer device to perform the image segmentation method described in the above embodiments.

[0374] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0375] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. An image segmentation method, characterized in that, The method includes: The first spatial features of the night scene image are extracted using an image segmentation model; The first image spatial domain features are transformed to obtain the first image frequency domain features, which include sub-frequency domain features of at least two frequency bands. Based on the difference in light sensitivity between the sub-frequency domain features of the at least two frequency bands, the sub-frequency domain features of the at least two frequency bands are fine-tuned using at least two different feature fine-tuning methods to obtain the second image frequency domain features; The second image frequency domain features are inversely transformed to obtain the second image spatial domain features; The image segmentation result corresponding to the night scene image is obtained by decoding the spatial features of the second image using the image segmentation model.

2. The method according to claim 1, characterized in that, The second image frequency domain feature is obtained by fine-tuning the sub-frequency domain features of the at least two frequency bands based on the difference in light sensitivity between the sub-frequency domain features of the at least two frequency bands using at least two different feature fine-tuning methods, including: Based on the at least two frequency bands, the first image frequency domain feature is divided into a first sub-frequency domain feature and a second sub-frequency domain feature. The frequency of the frequency band corresponding to the first sub-frequency domain feature is lower than the frequency of the frequency band corresponding to the second sub-frequency domain feature. The sensitivity of the first sub-frequency domain feature to illumination is lower than the sensitivity of the second sub-frequency domain feature to illumination. The first sub-frequency domain feature is fine-tuned using the first feature fine-tuning method to obtain the fine-tuned first sub-frequency domain feature; The second sub-frequency domain feature is fine-tuned using a second feature fine-tuning method to obtain the fine-tuned second sub-frequency domain feature. The second feature fine-tuning method is different from the first feature fine-tuning method. The second image frequency domain feature is obtained by fusing the finely adjusted first sub-frequency domain feature and the finely adjusted second sub-frequency domain feature.

3. The method according to claim 2, characterized in that, The step of fine-tuning the first sub-frequency domain feature using the first feature fine-tuning method to obtain the fine-tuned first sub-frequency domain feature includes: A dot product operation is performed on the first sub-frequency domain feature and the first label sequence corresponding to the first sub-frequency domain feature to obtain a first similarity map. The first label sequence is learned during the training of the image segmentation model. The first similarity map is used to characterize the feature similarity between the first sub-frequency domain feature and the first label sequence. Perform a dot product operation on the first label sequence after the first linear transformation and the first similarity map to obtain the first frequency domain fine-tuning feature; The first sub-frequency domain feature is finely adjusted based on the first frequency domain fine-tuning feature to obtain the finely adjusted first sub-frequency domain feature.

4. The method according to claim 3, characterized in that, The step of fine-tuning the first sub-frequency domain features based on the first frequency domain fine-tuning features to obtain the fine-tuned first sub-frequency domain features includes: The first frequency domain fine-tuning feature and the first sub-frequency domain feature are fused to obtain the first frequency domain fusion feature; The first frequency domain fusion feature after the second linear transformation is combined with the first sub-frequency domain feature to obtain the fine-tuned first sub-frequency domain feature.

5. The method according to claim 2, characterized in that, The step of fine-tuning the second sub-frequency domain feature using the second feature fine-tuning method to obtain the fine-tuned second sub-frequency domain feature includes: Randomization is performed on the second sub-frequency domain features to obtain the randomized second sub-frequency domain features; A dot product operation is performed on the randomized second sub-frequency domain features and the second label sequence corresponding to the second sub-frequency domain features to obtain a second similarity map. The second label sequence is learned during the training of the image segmentation model. The second similarity map is used to characterize the feature similarity between the randomized second sub-frequency domain features and the second label sequence. Perform a dot product operation on the second label sequence after the third linear transformation and the second similarity map to obtain the second frequency domain fine-tuning feature; The second sub-frequency domain feature after randomization is finely adjusted based on the second frequency domain fine-tuning feature to obtain the finely adjusted second sub-frequency domain feature.

6. The method according to claim 5, characterized in that, The randomization process performed on the second sub-frequency domain features to obtain the randomized second sub-frequency domain features includes: Based on the channel response threshold, the frequency domain features in each feature channel of the second sub-frequency domain feature are randomly assigned values ​​to obtain the randomized second sub-frequency domain feature. Wherein, the channel response value of the feature channel after randomization is not greater than the channel response threshold, and the channel response value is used to characterize the response characteristics of the frequency domain features within the feature channel.

7. The method according to claim 5, characterized in that, The first image frequency domain feature includes at least two second sub-frequency domain features, and each of the at least two second sub-frequency domain features corresponds to its respective second label sequence.

8. The method according to claim 5, characterized in that, The step of fine-tuning the randomized second sub-frequency domain features based on the second frequency domain fine-tuning features to obtain the fine-tuned second sub-frequency domain features includes: By fusing the second frequency domain fine-tuning feature with the randomized second sub-frequency domain feature, a second frequency domain fusion feature is obtained; The second frequency domain fusion feature after the fourth linear transformation is combined with the second sub-frequency domain feature after randomization to obtain the fine-tuned second sub-frequency domain feature.

9. The method according to any one of claims 2 to 8, characterized in that, The step of dividing the first image frequency domain features into a first sub-frequency domain feature and a second sub-frequency domain feature based on the at least two frequency bands includes: The first image frequency domain features are divided into frequency domain features of at least two frequency bands by using at least two bandpass filters; The frequency domain features corresponding to the frequency bands within the first frequency range are determined as the first sub-frequency domain features, and the frequency domain features corresponding to the frequency bands within the second frequency range are determined as the second sub-frequency domain features, wherein the frequencies of the first frequency range are lower than the frequencies of the second frequency range.

10. The method according to any one of claims 1 to 9, characterized in that, The step of transforming the spatial domain features of the first image to obtain the frequency domain features of the first image includes: The spatial domain features of the first image are transformed by discrete sine transform to obtain the frequency domain features of the first image; The step of performing an inverse transform on the frequency domain features of the second image to obtain the spatial domain features of the second image includes: The second image's frequency domain features are inversely transformed by the inverse discrete sine transform to obtain the second image's spatial domain features.

11. The method according to any one of claims 1 to 10, characterized in that, The first image spatial features are extracted by the image encoder in the image segmentation model; The image segmentation model includes at least two image encoders connected in series; wherein the input of the subsequent image encoder is the image spatial features output by the previous image encoder and finely processed.

12. The method according to any one of claims 1 to 11, characterized in that, The method further includes: The image segmentation model is used to extract the spatial features of the first sample image. The spatial domain features of the first sample image are transformed to obtain the frequency domain features of the first sample image, and the frequency domain features of the first sample image include sample sub-frequency domain features of at least two frequency bands. Based on the difference in light sensitivity between the sample sub-frequency domain features of the at least two frequency bands, the sample sub-frequency domain features of the at least two frequency bands are fine-tuned using at least two different feature fine-tuning methods to obtain the second sample image frequency domain features. The frequency domain features of the second sample image are inversely transformed to obtain the spatial domain features of the second sample image; The spatial features of the second sample image are decoded using the image segmentation model to obtain the sample image segmentation result corresponding to the sample image; The image segmentation model is trained based on the ground truth value of the image segmentation result corresponding to the sample image and the sample image segmentation result.

13. The method according to claim 12, characterized in that, The image segmentation model includes a first label sequence corresponding to the first sample sub-frequency domain features and a second label sequence corresponding to the second sample sub-frequency domain features; The step of training the image segmentation model based on the ground truth value of the image segmentation result corresponding to the sample image and the sample image segmentation result includes: Based on the ground truth value of the image segmentation result corresponding to the sample image and the sample image segmentation result, determine the image segmentation loss; Based on the image segmentation loss, update the first label sequence and the second label sequence.

14. An image segmentation apparatus, characterized in that, The device includes: The feature extraction module is used to extract the first image spatial features of the night scene image through an image segmentation model; The feature transformation module is used to transform the spatial domain features of the first image to obtain the frequency domain features of the first image, wherein the frequency domain features of the first image include sub-frequency domain features of at least two frequency bands. The feature fine-tuning module is used to fine-tune the sub-frequency domain features of the at least two frequency bands based on the difference in light sensitivity between the sub-frequency domain features of the at least two frequency bands using at least two different feature fine-tuning methods to obtain the second image frequency domain features. The feature inverse transformation module is used to perform an inverse transformation on the frequency domain features of the second image to obtain the spatial domain features of the second image. The feature decoding module is used to decode the spatial features of the second image through the image segmentation model to obtain the image segmentation result corresponding to the night scene image.

15. A computer device, characterized in that, The computer device includes a processor and a memory; the memory stores at least one instruction, which is executed by the processor to implement the image segmentation method as described in any one of claims 1 to 13.

16. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, which is executed by a processor to implement the image segmentation method as described in any one of claims 1 to 13.

17. A computer program product, characterized in that, The computer program product includes at least one instruction stored in a computer-readable storage medium; a processor of a computer device reads the at least one instruction from the computer-readable storage medium and executes the at least one instruction to cause the computer device to implement the image segmentation method as described in any one of claims 1 to 13.

Images