A zero-shot day-night domain adaptive training method and image classification method based on spectrum analysis

Abstract

The present invention discloses a zero-shot day / night domain adaptive training method based on spectral analysis, comprising the following steps: receiving a daytime image training set; obtaining an original training feature map based on the daytime image training set; determining an enhanced feature map based on the original training feature map; determining a supervised contrast loss function based on the original training feature map and the enhanced feature map; calculating an output descriptor based on the original training feature map; determining a cross-entropy loss function based on the enhanced feature map and the output descriptor; and updating network parameters based on the supervised contrast loss function and the cross-entropy loss function to obtain an adaptive classification model. The present invention also provides a zero-shot day / night domain adaptive image classification method based on spectral analysis, which can remove the influence of generalization-irrelevant factors and retain generalization-relevant object features, thereby improving the model's adaptability in the nighttime domain.

Description

Technical Field

[0001] The present invention belongs to the technical field of deep image recognition, and specifically relates to a zero-sample day / night domain adaptive training method and an image classification method based on spectrum analysis. Background Art

[0002] In practical applications, deep image recognition models are susceptible to illumination variations. For example, when a model trained on daytime scenes is applied to nighttime scenes, the model's performance often degrades significantly due to the domain shift. Therefore, zero-shot day / night domain adaptation remains an important research area. The goal of domain adaptation is to train a model on a source domain dataset so that it performs well on a different but similar target domain dataset. Domain adaptation can save expensive data collection and annotation costs. Existing deep domain adaptation methods can be mainly divided into two categories: those based on differential optimization and those based on adversarial learning. The zero-shot day / night domain adaptation task aims to generalize a model trained solely on daytime data to the nighttime domain. Compared to traditional domain adaptation, the zero-shot day / night domain adaptation task is more challenging due to the lack of access to target domain (nighttime scene) data during training. Existing zero-shot day / night domain adaptation methods focus on extracting color-invariant representations to mitigate the low-light effects of nighttime scenes, while neglecting the impact of other factors (such as texture and style) on generalization ability. This results in poor generalization performance and, in turn, poor adaptation performance in the nighttime domain. Summary of the Invention

[0003] In order to solve the above problems existing in the prior art, the present invention provides a zero-sample day and night domain adaptive training method and image classification method based on spectrum analysis. The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0004] A zero-sample day-night domain adaptive training method based on spectrum analysis includes the following steps:

[0005] Receive a training set of daytime images;

[0006] Obtaining an original training feature map according to the daytime image training set;

[0007] Determine an enhanced feature map based on the original training feature map;

[0008] Determining a supervised contrast loss function based on the original training feature map and the enhanced feature map;

[0009] Calculating an output descriptor based on the original training feature map;

[0010] Determining a cross entropy loss function based on the enhanced feature map and the output descriptor;

[0011] The network parameters are updated through training according to the supervised contrast loss function and the cross entropy loss function to obtain an adaptive classification model.

[0012] In one embodiment of the present invention, determining the enhanced feature map according to the original training feature map includes:

[0013] Determining a feature map spectrum feature based on the original training feature map;

[0014] Determine the high frequency component of the spectrum according to the spectrum characteristics of the characteristic graph;

[0015] Determine a high-pass characteristic map according to the high-frequency components of the frequency spectrum;

[0016] An enhanced feature map is determined according to the original training feature map and the high-pass feature map.

[0017] In one embodiment of the present invention, determining the feature graph spectral features according to the original training feature graph includes:

[0018] Performing Fourier transform on the original training feature map to obtain feature map frequency spectrum features.

[0019] In one embodiment of the present invention, determining the high-frequency component of the spectrum according to the spectrum characteristics of the characteristic graph includes:

[0020] The low-frequency components of the spectrum characteristics of the characteristic graph are removed by a high-pass filter to obtain the high-frequency components of the spectrum.

[0021] In one embodiment of the present invention, determining a high-pass feature map according to the high-frequency components of the spectrum includes:

[0022] Perform inverse Fourier transform on the high-frequency components of the spectrum to obtain a high-pass feature map.

[0023] In one embodiment of the present invention, determining an enhanced feature map according to the original training feature map and the high-pass feature map includes:

[0024] Batch splicing the original training feature map and the high-pass feature map to obtain a spliced feature map;

[0025] The concatenated feature map is input into the residual network for conversion, and an enhanced feature map is output.

[0026] In one embodiment of the present invention, the calculating the output descriptor according to the original training feature map includes:

[0027] Determine a dimensionality reduction feature map based on the original training feature map;

[0028] Determine a prototype set according to the dimensionality reduction feature map;

[0029] An output descriptor is calculated based on the set of prototypes.

[0030] In one embodiment of the present invention, the expression of the supervised contrast loss function is:

[0031]

[0032] Among them, y i and y j Represent the labels of anchor sample i and sample j respectively, B represents the batch size, Indicates that the label in the batch is y j The number of samples, l i≠j and Represents a similar indicator function, s i，j =v i T v j / ||v i ||||v j || represents the cosine similarity between anchor sample i and sample j, v i and v j They represent the high-level feature vectors of anchor samples i and j respectively, and t represents the temperature hyperparameter; the original training feature map is used as the anchor sample, the enhanced feature map is used as the positive sample, and the feature map of different categories from the current input is used as the negative sample.

[0033] In one embodiment of the present invention, the expression of the high-pass filter is:

[0034]

[0035] Where α and β represent the cutoff thresholds for removing low-frequency components, and (u, v) represents the coordinates of the center spectrum.

[0036] A second aspect of an embodiment of the present invention provides a zero-sample day / night domain adaptive image classification method based on spectrum analysis, comprising the following steps:

[0037] The pre-trained feature extractor extracts features from the dataset to be classified and outputs a feature map to be classified;

[0038] The feature map to be classified is input into the trained prototype compensation module to calculate the prototype, and the output descriptor to be classified is obtained according to the prototype calculation;

[0039] The feature map to be classified and the output descriptor to be classified are spliced together to output a spliced feature map to be classified;

[0040] The spliced feature map to be classified is input into the trained classification network, and the classification result is output;

[0041] The pre-trained feature extractor, the trained prototype compensation module and the trained classification network constitute an adaptive classification model obtained by training using the training method provided in the first aspect of the embodiment of the present invention.

[0042] Beneficial effects of the present invention:

[0043] The present invention compares the original training feature maps with the enhanced feature maps to reduce the influence of low-frequency components (e.g., image texture, style, and color), facilitating the extraction of generalized representations. Prototypes are calculated from underlying feature maps to promote the inclusion of rich semantic information in higher layers, enhancing the model's generalization capabilities. This improves the model's classification and detection capabilities in the nighttime domain, even when nighttime imagery is inaccessible during training.

[0044] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A flowchart of a zero-sample day-night domain adaptive training method based on spectrum analysis provided by an embodiment of the present invention;

[0046] Figure 2 An overall network framework diagram of a zero-sample day / night domain adaptive training method based on spectrum analysis provided by an embodiment of the present invention;

[0047] Figure 3 A schematic diagram of a knowledge filtering network for a zero-sample day / night domain adaptive training method based on spectrum analysis provided by an embodiment of the present invention;

[0048] Figure 4 A visualization image of the image classification method of the present invention and the nighttime domain detection results of the prior art;

[0049] Figure 5a Schematic diagram of feature visualization results using t-SNE on the CODaN dataset using the image classification method of the present invention and the prior art;

[0050] Figure 5b Schematic diagram of feature visualization results using t-SNE on the ShapeNet dataset using the image classification method of the present invention and the prior art;

[0051] Figure 6 Schematic diagram of the visualization results of the feature map of the CODaN dataset of the image classification method of the present invention and the prior art. DETAILED DESCRIPTION

[0052] The present invention will be further described in detail below with reference to specific examples, but the embodiments of the present invention are not limited thereto.

[0053] Example 1

[0054] like Figure 1 、 Figure 2 and Figure 3 As shown, this embodiment provides a zero-sample day and night domain adaptive training method based on spectrum analysis, including the following steps:

[0055] Step 10: Receive a daytime image training set.

[0056] Step 20: Obtain an original training feature map based on the daytime image training set.

[0057] In this step, the input daytime image training set is subjected to feature extraction by the pre-trained feature extractor, and the original training feature map is output. Where H, W, and C represent the height, width, and number of channels of the original training feature map, respectively. To reduce the influence of generalization-irrelevant factors and extract the generalized representation, frequency transformation is subsequently required to remove the low-frequency components of the extracted original training feature map.

[0058] Step 30: Determine an enhanced feature map based on the original training feature map. Specifically, determine the enhanced feature map based on the original training feature map. This step and step 40 are both performed by the comparison and filtering module.

[0059] Specifically, step 30 includes steps 31 to 34:

[0060] Step 31, determine the spectral characteristics of the feature map according to the original training feature map. Specifically, in this step, the original training feature map F b Perform Fourier transform to obtain the characteristic graph spectrum characteristics, which can be expressed as follows:

[0061]

[0062] in, Represents the i-th feature map of the original training feature map, i=1,...,C, represents Fourier transform, and (u, v) represents the coordinates of the center spectrum.

[0063] Step 32: Determine the high-frequency components of the spectrum according to the spectrum characteristics of the characteristic graph. Specifically, remove the low-frequency components of the spectrum characteristics of the characteristic graph through a high-pass filter to obtain the high-frequency components of the spectrum.

[0064] In this step, after Fourier transform, the low-frequency components of the image contain domain-specific information, and the high-frequency components contain domain-invariant information. Next, in order to reduce the impact of the low-frequency components, a high-pass filter is defined to remove the low-frequency components. The high-pass filter is:

[0065]

[0066] Among them, α and β represent the cutoff thresholds for removing low-frequency components. Then, the high-pass filter removes the low-frequency components of the feature map spectrum features by performing element-by-element multiplication with the feature map spectrum features to obtain the output spectrum high-frequency components:

[0067]

[0068] Among them, ⊙ represents the element-by-element product, Represents the high-frequency component of the spectrum output by the high-pass filter.

[0069] Step 33: Determine a high-pass feature map based on the high-frequency components of the spectrum. Specifically, perform an inverse Fourier transform on the high-frequency components of the spectrum to obtain a high-pass feature map:

[0070]

[0071] in, is the filtered high-pass feature map.

[0072] After passing through the high-pass filter, the output feature map contains less background information irrelevant to generalization than the original input feature map, and contains richer object-related content, which is conducive to enhancing the model's ability to extract robust features and improving the model's generalization performance.

[0073] Step 34: Determine an enhanced feature map based on the original training feature map and the high-pass feature map. This step includes steps 341 and 342:

[0074] Step 341 : batch-splicing the original training feature map and the high-pass feature map to obtain a spliced feature map.

[0075] In step 342, the concatenated feature map is input into the residual network for conversion, and an enhanced feature map F is output.

[0076] F=Φ([F a , F b ] B );

[0077] Among them, the residual network Φ consists of four residual network blocks, which are used to transform the connection results, [,] B Represents the concatenation operation of the batch size dimension.

[0078] Step 40: Determine a supervised contrast loss function based on the original training feature map and the enhanced feature map.

[0079] In order to encourage the filtered features to contain more information related to the object, supervised contrastive learning (Prannay Khosla et al. "Supervised contrastive learning". In: Advances in neural information processing systems 33 (2020), pp. 18661-18673.) is used to further enhance the filtered features. Specifically, the initially extracted features of the current input and the corresponding filtered features are used as anchor points and positive samples, respectively, while features of different categories from the current input are used as negative samples. The supervised contrast loss function can be determined:

[0080]

[0081] Among them, y i and y j Represent the labels of anchor sample i and sample j respectively, B represents the batch size, Indicates that the label in the batch is y j The number of samples. i≠j and is a similar indicator function, for example, if i≠j, l i≠j ∈{0,1}=1, otherwise l i≠j ∈{0,1}=0.s i，j =v i T v j / ||v i ||||v j || represents the cosine similarity between anchor sample i and sample j, where v i and v j denote the high-level feature vectors representing anchor samples i and j, respectively. t is the temperature hyperparameter, set to 0.07. The original training feature map is used as the anchor sample, the enhanced feature map is used as the positive sample, and the feature map of a different category from the current input is used as the negative sample.

[0082] Supervised contrast loss is used to narrow the semantic gap between the original features and the corresponding filtered features, which promotes the filtered features to contain rich object-related information and helps to alleviate the impact of low light in night scenes.

[0083] Step 50: Calculate the output descriptor based on the original training feature map. After obtaining the original training feature map, this step is performed in parallel with steps 30 and 40.

[0084] After being processed by the contrast filtering module (after being processed by steps 30 and 40), the extracted features contain rich object-related information, which improves the generalization ability of the model. However, the contrast filtering module may filter out some discriminative content that is crucial for identifying the corresponding object. To this end, a prototype compensation module can be used to calculate multiple prototypes from the underlying feature map to enhance the semantic information of high-level features. Specifically, step 50 is performed by the prototype compensation module, including steps 51 to 53:

[0085] Step 51: Determine a dimension-reduced feature map based on the original training feature map.

[0086] Step 52: Determine a prototype set based on the dimensionality reduction feature map.

[0087] Step 53: Calculate the output descriptor based on the prototype set.

[0088] Specifically, the original training feature map is firstly processed using the maximum pooling operation. Perform dimensionality reduction to obtain dimensionality reduction feature map Then the feature map F after dimensionality reduction is bm Input into the prototype compensation module, a set of prototypes is calculated (Aming Wu et al. "Universal-prototype enhancing for few-shot object detection". In: Proceedings of the IEEE / CVF International Conference on Computer Vision. 2021, pp. 9567-9576.). These prototypes constitute the prototype set Among them, p i is the prototype, and D is the number of prototypes. Next, the descriptor representing the image-level information is calculated based on the prototype set P:

[0089]

[0090] in, and is the convolution parameter, Represents the output descriptor, represents the residual operation that assigns visual features to corresponding prototypes.

[0091] Here, the contrast filtering module and the prototype compensation module constitute the knowledge filtering network.

[0092] Step 60: Determine a cross entropy loss function based on the enhanced feature map and the output descriptor.

[0093] In this step, specifically, F and The concatenated result is input into the classification network to obtain the predicted probability, thereby calculating the cross entropy loss function.

[0094] in, Represents the predicted probability:

[0095]

[0096] Ψ consists of two 1×1 convolutional layers with ReLU activation function to align the dimensions. yes The result of plastic surgery. and are the parameters of the fully connected layer. [,] Ｃ Represents the splicing operation of the channel dimension. Through the splicing operation, the descriptor containing the underlying discriminant information can be This is fused into feature F, enhancing F's discriminative power. Cls represents the classification network, a fully connected layer, where N is the number of categories in the daytime image training set. The model's ability to extract domain-invariant features can be further enhanced by using the prototype compensation module.

[0097] Step 70: Perform training based on the supervised contrast loss function and the cross entropy loss function to update the network parameters. After the training is completed, an adaptive classification model is obtained.

[0098] Specifically, training is performed using a supervised contrastive loss function and a cross-entropy loss function to update the parameters of the contrastive filtering module, prototype compensation module, and classification network. After training, the trained contrastive filtering module, prototype compensation module, and classification network are obtained. The trained prototype compensation module, trained classification network, and pre-trained feature extractor together form an adaptive classification model.

[0099] Specifically, the overall training objective (overall loss function ) by the cross entropy loss function and supervised contrast loss function composition.

[0100] The loss function of the model is:

[0101]

[0102] Here, λ represents a hyperparameter. Setting λ to 1 can balance the two loss terms.

[0103] The training method of this embodiment can effectively improve the model's image classification and object detection capabilities in the nighttime domain. The contrast filtering module can effectively remove the influence of irrelevant factors, reduce the impact of low-frequency components (e.g., style), and extract domain-invariant features. The prototype compensation module helps high-level layers incorporate rich semantic information, improving the model's generalization capabilities.

[0104] It should be noted that both the contrast filtering module and the prototype compensation module need to be used simultaneously during training. The network model structure during training includes a pre-trained feature extractor, a contrast filtering module, a prototype compensation module, and a classification network. However, during actual inference, by discarding the contrast filtering module and retaining the prototype compensation module, the model is generalized to unknown target domains to classify images. That is, after training is complete, the network without the trained contrast filtering module is used as the adaptive classification model. The resulting adaptive classification model then includes a pre-trained feature extractor, a trained prototype compensation module, and a trained classification network.

[0105] Example 2

[0106] This embodiment provides a zero-sample day / night domain adaptive image classification method based on spectrum analysis, which is applied to the adaptive classification model trained in the first embodiment. The adaptive classification model includes: a pre-trained feature extractor, a trained prototype compensation module, and a trained classification network.

[0107] Specifically, a zero-sample day-night domain adaptive image classification method based on spectrum analysis includes the following steps:

[0108] Step 1: The pre-trained feature extractor extracts features from the dataset to be classified and outputs a feature map to be classified.

[0109] Step 2: The feature map to be classified is input into the trained prototype compensation module to calculate the prototype, and the output descriptor to be classified is obtained based on the prototype.

[0110] Step 3: The feature map to be classified and the output descriptor to be classified are spliced together to output a spliced feature map to be classified.

[0111] Step 4: The spliced feature map to be classified is input into the trained classification network for classification, and the classification result is output.

[0112] like Figure 4 As shown, compared with the results of Faster R-CNN (shown in the first row), the image classification method of the present invention (shown in the second row) accurately detects objects in night images.

[0113] like Figure 5a The figure shows the results of feature visualization using t-SNE on the CODaN dataset by CIConv of the present invention and the prior art. Figure 5b The figure shows the results of feature visualization using t-SNE on the ShapeNet dataset by the present invention and the prior art CIConv. The colored dots represent different object categories. It can be seen that the method of the present invention can extract more discriminant features, thereby significantly improving the classification performance.

[0114] like Figure 6 As shown in FIG, compared with the feature map from CIConv (as shown in the second column), the features extracted by the method of the present invention (as shown in the third column) contain more object-related information and less background information, which is beneficial to improving the generalization performance.

[0115] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature identified as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, "plurality" means two or more, unless otherwise specifically defined.

[0116] In the description of this specification, the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification.

[0117] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. For those skilled in the art of the present invention, without departing from the concept of the present invention, several simple deductions or substitutions can be made, which should be considered to fall within the scope of protection of the present invention.

Claims

1. A zero-sample day-night domain adaptive training method based on spectrum analysis, characterized in that: The following steps are involved: Receive a training set of daytime images; Obtaining an original training feature map according to the daytime image training set; Determine an enhanced feature map based on the original training feature map; Determining a supervised contrast loss function based on the original training feature map and the enhanced feature map; Calculating an output descriptor based on the original training feature map; Determining a cross entropy loss function based on the enhanced feature map and the output descriptor; Training and updating network parameters according to the supervised contrast loss function and the cross entropy loss function to obtain an adaptive classification model; The determining of the enhanced feature map according to the original training feature map includes: Determining a feature map spectrum feature based on the original training feature map; Determine the high frequency component of the spectrum according to the spectrum characteristics of the characteristic graph; Determine a high-pass characteristic map according to the high-frequency components of the frequency spectrum; Determine an enhanced feature map according to the original training feature map and the high-pass feature map; The calculating the output descriptor according to the original training feature map includes: Determine a dimensionality reduction feature map based on the original training feature map; Determine a prototype set according to the dimensionality reduction feature map; An output descriptor is calculated based on the set of prototypes.

2. The zero-sample day-night domain adaptive training method based on spectrum analysis according to claim 1, characterized in that: The determining of the feature graph spectrum feature according to the original training feature graph includes: Performing Fourier transform on the original training feature map to obtain feature map frequency spectrum features.

3. The zero-sample day-night domain adaptive training method based on spectrum analysis according to claim 1, characterized in that: The determining of the high-frequency component of the spectrum according to the spectrum characteristics of the characteristic graph includes: The low-frequency components of the spectrum characteristics of the characteristic graph are removed by a high-pass filter to obtain the high-frequency components of the spectrum.

4. The zero-sample day-night domain adaptive training method based on spectrum analysis according to claim 1, characterized in that: Determining a high-pass characteristic map according to the high-frequency components of the spectrum includes: Perform inverse Fourier transform on the high-frequency components of the spectrum to obtain a high-pass feature map.

5. The zero-sample day-night domain adaptive training method based on spectrum analysis according to claim 4, characterized in that: Determining an enhanced feature map according to the original training feature map and the high-pass feature map includes: Batch splicing the original training feature map and the high-pass feature map to obtain a spliced feature map; The concatenated feature map is input into the residual network for conversion, and an enhanced feature map is output.

6. The zero-sample day-night domain adaptive training method based on spectrum analysis according to claim 1, characterized in that: The expression of the supervised contrast loss function is: ； in, and Represents anchor point samples respectively and samples Tags, represents the batch size, Indicates that the label in the batch is The number of samples, and represents a similar indicator function, Represents anchor point samples and samples The cosine similarity between and Represents anchor point samples respectively and samples The high-level feature vector of represents the temperature hyperparameter; the original training feature map is used as the anchor sample, the enhanced feature map is used as the positive sample, and the feature map of a different category from the current input is used as the negative sample.

7. The zero-sample day-night domain adaptive training method based on spectrum analysis according to claim 3, characterized in that: The expression of the high-pass filter is: ； in, and represents the cutoff threshold for removing low-frequency components, Indicates the coordinates of the center spectrum.

8. A zero-sample day-night domain adaptive image classification method based on spectrum analysis, characterized in that: The following steps are involved: The pre-trained feature extractor extracts features from the dataset to be classified and outputs a feature map to be classified; The feature map to be classified is input into the trained prototype compensation module to calculate the prototype, and the output descriptor to be classified is obtained according to the prototype calculation; The feature map to be classified and the output descriptor to be classified are spliced together to output a spliced feature map to be classified; The spliced feature map to be classified is input into the trained classification network, and the classification result is output; The pre-trained feature extractor, the trained prototype compensation module and the trained classification network constitute an adaptive classification model trained by the training method according to any one of claims 1 to 7.

Images