A domain-adaptive shadow detection method assisted by shadow image classification
By adding gradient inversion layer and domain classifier to the shadow detection model, and combining the shadow image classification task, the domain offset problem in shadow detection is solved, the model's shadow detection accuracy on the target data set is improved, and the effect of using shadowless images is achieved to improve detection performance.
Patent Information
- Application Number
- CN202210720961.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-06-16
- Publication Date
- 2025-05-02
- Estimated Expiration
- 2042-06-16
AI Technical Summary
The prior art has domain offset problems in shadow detection, which leads to poor performance of the model on the target data set and lacks methods to improve detection performance using shadowless images.
By adding gradient inversion layer and domain classifier to the shadow detection model, combined with shadow image classification task, the design training strategy uses the source dataset and the target dataset for training, reducing the impact of domain-independent features, thereby improving the model's shadow detection accuracy on the target dataset.
It effectively reduces the domain offset between data sets, improves the accuracy of the model's shadow detection on the target data set, and improves the detection performance with unshaded images.
Smart Images
Figure CN115063634B_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the field of image processing and relates to a domain adaptation shadow detection method assisted by shadow image classification. Background Art
[0002] The presence of shadows can interfere with some computer vision tasks, such as object detection, object tracking, and semantic segmentation. On the other hand, shadows also contain information such as the direction of light, camera position, and the geometric shape of the object. Therefore, image shadow detection is an important step.
[0003] Early shadow detection methods were designed based on imaging models or artificially designed features. These methods have high requirements on image quality and certain lighting conditions, such as Lambert surface and Planck light source. These methods need to meet specific conditions and are difficult to apply to different lighting conditions and more complex environments.
[0004] In the prior art, shadow detection methods based on convolutional neural networks (CNNs) have achieved higher accuracy and better generalization performance. CNN-based shadow detection methods require pixel-level image annotation, and collecting such a dataset is time-consuming and expensive. Existing shadow detection datasets, such as the ISTD and SBU datasets, contain 1,870 (1,330 pairs of samples are used as training sets and the remaining 540 pairs of samples are used as test sets) and 4,727 (4,089 pairs of samples are used as training sets and the remaining 638 pairs of samples are used as test sets) pairs of samples, respectively. Compared with the SBU dataset, the ISTD dataset has fewer samples, which to some extent limits the performance of the network model trained on the ISTD dataset.
[0005] In recent years, the method of rendering images and constructing datasets through computer software or game engines has received widespread attention. The advantage of this method is that since the rendering process can be controlled, all the data used to render a scene, such as shadow maps, can be obtained through computer technology, thus avoiding manual annotation of images. In this way, researchers can obtain a large number of training samples in a short period of time. However, due to certain differences between computer-generated images and images taken in the real world, models trained using computer-synthesized data will achieve poor results when tested on datasets obtained in the real world. This phenomenon is called domain shift.
[0006] On the other hand, the ISTD dataset contains shadow images and shadow-free images. How to use shadow-free images to improve the performance of shadow detection is a problem worth studying. Summary of the invention
[0007] In view of this, an object of the present invention is to provide a domain-adaptive shadow detection method for domain shift and a shadow detection method assisted by shadow image classification.
[0008] In order to achieve the above object, the present invention provides the following technical solutions:
[0009] A domain adaptation shadow detection method assisted by shadow image classification comprises the following steps:
[0010] S1: Adding gradient reversal layer and domain classifier to the shadow detection model to obtain domain adapted shadow detection network;
[0011] S2: Add the shadow image classification task to the domain-adapted shadow detection network to classify the shadow images;
[0012] S3: designing a training strategy to train the domain-adapted shadow detection network using the source dataset and the target dataset;
[0013] S4: Input the shadow image into the trained domain-adapted shadow detection network to obtain the prediction result of the shadow mask.
[0014] Furthermore, the adding of the gradient reversal layer in step S1 specifically includes: adding a gradient reversal layer (GRL) to the fifth layer of the feature extraction network of the shadow detection model, and using the output of the gradient reversal layer as the input of the domain classifier.
[0015] Furthermore, the output of the domain adaptation shadow detection network in step S1 indicates whether the pixel point (s, t) belongs to the shadow area, and the loss value at the pixel point (s, t) is calculated using binary cross entropy, as shown in the following formula:
[0016]
[0017] Where y represents the label value, Represents the shadow detection prediction result at the pixel point (s, t), The larger the value, the higher the probability that the pixel belongs to the shadow area. The total shadow detection loss is the sum of the loss values of all the points to be detected, as shown in the following formula:
[0018]
[0019] Where W and H represent the size of the input image;
[0020] The output of the domain classifier is a feature map of size 13×13×1. Each feature point on the feature map represents the classification result of the domain classifier for the domain label of the input image.
[0021] The loss function at each feature point (u, v) can be expressed as:
[0022]
[0023] Among them, y represents the domain label of the input image, represents the predicted probability of the domain label of the domain classifier at this feature point; the total domain adaptation loss is the sum of the loss values at all feature points, as shown in the following formula:
[0024]
[0025] Among them, M and N represent the size of the feature map.
[0026] Furthermore, the shadow image classification task in step S2 is to add a fully connected layer to the basic feature extraction network, convert the network into a binary classification network, add a binary classification cross entropy loss function, and realize shadow image classification.
[0027] Further, step S2 specifically includes:
[0028] After the feature extraction network, an average pooling layer with a step size of 1×1 and a window size of 7×7 is added to obtain a 1×1×2048 feature map. This feature map is used as the input of the 1000 classification fully connected layer of the ResNeXt101 network to obtain a 1000-dimensional vector. This vector is used as the input of the fully connected layer with an input dimension of 1000 and an output dimension of 2 to achieve image binary classification.
[0029] The shadow image classification task uses binary cross entropy as the loss function as follows:
[0030]
[0031] Where y is the true label of the input image. When y is 1, it means that the input image is a shadow image, and when y is 0, it means that the input image is a shadow-free image. Represents the predicted probability of the category to which the input image belongs, and ω is the weighting coefficient of the loss function.
[0032] Furthermore, the training strategy described in step S3 includes: freezing the parameters of the fourth and fifth layers of the feature extraction network during the domain adaptation process; freezing the parameters of the first to fourth layers of the feature extraction network when training the classification task; using shadow images and shadow labels in the source domain to train the shadow detection task; and using shadow images in the source domain and the target domain for domain adaptation training.
[0033] The beneficial effect of the present invention is that: the present invention uses a deep neural network to learn domain-independent features by adding a gradient reversal layer and a shadow image classification task to solve the domain offset problem between data sets, thereby improving the shadow detection accuracy of the model on the target data set.
[0034] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:
[0036] Figure 1 It is a schematic diagram of the process flow of the domain adaptation shadow detection method assisted by shadow image classification according to the present invention;
[0037] Figure 2 Domain adaptation shadow detection network structure diagram;
[0038] Figure 3 Domain-adaptive shadow detection architecture aided by shadow image classification;
[0039] Figure 4 The domain adaptation shadow detection result image when freezing the parameters of different layers of the feature extraction network;
[0040] Figure 5 Visual comparison results of several shadow detection methods. DETAILED DESCRIPTION
[0041] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.
[0042] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.
[0043] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.
[0044] like Figure 1-2 As shown, an example of the present invention provides a domain adaptation shadow detection method assisted by shadow image classification, the method comprising:
[0045] Step 1: Add a gradient reversal layer and a domain classifier to the shadow detection model to achieve domain-adaptive shadow detection; add a gradient reversal layer (GRL) after the fifth layer of the feature extraction network, and use the output of the gradient reversal layer as the input of the domain classifier. The domain-adaptive shadow detection network training consists of two steps:
[0046] (1) Use the shadow images and shadow labels in the source domain to perform shadow detection training. At this time, the output of the network is the prediction result for the shadow area in the image.
[0047] (2) Use shadow images in the source domain and the target domain for domain adaptation training. At this time, the output of the network is the prediction result for the domain category of the input image.
[0048] The output of the shadow detection network indicates whether the pixel point (s, t) belongs to the shadow area, so the loss value at the pixel point (s, t) is calculated using binary cross entropy, as shown in the following formula:
[0049]
[0050] Where y represents the label value, Represents the shadow detection prediction result at the pixel point (s, t), The larger the value, the higher the probability that the pixel belongs to the shadow area. The total shadow detection loss is the sum of the loss values of all points to be detected, as shown in the following formula:
[0051]
[0052] Where W and H represent the size of the input image.
[0053] The output of the domain classifier is a feature map of size 13×13×1. Each feature point on the feature map represents the classification result of the domain classifier for the domain label of the input image. Compared with the prediction result of size 1×1, the prediction result of size 13×13 is mapped to the input image as the domain classification result of multiple image regions, thereby indirectly obtaining more training samples. Similar to the loss function of the shadow detection task, the loss function on each feature point (u,v) can be expressed as:
[0054]
[0055] Among them, y represents the domain label of the input image, Represents the predicted probability of the domain label of the domain classifier at this feature point. The total domain adaptation loss is the sum of the loss values at all feature points, as shown in the following formula:
[0056]
[0057] Among them, M and N represent the size of the feature map.
[0058] Considering that the differences between the source dataset and the target dataset are mainly manifested in the inconsistency of shallow features, such as the brightness of colors, the parameters of the fourth and fifth layers of the feature extraction network are frozen during the domain adaptation process to achieve domain adaptation on shallow features.
[0059] Step 2: Add the shadow image classification task to the shadow detection network; the feature extraction network in the shadow detection network assisted by shadow image classification is the pre-trained model of ResNeXt101 on ImageNet. In order to convert the original ResNeXt101 network from a 1000 classification network to a 2-classification (shadow image and non-shadow image) network, an average pooling layer with a step size of 1×1 and a window size of 7×7 is added after the feature extraction network to obtain a 1×1×2048 feature map, which is used as the input of the 1000 classification fully connected layer of the ResNeXt101 network to obtain a 1000-dimensional vector, and the vector is used as the input of the fully connected layer with an input dimension of 1000 and an output dimension of 2 to achieve image binary classification. Determining whether an image is a shadow image is a binary classification problem, so the shadow image classification task uses binary cross entropy as the loss function, as shown below:
[0060]
[0061] Where y is the true label of the input image. When y is 1, it means that the input image is a shadow image, and when y is 0, it means that the input image is a shadow-free image. Represents the predicted probability of the category to which the input image belongs, and ω is the weighting coefficient of the loss function, which is set to 0.1 in the experiment. Since the shadow image classification task is domain-dependent, the feature extraction network will tend to learn the features in the source dataset. In order to solve this problem, when training the classification task, the parameters of the first to fourth layers of the feature extraction network are frozen.
[0062] Step 3: Combine the methods in step 1 and step 2 to design a suitable training strategy and train the model using the source dataset and the target dataset.
[0063] Step 4: Input the shadow image into the model trained in step 3 to obtain the prediction result of the shadow mask.
[0064] like Figure 3 As shown in Figure 3, when performing domain adaptation training, freezing the parameters of the fourth and fifth layers of the feature extraction network has the best time domain adaptation effect.
[0065] like Figure 4 As shown in Figure 2, the domain adaptation shadow detection method assisted by shadow image classification can detect the shadow area in the image more accurately. The reason for the failure of shadow detection in the snow shadow image in the first column of the second row is that the GTAVS dataset lacks training samples under snowy scenes.
[0066] Freeze the parameters of the first to fourth layers of the feature extraction network when training the classification task.
[0067] In order to verify the effectiveness of the domain adaptation shadow detection method assisted by shadow image classification provided by the example of the present invention, a separate domain adaptation shadow detection experiment and a domain adaptation shadow detection experiment assisted by shadow image classification were conducted; the operating system of the test platform is Ubuntu 20.04.3LTS. The main configuration is: the CPU model is Intel(R) Core(TM) i9-10920X CPU@3.50GHz, the GPU model is GeForce RTX 3090×1, the memory model is DDR4 16GB×8, and pytorch1.8 is used as the deep learning framework.
[0068] In order to quantitatively evaluate the results of shadow detection, the balanced error rate (BER) is used as the evaluation indicator, which is calculated as follows:
[0069]
[0070] In addition, the detection accuracy of the shadow area and the detection accuracy of the non-shadow area are calculated as follows:
[0071]
[0072]
[0073] Where T p ,N p ,T n and N n They represent the number of pixels correctly detected as shadow areas, the number of pixels in shadow areas in the label, the number of pixels correctly detected as non-shadow areas, and the number of pixels in non-shadow areas in the label.
[0074] Two datasets are used in the experiment, namely the GTAVS shadow detection dataset and the SBU shadow detection dataset. In the following description, the GTAVS dataset is called the source domain and the SBU dataset is called the target domain.
[0075] The GTAVS dataset is a dataset collected from open world games using computer rendering technology. The dataset contains 2090 pairs of shadow images and shadow masks in complex scenes.
[0076] Most of the shadow images in the SBU dataset are collected in natural scenes. The dataset contains 4727 samples. 4089 samples are used as training data, and the other 638 samples are used as test data. It should be noted that when training the proposed network model, only the shadow images in the SBU training set are used, and no shadow masks are used.
[0077] In order to verify the effectiveness of the proposed domain adaptation shadow detection method, the BDRAR shadow detection network is first trained using the GTAVS dataset to perform separate shadow detection training. Then the GTAVS dataset is used as the source domain, the SBU dataset is used as the target domain, and the domain adaptation module is added to train the domain adaptation shadow detection network. The test results are shown in Table 1.
[0078] Table 1
[0079]
[0080]
[0081] It can be seen from the above table that through domain adaptation training, the accuracy of the model on the source dataset will decrease, but the accuracy of the model on the target dataset has been greatly improved.
[0082] In order to verify the parameter freezing strategy during domain adaptation training, the parameters of the feature extraction network are frozen during domain adaptation training. The results are as follows:
[0083] Table 2
[0084]
[0085] It can be seen from Table 2 that the best shadow detection effect can be achieved when the parameters of the fourth and fifth layers of the feature extraction network are frozen.
[0086] like Figure 5 As shown in Figure 3, in order to illustrate that the shadow image classification task can be effectively combined with domain adaptation to improve the model accuracy, the GTAVS dataset and the SBU dataset are used to train the domain adaptation shadow detection network assisted by shadow image classification. The test results are compared with other methods, and the results are shown in Table 3:
[0087] Table 3
[0088]
[0089] Although the detection effect of the proposed method is not as good as that of BDRAR and AGPE, it is worth noting that both methods use SBU as training and test sets. The training set of the proposed method is the GTAVS training set, and the domain adaptation training method is adopted, which makes the accuracy of the model better than the first four methods in the table.
[0090] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.
Claims
1. A domain adaptation shadow detection method assisted by shadow image classification, characterized by: The following steps are involved: S1: Adding gradient reversal layer and domain classifier to the shadow detection model to obtain domain adapted shadow detection network; The adding of the gradient reversal layer specifically includes: adding a gradient reversal layer to the fifth layer of the feature extraction network of the shadow detection model, and using the output of the gradient reversal layer as the input of the domain classifier; the domain adaptation shadow detection network training includes two steps: (1) Use the shadow image and shadow label in the source domain to perform shadow detection training. At this time, the output of the network is the prediction result of the shadow area in the image; (2) Use the shadow images in the source domain and the target domain for domain adaptation training. At this time, the output of the network is the prediction result of the input image domain category; The output of the domain adaptation shadow detection network indicates whether the pixel point (s, t) belongs to the shadow area. The loss value at the pixel point (s, t) is calculated using binary cross entropy, as shown in the following formula: Where y represents the label value, Represents the shadow detection prediction result at the pixel point (s, t), The larger the value, the higher the probability that the pixel belongs to the shadow area. The total shadow detection loss is the sum of the loss values of all the points to be detected, as shown in the following formula: Where W and H represent the size of the input image; The output of the domain classifier is a feature map of size 13×13×1. Each feature point on the feature map represents the classification result of the domain classifier for the domain label of the input image. The loss function at each feature point (u, v) can be expressed as: Where y' represents the domain label of the input image, represents the predicted probability of the domain label of the domain classifier at this feature point; the total domain adaptation loss is the sum of the loss values at all feature points, as shown in the following formula: Among them, M and N represent the size of the feature map; S2: Add a shadow image classification task to the domain-adapted shadow detection network to classify the shadow image; the shadow image classification task is to add a fully connected layer to the basic feature extraction network, convert the network into a binary classification network, add a binary classification cross entropy loss function, and realize shadow image classification; the feature extraction network in the shadow detection network assisted by shadow image classification is a pre-trained model of ResNeXt101 on ImageNet; after the feature extraction network, add an average pooling layer with a step size of 1×1 and a window size of 7×7 to obtain a 1×1×2048 feature map, and use the feature map as the input of the ResNeXt101 network 1000 classification fully connected layer to obtain a 1000-dimensional vector, and use the vector as the input of the fully connected layer with an input dimension of 1000 and an output dimension of 2 to realize image binary classification; The shadow image classification task uses binary cross entropy as the loss function as follows: Where y" is the true label of the input image. When y" is 1, it means the input image is a shadow image. When y" is 0, it means the input image is a shadow-free image. Represents the predicted probability of the category to which the input image belongs, and ω is the weighting coefficient of the loss function; S3: Design a training strategy to train the domain-adapted shadow detection network using the source data set and the target data set; the training strategy includes: freezing the parameters of the fourth and fifth layers of the feature extraction network during the domain adaptation process; freezing the parameters of the first to fourth layers of the feature extraction network when training the classification task; using the shadow images and shadow labels in the source domain to train the shadow detection task; and using the shadow images in the source domain and the target domain for domain adaptation training; S4: Input the shadow image into the trained domain-adapted shadow detection network to obtain the prediction result of the shadow mask.