An unsupervised low-light domain adaptive training method and detection method

Abstract

The present invention discloses an unsupervised low-light domain adaptive training method and detection method. The method comprises the following steps: 1) collecting labeled normal-light training data, unlabeled low-light training data, and a pre-trained model; connecting a multi-layer perceptron after the feature extractor of the pre-trained model to obtain a first model; 2) using normal-light data to train the multi-layer perceptron in the first model; 3) constructing a deep concave curve model and placing it before the feature extractor in the first model to obtain a second model; 4) using low-light data to train the deep concave curve model in the second model; 5) using the deep concave curve model to brighten the low-light data and then input it into the pre-trained model, using the predicted label as a pseudo-label for the low-light data; 6) using normal-light data and low-light data with pseudo-labels to train and fine-tune the pre-trained model; 7) brightening the low-light image to be processed and inputting the fine-tuned pre-trained model to output the corresponding detection result.

Description

Technical Field

[0001] The present invention belongs to the field of digital image low-light enhancement and machine vision, and relates to an unsupervised low-light domain adaptive training method and a detection method based on a deep concave curve. Background Art

[0002] Low light is a common type of image degradation. Insufficient lighting is usually caused by low-light shooting environments, camera failures, incorrect parameter settings, and other reasons. Visual tasks in low-light environments, including object classification, face detection, behavior recognition, and optical flow estimation, have always attracted the attention of academia and industry. Traditional low-light visual task model training requires large-scale labeled training sets, but in low-light environments, data is difficult to label, and the industry already has a large number of normal-light training datasets and pre-trained models. Building new low-light datasets and retraining models will repeatedly consume manpower and material resources. How to make full use of existing labeled normal-light training datasets and normal-light pre-trained models, and train models that can be applied in low-light environments without additional low-light annotations, that is, migrating normal-light pre-trained models to low-light environments through unsupervised domain adaptation methods, has broad practical significance and application value.

[0003] Traditional unsupervised low-light domain adaptation methods can be divided into three categories. Brightening-based methods brighten low-light images, thereby improving the performance of models trained on normal-light images. Feature transfer-based methods use contrastive learning to align the features of normal-light and low-light images, enabling the model to be used in low-light environments. Adversarial learning-based methods generate low-light images using a generative adversarial network and use pseudo-labels to transfer the model to low-light environments.

[0004] However, brightening-based methods ignore the differences between human and machine vision, feature transfer-based methods overlook the importance of pixel-level adjustments, and adversarial learning-based methods require data from multiple domains and ignore the characteristics of the input image itself. Existing unsupervised domain adaptation methods have been ineffective and unable to meet the needs of practical applications. Summary of the Invention

[0005] To address the above issues, the present invention aims to provide an unsupervised low-light adaptive training method and detection method based on deep concave curves. This invention uses a self-supervised training strategy to train a deep concave curve model for brightness enhancement, comprehensively improving the model's performance in low-light environments.

[0006] The technical solution adopted in the present invention is as follows:

[0007] An unsupervised low-light domain adaptive training method, the steps of which include:

[0008] 1) Collecting labeled normal-light training data, unlabeled low-light training data, and a pre-trained model; the pre-trained model is a visual task model trained on the labeled normal-light training data; connecting a multilayer perceptron to the feature extractor of the pre-trained model to obtain a first model; the multilayer perceptron is used to map features extracted by the feature extractor to the representation space of the self-supervised task; in the rotation puzzle-based self-supervised learning adopted in this scheme, the output of the multilayer perceptron is a 30-dimensional vector representing the lexicographical index of the scrambled image in all image block permutation combinations;

[0009] 2) training the first model using the labeled normal illumination training data, locking the parameters of the feature extractor during the training process, and only training the multilayer perceptron;

[0010] 3) constructing a deep concave curve model for predicting the pixel value of each pixel of the input image after brightness enhancement; placing the deep concave curve model before the feature extractor in the first model to obtain a second model;

[0011] 4) training the second model using the low-light training data; during the training process, locking the parameters of the feature extractor and the parameters in the multi-layer perceptron, and only training the deep concave curve model;

[0012] 5) brightening the low-light training data using the trained deep concave curve model and then inputting the brightened data into the pre-trained model to predict labels for the low-light training data; and using the predicted labels as pseudo labels for the low-light training data;

[0013] 6) Using the labeled normal light training data and the pseudo-labeled low light training data, the pre-trained model is trained to obtain a fine-tuned pre-trained model.

[0014] Furthermore, the multilayer perceptron adopts a network structure of "fully connected layer-batch normalization layer-linear rectification function-fully connected layer"; the method of using the normal illumination training data to train the first model is: first, the normal illumination training data is rotated and divided into blocks in sequence to obtain multiple image blocks; then the order of the image blocks is disrupted and input into the feature extractor for feature extraction, and the extracted features are sent to the multilayer perceptron; the multilayer perceptron predicts the order of each image block based on the input feature data; wherein the loss function used to train the first model is L C is the cross entropy loss function, is the dictionary number of the disrupted order of the normal illumination image N in all image block permutation and combination schemes, is the puzzle order predicted by the multilayer perceptron.

[0015] Furthermore, the deep concave curve model sequentially includes a downsampling layer, a U-net network, a convolutional layer, a global pooling layer and a fully connected layer; wherein the downsampling layer is used to downsample the input image and input it into the U-net network, the U-net network is used to extract features of the input data and input it into the convolutional layer, the convolutional layer further extracts features from the input feature data and inputs the extracted features into the global pooling layer and the fully connected layer in sequence to obtain a prediction result.

[0016] Furthermore, the method of training the second model using the low-light training data is as follows: first, the low-light training data is brightened using the deep concave curve model to obtain a brightened image; then, the brightened image is rotated and divided into blocks in sequence to obtain multiple image blocks; then, the order of the image blocks is disrupted and input into the feature extractor for feature extraction, and the extracted features are sent to the multi-layer perceptron; the multi-layer perceptron predicts the order of each image block based on the input feature data; wherein the loss function used for training the second model is L C is the cross entropy loss function, is the dictionary number of the disrupted order of the low-light image L in all image block permutation and combination schemes, is the puzzle order predicted by the multilayer perceptron.

[0017] Furthermore, the deep concave curve model includes two convolutional layers; that is, the deep concave curve model sequentially includes a downsampling layer, a U-net network, a first convolutional layer, a second convolutional layer, a global pooling layer and a fully connected layer.

[0018] Furthermore, for the classification task, the pre-training model adopts ResNet-18; for the face detection task, the pre-training model adopts DSFD; for the behavior recognition task, the pre-training model adopts I3D; for the optical flow estimation task, the pre-training model adopts PWC-Net.

[0019] An unsupervised low-light domain image visual task detection method, the steps of which include:

[0020] 1) Collecting labeled normal-light training data, unlabeled low-light training data, and a pre-trained model; the pre-trained model is a visual task model trained on the labeled normal-light training data; connecting a multilayer perceptron after the feature extractor of the pre-trained model to obtain a first model; the multilayer perceptron adopts a network structure of "fully connected layer-batch normalization layer-linear rectification function-fully connected layer";

[0021] 2) training the first model using the labeled normal illumination training data, locking the parameters of the feature extractor during the training process, and only training the multilayer perceptron;

[0022] 3) constructing a deep concave curve model for predicting the pixel value of each pixel of the input image after brightness enhancement; placing the deep concave curve model before the feature extractor in the first model to obtain a second model;

[0023] 4) training the second model using the low-light training data; during the training process, locking the parameters of the feature extractor and the parameters in the multi-layer perceptron, and only training the deep concave curve model;

[0024] 5) brightening the low-light training data using the trained deep concave curve model and then inputting the brightened data into the pre-trained model to predict labels for the low-light training data; and using the predicted labels as pseudo labels for the low-light training data;

[0025] 6) using the labeled normal light training data and the pseudo-labeled low light training data to train the pre-trained model to obtain a fine-tuned pre-trained model;

[0026] 7) For the low-light image to be processed, it is input into the trained deep concave curve model for brightening and then input into the fine-tuned pre-trained model to output the corresponding visual task detection result.

[0027] A server, characterized in that it includes a memory and a processor, the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program includes instructions for executing each step in the above method.

[0028] A computer-readable storage medium stores a computer program thereon, wherein the computer program implements the steps of the above method when executed by a processor.

[0029] Compared with the prior art, the present invention has the following positive effects:

[0030] The present invention significantly improves the performance of the normal illumination model in low-light environments. On the CODaN low-light classification benchmark test set, the accuracy of the general classification model ResNet-18 can be increased from 60.96% to 63.92%; on the DarkFace low-light face detection benchmark test set, the mean average precision of the general face detector Dual Shot Face Detector can be increased from 44.44 to 46.91; on the low-light behavior recognition benchmark test set ARID, the recognition accuracy can be increased from 50.18% to 52.13%; on the low-light optical flow estimation benchmark test machine VBOF, the end-point error can be reduced from 8.99 to 7.44. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is the structural diagram of the deep concave curve model.

[0032] Figure 2 This is the training flow chart of the deep concave curve model.

[0033] Figure 3 Flowchart for migrating pre-trained models to low-light domain. DETAILED DESCRIPTION

[0034] In order to make the above features and advantages of the present invention more clearly understood, embodiments are given below and described in detail with reference to the accompanying drawings.

[0035] This embodiment discloses an unsupervised low-light domain adaptation method applied to low-light classification tasks, which is described in detail as follows:

[0036] Step 1: Collect annotated normal lighting images to form a training dataset {X N , Y N}; Collect low-light images to form a low-light training dataset {X L}. Among them, the sample X in the normal lighting training dataset N Must contain category information Y N , the samples in the low-light training dataset do not need to contain category information. Obtain a pre-trained model on normal-light images that includes a feature extractor. The pre-trained model uses the residual convolutional network ResNet-18, but other pre-trained models are also acceptable. For classification tasks, the pre-trained model uses ResNet-18; for face detection, the pre-trained model uses DSFD; for action recognition, the pre-trained model uses I3D; and for optical flow estimation, the pre-trained model uses PWC-Net.

[0037] Step 2: Build and train a multilayer perceptron. The multilayer perceptron uses a network structure of "fully connected layer - batch normalization layer - linear rectifier function - fully connected layer". Fix the parameters of the feature extractor obtained in step 1, connect the multilayer perceptron to the feature extractor, and use the self-supervised training method to train the normal illumination dataset {X L Train a multi-layer perceptron. The self-supervised training method can use the rotation puzzle strategy, which involves first rotating the image, then dividing it into nine 3×3 blocks, shuffling the blocks, and training the multi-layer perceptron to restore the original order. The loss function for this step is:

[0038]

[0039] Where, L C is the cross entropy loss function, is the dictionary number of the disrupted order of the normal illumination image N in all image block permutation and combination schemes, is the puzzle order predicted by the multilayer perceptron. The training batch size is 64, and the learning rate is used for 150,000 iterations, followed by 150,000 iterations with a learning rate of 0.001.

[0040] Step 3: Construct a deep concave curve model, which takes an unlabeled low-light image as input to predict the mapping g. g is a mapping from the pixel value of the original image to the pixel value of the new image. For example, for an 8-bit grayscale image, since there are 256 values ​​in the color domain, g is a 256-dimensional vector. The output of the deep concave curve model is the inverse of the discrete second-order derivative of g before normalization, which is a 255-dimensional vector. Based on this output, g can be integrated and normalized to obtain g. For an 8-bit three-channel color image, the deep concave curve model predicts g corresponding to the three channels separately, that is, the output is a 765-dimensional vector. The last layer of the deep concave curve model is a rectified linear function to ensure that the output is a non-negative value, thereby ensuring that g is a concave curve. The detailed structure of the deep concave curve model is shown in the attached figure. Figure 1 As shown in the figure, the network consists of a downsampling layer, a U-net network, two 3×3 convolutional layers, a global pooling layer, and a fully connected layer. The downsampling layer reduces the resolution of the input image to 16×16. The U-net network uses the output of the downsampling layer as input to extract features, with the output being of the same size as the input. The two convolutional layers use the output of the U-net network as input to further extract features. The global pooling layer and the fully connected layer use the output of the convolutional layer as input and output the prediction result of the deep concave curve model, namely a 765-dimensional vector.

[0041] Step 4: Train the deep concave curve model. The flowchart is as shown in the attached figure. Figure 2In this step, the parameters of the feature extractor and multi-layer perceptron remain unchanged, and only the deep concave curve model is trained. The training uses the low-light dataset {X L}, using a self-supervised paradigm. The self-supervised training method can use a rotation puzzle strategy, which involves first rotating the image, then dividing the image into nine 3×3 blocks, shuffling the blocks, and training the model to restore the original order. The loss function for this step is:

[0042]

[0043] Where, L C is the cross entropy loss function, is the dictionary number of the disrupted order of the low-light image L in all image block permutation and combination schemes, is the puzzle order predicted by the multilayer perceptron. The training batch size is 64, the initial learning rate is set to 0.01, and a total of 20,000 iterations are performed. The learning rate is decayed by a rate of 0.1 after the 5,000th and 10,000th iterations.

[0044] Step 5: Obtain pseudo labels for low-light training data. In this step, firstly transform the low-light image dataset {X L} Input the deep concave curve prediction model to obtain the brightened low-light image dataset {E(X L )}, and then the low-light image dataset {E(X L )} Input the pre-trained model obtained in step 1 to predict the label Among the obtained labels, labels with confidence lower than 0.98 will be discarded. This step obtains a low-light dataset containing pseudo labels.

[0045] Step 6: Use the normal light dataset with labels collected in step 1 and the low light dataset with pseudo labels obtained in step 6 to migrate the pre-trained model to the low light domain. The specific flow chart is as follows Figure 3 The pre-trained model was trained using the cross-entropy loss function with a batch size of 64 for 7000 iterations. The initial learning rate was 0.001, which was decayed by a rate of 0.1 after the 2000th, 4000th, and 6000th iterations. The training used the SGD optimizer with a momentum of 0.9 and a weight decay of 0.00001. Data augmentation methods used in training included random crops, horizontal flips, color jittering, and random rotations.

[0046] Step 7: In the inference phase, for the low-light image to be classified, first use the deep concave curve prediction obtained in step 4 to brighten it, then input the low-light classification model trained in step 6 (that is, the fine-tuned pre-trained model) to obtain the prediction result, which is a vector with a size equal to the number of classification categories in the dataset.

[0047] For the face detection task, the pre-trained model uses the dual detection face recognizer DSFD. For the low-light image to be detected, the deep concave curve prediction obtained in step 4 is first used for brightening, and then the DSFD trained and fine-tuned in step 6 is input to obtain the predicted face detection frame coordinates.

[0048] For the action recognition task, the pre-trained model uses a two-stream expanded 3D convolutional network (I3D). For low-light images to be detected, the deep concave curve predictions obtained in step 4 are first used for brightening. Then, the I3D trained and fine-tuned in step 6 is input to obtain the action recognition prediction results for each frame of the video, which is a vector with a size equal to the number of action types in the dataset.

[0049] For the optical flow estimation task, the pre-trained model uses the pyramid-deformation-stereo matching optical flow estimation network PWC-Net. For the low-light image to be detected, the deep concave curve prediction obtained in step 4 is first used for brightening. Then, the fine-tuned PWC-Net trained in step 6 is input to obtain the position offset of each pixel in the image at the next moment.

[0050] The above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the same. Those skilled in the art may modify or make equivalent substitutions for the technical solutions of the present invention without departing from the spirit and scope of the present invention. The scope of protection of the present invention shall be based on the claims.

Claims

1. An unsupervised low-light domain adaptive training method, comprising the following steps: 1) Collecting labeled normal-light training data, unlabeled low-light training data, and a pre-trained model; the pre-trained model is a visual task model trained on the labeled normal-light training data; connecting a multi-layer perceptron after the feature extractor of the pre-trained model to obtain a first model; the multi-layer perceptron is used to map features extracted by the feature extractor to the representation space of the self-supervised task; 2) training the first model using the labeled normal illumination training data, locking the parameters of the feature extractor during the training process, and only training the multilayer perceptron; 3) constructing a deep concave curve model for predicting the pixel value of each pixel of the input image after brightness enhancement; placing the deep concave curve model before the feature extractor in the first model to obtain a second model; 4) training the second model using the low-light training data; During the training process, the parameters of the feature extractor and the parameters in the multilayer perceptron are locked, and only the deep concave curve model is trained; 5) brightening the low-light training data using the trained deep concave curve model and then inputting the brightened data into the pre-trained model to predict labels for the low-light training data; and using the predicted labels as pseudo labels for the low-light training data; 6) Using the labeled normal light training data and the pseudo-labeled low light training data, the pre-trained model is trained to obtain a fine-tuned pre-trained model.

2. The method according to claim 1, characterized in that The multilayer perceptron adopts a network structure of "fully connected layer-batch normalization layer-linear rectification function-fully connected layer"; the method of using the normal illumination training data to train the first model is: first, the normal illumination training data is rotated and divided into blocks in sequence to obtain multiple image blocks; then the order of the image blocks is disrupted and input into the feature extractor for feature extraction, and the extracted features are sent to the multilayer perceptron; the multilayer perceptron predicts the order of each image block based on the input feature data; wherein, the loss function used in training the first model is L C is the cross entropy loss function, is the dictionary number of the disrupted order of the normal illumination image N in all image block permutation and combination schemes, is the puzzle order predicted by the multilayer perceptron.

3. The method according to claim 1 or 2, characterized in that The deep concave curve model sequentially includes a downsampling layer, a U-net network, a convolutional layer, a global pooling layer and a fully connected layer; wherein the downsampling layer is used to downsample the input image and input it into the U-net network, the U-net network is used to extract features from the input data and input it into the convolutional layer, the convolutional layer further extracts features from the input feature data and sequentially inputs the extracted features into the global pooling layer and the fully connected layer to obtain a prediction result.

4. The method according to claim 3, characterized in that The method for training the second model using the low-light training data is as follows: first, the low-light training data is brightened using the deep concave curve model to obtain a brightened image; then, the brightened image is rotated and divided into blocks in sequence to obtain multiple image blocks; then, the order of the image blocks is disrupted and input into the feature extractor for feature extraction, and the extracted features are sent to the multi-layer perceptron; the multi-layer perceptron predicts the order of each image block based on the input feature data; wherein the loss function used for training the second model is L C is the cross entropy loss function, is the dictionary number of the disrupted order of the low-light image L in all image block permutation and combination schemes, is the puzzle order predicted by the multilayer perceptron.

5. The method according to claim 3, characterized in that The deep concave curve model includes two convolutional layers; that is, the deep concave curve model sequentially includes a downsampling layer, a U-net network, a first convolutional layer, a second convolutional layer, a global pooling layer and a fully connected layer.

6. The method according to claim 1, characterized in that For the classification task, the pre-trained model uses ResNet-18; for the face detection task, the pre-trained model uses DSFD; for the behavior recognition task, the pre-trained model uses I3D; for the optical flow estimation task, the pre-trained model uses PWC-Net.

7. An unsupervised low-light domain image visual task detection method, comprising the following steps: 1) Collecting labeled normal-light training data, unlabeled low-light training data, and a pre-trained model; the pre-trained model is a visual task model trained on the labeled normal-light training data; connecting a multilayer perceptron after the feature extractor of the pre-trained model to obtain a first model; the multilayer perceptron adopts a network structure of "fully connected layer-batch normalization layer-linear rectifier function-fully connected layer"; 2) training the first model using the labeled normal illumination training data, locking the parameters of the feature extractor during the training process, and only training the multilayer perceptron; 3) constructing a deep concave curve model for predicting the pixel value of each pixel of the input image after brightness enhancement; placing the deep concave curve model before the feature extractor in the first model to obtain a second model; 4) training the second model using the low-light training data; During the training process, the parameters of the feature extractor and the parameters in the multilayer perceptron are locked, and only the deep concave curve model is trained; 5) brightening the low-light training data using the trained deep concave curve model and then inputting the brightened data into the pre-trained model to predict labels for the low-light training data; and using the predicted labels as pseudo labels for the low-light training data; 6) using the labeled normal light training data and the pseudo-labeled low light training data to train the pre-trained model to obtain a fine-tuned pre-trained model; 7) For the low-light image to be processed, it is input into the trained deep concave curve model for brightening and then input into the fine-tuned pre-trained model to output the corresponding visual task detection result.

8. A server, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program includes instructions for executing each step of the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images