Image Classification Method under Complex Illumination Imaging Based on Deep Learning

Abstract

The present invention relates to the technical field of computer vision and digital image processing, specifically an image classification method under complex illumination imaging based on deep learning, including a collection module and a processing module; the present invention performs segmentation processing on the source image, classifies the segmented images, and after classification, outputs the probability of the corresponding category of the image. The classification network will output richer image information. By performing segmentation processing on the source image, the segmented pictures are input into the classification network to obtain the probability corresponding to each small piece of image. The obtained results are formed into a result matrix and input into the decision network. The decision network learns the probability information about different positions of the source image, effectively avoiding the interference caused by local overexposure. The picture segmentation processing avoids the interference of similar parts between the target and the background, improving the classification accuracy. The small piece picture classification network outputs the probabilities of four categories, effectively avoiding the influence of uneven illumination on picture classification.

Description

Technical Field

[0001] The present invention relates to a classification method, specifically an image classification method under complex illumination imaging based on deep learning, belonging to the technical fields of computer vision and digital image processing. Background Art

[0002] Under the requirements of modern production conditions, in order to ensure the safety of construction, many outdoor construction sites are required to install cameras. This makes camera imaging affected by sunlight and rainy weather, with complex imaging brightness and often overexposure phenomena. In the overexposed part, a large amount of image information is seriously lost, which makes it impossible for the neural network to extract features in part, and the accuracy of traditional detection and classification algorithms is severely affected. Secondly, due to the influence of strong light, there will be a Tyndall effect in the image, which will further blur the image features. Especially in the case of night imaging, this phenomenon is particularly obvious when there is strong light shining on the camera. Traditional classification methods will first perform image enhancement processing when dealing with such data, but the data after image enhancement cannot fill the features of the overexposed part, and since the monitoring is carried out 24 hours a day, the brightness difference between day and night imaging is huge, and ordinary data enhancement methods cannot adapt. Therefore, traditional classification networks are not suitable for such scenarios.

[0003] For the existing methods, Chinese Patent CN202110266753.7 discloses a ship detection and classification method in a complex marine environment. The steps of this method include: training a deep convolutional neural network model through a dataset of ship samples; predicting the category and position information of ships through the trained deep convolutional neural network model; the deep convolutional neural network model includes: a backbone feature extraction network for generating four feature layers of different scales; a multi-scale feature fusion network for increasing the receptive field of small ships; a detection network for generating a prediction picture of the ship, connected to the multi-scale feature fusion network. This patent uses the method of multi-scale feature fusion to detect the target, but lacks the enhancement processing of features, and the accuracy decreases on data with complex lighting conditions;

[0004] Chinese Patent CN202010522217.4 discloses a method for detecting ships at sea based on an improved YOLOV3 algorithm. The steps of this method include: Step 1, designing a network structure for multi-scale feature fusion; Step 2, designing a network structure for feature information interaction; Step 3, optimizing the model loss function; Step 4, dataset balancing processing and prior box clustering; Step 5, model training: the model includes a data preprocessing module and an improved YOLOV3 network structure; Step 6, using the trained model to predict the category and position information of ships for the target image. The disadvantage of this method is that it does not perform feature enhancement for lighting factors, and the network cannot adapt to the detection task under complex lighting.

[0005] Chinese Patent CN201910514333.9 discloses a method for detecting and identifying ships in day and night images. The specific steps of this method are as follows: Step 1, use a light sensor to detect the illuminance of ship images at different times, and divide them into two categories: day images and night images according to different illuminance ranges of the ship images; Step 2, for day images, first detect all objects that appear within the detection range, and then screen out ship-like objects from them; Step 3, for night images, first detect the significant targets in the night images, and screen out ship-like objects from them; Step 4, based on the screened ship-like objects, obtain the real-time positions and category information of all ships in the current video frame. The limitation of this method is that when processing images with different illuminances, the images are manually separated, and the manual processing method is not intelligent enough. The classification criteria for datasets under different lighting conditions are different. When changing the usage scenario of the algorithm, data needs to be reselected, which increases the cost of using the algorithm.

[0006] In view of this, a method for image classification under complex illumination imaging based on deep learning is specifically proposed to help solve the above problems. Summary of the Invention

[0007] The object of the present invention is to provide a method for image classification under complex illumination imaging based on deep learning, which can construct a local image class network adapted to complex illumination scenarios and can construct a decision network based on position information and category probability information to solve the problem of low classification accuracy caused by serious loss or fuzziness of image features under complex illumination conditions.

[0008] The present invention realizes the above object through the following technical solutions. A method for image classification under complex illumination imaging based on deep learning includes a collection module and a processing module. The collection module includes the following steps:

[0009] S1. Use a light sensor to detect the illuminance of the required object images at different times, and classify different illuminance ranges of the required object images;

[0010] S2. Process the images in the first range through the object detection algorithm FAster R-CNN based on a deep convolutional neural network;

[0011] S3. Process the images in the second range through the convolutional neural network algorithm guided by regional covariance;

[0012] S4. Based on the screened required objects, obtain the real-time positions and category information of all the objects in the current video frame;

[0013] S5. Use AUC and MAE to evaluate the detection results,

[0014] The processing module corresponds to the collection module.

[0015] Further, step S1 specifically includes:

[0016] S11. Collect pictures of the same scene at a large number of different time periods, statistically analyze the illuminance ranges of each time period, and create a reference comparison table of illuminance ranges;

[0017] S12. Detect the illuminance of the required object image transmitted by the camera through a light sensor element, compare it with the reference comparison table of illuminance ranges, determine whether the image is an image in the first range or the second range, and classify the images according to the different illuminances of the ship images, which is convenient for subsequent different processing of images in different ranges and improves the classification quality and effect.

[0018] Further, step S2 specifically includes:

[0019] S21. Calculate the first convolutional feature map of the first range to be detected;

[0020] S22. Process the first convolutional feature map;

[0021] S23. Extract features and classify the first convolutional feature map. For the images in the first range, that is, daytime images, use an object detection algorithm based on a deep convolutional neural network for processing, which is convenient for subsequent classification work.

[0022] Further, step S3 uses a convolutional neural network algorithm guided by regional covariance to process the images in the second range, specifically including:

[0023] S31. Extract low-level features of the images in the second range in units of pixels;

[0024] S32. Construct regional covariance based on multi-dimensional feature vectors;

[0025] S33. Construct a convolutional neural network model with the covariance matrix as the training sample;

[0026] S34. Calculate the image saliency based on the principles of local and global contrast;

[0027] S35. Frame the significant ship targets, obtain the ship positions. By using a convolutional neural network algorithm guided by regional covariance to process the images in the second range, that is, night-time images, it is convenient for subsequent classification work and improves the classification effect.

[0028] Further, the processing module includes source image preprocessing, image slicing, classification by a classification network, decision-making by a decision-making network, and output of results, specifically including the following steps:

[0029] A. Source image preprocessing: The size of the source image is 2160*3840. First, the region of interest is cropped according to the needs of the algorithm to obtain an image of 2160*1720, and then it is resized to an image of 1080*860 for subsequent use;

[0030] B. Image slicing: The 1080*860 image is sliced into 357 (21*17) small images of 50*50;

[0031] C. Classification by the classification network and obtaining the classification tensor: The small images are input into the classification network to obtain the classification results of each small image, and the classification results are combined into a 21*17 classification tensor;

[0032] D. Decision-making by the decision network: The 21*17 classification tensor is input into the decision network to obtain the final decision result;

[0033] E. Output result: The decision result is output. Through source image preprocessing, image slicing, classification by the classification network, decision-making by the decision network, and output result, it is convenient to process, classify, and output the source image, making the image information richer when output. These information guide the final decision result to improve the classification effect.

[0034] Furthermore, the collection module collects the processed images as the source images of the processing module. Through the combined use of the collection module and the processing module, it is convenient to process the collected source images, effectively avoiding the interference caused by local overexposure.

[0035] Furthermore, when training the classification network data in step C, it specifically includes:

[0036] C11. Each image has a size of 50*50, and four values are marked, corresponding to the probabilities of the corresponding categories of the image;

[0037] C12. The loss function of the classification network uses the KL divergence loss, and its calculation formula is as follows: The calculation formula of the KL divergence for discrete probability distributions:

[0038]

[0039] By splitting the source image, classifying the sliced images, and after classification, the probability of the corresponding category of the image will be output. Thus, the classification network will output richer image information, and these information will guide the final decision result. The image slicing process avoids the interference of similar parts between the target and the background, improving the classification accuracy.

[0040] Furthermore, when training the decision network data in step D, it specifically includes:

[0041] D11: Each picture has a size of 1080*860, and the label indicates whether there is a ship in the picture;

[0042] D12: When training the decision network, first cut the data into pictures of 50*50, input them into the classification network, obtain 21*17 tensors of 1*4, and splice these tensors into a tensor of 21*17*4 as the input of the decision network;

[0043] D13: Input the tensor of 21*17*4 into the network for training, and use cross-entropy loss as the loss function. Its calculation formula is as follows:

[0044]

[0045] By inputting the sliced pictures into the classification network, the probability corresponding to each small piece of image is obtained. The obtained results are formed into a result matrix and input into the decision network. The decision network learns the probability information about different positions of the source image, effectively avoiding the interference caused by local overexposure. The small-piece picture classification network outputs the probabilities of four categories, effectively avoiding the influence of uneven illumination on picture classification.

[0046] Technical effects and advantages of the present invention:

[0047] 1. The present invention proposes to perform slicing processing on the source image, classify the sliced images, and after classification, the probability of the corresponding category of the image will be output. Thus, the classification network will output richer image information, and this information will guide the final decision result. The picture slicing processing avoids the interference of similar parts between the target and the background, and improves the classification accuracy.

[0048] 2. The present invention proposes to perform slicing processing on the source image, input the sliced pictures into the classification network, obtain the probability corresponding to each small piece of image, form the obtained results into a result matrix, and input it into the decision network. The decision network learns the probability information about different positions of the source image, effectively avoiding the interference caused by local overexposure. The small-piece picture classification network outputs the probabilities of four categories, effectively avoiding the influence of uneven illumination on picture classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a schematic diagram of the basic process of the present invention;

[0050] Figure 2 It is a schematic diagram of the basic process of the collection module in the present invention;

[0051] Figure 3 It is a schematic diagram of the basic process of the processing module in the present invention; DETAILED DESCRIPTION OF THE INVENTION

[0052] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0053] Please refer to Figures 1-3 shown, an image classification method under complex illumination imaging based on deep learning, including a collection module and a processing module. The collection module includes the following steps:

[0054] S1. Use a light sensor to measure the illuminance of the required object images at different time periods, and classify the different illuminance ranges of the required object images;

[0055] S2. Process the images in the first range through the object detection algorithm FAster R-CNN based on a deep convolutional neural network;

[0056] S3. Process the images in the second range through a convolutional neural network algorithm guided by regional covariance;

[0057] S4. Based on the selected required objects, obtain the real-time positions and category information of all objects in the current video frame;

[0058] S5. Use AUC and MAE to evaluate the detection results,

[0059] The processing module corresponds to the collection module.

[0060] Step S1 specifically includes:

[0061] S11. Collect a large number of pictures of the same scene at different time periods, statistically analyze the illuminance ranges of each time period, and make a reference comparison table of illuminance ranges;

[0062] S12. Detect the illuminance of the required object images transmitted by the camera through a light sensor, compare with the reference comparison table of illuminance ranges, and judge whether the image is in the first range or the second range,

[0063] Step S2 specifically includes:

[0064] S21. Calculate the first convolutional feature map of the first range to be detected;

[0065] S22. Process the first convolutional feature map;

[0066] S23. Extract features and classify the first convolutional feature map,

[0067] Step S3 processes the image in the second range using a convolutional neural network algorithm guided by regional covariance, specifically including:

[0068] S31. Extract low-level features of the image in the second range in units of pixels;

[0069] S32. Construct regional covariance based on multi-dimensional feature vectors;

[0070] S33. Construct a convolutional neural network model using the covariance matrix as training samples;

[0071] S34. Calculate image saliency based on local and global contrast principles;

[0072] S35. Frame the significant ship targets to obtain the ship positions. The image in the first range is the daytime image, and the image in the second range is the nighttime image. By classifying the images based on the different illuminations of the hull images and using different processing strategies for the classified daytime and nighttime images respectively, the classification effect is improved.

[0073] The processing module includes source image preprocessing, image slicing, classification by the classification network, decision-making by the decision network, and output of results, specifically including the following steps:

[0074] A. Source image preprocessing: The size of the source image is 2160*3840. First, crop the region of interest according to the needs of the algorithm to obtain a 2160*1720 picture, and then resize it to a 1080*860 picture for subsequent use;

[0075] B. Image slicing: Slice the 1080*860 picture into 357 (21*17) small 50*50 pictures;

[0076] C. Classification by the classification network and obtain the classification tensor: Input the small pictures into the classification network to obtain the classification results of each small picture, and form a 21*17 classification tensor with the classification results;

[0077] D. Decision-making by the decision network: Input the 21*17 classification tensor into the decision network to obtain the final decision result;

[0078] E. Output of results: Output the decision result,

[0079] The collection module collects the processed images as the source images of the processing module.

[0080] When training the data of the classification network in step C, it specifically includes:

[0081] C11. The size of each picture is 50*50, and four values are marked, corresponding to the probabilities of the corresponding categories for the picture;

[0082] In C12, the KL divergence loss is used as the classification network loss function, and its calculation formula is as follows:

[0083] The calculation formula for the KL divergence of discrete probability distributions:

[0084] The source image is segmented, and the segmented images are classified. After classification, the probability of the corresponding class of the image will be output. Thus, the classification network will output richer image information, and this information will guide the final decision result. The image segmentation process avoids the interference of similar parts between the target and the background and improves the classification accuracy.

[0085] When training the decision network data in step D, it specifically includes:

[0086] D11: The size of each image is 1080*860, and the label indicates whether there is a ship in the image;

[0087] D12: When training the decision network, first segment the data into 50*50 images, input them into the classification network, obtain 21*17 tensors of 1*4, and splice these tensors into a 21*17*4 tensor as the input of the decision network;

[0088] D13: Input the 21*17*4 tensor into the network for training, and use the cross-entropy loss as the loss function. Its calculation formula is as follows:

[0089] Input the segmented images into the classification network to obtain the probability corresponding to each small piece of image. The obtained results are formed into a result matrix and input into the decision network. The decision network learns the probability information about different positions of the source image, effectively avoiding the interference caused by local overexposure. The small piece image classification network outputs the probabilities of four classes, effectively avoiding the influence of uneven illumination on image classification.

[0090] First, perform the image collection work. The collection is carried out through the collection module. First, use the photosensitive element to measure the illuminance of the hull images at different times, and classify the different illuminance ranges of the hull images. Collect a large number of pictures of the same scene at different time periods, statistically analyze the illuminance ranges of each time period, and make a reference comparison table of illuminance ranges. Detect the illuminance of the required object images transmitted by the camera through the photosensitive element, and compare it with the reference comparison table of illuminance ranges to determine whether the category of the hull image is an image in the first range or an image in the second range. For the images in the first range, process them through the object detection algorithm FAster R-CNN based on the deep convolutional neural network. For the images in the second range, process them through the convolutional neural network algorithm guided by regional covariance. When processing the images in the first range, calculate the first convolutional feature map of the first range to be detected, process the first convolutional feature map, extract features and classify the first convolutional feature map. Classify the images according to the different illuminances of the hull images, which is convenient for subsequent different processing of images in different ranges, and improves the classification quality and effect. By using the object detection algorithm based on the deep convolutional neural network to process the images in the first range, that is, the daytime images, it is convenient for subsequent classification work. When processing the images in the second range, extract the low-level features of the images in the second range with pixels as units, construct the regional covariance based on multi-dimensional feature vectors, construct a convolutional neural network model with the covariance matrix as the training sample, calculate the image saliency based on the local and global contrast principles, frame the significant ship targets, and obtain the ship positions. By using the convolutional neural network algorithm guided by regional covariance to process the images in the second range, that is, the night images, it is convenient for subsequent classification work and improves the classification effect. Based on the selected required objects, obtain the real-time positions and category information of all the objects in the current video frame, and use AUC and MAE to evaluate the detection results. Take the images collected by the collection module as the source images, and preprocess the source images. The size of the source images is 2160*3840. First, crop the region of interest according to the needs of the algorithm to obtain a 2160*1720 picture, and then resize it into a 1080*860 picture for subsequent use. Then slice the images. Cut the 1080*860 picture into 357 (21*17) small 50*50 pictures, perform classification processing on the sliced images, input the small pictures into the classification network to obtain the classification results of each small picture, and form a 21*17 classification tensor with the classification results. The size of each picture is 50*50, and four values are marked, corresponding to the probabilities of the corresponding categories when corresponding to the pictures. After classification, the probability of the corresponding category of the image will be output. Input the 21*17 classification tensor into the decision network. The size of each picture is 1080*860, and the label indicates whether there is a ship in the picture. When training the decision network, first cut the data into 50*50 pictures, input them into the classification network, and obtain 21*17 tensors of 1*4.And splice these tensors into a tensor of 21*17*4 as the input of the decision network to obtain the final decision result. Thus, the classification network will output richer image information, which will guide the final decision result. The picture is processed by slicing to avoid the interference of similar parts between the target and the background and improve the classification accuracy. The decision result is output, the source image is sliced, and the sliced pictures are input into the classification network to obtain the probabilities corresponding to each small piece of image. The obtained results are combined into a result matrix and input into the decision network. The decision network learns the probability information about different positions of the source image, effectively avoiding the interference caused by local overexposure. The small picture classification network outputs the probabilities of four categories, effectively avoiding the impact of uneven illumination on picture classification.

[0091] For those skilled in the art, it is obvious that the present invention is not limited to the details of the above exemplary embodiments, and without departing from the spirit or basic characteristics of the present invention, the present invention can be implemented in other specific forms. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting. The scope of the present invention is defined by the appended claims rather than the above description. Therefore, all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be embraced within the present invention. Any reference signs in the claims should not be regarded as limiting the claimed claims.

[0092] In addition, it should be understood that although this specification is described according to embodiments, not every embodiment only contains an independent technical solution. This narrative way of the specification is only for clarity. Those skilled in the art should regard the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. An image classification method under complex illumination imaging based on deep learning, comprising a collection module and a processing module, characterized in that: The collection module includes the following steps: S1. Use a light sensor to measure the illuminance of the required object images at different times, and classify different illuminance ranges of the required object images; S2. Process the images in the first range through the object detection algorithm FAster R-CNN based on a deep convolutional neural network; S3. Process the images in the second range through the convolutional neural network algorithm guided by regional covariance; S4. Based on the selected required objects, obtain the real-time positions and category information of all the objects in the current video frame; S5. Use AUC and MAE to evaluate the detection results, The processing module corresponds to the collection module; Step S1 specifically includes: S11. Collect a large number of pictures of the same scene at different times, statistically analyze the illuminance ranges of each time period, and make an illuminance range reference comparison table; S12. Detect the illuminance of the required object images transmitted by the camera through a light sensor, compare with the illuminance range reference comparison table, and determine whether the category of the image is an image in the first range or an image in the second range; The processing module includes source image preprocessing, image slicing, classification by a classification network, decision-making by a decision network, and output of results, and specifically includes the following steps: A. Source image preprocessing: The size of the source image is 2160*3840. First, crop the region of interest according to the needs of the algorithm to obtain a 2160*1720 picture, and then resize it to a 1080*860 picture for subsequent use; B. Image slicing: Slice the 1080*860 picture into 357 (21*17) small pictures of 50*50; C. Classification by a classification network and obtain a classification tensor: Input the small pictures into the classification network to obtain the classification results of each small picture, and form a 21*17 classification tensor with the classification results; D. Decision-making by a decision network: Input the 21*17 classification tensor into the decision network to obtain the final decision result; E. Output results: Output the decision result.

2. The image classification method under complex illumination imaging based on deep learning according to claim 1, wherein: Step S2 specifically includes: S21. Calculate the first convolutional feature map of the first range to be detected; S22. Process the first convolutional feature map; S23. Extract features and classify the first convolutional feature map.

3. The image classification method under complex illumination imaging based on deep learning according to claim 2, wherein: Step S3 uses the convolutional neural network algorithm guided by regional covariance to process the images in the second range, specifically including: S31. Extract low-level features of the images in the second range with pixels as units; S32. Construct regional covariance based on multi-dimensional feature vectors; S33. Construct a convolutional neural network model with the covariance matrix as training samples; S34. Calculate the image saliency based on the local and global contrast principles; S35. Frame the significant ship targets and obtain the ship positions.

4. The image classification method under complex illumination imaging based on deep learning according to claim 3, characterized in that: The collection module collects the processed images as the source images of the processing module.

5. The image classification method under complex illumination imaging based on deep learning according to claim 4, characterized in that: When training data for the classification network in step C, it specifically includes: C11. The size of each picture is 50*50, and four values are marked, corresponding to the probabilities of the corresponding categories for the pictures; C12. The loss function of the classification network uses the KL divergence loss, and its calculation formula is as follows: KL divergence calculation formula for discrete probability distribution:

6. The image classification method under complex illumination imaging based on deep learning according to claim 5, characterized in that: When performing decision network training data in step D, it specifically includes: D11: The size of each picture is 1080*860, and the label indicates whether there is a ship in the picture; D12: When training the decision network, first cut the data into 50*50 pictures, input them into the classification network, obtain 21*17 tensors of 1*4, and splice these tensors into a tensor of 21*17*4 as the input of the decision network; D13: Input the tensor of 21*17*4 into the network for training, and use cross-entropy loss as the loss function. Its calculation formula is as follows:

Images