A low-illumination image description method based on RDE-Net

Abstract

The application discloses a low-illumination image description method based on RDE-Net, and the method comprises the following steps: 1) processing and dividing an image description data set; 2) constructing an FEX network to extract global features, regional features and detail features from a feature map; 3) constructing an ENH network to strengthen the regional features by using the detail features and to enhance the perception ability of the regional features to details; and 4) constructing a two-stage prediction module and a CC module to jointly describe an image. In the method, the regional features strengthened by details and object category words generated by a low-illumination target detector are utilized, so that similar objects can be well distinguished under low illumination, and confusion is avoided.

Description

TECHNICAL FIELD

[0001] The present application relates to deep learning, low-illumination image processing and image description technology, and in particular to a low-illumination image description method based on RDE-Net. BACKGROUND

[0002] Image description is one of the important tasks combining computer vision and natural language processing, two artificial intelligence fields, and has broad application prospects. In the image description scene, illumination is an important factor affecting the accuracy of description.

[0003] Low-illumination images refer to pictures taken at night or in poor lighting conditions. Insufficient light can seriously damage and reduce the visual quality of images. Describing images with low visibility, poor contrast and color deviation not only has certain difficulty for humans, but also is a great challenge for computers. Many existing image description models often work normally under sufficient lighting conditions, but when processing low-illumination images, the accuracy is not satisfactory. In order to solve the problem of describing images taken under low illumination, an effective low-illumination image description method is urgently needed. SUMMARY

[0004] The purpose of the present application is to overcome the shortcomings of the prior art and provide a low-illumination image description method based on RDE-Net. This method can reduce the description errors caused by insufficient features in low-illumination pictures and achieve accurate description of low-illumination images.

[0005] The technical solution to achieve the purpose of the present application is:

[0006] A low-illumination image description method based on RDE-Net, comprising the following steps:

[0007] 1) Process and divide the image description data set: the process is:

[0008] 1-1) Use the COCO data set, which contains 123287 images. Divide this data set into two parts in the ratio of 8:2, of which 80% is the training set, a total of 98630 pictures, and 20% is the test set, a total of 25057 pictures. The collection and division are shown in Table 1, Table 1:

[0009]

[0010] 1-2) Use the existing CycleGAN model to process the pictures in the data set: reduce the light of the pictures, convert the normal-illumination pictures in the data set to low-illumination pictures, and then uniformly scale the image size to 256*256;

[0011] 1-3) The existing IA-YOLO model is used to perform object detection on low-light images to obtain the category and corresponding location of objects in each image. The category of an object is represented by a category word, i.e., the object category text information, and the location is represented by the bounding box coordinates. IA-YOLO is a model based on YOLOv3 that can perform low-light object detection and has a good detection effect on low-light images.

[0012] 2) Constructing a FEX network to extract global features, regional features, and detail features from the feature map: The process is as follows:

[0013] 2-1) The FEX network sets up mapping operations for three feature maps and calls three modules. The bounding boxes obtained in steps 1-3) are scaled down proportionally to obtain their corresponding positions in the three feature maps. The three feature maps are the three feature maps fed into the detection head in IA-YOLO, denoted as C3, C4, and C5 respectively. Then, for each bounding box, the feature map used to generate its coordinates is selected, and the region feature sub-map is obtained by mapping. Subsequently, the RFE module, DFE module, and GEX module are called. Based on the object detection results, one of the three feature maps is selected for mapping the region feature sub-map. The reason for this is that bounding boxes have different sizes and feature maps have different depths. Shallow feature maps are used to map bounding boxes with smaller areas, and deep feature maps are used to map feature maps with larger areas, effectively completing the bottom-up focus of attention on the object in the image.

[0014] 2-2) Input the region feature sub-map into the RFE and DFE modules, and input C3, C4, and C5 into the GEX module. The RFE and DFE modules extract region feature vectors and detail feature vectors from the region feature sub-map, and the GEX module extracts global feature vectors from the feature map. The outputs of the GEX, RFE, and DFE modules are all 512-dimensional feature vectors. The purpose of using three modules to extract feature vectors is to extract features from the object, details, and background of the image respectively. Each of the three features has its own advantages and disadvantages. By using them reasonably, their advantages can complement each other and improve the accuracy of the description.

[0015] 3) Constructing an ENH network to enhance regional features based on detailed features: The process is as follows:

[0016] 3-1) In the ENH network, the SIMCAL module and the RFEN module are set. For the input of the network, the SIMCAL module is called to calculate the weight first, and then the RFEN module is called for feature enhancement. When the SIMCAL module is called, the region features and the detail features are sent into the SIMCAL module for weight calculation, and the output is the effective weight and the similarity weight. Each detail feature corresponds to an effective weight, and the effective weight reflects how much detail information is contained in the detail feature. Each feature vector corresponds to a similarity weight, and the similarity weight reflects the size of the role played by each feature vector in the strengthening process;

[0017] 3-2) The region features, the detail features and the output of the SIMCAL module are sent into the RFEN module together. The perception ability of the region features to the details is enhanced by the detail features. Due to the noise that exists universally in the low-illumination environment, the details of the objects in the picture become difficult to observe. Therefore, if the feature extraction method under normal illumination is still used under low illumination, it will not be able to produce an accurate description due to insufficient extracted features. Therefore, the object features of the region as a whole are strengthened by the detail features that reflect the detail information of each part of the region. After strengthening, the details of each part in the region can be perceived more clearly, so that the possibility of misjudgment in the description process can be reduced, thereby improving the accuracy of the description;

[0018] 4) A two-stage prediction module and a CC module are constructed to describe the image together. The process is as follows:

[0019] 4-1) The region features obtained in step 3-2) and the global features obtained in step 2-2) are used for the first stage description, that is, they are sent into a Transformer structure together, which is denoted as Tr-s1. The Tr-s1 is provided with an encoder and a decoder. Three encoding layers and three decoding layers are respectively arranged in the encoder and the decoder. The output of the Tr-s1 is the global feature vector generated by the self-attention operation of the last encoding layer and the calculation result generated by the last decoding layer, which are denoted as F g and F h , respectively. The region features and the global features are sent into the encoder of the Tr-s1 for feature fusion. On the one hand, it is necessary to have a certain understanding of the global and local features in the decoding process to increase the accuracy of the decoding. On the other hand, the self-attention mechanism arranged in the encoder is used to make the global features and the region features fully communicate, so as to prepare for the prediction of the second stage;

[0020] 4-2) A linear layer is arranged to receive the F h obtained in step 4-1), and then a softmax function is arranged. The output is the probability of each word in the word table. The word with the maximum probability is selected at each time step as the result. The cross-entropy loss is used as the loss function, which is denoted as where y1:c-1 represents the first c-1 words that have been predicted, y c represents the cth word to be predicted, P v represents the probability;

[0021] 4-3) Encode the object category text information obtained in step 1-3), each word corresponds to a 512-dimensional word vector, and then form a word matrix E according to the word vectors, each column of E represents a word vector of a word;

[0022] 4-4) The second stage description also adopts a Transformer structure, denoted as Tr-s2, and the word matrix E obtained in step 4-3) and the first stage calculation result F h is input into Tr-s2, and after encoding and decoding, the second stage calculation result is obtained, and the second stage calculation result is input into the linear layer trained in step 4-2), and the result is input into the softmax function to obtain the word probability predicted by the second stage, denoted as F th The second stage prediction is to adjust the sentence generated based solely on image features using category text information. To avoid deterioration of the adjusted results, the classification head trained in the first stage is used to process the output of the second stage decoder. The category text information is used because low-light target detection can produce relatively accurate detection results, and the results of target detection often appear in the sentences generated by image description. By utilizing the results of target detection, the possibility of misjudgment by the image description model for low-light images can be reduced.

[0023] 4-5) A CC module is set after each decoding layer in the decoder in Tr-2. The CC module is used to predict the copy probability of each word in the category text. In the prediction process, the output of each decoding layer and the global feature vector are used. Since a CC module needs to be set after each of the three decoding layers, three sets of copy probabilities will be obtained. The average of the three sets of probabilities is taken as the final copy probability. The copy probability is the probability that the category word is directly used as the prediction result of the word at the current time step. In low-light conditions, insufficient features may cause the model to mistake many similar objects for the same object. However, a low-light target detection model has good recognition ability for the categories of objects in a picture. Therefore, by calculating the object category words generated by the low-light target detection model into copy probabilities, the ability to distinguish similar objects in the low-light image description task can be improved, and the description accuracy can be improved.

[0024] 4-6) Process the results of the CC module and the second described results to generate the final prediction results of the current time step, that is, splice the second stage word prediction probability obtained in step 4-4) and the word copy probability obtained in step 4-5), and then select the highest probability word from them as the prediction result of the current time step, and use cross-entropy loss as the loss function, denoted as where y 1:c-1 represents the first c-1 words that have been predicted, y c represents the cth word to be predicted, P tg represents the probability;

[0025] 4-7) In the training process of the overall model, the final loss function for training is the sum of the two-stage loss functions, and the formula of the loss function is shown in (1):

[0026]

[0027] where y 1:c-1 represents the first c-1 words that have been predicted, y c represents the cth word to be predicted, P v represents the probability, P tg represents the probability.

[0028] The GEX module, the RFE module, and the DFE module described in step 2-2) are respectively:

[0029] The GEX module is used for global feature extraction, that is, C3, C4 and C5 are sent into the GEX module, and the three feature maps sequentially pass through Rolpool layer, maximum pooling layer and 8*8 convolution layer in the module, and are converted into three 512-dimensional global feature vectors as the output of the module. In the process of describing the picture, the scene where the object is located is an important information. The scene information often appears in the finally generated sentence, and the information can also be used to make certain reasoning on the relationship between the objects in the picture. In low light, due to insufficient features in the picture, the method of relying on the communication between regional features to obtain scene information in normal light description is no longer applicable. In order to solve the problem of insufficient scene information in low light description, the module is set to convert the whole feature map into a feature vector. The feature vector extracted in this way has a field of view covering the whole picture, that is, a global feature vector. In order to obtain the global feature, RolPool layer is set to make reasoning based on the existing information, and supplement the features lost due to low light environment. After passing through the layer, the width and height of the feature map will be 1.25 times of the original. The subsequent maximum pooling layer converts the feature map into 8*8 size. This process is to reduce the interference information that may exist in the feature map. Finally, an 8*8 convolution layer is used to convert the feature map into a feature vector. 512 convolution kernels are set in the convolution layer, so the output of the module is a 512-dimensional global feature vector.

[0030] The DFE module is used for detail feature extraction, which aims to extract the features of each part of the region where the object is located, that is, for each region feature sub-map mapped from the object region, 4 small rectangles are uniformly divided on it. Each small rectangle is mapped again as a detail feature sub-map. The detail feature sub-map is sent to Rolpool layer, maximum pooling layer and 4*4 convolution layer to obtain a 512-dimensional detail feature vector. Since the details of the object are not easy to observe under low light, the division of small rectangles is used to supplement the details. Each small rectangle only covers part of the object, so the features extracted therefrom must pay little attention to the category information of the object, but more to the perception of details, reflecting the detail information of this part, so it is called detail feature. In order to better extract the detail feature, the detail feature sub-map is sent to RolPool layer to convert it into the original size of the feature map to which it belongs. In this process, the missing features in the low light image are supplemented by using the existing information. The subsequent maximum pooling layer unifies the detail feature sub-map into 4*4 size, which reduces the interference information that may be contained therein in this process. Finally, a 4*4 convolution layer is used to convert the feature map into a feature vector. 512 convolution kernels are set in the convolution layer, so the output of the module is a 512-dimensional global feature vector.

[0031] The RFE module is used for extracting regional features. In this process, the regional feature subgraph is sequentially sent into a Rolpool layer, a maximum pooling layer and an 8*8 convolution layer, and finally a 512-dimensional regional feature vector is obtained. In this process, the regional feature subgraph is sequentially sent into a Rolpool layer, a maximum pooling layer and an 8*8 convolution layer. In the Rolpool layer, the regional feature subgraph is converted into the original size of the feature map, and the feature information that may be lost in the low-illumination image is supplemented. In the maximum pooling layer, all regional feature subgraphs are converted into 8*8 size, and the interference information is reduced. In the 8*8 convolution layer, the regional feature subgraph is converted into a regional feature vector by using a convolution layer provided with 512 convolution kernels.

[0032] The SIMCAL module described in step 3-1) is:

[0033] 3-1-1) The regional features are respectively sent into two different linear layers to obtain two 64-dimensional vectors, which are respectively used as query and key. A new linear layer is arranged to receive the detail features. After the linear layer, each detail feature corresponds to a 64-dimensional vector, which is used as key. Two different linear layers are used to process the regional features and the detail features, so that the two types of features still have their respective characteristics after processing, that is, the former focuses on the category, and the latter focuses on the details.

[0034] 3-1-2) The query and each key are calculated by the dot product to calculate the similarity, and then the obtained similarity is sent into a sigmoid layer to calculate the similarity weight.

[0035] 3-1-3) The detail features are sent into a linear layer to be converted into 64-dimensional vectors, and then the elements in the vectors are summed to be used as the effective score of the detail features. Finally, the effective scores of all the detail features are sent into a sigmoid layer to calculate the effective weight. Since the detail information of the object in the low-illumination image is insufficient, it cannot be guaranteed that the detail features extracted from each small rectangle can perceive sufficient detail information. Therefore, the effective weight of each detail feature is calculated. The vector carrying more detail information has a higher weight, and the vector carrying less detail information has a lower weight.

[0036] The RFEN module described in step 3-2) is:

[0037] 3-2-1) The regional features and the detail features are sent into a linear layer. After the linear layer, each feature corresponds to a 64-dimensional vector, which is denoted as value. For the value corresponding to the detail feature, the value is multiplied by the effective weight received from the SIMCAL module.

[0038] 3-2-2) According to the similarity weight received from the SIMCAL module, the value is weighted and summed, and the sum is taken as the enhanced regional feature vector, which is the output of the module. The high and low of the effective weight reflects how much detail information is contained in the vector. The value is multiplied by the effective weight first, and then weighted and summed using the similarity weight, which ensures the effectiveness of the detailed vector and avoids the interference of the less detailed vector.

[0039] The CC module described in step 4-5) is:

[0040] 4-5-1) F is obtained from step 4-1) g As the global feature after fusion, the global feature contains two parts, key and value, denoted as and

[0041] 4-5-2) A m is taken from the output of the decoding layer, and is taken from the output of the decoding layer, and is taken from the output of the decoding layer, and th is taken from the output of the decoding layer, and thg is taken from the output of the decoding layer, and

[0042] 4-5-3) F thg is taken into the linear layer, and the output is denoted as F k thg , and the word vector of each word is taken into the linear layer and multiplied by F thg , and the result is the score of the word. By calculating the score of the word, the use of the object class text detected in the picture is completed. According to the object class text to assist in the generation of low-light image description, the possibility of misjudgment of the low-light image description model can be reduced.

[0043] 4-5-4) The calculation formula of each word score is as follows:

[0044]

[0045] The technical scheme adopts a low-illumination target detection model to extract feature information of a low-illumination image, extracts regional features, detail features and global features on a feature map, uses the detail features to enhance the regional features, strengthens the perception ability of the regional features to object details, uses multiple global features to focus on background information of the picture, adopts a two-stage manner to generate a prediction probability of a word in a word table and a copy probability of a category word, uses the regional features with strengthened detail perception ability and the category word obtained through target detection, reduces the possibility of misjudgment of the model to an object in a low-illumination image, and thus improves the accuracy of description.

[0046] The method can reduce the description error problem caused by insufficient features in a low-illumination picture, and can realize accurate description of a low-illumination image. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 It is a schematic diagram of the RDE-Net network model in the embodiment.

[0048] Figure 2 It is a structural schematic diagram of the FEX in the embodiment.

[0049] Figure 3 It is a structural schematic diagram of the RFE in the embodiment.

[0050] Figure 4 It is a schematic diagram of the DFE in the embodiment.

[0051] Figure 5 It is a structural schematic diagram of the GEX in the embodiment.

[0052] Figure 6 It is a structural schematic diagram of the ENH in the embodiment.

[0053] Figure 7 It is a schematic diagram of the second-stage description in the embodiment.

[0054] Figure 8 It is a structural schematic diagram of the CC in the embodiment. DETAILED DESCRIPTION

[0055] The content of the present application will be further described below in combination with the drawings and embodiments, but is not limited to the present application.

[0056] Embodiment:

[0057] Reference Figure 1 A low-illumination image description method based on RDE-Net, comprising the following steps:

[0058] 1) Processing and dividing the image description data set: the process is:

[0059] 1-1) Using COCO dataset, the COCO dataset contains a total of 123287 images, divide this dataset into two parts in the ratio of 8:2, 80% of which is the training set, a total of 98630 pictures, 20% of which is the test set, a total of 25057 pictures, the collection and division are shown in Table 1, Table 1:

[0060]

[0061] 1-2) Using the existing CycleGAN model to process the pictures in the dataset: reducing the illumination of the pictures, converting the normal illumination pictures in the dataset into low-illumination pictures, and then uniformly scaling the image size to 256*256;

[0062] 1-3) Using the existing IA-YOLO model to detect the objects in the low-illumination pictures, obtaining the class and corresponding position of the objects in each picture, the class of the object is represented as a class word, that is, the object class text information, and the position is represented as the bounding box coordinates. IA-YOLO is a model based on YOLOv3 that can perform low-illumination object detection and has good detection effect on low-illumination pictures;

[0063] 2) As shown in Figure 2 , the FEX network is constructed to extract global features, regional features and detail features from the feature maps: the process is as follows:

[0064] 2-1) The FEX network is set to have mapping operations for three feature maps and the calling of three modules. The bounding boxes obtained in step 1-3) are reduced in size, and the corresponding positions in the three feature maps are obtained. The three feature maps are the three feature maps sent into the detection head in IA-YOLO, which are denoted as C3, C4 and C5. Then, for each bounding box, the feature map used to generate its coordinates is selected, and the regional feature subgraph is obtained using the mapping method. Then, the RFE module, the DFE module and the GEX module are called. Based on the results of object detection, one of the three feature maps is selected for the mapping of the regional feature subgraph. The reason is that the bounding boxes have different sizes, and the feature maps have different depths. The shallow feature map is used to map the small area bounding box, and the deep feature map is used to map the large area feature map, effectively completing the bottom-up attention to the objects in the picture;

[0065] 2-2) Input the regional feature subgraph into the RFE module and the DFE module, and input C3, C4 and C5 into the GEX module. The RFE module and the DFE module extract the regional feature vector and the detail feature vector from the regional feature subgraph, and the GEX module extracts the global feature vector from the feature map, as shown in Figure 3 、 Figure 4 、 Figure 5As shown, the outputs of the GEX module, the RFE module, and the DFE module are all 512-dimensional feature vectors. The three modules are used to extract features from the object, the details, and the background in the image, respectively. Each of the three features has its own advantages and disadvantages. By using them properly, their advantages can be complementary, and the accuracy of the description can be improved.

[0066] 3) As shown in Figure 6 , the ENH network is constructed to enhance the region features based on the detail features. The process is as follows:

[0067] 3-1) In the ENH network, the SIMCAL module and the RFEN module are set. For the input of the network, the SIMCAL module is called first to calculate the weight, and then the RFEN module is called for feature enhancement. When the SIMCAL module is called, the region features and the detail features are sent to the SIMCAL module for weight calculation. The output is the effective weight and the similarity weight. Each detail feature corresponds to an effective weight, and the effective weight reflects the amount of detail information contained in the detail feature. Each feature vector corresponds to a similarity weight, and the similarity weight reflects the role of each feature vector in the enhancement process.

[0068] 3-2) The region features, the detail features, and the output of the SIMCAL module are sent to the RFEN module together. The region features are enhanced based on the detail features to improve the perception of details. In low-light environments, noise is common, and the details of the objects in the image are not easy to observe. Therefore, if the feature extraction method used in normal lighting is still used in low-light environments, the extracted features may not be sufficient to produce an accurate description. To solve this problem, the detail features that reflect the detail information of each part of the region are used to enhance the object features that reflect the whole region. After enhancement, the details of each part in the region can be perceived more clearly, which can reduce the possibility of misjudgment in the description process and improve the accuracy of the description.

[0069] 4) As shown in Figure 7 , Figure 8 , the two-stage prediction module and the CC module are constructed to describe the image together. The process is as follows:

[0070] 4-1) The region features obtained in step 3-2) and the global features obtained in step 2-2) are used for the first-stage description. That is, they are sent to a Transformer structure together, which is denoted as Tr-s1. Tr-s1 has an encoder and a decoder. The encoder and the decoder have three encoding layers and three decoding layers, respectively. The output of Tr-s1 is the global feature vector generated by the last encoding layer after self-attention operation and the calculation result generated by the last decoding layer, which are denoted as F g and F hThe regional features and the global features are jointly fed into the encoder of Tr-s1 for feature fusion, so that the decoder can have a certain understanding of the global and local features, thereby increasing the decoding accuracy, and the self-attention mechanism in the encoder is used to make the global features and the regional features fully communicate and prepare for the prediction in the second stage;

[0071] 4-2) A linear layer is arranged to receive the F obtained in step 4-1), and a softmax function is arranged after the linear layer, and the output is the probability of each word in the word table, and the word with the maximum probability is selected as the result at each time step, and the cross-entropy loss is used as the loss function, denoted as h Where y 1:c-1 represents the first c-1 predicted words, y c represents the cth word to be predicted, and P v represents the probability.

[0072] 4-3) The object category text information obtained in step 1-3) is encoded, and each word corresponds to a 512-dimensional word vector, and a word matrix is formed according to the word vector, denoted as E, each column in E represents a word vector of a word.

[0073] 4-4) Referring to Figure 7 , the second stage description also adopts the Transformer structure, denoted as Tr-s2, the word matrix E obtained in step 4-3) and the first stage calculation result F h obtained in step 4-1) are fed into Tr-s2, and the second stage calculation result is obtained after encoding and decoding, and the second stage calculation result is fed into the linear layer trained in step 4-2), and the result is fed into the softmax function to obtain the word probability predicted by the second stage, denoted as F th The second stage prediction is to adjust the sentence generated based on the image features by using the category text information, and in order to avoid the deterioration of the adjusted result, the classification head trained in the first stage is used to process the output of the second stage decoder, and the category text information is used because the low-illumination target detection can obtain relatively accurate detection results, and the target detection results often appear in the sentences generated by image description, and by using the target detection results, the possibility of misjudgment of the image description model for low-illumination images can be reduced.

[0074] 4-5) A CC module is arranged after each decoding layer of the decoder in Tr-2, referring to Figure 8 ​, the CC module is to predict the copy probability of each word in the category text. In the process of prediction, the output of each decoding layer and the global feature vector will be used. Since the CC module needs to be set after the three decoding layers, three sets of copy probabilities will be obtained. The average of the three sets of probabilities is taken as the final copy probability. The so-called copy probability is the probability that the category word is directly used as the prediction result of the word at the current time step. Under low illumination, insufficient features may cause the model to mistake many similar objects as the same object. The low-illumination object detection model has good recognition ability for the categories of objects in the picture. Therefore, by calculating the object category words generated by the low-illumination object detection model into copy probability, the ability to distinguish similar objects in the low-illumination image description task can be improved, and the description accuracy can be improved.

[0075] 4-6) Process the results of the CC module and the second description to generate the final prediction result of the current time step, that is, splice the second stage word prediction probability obtained in step 4-4) and the word copy probability obtained in step 4-5), and then select the word with the highest probability as the prediction result of the current time step. Cross-entropy loss is used as the loss function, denoted as where y 1:c-1 represents the first c-1 words that have been predicted, y c represents the cth word to be predicted, P tg represents the probability.

[0076] 4-7) In the training process of the overall model, the final training loss function is the sum of the two-stage loss functions, and the formula of the loss function is shown in (1):

[0077]

[0078] where y 1:c-1 represents the first c-1 words that have been predicted, y c represents the cth word to be predicted, P v represents the probability, P tg represents the probability.

[0079] The GEX module, RFE module, and DFE module described in step 2-2) are:

[0080] The GEX module is used for global feature extraction, that is, C3, C4 and C5 are sent into the GEX module, and the three feature maps sequentially pass through Rolpool layer, maximum pooling layer and 8*8 convolution layer in the module, and are converted into three 512-dimensional global feature vectors as the output of the module. In the process of describing the picture, the scene in which the object is located is an important information. The scene information often appears in the finally generated sentence, and the information can also be used to make certain reasoning on the relationship between the objects in the picture. In low light, the features in the picture are insufficient, so the method of relying on the communication between the regional features to obtain the scene information in the normal light description is no longer applicable. In order to solve the problem of insufficient scene information in low light description, the module is set to convert the whole feature map into a feature vector. The feature vector extracted in this way has a field of view covering the whole picture, that is, a global feature vector. In order to obtain the global feature, RolPool layer is set to make reasoning based on the existing information, and supplement the features lost due to low light environment. After passing through the layer, the width and height of the feature map will be 1.25 times of the original. The subsequent maximum pooling layer converts the feature map into 8*8 size. This process is to reduce the interference information that may exist in the feature map. Finally, an 8*8 convolution layer is used to convert the feature map into a feature vector. 512 convolution kernels are set in the convolution layer, so the output of the module is a 512-dimensional global feature vector.

[0081] The DFE module is used for detail feature extraction, which aims to extract the features of each part of the region where the object is located, that is, for each region feature sub-map mapped from the object region, 4 small rectangles are uniformly divided thereon, and each small rectangle is again mapped to serve as a detail feature sub-map. The detail feature sub-map is sent to Rolpool layer, maximum pooling layer and 4*4 convolution layer to obtain a 512-dimensional detail feature vector. Since the details of the object are not easy to observe under low light, a small rectangle division method is used to supplement the details. Each small rectangle only covers part of the object, so the features extracted therefrom must pay little attention to the category information of the object, but more to the perception of details to reflect the detail information of the part, so it is called detail feature. In order to better extract the detail feature, the detail feature sub-map is sent to RolPool layer to convert it into the original size of the respective feature map. In this process, the missing features in the low light image are supplemented by using the existing information. The subsequent maximum pooling layer unifies the detail feature sub-map into 4*4 size, which reduces the interference information that may be contained therein in this process. Finally, a 4*4 convolution layer is used to convert the feature map into a feature vector. 512 convolution kernels are set in the convolution layer, so the output of the module is a 512-dimensional global feature vector.

[0082] The RFE module is used for extracting regional features. In this process, the regional feature subgraph is sequentially sent into the Rolpool layer, the maximum pooling layer and the 8*8 convolution layer, and finally a 512-dimensional regional feature vector is obtained. In this process, the regional feature subgraph is sequentially sent into the Rolpool layer, the maximum pooling layer and the 8*8 convolution layer. In the Rolpool layer, the regional feature subgraph is converted into the original size of the feature map, and the feature information that may be lost in the low-illumination image is supplemented. In the maximum pooling layer, all regional feature subgraphs are converted into 8*8 size, and the interference information is reduced. In the 8*8 convolution layer, the regional feature subgraph is converted into a regional feature vector by using the convolution layer provided with 512 convolution kernels.

[0083] The SIMCAL module described in step 3-1) is:

[0084] 3-1-1) The regional features are respectively sent into two different linear layers to obtain two 64-dimensional vectors, which are respectively used as query and key. A new linear layer is arranged to receive the detail features. After passing through the linear layer, each detail feature corresponds to a 64-dimensional vector, which is used as key. Two different linear layers are used to process the regional features and the detail features, so that the two types of features still have their respective characteristics after processing, that is, the former focuses on the category, and the latter focuses on the details;

[0085] 3-1-2) The query and each key are calculated by the dot product to calculate the similarity, and then the obtained similarity is sent into the sigmoid layer to calculate the similarity weight;

[0086] 3-1-3) The detail features are sent into the linear layer to be converted into 64-dimensional vectors, and then the elements in the vectors are summed to be the effective score of the detail features. Finally, the effective scores of all the detail features are sent into the sigmoid layer to calculate the effective weight. Since the detail information of the object in the low-illumination image is insufficient, it cannot be guaranteed that the detail features extracted from each small rectangle can perceive sufficient detail information. Therefore, the effective weight of each detail feature is calculated. The vector carrying more detail information has a higher weight, and the vector carrying less detail information has a lower weight.

[0087] The RFEN module described in step 3-2) is:

[0088] 3-2-1) The regional features and the detail features are sent into the linear layer. After passing through the linear layer, each feature corresponds to a 64-dimensional vector. These vectors are denoted as value. For the value corresponding to the detail feature, the value is multiplied by the effective weight received from the SIMCAL module;

[0089] 3-2-2) According to the similarity weight received from the SIMCAL module, the value is weighted and summed, and the sum is taken as the enhanced regional feature vector, which is the output of the module. The high and low of the effective weight reflects how much detail information is contained in the vector. The value is multiplied by the effective weight first, and then weighted and summed using the similarity weight, which ensures the effectiveness of the detailed vector and avoids the interference of the less detailed vector.

[0090] The CC module described in step 4-5) is:

[0091] 4-5-1) F is obtained from step 4-1) g As the global feature after fusion, the global feature contains two parts, key and value, denoted as and

[0092] 4-5-2) A m is taken from the output of the decoding layer, and is taken from the output of the decoding layer, and is taken from the output of the decoding layer, and th is summed, and the sum is denoted as F thg Since the global feature vector represents the characteristics of the background words, it is considered in the process of calculating the category word duplication probability. Specifically, the weights of the three global feature vectors are calculated by taking the inner product of the semantic information of the predicted word and the global feature vector. The size of the weight represents the size of the global feature vector at different depths in this prediction. The global information is utilized by adding the global feature vector to the output of the current decoding layer.

[0093] 4-5-3) F thg is sent to a linear layer, and the output is denoted as F k thg The word vector of each word is taken after the linear layer and the F thg is taken, and the result is the score of the word. By calculating the score of the word, the use of the object category text detected in the picture is completed. According to the object category text to assist the generation of low-light image description, the possibility of misjudgment of the low-light image description model can be reduced.

[0094] 4-5-4) The calculation formula of each word score is as follows:

[0095]

Claims

1. A low-light image description method based on RDE-Net, characterized in that, Includes the following steps: 1) Processing and partitioning the image description dataset: The process is as follows: 1-1) The COCO dataset is used. The COCO dataset contains a total of 123,287 images. This dataset is divided into two parts in an 8:2 ratio, with 80% being the training set (98,630 images) and 20% being the test set (25,057 images). 1-2) Use the existing CycleGAN model to process the images in the dataset: reduce the illumination of the images, convert the normal illumination images in the dataset into low illumination images, and then uniformly scale the image size to 256*256; 1-3) Use the existing IA-YOLO model to perform object detection on low-light images, and obtain the category and corresponding location of objects in each image. The category of an object is represented by a category word, i.e., the object category text information, and the location is represented by bounding box coordinates. 2) Constructing the FEX network feature map to extract global features, regional features, and detail features: The process is as follows: 2-1) The FEX network has a mapping operation for three feature maps and calls three modules. The bounding boxes obtained in steps 1-3) are scaled down proportionally to obtain the corresponding positions in the three feature maps. The three feature maps are the three feature maps fed into the detection head in IA-YOLO, which are denoted as C3, C4 and C5 respectively. Then, for each bounding box, the feature map used when generating its coordinates is selected, and the region feature sub-map is obtained by mapping. Then, the RFE module, DFE module and GEX module are called. 2-2) Input the region feature sub-map into the RFE module and DFE module, and input C3, C4 and C5 into the GEX module. The RFE module and DFE module extract the region feature vector and detail feature vector from the region feature sub-map, and the GEX module extracts the global feature vector from the feature map. The outputs of the GEX module, RFE module and DFE module are all 512-dimensional feature vectors. 3) Constructing an ENH network to enhance regional features based on detailed features: The process is as follows: 3-1) The ENH network has a SIMCAL module and a RFEN module. For the network input, the SIMCAL module is called first, and then the RFEN module is called. When calling the SIMCAL module, the regional features and detailed features are fed into it for weight calculation. The output is the effective weight and similarity weight. 3-2) The region features, detail features, and the output of the SIMCAL module are fed into the RFEN module to enhance the region features' ability to perceive details based on the detail features; 4) Construct a two-stage prediction module and a CC module to jointly describe the image: The process is as follows: 4-1) The first stage description uses the regional features and global features obtained in step 2-2), that is, both are fed into a Transformer structure, denoted as Tr-s1. Tr-s1 contains an encoder and a decoder, with three encoding layers and three decoding layers in each layer. The output of Tr-s1 is the global feature vector generated by the last encoding layer after self-attention operation and the calculation result generated by the last decoding layer, denoted as F, respectively. g 'and F h ; 4-2) Set up a linear layer to receive the F obtained in step 4-1). h This is followed by a softmax function, which outputs the probability of each word in the vocabulary. At each time step, the word with the highest probability is selected as the result, and cross-entropy loss is used as the loss function, denoted as . Where y 1:c-1 This indicates the first c-1 words that have been predicted, y c P represents the c-th word to be predicted. v Represents probability; 4-3) Encode the object category text information obtained in step 1-3). Each word corresponds to a 512-dimensional word vector. Then construct a word matrix based on the word vectors, denoted as E. Each column in E represents the word vector of a word. 4-4) The second stage description also uses the Transformer structure, denoted as Tr-s2, combining the word matrix E obtained in step 4-3) and the first stage calculation result F obtained in step 4-1). h The result is fed into Tr-s2, where it is encoded and decoded to obtain the second-stage calculation result. This second-stage result is then fed into the linear layer trained in step 4-2), and the result is fed into the softmax function to obtain the word probability predicted in the second stage, denoted as F. th ; 4-5) In Tr-2, a CC module is set after each decoding layer of the decoder. A CC module needs to be set after each of the three decoding layers. Finally, three sets of replication probabilities will be obtained. The average of the three sets of probabilities is taken as the final replication probability. 4-6) Process the results of the CC module and the second description to generate the final prediction result for the current time step. This involves concatenating the word prediction probabilities obtained in step 4-4) and the word duplication probabilities obtained in step 4-5), and then selecting the word with the highest probability as the prediction result for the current time step. Cross-entropy loss is used as the loss function, denoted as . Where y 1:c-1 This indicates the first c-1 words that have been predicted, y c P represents the c-th word to be predicted. tg Represents probability; 4-7) During the training process of the overall model, the final training loss function is the sum of the loss functions of the two stages, and the formula of the loss function is shown in (1): Where y 1:c-1 This indicates the first c-1 words that have been predicted, y c P represents the c-th word to be predicted. v P represents the probability predicted in step 4-2). tg This represents the probability predicted in steps 4-6).

2. The low-light image description method based on RDE-Net according to claim 1, characterized in that, The GEX module, RFE module, and DFE module mentioned in step 2-2) are respectively: The GEX module is used for global feature extraction. Specifically, C3, C4, and C5 are fed into the GEX module. Inside the module, the three feature maps are sequentially processed through a Rolpool layer, a max pooling layer, and an 8*8 convolutional layer, and transformed into three 512-dimensional global feature vectors as the output of the module. The DFE module is used for detail feature extraction. For each region feature sub-map obtained by mapping the object region, it is evenly divided into 4 small rectangles. Each small rectangle is mapped again and used as a detail feature sub-map. The detail feature sub-map is then fed into the Rolpool layer, the max pooling layer and the 4*4 convolutional layer to obtain a 512-dimensional detail feature vector. The RFE module is used to extract region features. In this process, the region feature sub-map is fed into the Rolpool layer, the max pooling layer and the 8*8 convolutional layer in sequence to finally obtain a 512-dimensional region feature vector.

3. The low-light image description method based on RDE-Net according to claim 1, characterized in that, The SIMCAL module mentioned in step 3-1) is: 3-1-1) The regional features are fed into two different linear layers to obtain two 64-dimensional vectors, which are used as query and key respectively. Then a new linear layer is set up to receive the detailed features. After passing through the linear layer, each detailed feature corresponds to a 64-dimensional vector, which is used as key. 3-1-2) Calculate the similarity between query and each key by using the dot product, and then feed the obtained similarity into the sigmoid layer to calculate the similarity weight; 3-1-3) The detailed features are fed into a linear layer and transformed into a 64-dimensional vector. Then, the elements in the vector are summed to obtain the effective score of the detailed features. Finally, the effective scores of all detailed features are fed into a sigmoid layer to calculate the effective weights. Each detailed feature corresponds to one effective weight.

4. The low-light image description method based on RDE-Net according to claim 1, characterized in that, The RFEN module mentioned in step 3-2) is: 3-2-1) Feed the region features and detail features into the linear layer. After passing through the linear layer, each feature corresponds to a 64-dimensional vector. These vectors are denoted as values. For the value corresponding to the detail feature, let the value be multiplied by the effective weights received from the SIMCAL module. 3-2-2) Based on the similarity weights received from the SIMCAL module, each value is weighted and summed. The sum is used as the enhanced region feature vector and is used as the output of the module.

5. The low-light image description method based on RDE-Net according to claim 1, characterized in that, The CC module mentioned in steps 4-5) is: 4-5-1) F obtained from step 4-1) g As a fused global feature, the global feature consists of two parts: a key and a value, denoted as ' and 4-5-2) The A output from the decoding layer m and Perform inner products sequentially to obtain a similarity vector Sim of length 3. Then, feed Sim into the sigmoid function to calculate its weights. The weights are then compared with... After multiplying them separately, combine them with F obtained in step 4-4). th Summation, the result of which is denoted as F. thg ; 4-5-3)F thg The input is fed into a linear layer, and the output is denoted as F. k thg The word vector of each word is passed through a linear layer and then compared with F. thg Perform the inner product operation, and the result is the score for that word; 4-5-4) The formula for calculating the score for each word is as follows:

Images