Nighttime thermal infrared image traffic light coloring method

Abstract

The application discloses a night thermal infrared image traffic light coloring method, which comprises the following steps: firstly, predicting a binary traffic light mask of a night thermal infrared image and an arbitrary daytime color image; secondly, dividing the mask into an upper mask and a lower mask according to the center height of each instance; thirdly, dividing the mask into a bright mask and a dark mask according to the brightness and chroma values; fourthly, calculating the color features of the bright areas of the traffic lights in the daytime color image and the coloring result of the model output; fifthly, calculating the brightness difference between the bright areas and the dark areas of the traffic lights in the coloring result as a brightness loss function; sixthly, calculating the color feature distance between the coloring result and the daytime color image at the corresponding position of the traffic lights as a color loss function; and finally, training the model by combining the brightness loss function and the color loss function to complete the coloring of the traffic lights in the night thermal infrared image. The coloring method is designed by taking into account the evaluation of brightness and color, the apparent credibility of the traffic lights in the coloring result of the night thermal infrared image is enhanced, and the traffic light detection effect in the glare area in the night scene is improved.

Description

Technical Field

[0001] This invention belongs to the fields of computer vision and deep learning technology, and specifically relates to a method for coloring traffic lights in nighttime thermal infrared images. Background Technology

[0002] All-weather scene perception is crucial for numerous applications, including autonomous driving, intelligent monitoring, and rescue robots. Among these, thermal infrared cameras, due to their stability in imaging glare and low-light environments, have become a vital tool for visual acquisition of nighttime scenes. However, thermal infrared imaging results are typically presented as grayscale images, suffering from drawbacks such as blurred boundaries, low contrast, and poor detail representation. This not only affects human observers' perception but also hinders the transfer and application of existing scene perception models based on visible light images. Therefore, research on nighttime thermal infrared image colorization, by colorizing nighttime thermal infrared images into consistent daytime color images, holds promise for promoting the application of related computer vision methods in all-weather scenes and multimodal data.

[0003] In traffic scenarios, due to the difficulty in acquiring paired daytime color images, nighttime thermal infrared image colorization tasks typically utilize unsupervised image-to-image transformation methods based on deep neural networks. Unsupervised image-to-image transformation aims to map a source domain image to a target domain without paired images, while preserving the original image's content information. However, due to the lack of pixel-level semantic annotation, this method struggles to guarantee semantic consistency in nighttime thermal infrared image colorization. Furthermore, due to the blurred boundaries and extremely low contrast of thermal infrared images, existing colorization methods perform poorly on small targets such as traffic lights. In addition, traffic light detection in glare regions is extremely challenging for detection models based on visible light images. Therefore, enhancing the color reliability of traffic lights in thermal infrared image colorization results will help improve traffic light detection performance in nighttime scenes.

[0004] Although traffic light signals are not uniform in color, they are strongly correlated with their spatial location. For example, signals emitting light in similar locations usually exhibit the same color. Therefore, designing appropriate thermal infrared image coloring methods based on the color distribution characteristics of traffic lights will help enhance the visual effect of traffic light coloring, thereby improving the accuracy of nighttime traffic light detection. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a method for coloring traffic lights in nighttime thermal infrared images. The method is designed based on the color distribution characteristics of traffic lights, guiding a neural network model to learn the mapping relationship between the emission position and color of traffic light signals during training. This enhances the visual effect of traffic light coloring and improves the accuracy of nighttime traffic light detection.

[0006] The technical solution adopted in this invention is: a method for coloring traffic lights in nighttime thermal infrared images, the specific steps of which are as follows:

[0007] S1. Predict the binarized traffic light mask of nighttime thermal infrared image and any daytime color image;

[0008] S2. Based on the center height of each traffic light instance in the mask, divide the traffic light mask of the nighttime thermal infrared image and the traffic light mask of the daytime color image into upper and lower masks respectively.

[0009] S3. Based on the average brightness, the traffic light mask of the nighttime thermal infrared image in step S1 is divided into bright and dark masks. Based on the chromaticity value, the traffic light mask of the daytime color image in step S1 is divided into bright and dark masks.

[0010] S4. First, take the intersection of the traffic light mask of the night thermal infrared image in step S2 and the upper and lower masks of the traffic light mask of the daytime color image with the bright mask in step S3. Then, calculate the color features of the bright areas of the traffic lights at different positions in the color model output of the daytime color image and the night thermal infrared image respectively.

[0011] S5. Using the bright and dark masks of traffic lights in the nighttime thermal infrared image obtained in step S3, calculate the brightness difference between the bright and dark areas of the traffic lights in the coloring result, and use it as the brightness loss function.

[0012] S6. Using the color features of bright areas at different locations obtained in step S4, calculate the color distance between the coloring result and the traffic light color in similar spatial locations of the daytime color image, and use it as the color loss function.

[0013] S7. Combine brightness and color loss functions to train a nighttime thermal infrared image coloring model to complete the coloring of traffic lights in nighttime thermal infrared images.

[0014] Furthermore, step S1 is specifically as follows:

[0015] S11. Using a semantic segmentation model pre-trained on a daytime visible light dataset, predict the semantic segmentation mask of the daytime color image, and then extract the binarized traffic light mask.

[0016] S12. Select an unsupervised image-to-image conversion model as the baseline model to achieve mutual conversion between nighttime thermal infrared images and daytime color images;

[0017] The baseline model's network structure includes two generators and two discriminators: a generator that converts nighttime thermal infrared images into pseudo-daytime color images and a generator that converts daytime color images into pseudo-nighttime thermal infrared images; a discriminator that distinguishes between true and pseudo-nighttime thermal infrared images and a discriminator that distinguishes between true and pseudo-daytime color images.

[0018] S13. Using the pseudo-nighttime thermal infrared image converted from the daytime color image and the semantic mask of the daytime color image obtained in step S11, train a nighttime thermal infrared image segmentation network online.

[0019] S14. Use a nighttime thermal infrared image segmentation network to predict the semantic mask of the nighttime thermal infrared image, and then extract the corresponding binarized traffic light mask.

[0020] Furthermore, step S2 is specifically as follows:

[0021] S21. Determine the number of connected branches and the corresponding regions of the traffic light mask based on connectivity;

[0022] S22. Define the connected branch region with a pixel count greater than a set threshold as a traffic light instance;

[0023] S23. Find the minimum row and the maximum row of the binarized mask corresponding to the traffic light instance, and take the average as the center height of the instance;

[0024] S24. Use the binarized mask above the row where the center height is located as the upper mask and the mask below it as the lower mask.

[0025] Furthermore, step S3 is specifically as follows:

[0026] S31. Multiply the traffic light mask of the nighttime thermal infrared image obtained in step S1 with the original nighttime thermal infrared image element by element to obtain the traffic light region in the nighttime thermal infrared image.

[0027] S32. Divide the sum of the brightness of the traffic light area by the sum of the corresponding binary masks to obtain the average brightness of the traffic light area;

[0028] S33. Compare the brightness of each position in the traffic light area with the average brightness. If it is greater than the average brightness, the value of that position in the bright mask is 1; otherwise, the value of that position in the dark mask is 1.

[0029] S34. The binarized mask obtained after traversing all pixels is used as the bright and dark mask of traffic lights in the nighttime thermal infrared image.

[0030] S35. Convert the RGB format daytime color image to HSV format;

[0031] S36. Traverse all pixels and set the value of the position that matches the definition of the red area to 1. The resulting binary mask is used as the red area mask of the traffic light.

[0032] The definition of the red area is set according to actual needs.

[0033] S37. Set the value of the position that conforms to the definition of green area to 1, and use the resulting binarized mask as the green area mask of the traffic light.

[0034] The definition of the green area is set according to actual needs.

[0035] S38. For each traffic light instance, if the area of ​​its corresponding red area mask is greater than that of the green area mask, then the red area mask is used as the bright mask of the instance; otherwise, the green area mask is used as the bright mask of the instance.

[0036] S39. Take the union of the bright masks of all instances as the bright mask of the daytime color image, and use its complement as the dark mask.

[0037] Furthermore, step S4 is specifically as follows:

[0038] S41. Take the intersection of the daytime color image of the traffic light mask obtained in step S2 and the traffic light illumination mask obtained in step S3.

[0039] S42. Multiply the intersection with the daytime color image element by element;

[0040] S43. Summing the product by channel and dividing by the sum of the intersection masks, we finally obtain a 1×3 one-dimensional vector, which serves as the average color feature of the bright area above the traffic light.

[0041] S44. Replace the upper mask in step S41 with the lower mask, and repeat steps S41 to S43 to calculate the average color features of the bright area below the traffic light.

[0042] S45. Replace the daytime color image and corresponding mask from steps S41 to S44 with the coloring result output by the coloring model and the corresponding mask. Repeat steps S41 to S44 and calculate the average color features of the bright areas above and below the traffic lights in the coloring result using the same method.

[0043] Furthermore, step S5 is specifically as follows:

[0044] S51. First, use the bright and dark masks of the traffic lights in the nighttime thermal infrared image obtained in step S3, and multiply them element-wise with the coloring results output by the nighttime thermal infrared image coloring model to obtain the bright and dark areas of the traffic lights in the coloring results.

[0045] S52. Calculate the minimum brightness of the traffic light illuminated area in the coloring result, denoted as δ. br ;

[0046] S53. Calculate the average brightness of the dark areas of the traffic lights in the coloring results, expressed as μ. dr ;

[0047] S54. Calculate the difference between the average brightness of the dark area and the lowest brightness of the bright area. The result, after non-negativity constraint and adaptive scaling, is used as the brightness loss function. The calculation method is as follows:

[0048]

[0049] Furthermore, step S6 is specifically as follows:

[0050] S61. The average color features of the upper and lower bright areas of the traffic lights in the daytime color image obtained in step S4 are respectively represented as:

[0051] S62. The average color features of the upper and lower bright areas in the traffic light obtained in step S4 are respectively represented as:

[0052] S63. Calculate the color feature distance between the colorization result and the bright area above the traffic light in the daytime color image, denoted as d. uu The calculation method is as follows:

[0053]

[0054] Where k represents the index of the RGB image color channel, with an upper bound of 3, representing the number of color channels.

[0055] Similarly, using features With features Calculate the color feature distance d of the bright area below. ll and the use of features With features Calculate the inter-class distance d of color features lu .

[0056] S64, Utilizing intra-class distance d ll Inter-class distance d lu Calculate the adaptive range gain factor, denoted as β. g The calculation method is as follows:

[0057]

[0058] Here, τ represents a constant that prevents the denominator from being zero and controls the range of output values.

[0059] S65. The weighted sum of the colorization result and the color feature distances of the daytime color image above and below the traffic light is used as the color loss function, calculated as follows:

[0060] Color loss function = d uu +β g ×d ll

[0061] Furthermore, step S7 is specifically as follows:

[0062] S71. Keeping the network structure, optimizer, and learning rate setting rules of the baseline model selected in step S12 unchanged, the brightness loss function in step S54 and the color loss function in step S65 are added to the original loss function of the baseline model to train the model.

[0063] S72. When the loss function value converges, the model training ends and the nighttime thermal infrared image traffic light coloring is completed.

[0064] Among them, the generator that converts the trained nighttime thermal infrared images into pseudo-daytime color images serves as the final nighttime thermal infrared image colorization model.

[0065] The beneficial effects of this invention are as follows: The method of this invention first predicts the binarized traffic light masks of nighttime thermal infrared images and arbitrary daytime color images. Based on the center height of each instance, the masks are divided into upper and lower masks. Then, based on luminance and chromaticity values, the masks are further divided into bright and dark masks. The color features of bright areas at different positions of the traffic lights in the daytime color image and the model's output colorization result are calculated. Then, the luminance difference between the bright and dark areas of the traffic lights in the colorization result is calculated as a luminance loss function. The distance between the colorization result and the daytime color image at corresponding positions of the traffic light color features is calculated as a color loss function. Finally, the model is trained by combining the luminance and color loss functions to complete the colorization of traffic lights in nighttime thermal infrared images. This invention designs traffic light luminance and color loss functions based on the color distribution characteristics of traffic lights. Under conditions without paired image supervision, it guides the neural network model to learn the mapping relationship between the emission position and color of traffic light signals during training. It considers both luminance and color evaluation in the colorization method, enhancing the apparent reliability of traffic lights in the nighttime thermal infrared image colorization result and improving the detection effect of traffic lights in glare areas in nighttime scenes. Attached Figure Description

[0066] Figure 1 This is a flowchart of a nighttime thermal infrared image traffic light coloring method according to the present invention.

[0067] Figure 2 This is a schematic diagram of the training and testing phase of the nighttime thermal infrared image traffic light coloring method in an embodiment of the present invention.

[0068] Figure 3 This is a comparison example of the coloring results in this embodiment of the invention with other methods. Detailed Implementation

[0069] The method of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0070] like Figure 1 The flowchart of a nighttime thermal infrared image traffic light coloring method of the present invention is shown below, and the specific steps are as follows:

[0071] S1. Predict the binarized traffic light mask of nighttime thermal infrared image and any daytime color image;

[0072] S2. Based on the center height of each traffic light instance in the mask, divide the traffic light mask of the nighttime thermal infrared image and the traffic light mask of the daytime color image into upper and lower masks respectively.

[0073] S3. Based on the average brightness, the traffic light mask of the nighttime thermal infrared image in step S1 is divided into bright and dark masks. Based on the chromaticity value, the traffic light mask of the daytime color image in step S1 is divided into bright and dark masks.

[0074] S4. First, take the intersection of the traffic light mask of the night thermal infrared image in step S2 and the upper and lower masks of the traffic light mask of the daytime color image with the bright mask in step S3. Then, calculate the color features of the bright areas of the traffic lights at different positions in the color model output of the daytime color image and the night thermal infrared image respectively.

[0075] S5. Using the bright and dark masks of traffic lights in the nighttime thermal infrared image obtained in step S3, calculate the brightness difference between the bright and dark areas of the traffic lights in the coloring result, and use it as the brightness loss function.

[0076] S6. Using the color features of bright areas at different locations obtained in step S4, calculate the color distance between the coloring result and the traffic light color in similar spatial locations of the daytime color image, and use it as the color loss function.

[0077] S7. Combine brightness and color loss functions to train a nighttime thermal infrared image coloring model to complete the coloring of traffic lights in nighttime thermal infrared images.

[0078] In this embodiment, step S1 is specifically as follows:

[0079] S11, such as Figure 2 As shown, in this embodiment, the semantic segmentation model HMSANet, which is pre-trained on the Mapillary Vistas dataset, is first used to predict the semantic segmentation mask of the daytime color image, and then the binarized traffic light mask is extracted.

[0080] S12. Using the ToDayGAN model as the baseline model, the mutual conversion between nighttime thermal infrared images and daytime color images is realized;

[0081] The baseline model's network structure includes two generators and two discriminators: a generator that converts nighttime thermal infrared images into pseudo-daytime color images and a generator that converts daytime color images into pseudo-nighttime thermal infrared images; a discriminator that distinguishes between true and pseudo-nighttime thermal infrared images and a discriminator that distinguishes between true and pseudo-daytime color images.

[0082] S13. Using the pseudo-nighttime thermal infrared image converted from the daytime color image and the semantic mask of the daytime color image obtained in step S11, train a nighttime thermal infrared image segmentation network online.

[0083] S14. Use a nighttime thermal infrared image segmentation network to predict the semantic mask of the nighttime thermal infrared image, and then extract the corresponding binarized traffic light mask.

[0084] In this embodiment, step S2 is specifically as follows:

[0085] S21. Use the measure.label() function of the scikit-image library to determine the number of connected components and the corresponding regions of the traffic light mask;

[0086] S22. Define connected branch regions with more than 50 pixels as traffic light instances;

[0087] S23. Find the minimum row and the maximum row of the binarized mask corresponding to the traffic light instance, and take the average as the center height of the instance;

[0088] S24. Use the binarized mask above the row where the center height is located as the upper mask and the mask below it as the lower mask.

[0089] In this embodiment, step S3 is specifically as follows:

[0090] S31. Multiply the traffic light mask of the nighttime thermal infrared image obtained in step S1 with the original nighttime thermal infrared image element by element to obtain the traffic light region in the nighttime thermal infrared image.

[0091] S32. Divide the sum of the brightness of the traffic light area by the sum of the corresponding binary masks to obtain the average brightness of the traffic light area;

[0092] S33. Compare the brightness of each position in the traffic light area with the average brightness. If it is greater than the average brightness, the value of that position in the bright mask is 1; otherwise, the value of that position in the dark mask is 1.

[0093] S34. The binarized mask obtained after traversing all pixels is used as the bright and dark mask of traffic lights in the nighttime thermal infrared image.

[0094] S35. Convert the RGB format daytime color image to HSV format;

[0095] S36. Traverse all pixels and set the value of the position that satisfies the conditions 0≤H<25 and 155<H≤180, S>42, V>45 to 1. Use the resulting binarized mask as the red area mask of the traffic light.

[0096] Among them, hue (H), saturation (S), and brightness (V).

[0097] S37. Set the value of the position that satisfies the conditions 67≤H<110, S>42, V>45 to 1, and use the resulting binarized mask as the green area mask of the traffic light.

[0098] S38. For each traffic light instance, if the area of ​​its corresponding red area mask is greater than that of the green area mask, then the red area mask is used as the bright mask of the instance; otherwise, the green area mask is used as the bright mask of the instance.

[0099] S39. Take the union of the bright masks of all instances as the bright mask of the daytime color image, and use its complement as the dark mask.

[0100] In this embodiment, step S4 is specifically as follows:

[0101] S41. Take the intersection of the daytime color image of the traffic light mask obtained in step S2 and the traffic light illumination mask obtained in step S3.

[0102] S42. Multiply the intersection with the daytime color image element by element;

[0103] S43. Summing the product by channel and dividing by the sum of the intersection masks, we finally obtain a 1×3 one-dimensional vector, which serves as the average color feature of the bright area above the traffic light.

[0104] S44. Replace the upper mask in step S41 with the lower mask, and repeat steps S41 to S43 to calculate the average color features of the bright area below the traffic light.

[0105] S45. Replace the daytime color image and corresponding mask from steps S41 to S44 with the coloring result output by the coloring model and the corresponding mask. Repeat steps S41 to S44 and calculate the average color features of the bright areas above and below the traffic lights in the coloring result using the same method.

[0106] In this embodiment, step S5 is specifically as follows:

[0107] S51. First, use the bright and dark masks of the traffic lights in the nighttime thermal infrared image obtained in step S3, and multiply them element-wise with the coloring results output by the nighttime thermal infrared image coloring model to obtain the bright and dark areas of the traffic lights in the coloring results.

[0108] S52. Calculate the minimum brightness of the traffic light illuminated area in the coloring result, denoted as δ. br ;

[0109] S53. Calculate the average brightness of the dark areas of the traffic lights in the coloring results, expressed as μ. dr ;

[0110] S54. Calculate the difference between the average brightness of the dark area and the lowest brightness of the bright area. The result, after non-negativity constraint and adaptive scaling, is used as the brightness loss function. The calculation method is as follows:

[0111]

[0112] like Figure 2 As shown, in this embodiment, step S6 is specifically as follows:

[0113] S61. The average color features of the upper and lower bright areas of the traffic lights in the daytime color image obtained in step S4 are respectively represented as:

[0114] S62. The average color features of the upper and lower bright areas in the traffic light obtained in step S4 are respectively represented as:

[0115] S63. Calculate the color feature distance between the colorization result and the bright area above the traffic light in the daytime color image, denoted as d. uu The calculation method is as follows:

[0116]

[0117] Where k represents the index of the RGB image color channel, with an upper bound of 3, representing the number of color channels.

[0118] Similarly, using features With features Calculate the color feature distance d of the bright area below. ll and the use of features With features Calculate the inter-class distance d of color features lu .

[0119] S64, Utilizing intra-class distance d ll Inter-class distance d lu Calculate the adaptive range gain factor, denoted as β. g The calculation method is as follows:

[0120]

[0121] Wherein, τ represents a constant that prevents the denominator from being 0 and controls the range of output values, which is empirically set to 0.05 in this embodiment.

[0122] S65. The weighted sum of the colorization result and the color feature distances of the daytime color image above and below the traffic light is used as the color loss function, calculated as follows:

[0123] Color loss function = d uu +β g ×d ll

[0124] In this embodiment, step S7 is specifically as follows:

[0125] S71. Keeping the network structure, optimizer, and learning rate setting rules of the baseline model selected in step S12 unchanged, the brightness loss function in step S54 and the color loss function in step S65 are added to the original loss function of the baseline model to train the model.

[0126] S72. When the loss function value converges, the model training ends and the nighttime thermal infrared image traffic light coloring is completed.

[0127] Among them, the generator that converts the trained nighttime thermal infrared images into pseudo-daytime color images serves as the final nighttime thermal infrared image colorization model.

[0128] The method of this invention was tested in ablation experiments on the FLIR and Brno datasets. Extensive experimental results demonstrate that incorporating the designed luminance and color loss functions during model training can enhance the apparent credibility of traffic lights in the colorized results of nighttime thermal infrared images. Furthermore, in the downstream semantic segmentation task of the colorized results, the segmentation performance of traffic lights can be significantly improved. For example... Figure 3 As shown, compared with other comparative methods, the coloring result of the method of the present invention can not only preserve the traffic light structure of the original image, but also predict the color of the traffic light signal, which is beneficial to the subsequent realization of reliable all-weather scene perception by computer vision algorithms based on visible light.

[0129] In summary, the method of this invention designs traffic light brightness and color loss functions based on the color distribution characteristics of traffic lights. Under the condition of no paired image supervision, it guides the neural network model to learn the mapping relationship between the emission position and color of traffic light signals during training. It also designs a coloring method that takes into account both brightness and color evaluation, thereby enhancing the apparent reliability of traffic lights in the coloring results of nighttime thermal infrared images and improving the detection effect of traffic lights in glare areas in nighttime scenes.

[0130] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of this invention.

Claims

1. A method for coloring traffic lights in nighttime thermal infrared images, comprising the following steps: S1. Predict the binarized traffic light mask of nighttime thermal infrared image and any daytime color image; S2. Based on the center height of each traffic light instance in the mask, divide the traffic light mask of the nighttime thermal infrared image and the traffic light mask of the daytime color image into upper and lower masks respectively. S3. Based on the average brightness, the traffic light mask of the nighttime thermal infrared image in step S1 is divided into bright and dark masks. Based on the chromaticity value, the traffic light mask of the daytime color image in step S1 is divided into bright and dark masks. S4. First, take the intersection of the traffic light mask of the night thermal infrared image in step S2 and the upper and lower masks of the traffic light mask of the daytime color image with the bright mask in step S3. Then, calculate the color features of the bright areas of the traffic lights at different positions in the color model output of the daytime color image and the night thermal infrared image respectively. S5. Using the bright and dark masks of traffic lights in the nighttime thermal infrared image obtained in step S3, calculate the brightness difference between the bright and dark areas of the traffic lights in the coloring result, and use it as the brightness loss function. Step S5 is as follows: S51. First, use the bright and dark masks of the traffic lights in the nighttime thermal infrared image obtained in step S3, and multiply them element-wise with the coloring results output by the nighttime thermal infrared image coloring model to obtain the bright and dark areas of the traffic lights in the coloring results. S52. Calculate the minimum brightness of the traffic light illuminated area in the coloring result, expressed as: ; S53. Calculate the average brightness of the dark areas of the traffic lights in the coloring results, expressed as: ; S54. Calculate the difference between the average brightness of the dark area and the lowest brightness of the bright area. The result, after non-negativity constraint and adaptive scaling, is used as the brightness loss function. The calculation method is as follows: ； S6. Using the color features of bright areas at different locations obtained in step S4, calculate the color distance between the coloring result and the traffic light color in similar spatial locations of the daytime color image, and use it as the color loss function. S7. Combine brightness and color loss functions to train a nighttime thermal infrared image coloring model to complete the coloring of traffic lights in nighttime thermal infrared images.

2. The method for coloring traffic lights in nighttime thermal infrared images according to claim 1, characterized in that, The specific steps of S1 are as follows: S11. Using a semantic segmentation model pre-trained on a daytime visible light dataset, predict the semantic segmentation mask of the daytime color image, and then extract the binarized traffic light mask. S12. Select an unsupervised image-to-image conversion model as the baseline model to achieve mutual conversion between nighttime thermal infrared images and daytime color images; The baseline model's network structure includes two generators and two discriminators: a generator for converting nighttime thermal infrared images into pseudo-daytime color images, and a generator for converting daytime color images into pseudo-nighttime thermal infrared images; a discriminator for distinguishing between true and pseudo-nighttime thermal infrared images, and a discriminator for distinguishing between true and pseudo-daytime color images. S13. Using the pseudo-nighttime thermal infrared image converted from the daytime color image and the semantic mask of the daytime color image obtained in step S11, train a nighttime thermal infrared image segmentation network online. S14. Use a nighttime thermal infrared image segmentation network to predict the semantic mask of the nighttime thermal infrared image, and then extract the corresponding binarized traffic light mask.

3. The method for coloring traffic lights in nighttime thermal infrared images according to claim 1, characterized in that, Step S2 is as follows: S21. Determine the number of connected branches and the corresponding regions of the traffic light mask based on connectivity; S22. Define the connected branch region with a pixel count greater than a set threshold as a traffic light instance; S23. Find the minimum row and the maximum row of the binarized mask corresponding to the traffic light instance, and take the average as the center height of the instance; S24. Use the binarized mask above the row where the center height is located as the upper mask and the mask below it as the lower mask.

4. The method for coloring traffic lights in nighttime thermal infrared images according to claim 1, characterized in that, Step S3 is as follows: S31. Multiply the traffic light mask of the nighttime thermal infrared image obtained in step S1 with the original nighttime thermal infrared image element by element to obtain the traffic light region in the nighttime thermal infrared image. S32. Divide the sum of the brightness of the traffic light area by the sum of the corresponding binary masks to obtain the average brightness of the traffic light area; S33. Compare the brightness of each position in the traffic light area with the average brightness. If it is greater than the average brightness, the value of that position in the bright mask is 1; otherwise, the value of that position in the dark mask is 1. S34. The binarized mask obtained after traversing all pixels is used as the bright and dark mask of traffic lights in the nighttime thermal infrared image. S35. Convert the RGB format daytime color image to HSV format; S36. Traverse all pixels and set the value of the position that matches the definition of the red area to 1. The resulting binary mask is used as the red area mask of the traffic light. The definition of the red area is set according to actual needs; S37. Set the value of the position that conforms to the definition of green area to 1, and use the resulting binarized mask as the green area mask of the traffic light. The definition of the green area is set according to actual needs; S38. For each traffic light instance, if the area of ​​its corresponding red area mask is greater than that of the green area mask, then the red area mask is used as the bright mask of the instance; otherwise, the green area mask is used as the bright mask of the instance. S39. Take the union of the bright masks of all instances as the bright mask of the daytime color image, and use its complement as the dark mask.

5. The method for coloring traffic lights in nighttime thermal infrared images according to claim 1, characterized in that, Step S4 is as follows: S41. Take the intersection of the daytime color image of the traffic light mask obtained in step S2 and the traffic light illumination mask obtained in step S3. S42. Multiply the intersection with the daytime color image element by element; S43. Summing the product by channel and dividing by the sum of the intersection masks, we finally obtain a 1×3 one-dimensional vector, which serves as the average color feature of the bright area above the traffic light. S44. Replace the upper mask in step S41 with the lower mask, and repeat steps S41 to S43 to calculate the average color features of the bright area below the traffic light. S45. Replace the daytime color image and corresponding mask from steps S41 to S44 with the coloring result output by the coloring model and the corresponding mask. Repeat steps S41 to S44 and calculate the average color features of the bright areas above and below the traffic lights in the coloring result using the same method.

6. The method for coloring traffic lights in nighttime thermal infrared images according to claim 1, characterized in that, Step S6 is as follows: S61. The average color features of the upper and lower bright areas of the traffic lights in the daytime color image obtained in step S4 are respectively represented as: , ; S62. The average color features of the upper and lower bright areas in the traffic light obtained in step S4 are respectively represented as: , ; S63. Calculate the color feature distance between the coloring result and the bright area above the traffic light in the daytime color image, expressed as: The calculation method is as follows: ； Where k represents the index of the RGB image color channel, with an upper bound of 3, representing the number of color channels; Similarly, using features With features Calculate the color feature distance of the bright area below. and the use of features With features Calculate the inter-class distance of color features ; S64, Utilizing intra-class spacing Inter-class distance Calculate the adaptive range gain factor, expressed as The calculation method is as follows: ； in, This represents a constant that prevents the denominator from being zero and controls the range of output values. S65. The weighted sum of the colorization result and the color feature distances of the daytime color image above and below the traffic light is used as the color loss function, calculated as follows: 。 7. The method for coloring traffic lights in nighttime thermal infrared images according to claim 1, characterized in that, Step S7 is as follows: S71. Keeping the network structure, optimizer, and learning rate setting rules of the baseline model selected in step S12 unchanged, the brightness loss function in step S54 and the color loss function in step S65 are added to the original loss function of the baseline model to train the model. S72. When the loss function value converges, the model training ends and the nighttime thermal infrared image traffic light coloring is completed. Among them, the generator that converts the trained nighttime thermal infrared images into pseudo-daytime color images serves as the final nighttime thermal infrared image colorization model.

Images