Traffic light coloring method, device, equipment and storage medium

Abstract

The present application discloses a traffic light coloring method, apparatus, device, and storage medium, including: obtaining a set of image frames to be processed corresponding to a video stream to be processed, performing color recognition conversion on each of the image frames in the set to be processed, and generating a set of auxiliary image frames corresponding to the set to be processed; the lit pixels in each auxiliary frame in the set of auxiliary image frames are green pixels or preset non-green pixels; determining each image frame to be processed and the corresponding auxiliary frame as a set of image frames to be processed based on the corresponding relationship; inputting each set of image frames to be processed into a pre-trained traffic light coloring model to determine a set of preliminary coloring image frames; and performing color restoration on each preliminary coloring image frame based on the pixel category information in each preliminary coloring image frame and the traffic light position information in the image frame to be processed. This reduces the confusion in the recognition of red and yellow lights in traffic lights and improves the accuracy of traffic light coloring.

Description

Technical Field

[0001] The present application relates to the field of computer vision recognition technology, and in particular to a traffic light coloring method, device, equipment, and storage medium. Background Art

[0002] With the rapid development of science and technology, new deep learning platforms such as checkpoint electric police and smart transportation are increasingly being used in daily life. Traffic light color, as an important basis for traffic light status identification, is widely used in smart transportation.

[0003] Based on the advancements in deep learning in computer vision, neural networks offer superior recognition performance compared to traditional methods of manually extracting features and performing pattern recognition. Consequently, deep learning has been widely adopted in traffic light colorization. Existing methods often rely on neural network models trained on multiple templates to identify traffic light color and position, or directly use neural networks to identify red, green, and yellow lights.

[0004] However, templates cannot cover all traffic light situations, and multi-template training requires a massive number of templates, consuming significant computing resources. Furthermore, in real-world environments, scenes and lighting are complex. For example, at dusk, yellow and red lights appear similar in real-time images captured by traffic light alarms and smart transportation systems. Even with neural network models, there's still a high probability of recognition errors, impacting the accuracy of traffic light alarms and smart transportation systems. Summary of the Invention

[0005] The present application provides a traffic light coloring method, apparatus, device, and storage medium, which reduce the scene dependency of a neural network model used in computer vision to identify traffic light colors, reduce confusion between red and yellow lights in traffic lights, and enable traffic light coloring with high recognition accuracy in different scenes, at different time periods, and under different lighting conditions.

[0006] In a first aspect, an embodiment of the present application provides a traffic light coloring method, comprising:

[0007] Obtaining a set of image frames to be processed corresponding to the video stream to be processed, and performing color recognition conversion on each of the image frames to be processed in the set to be processed to generate a set of auxiliary image frames corresponding to the set of image frames to be processed; wherein the lit pixels in each of the auxiliary image frames in the set of auxiliary image frames are green pixels or preset non-green pixels;

[0008] Determine each to-be-processed image frame and its corresponding auxiliary image frame as an to-be-processed image frame group based on the corresponding relationship, input each to-be-processed image frame group into a pre-trained traffic light coloring model, and determine a preliminary coloring image frame set;

[0009] Based on the pixel category information in each preliminary colorized image frame and the traffic light position information in the image frames to be processed corresponding to each preliminary colorized image frame, color restoration is performed on each preliminary colorized image frame to generate a colorized image frame set corresponding to the image frame set to be processed.

[0010] In a second aspect, an embodiment of the present application further provides a traffic light coloring device, comprising:

[0011] An image frame acquisition module is configured to acquire a set of image frames to be processed corresponding to a video stream to be processed, and perform color recognition conversion on each of the image frames to be processed in the set to be processed to generate a set of auxiliary image frames corresponding to the set of image frames to be processed; wherein the lit pixels in each of the auxiliary image frames in the set of auxiliary image frames are green pixels or preset non-green pixels;

[0012] A preliminary coloring module is used to determine each to-be-processed image frame and the corresponding auxiliary image frame as a to-be-processed image frame group based on the corresponding relationship, and input each to-be-processed image frame group into a pre-trained traffic light coloring model to determine a preliminary coloring image frame set;

[0013] The color-drawing image generation module is used to perform color restoration on each preliminary color-drawing image frame based on the pixel category information in each preliminary color-drawing image frame and the traffic light position information in the image frame to be processed corresponding to each preliminary color-drawing image frame, and generate a set of color-drawing image frames corresponding to the set of image frames to be processed.

[0014] In a third aspect, an embodiment of the present application further provides a traffic light coloring device, comprising:

[0015] at least one processor; and

[0016] a memory communicatively connected to at least one processor; wherein,

[0017] The memory stores a computer program that can be executed by at least one processor. The computer program is executed by at least one processor so that the at least one processor can execute the traffic light coloring method provided in the embodiment of the present application.

[0018] In a fourth aspect, an embodiment of the present application further provides a storage medium comprising computer-executable instructions, which, when executed by a computer processor, are used to execute the traffic light coloring method provided in an embodiment of the present application.

[0019] The embodiments of the present application provide a traffic light coloring method, apparatus, device, and storage medium. The method obtains a set of image frames to be processed corresponding to a video stream to be processed, and performs color recognition conversion on each image frame to be processed in the set of image frames to be processed to generate a set of auxiliary image frames corresponding to the set of image frames to be processed. The lighted pixels in each auxiliary image frame in the set of auxiliary image frames are green pixels or preset non-green pixels. Each image frame to be processed and the corresponding auxiliary image frame are determined as a set of image frames to be processed based on the corresponding relationship. Each set of image frames to be processed is input into a pre-trained traffic light coloring model to determine a set of preliminary coloring image frames. The method performs color restoration on each preliminary coloring image frame based on pixel category information in each preliminary coloring image frame and traffic light position information in the image frames to be processed corresponding to each preliminary coloring image frame to generate a set of coloring image frames corresponding to the set of image frames to be processed. By adopting the above technical solution, the image frames to be processed corresponding to the acquired video stream to be processed are preprocessed so that the illuminated portion of the traffic lights in each set of image frames to be processed can be processed to a predetermined non-green color or green based on the actual detected color. That is, the pixels corresponding to the easily confused yellow and red lights are all processed to red pixels. This ensures that the illuminated colors of the traffic lights displayed in the processed auxiliary image frames are easily distinguishable red and green, facilitating the subsequent correct recognition of the traffic light colors by the model. The image frames to be processed and the corresponding auxiliary image frames are then grouped and input into a pre-trained traffic light colorization model for processing. The set of images are compared with each other to determine the pixel category information corresponding to each pixel therein. The images with the determined pixel category information are then determined as preliminary colorization image frames. Based on the pixel category information in each preliminary colorization image frame, the illuminated color of the traffic light can be determined. Then, the illuminated color of the traffic light in the preliminary colorization image frames can be restored according to the red, yellow, and green lighting logic of the traffic light, resulting in a final colorization image frame set corresponding to the image frames to be processed. Since the pre-trained traffic light coloring model only needs to distinguish between the red and green colors with obvious distinction, it is less affected by the scene in which the traffic light is located. Therefore, when training the traffic light coloring model, there is no need to provide different templates for different scenes to train it, which reduces the amount of data required for training and avoids traffic light coloring errors caused by insufficient scene templates. The amount of data calculation is reduced during use. After accurate category division, the lighting color of the traffic light in the preliminary coloring image frame is restored according to the lighting logic of the red, yellow and green lights in the traffic light, so that the yellow light in the traffic light can be accurately restored, thereby improving the accuracy of traffic light coloring.

[0020] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0022] Figure 1 A flow chart of a traffic light coloring method provided in one embodiment of the present application;

[0023] Figure 2 A flow chart of a traffic light coloring method provided in one embodiment of the present application;

[0024] Figure 3 A schematic diagram of the structure of a traffic light color depiction device provided in an embodiment of the present application;

[0025] Figure 4 A schematic structural diagram of a traffic light color depiction device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0028] Figure 1This is a flowchart of a traffic light coloring method provided in one embodiment of the present application. This embodiment of the present application is applicable to situations where, when platforms such as checkpoint electronic police and smart transportation capture videos containing traffic lights, they need to correctly distinguish and depict the colors of traffic lights in the video. This method can be performed by a traffic light coloring device, which can be implemented by software and / or hardware and can be configured in a traffic light coloring device. Optionally, the traffic light coloring device can be an electronic device, such as a laptop, desktop computer, or smart tablet, which is not limited in this embodiment of the present application.

[0029] like Figure 1 As shown, a traffic light coloring method provided in an embodiment of the present application specifically includes the following steps:

[0030] S101 : Obtain a set of image frames to be processed corresponding to a video stream to be processed, perform color recognition conversion on each image frame in the set of image frames to be processed, and generate a set of auxiliary image frames corresponding to the set of image frames to be processed.

[0031] The lighted pixels in each auxiliary image frame in the auxiliary image frame set are green pixels or preset non-green pixels.

[0032] In this embodiment, the video stream to be processed can be specifically understood as video data captured by video capture devices included in intelligent platforms such as checkpoints, electronic police stations, and smart transportation platforms. Generally, the video stream to be processed is video captured by video equipment at intersections to determine whether vehicles at the intersection have committed traffic violations. Since traffic violations include running red lights, the traffic lights at the intersection are typically captured in the video stream to be processed. The image frame set to be processed can be specifically understood as all video frames in the video stream to be processed, or a collection of multiple video frames obtained by extracting frames from the video stream to be processed. Each video frame in the collection can be considered to include a traffic light, requiring classification of the traffic light's lighting status and color rendering of the illuminated light. It is understood that the video stream to be processed can be a video stream from a historical time period or a video stream acquired in real time. For real-time processing scenarios, the image frame to be processed corresponding to the current moment is the image frame in the image frame set to be processed that currently requires color recognition conversion. In other words, the corresponding auxiliary image frame is obtained by performing real-time color recognition conversion on the image frame to be processed at the current moment, acquired in real time.

[0033] In this embodiment, the auxiliary image frame can be specifically understood as the image frame to be processed after color conversion is completed. The lit pixels can be specifically understood as the pixels corresponding to the lit part of the traffic light in the image frame to be processed and the auxiliary image frame.

[0034] Specifically, when the video stream to be processed is obtained, each video frame in the video stream to be processed is extracted according to actual needs to obtain a set of image frames to be processed corresponding to the video stream to be processed. For each image frame to be processed in the set of image frames to be processed, since when the traffic light is on, it can be considered that its lit part is in a state that is easier to distinguish in the image frame, it is possible to determine the pixels in the image frame to be processed that belong to the traffic light and have been lit, and use them as lit pixels and identify the color of each lit pixel. Because the yellow and red lights in the traffic lights are easily affected by the environmental scene and light when they are lit and are difficult to distinguish, and the green light is less difficult to identify when it is lit than the red and yellow lights, so in the embodiment of the present application, when performing color recognition on each lit pixel, only whether the lit pixel is displayed as green is distinguished. If it is determined that the displayed color is not green, the illuminated pixels are uniformly converted to a preset non-green color that contrasts strongly with green and the background color. Otherwise, the illuminated pixel color remains unchanged or is converted to a uniform green pixel value. The image obtained after completing the illuminated pixel color conversion is determined as the auxiliary image frame corresponding to the image frame to be processed. In other words, an auxiliary image frame is obtained in which the illuminated pixels in the traffic light area can only be represented as green pixels or preset non-green pixels. After obtaining the auxiliary image frames that correspond one-to-one with each image frame to be processed, the auxiliary image frame set consisting of each auxiliary image frame can also be considered to have the same correspondence relationship with the image frame set to be processed.

[0035] Optionally, the color of each lit pixel in the image frame to be processed can be determined by identifying the HSV color of each pixel in the image frame to be processed. When the lit pixel is determined to be green, each lit pixel can be assigned a value of [0, 255, 0]. When the lit pixel is determined to be non-green, each lit pixel can be assigned a value of [255, 0, 0]. The values ​​assigned to each lit pixel here are for illustration purposes only, and other non-green pixel values ​​can also be assigned. The HSV color reference range is shown in the following table:

[0036]

[0037]

[0038] In an embodiment of the present application, by performing color recognition conversion on each image frame to be processed in the set of image frames to be processed, the traffic light part is divided into an easily distinguishable green color and a preset non-green color, thereby improving the distinction of the traffic light display color under different scenes and different light conditions, so as to facilitate the subsequent preliminary traffic light coloring of the image frames to be processed, thereby improving the accuracy of the preliminary coloring.

[0039] S102 : Determine each to-be-processed image frame and the corresponding auxiliary image frame as an to-be-processed image frame group according to the corresponding relationship, input each to-be-processed image frame group into a pre-trained traffic light coloring model, and determine a preliminary coloring image frame set.

[0040] In this embodiment, the traffic light coloring model can be specifically understood as a neural network model that performs feature extraction and classification on the image combination input thereto, and outputs the confidence level of each pixel in the image combination corresponding to each category that the model can divide. Optionally, in the embodiment of the present application, the categories divided by the traffic light coloring model may include background, red light, and green light. The preliminary coloring image frame can be specifically understood as an image frame that contains pixel category information corresponding to each pixel in the image frame group to be processed, wherein the size of the preliminary coloring image frame is the same as that of the image frame to be processed and the auxiliary image frame, and the pixel category information corresponding to each pixel in the preliminary coloring image frame is the pixel category information of the pixel at the same position in the image frame to be processed and the auxiliary image frame.

[0041] Specifically, based on the correspondence between the image frame to be processed and the auxiliary image frame, the two corresponding image frames to be processed and the auxiliary image frame are grouped as a group of image frames to be processed. That is, the number of groups of image frames to be processed is the same as the number of image frames to be processed in the set of image frames to be processed. Each group of image frames to be processed is input into a pre-trained traffic light colorization model. The traffic light colorization model is used to extract corresponding features and classify the two image frames in the group of image frames to be processed. The confidence level of each pixel in the group of image frames to be processed for each category categorized by the traffic light colorization model is output. Based on the confidence level, the pixel category information corresponding to each pixel is determined. An image frame formed by arranging the pixels containing the pixel category information in corresponding positions is determined as a preliminary colorization image frame. A preliminary colorization image frame set is constructed based on the correspondence between each group of image frames to be processed and the set of image frames to be processed, thereby obtaining a preliminary colorization image frame set corresponding to the set of image frames to be processed.

[0042] S103 , performing color restoration on each of the preliminary colorized image frames based on pixel category information in each of the preliminary colorized image frames and traffic light position information in the image frames to be processed corresponding to each of the preliminary colorized image frames, to generate a set of colorized image frames corresponding to the set of image frames to be processed.

[0043] In this embodiment, pixel category information can be specifically understood as the category information of the subject to which the pixel in the processed image frame belongs, determined after classification by the traffic light color rendering model. This is also one of the categories that can be classified by the traffic light color rendering model. In this embodiment of the present application, the pixel category information can be one of background, red light, and green light. Traffic light location information can be specifically understood as the image area corresponding to the traffic light in the processed image frame, or can be understood as the set of pixels in the processed image frame that are primarily the traffic light. The rendered image frame can be specifically understood as the image frame that has completed the restoration of the color of the traffic light.

[0044] Specifically, for each preliminary painted image frame in the set of preliminary painted image frames, a target analysis is first performed on the corresponding image to be processed to determine multiple pixels in the preliminary painted image frame corresponding to the traffic light location. Since each pixel in the preliminary painted image frame contains pixel category information, the pixel category information of the multiple pixels corresponding to the traffic light location can be used to determine the traffic light color of the preliminary painted image frame before color restoration. Since traffic light illumination follows the order "green -> yellow -> red -> green," with yellow and red both identified as red light pixel category information due to image preprocessing, and there is temporal continuity between the image frames to be processed, when the pixel category information is determined to be red, the traffic light color of the currently processed preliminary painted image frame can be determined based on the restoration results of the preliminary painted image frames preceding the preliminary painted image frame, combined with the logical order of traffic light illumination. Color restoration is then performed on the preliminary painted image frame based on the determination result, resulting in a painted image frame corresponding to the preliminary painted image frame. After completing the processing of all preliminary colorized image frames, the obtained colorized image frames are put into a set in the same order as the image frames in the set of image frames to be processed, thereby generating a colorized image frame set corresponding to the set of image frames to be processed.

[0045] The technical solution of this embodiment preprocesses the set of unprocessed image frames corresponding to the acquired video stream to be processed, so that the illuminated portions of the traffic lights in each set of unprocessed image frames can be processed to a preset non-green color or green based on the actual detected color. That is, the pixels corresponding to the easily confused yellow and red lights are all processed to preset non-green pixels, so that the illuminated colors displayed by the traffic lights in the processed auxiliary image frame sets are two easily distinguishable pixel colors. The unprocessed image frames and the corresponding auxiliary image frames are then grouped and input into a pre-trained traffic light colorization model for processing. The set of images are compared with each other to determine the pixel category information corresponding to each pixel therein, and the images after the pixel category information is determined are determined as preliminary colorization image frames. Based on the pixel category information in each preliminary colorization image frame, the illuminated color of the traffic light and the location information of the illuminated pixels therein can be determined. Then, the illuminated color of the traffic light in the preliminary colorization image frames can be restored according to the red, yellow, and green lighting logic of the traffic light, to obtain a final colorization image frame set corresponding to the set of unprocessed image frames. Since the pre-trained traffic light coloring model only needs to distinguish between preset non-green colors and green with obvious distinction, it is less affected by the scene in which the traffic light is located. Therefore, when training the traffic light coloring model, there is no need to provide different templates for different scenes to train it, which reduces the amount of data required for training and avoids traffic light coloring errors caused by insufficient scene templates. The amount of data calculation is reduced during use. After accurate category division, the lighting color of the traffic light in the preliminary coloring image frame is restored according to the lighting logic of the red, yellow and green lights in the traffic light, so that the yellow light in the traffic light can be accurately restored, thereby improving the accuracy of traffic light coloring.

[0046] Figure 2A flowchart of a traffic light coloring method is provided for one embodiment of the present application. The technical solution of this embodiment of the present application is further optimized based on the aforementioned optional technical solutions. By performing target recognition on a to-be-processed image frame to obtain lit pixels therein, the to-be-processed image frame is then color-recognized, and the lit pixels are reassigned based on the color recognition results. Pixels corresponding to the lit portion of the traffic light are converted into easily distinguishable preset non-green or green pixels to obtain corresponding auxiliary image frames. The to-be-processed image frame and the corresponding auxiliary image frame are then combined into a to-be-processed image frame set, which is then input into a traffic light coloring model comprising at least a feature extraction layer and a classification layer for multi-scale feature extraction and category classification. The result is an image of the same size as the to-be-processed image frame set, in which each pixel has at least one candidate pixel category with a confidence score. Each pixel in the output image is processed based on the confidence score to obtain a preliminary coloring image frame with clear pixel category information for each pixel. Since the order of the frames in the preliminary coloring image frame set is the same as that in the to-be-processed image frame set, it can be understood that the two have the same temporal order, and the lighting of the traffic light also has its corresponding temporal logic. Therefore, each preliminary color-decorated image frame can be processed as the current image frame in sequence. Based on the pixel category information corresponding to the set of traffic light pixels in the current image frame, it is determined whether the traffic light is on and the color of the light when it is on. If the traffic light is on, the color that the actual traffic light should display in the current image frame is determined based on the light color and the cumulative number of yellow light frames, the cumulative number of green light frames, and the traffic light position information before the current image frame, thereby achieving color restoration of the current image frame. By reassigning the color of the traffic light in the image frame to be processed to a preset non-green color or green, the pixel category division of each pixel in the preliminary color-decorated image frame is ensured to be clear. Then, based on the color change logic of the traffic light, the cumulative number of yellow light frames, the cumulative number of green light frames, and the traffic light position information, the color of each preliminary color-decorated image frame is restored, ensuring the accuracy of the yellow light restoration and avoiding the confusion of yellow and red lights caused by insufficient model templates or environmental factors when directly identifying the traffic light color.

[0047] like Figure 2 As shown, a traffic light coloring method provided in an embodiment of the present application specifically includes the following steps:

[0048] S201 , obtaining a set of image frames to be processed corresponding to a video stream to be processed, performing target recognition on each image frame to be processed in the set of image frames to be processed, and determining lit pixels in the image frame to be processed.

[0049] Specifically, when a video stream to be processed is acquired, each video frame in the video stream to be processed is extracted according to actual needs to obtain a set of image frames to be processed corresponding to the video stream to be processed. Each image frame in the set of image frames to be processed can be considered an image containing a traffic light. Because a traffic light differs significantly from its surroundings when lit, and the lit area of ​​a traffic light is generally a fixed shape, target recognition can be performed on the image frame to be processed based on the hue, saturation, brightness, and shape of the lit area in the lit state. The lit area is then determined, and the pixels corresponding to the lit area are identified as lit pixels.

[0050] S202 , performing color recognition on the lit pixel, and reassigning the lit pixel to a green pixel or a preset non-green pixel according to the color recognition result.

[0051] Specifically, by identifying the HSV color of each illuminated pixel, since the hue, saturation, and brightness are within a certain range, the illuminated pixel can be considered to appear green. However, the color displayed during actual display may be varying shades of green. To facilitate the classification of the subsequent traffic light color model, the color of each illuminated pixel can be reassigned so that the displayed color of each illuminated pixel is the same and easy to distinguish. That is, when the HSV color of each illuminated pixel is within the green range, each illuminated pixel is uniformly reassigned to a green pixel. When the HSV color of each illuminated pixel is not within the green range, it can be considered that the traffic light corresponding to each illuminated pixel is yellow or red. In this case, each illuminated pixel is uniformly reassigned to a preset non-green pixel that is significantly different from the green and background colors. In other words, the yellow light and the red light are collectively treated as the preset non-green color for subsequent identification. In an optional embodiment, each illuminated pixel that is not within the green range can be uniformly reassigned to a red pixel. In other words, the yellow light and the red light are treated as red lights for subsequent identification.

[0052] S203 : Determine the image frame to be processed after pixel re-assignment as the auxiliary image frame corresponding to the image frame to be processed, and determine the set consisting of the auxiliary image frames as the auxiliary image frame set corresponding to the image frame set to be processed.

[0053] Specifically, the image frame to be processed after pixel reassignment is determined as the auxiliary image frame corresponding to the image frame to be processed, and the auxiliary image frames are arranged into a set in the same order as the image frame to be processed, so as to obtain the auxiliary image frame set corresponding to the image frame set to be processed.

[0054] S204 : Determine each to-be-processed image frame and the corresponding auxiliary image frame as a to-be-processed image frame group according to the corresponding relationship.

[0055] S205 , for a to-be-processed image frame group, input the to-be-processed image frame group into a feature extraction layer in a pre-trained traffic light coloring model to perform multi-scale feature extraction, and determine an intermediate feature map that corresponds to the to-be-processed image frame group and has the same size.

[0056] In this embodiment, the feature extraction layer can be specifically understood as a set of neural network layers for performing multi-scale feature extraction and fusion on the input image. Optionally, the feature extraction layer may include a convolutional layer, a batch normalization (BN) layer, and a pooling layer.

[0057] Specifically, each image frame group to be processed is input into the pre-trained traffic light coloring model for processing. Taking one of the image frame groups to be processed as an example, when input into the traffic light coloring model, the feature extraction layer in the traffic light coloring model will first extract image features of the image frame group to be processed. When performing the feature extraction task, the image frame group to be processed can be downsampled to obtain feature maps of different scales, and then upsampled and merged in turn, and the extracted multi-scale feature maps are fused to obtain an intermediate feature map corresponding to the image frame group to be processed and with the same size.

[0058] Exemplarily, the feature map after four downsamplings can be upsampled once, the result obtained can be merged with the result of downsampling three times, upsampling can be performed again, the upsampling result can be merged with the result of downsampling twice, upsampling can be performed again, the upsampling result can be merged with the result of downsampling once, and upsampling can be performed again to obtain a feature map with the same size as the input image.

[0059] Optionally, the residual structure of the feature extraction layer is a convolution residual structure and an identity mapping residual structure.

[0060] Specifically, in the embodiment of the present application, the residual structure of the feature extraction layer is set to include a 1*1 convolution residual structure and an identity residual structure. Since the residual structure has multiple branches, it is equivalent to adding multiple gradient flow paths to the feature extraction layer network, and all network layers can be converted into 3*3 convolutions through node fusion to facilitate model deployment and acceleration. Since the 1*1 convolution is equivalent to a special 3*3 convolution with many 0s in the convolution kernel, and the identity mapping is a special 1*1 convolution with the unit matrix as the convolution kernel, the embodiment of the present application can relatively easily realize the residual structure in which the convolution residual structure and the identity mapping residual structure coexist in the model inference stage, thereby realizing model integration.

[0061] S206: Input the intermediate feature map into the classification layer in the traffic light color model for category division, and determine at least one candidate pixel category corresponding to each pixel in the output image, as well as the category confidence of the candidate pixel category.

[0062] In this embodiment, the classification layer can be specifically understood as a neural network layer for classifying the input feature image and outputting the corresponding probabilities of each category. In the embodiment of the present application, the classification layer can be a softmax layer, and its corresponding categories are background, red light, and green light. The candidate pixel category can be specifically understood as the pixel category with a non-zero probability corresponding to each pixel in the output image after the intermediate feature map is classified by the classification layer. For example, for a pixel in the output image, the classification layer outputs a background probability of 80%, a red light probability of 20%, and a green light probability of 0%, then the background and red light can be considered as the candidate pixel categories of the pixel. The category confidence can be specifically understood as the probability value that the probability of the candidate pixel category is true.

[0063] Specifically, the intermediate feature map is used as the input of the classification layer in the traffic light coloring model, and the classification layer is used to classify each pixel in the intermediate feature map into categories, and the probabilities of multiple pixel categories pre-trained in the classification layer are determined. The pixel categories with non-zero probabilities are used as candidate pixel categories corresponding to the pixels, and the category confidence of each candidate pixel category is determined. The image containing the candidate pixel category and category confidence of each pixel is used as the output image of the traffic light coloring model.

[0064] S207 : Determine the candidate pixel category with the largest category confidence as the pixel category information of the pixel, and determine the output image after the pixel category information determination is completed as the preliminary coloring image frame corresponding to the image frame group to be processed.

[0065] Specifically, for each pixel in the output image, the probability of each candidate pixel category and the confidence of each category in the pixel are compared, and the candidate pixel category with the largest probability and the largest category confidence is determined as the pixel category information of the pixel. After the pixel category information of each pixel in the output image is confirmed, the output image in which each pixel contains only one corresponding pixel category information is determined as the preliminary color image frame corresponding to the image frame group to be processed.

[0066] S208 : Determine a set of preliminary colorized image frames corresponding to each to-be-processed image frame group as a preliminary colorized image frame set.

[0067] Specifically, by arranging the preliminary color image frames into a set in the same order as the image frame group to be processed, a set of auxiliary image frames corresponding to the image frame group to be processed can be obtained. This can be understood as the arrangement order of the preliminary color image frames in the preliminary color image frame set being the same as the acquisition order of the image frames to be processed in the video stream to be processed, so as to facilitate the subsequent color restoration of the lighting conditions of the traffic lights in the preliminary color image frames according to the lighting order of the traffic lights.

[0068] S209 , taking each of the preliminary colorized image frames as a current image frame, and determining a set of traffic light pixels in the current image frame according to the traffic light position information in the image frame to be processed corresponding to the current image frame.

[0069] Specifically, each preliminary colorized image frame is processed sequentially as the current image frame based on the order in which the preliminary colorized image frames are arranged in the set. Because the size of the current image frame is consistent with that of the image frame to be processed, it can be assumed that there is a one-to-one correspondence between the positions of each pixel in the current image frame and the positions of each pixel in the image frame to be processed. Therefore, target recognition can be performed on the image frame to be processed corresponding to the current image frame to determine the traffic light position information corresponding to the traffic light in the image frame to be processed. This is the set of pixels corresponding to the traffic light position in the image frame to be processed, and the set of traffic light pixels in the current image frame can be reversely determined using this information. Alternatively, the current image frame can be a preliminary colorized image frame corresponding to the image frame to be processed, acquired in real time.

[0070] S210: Determine the set of lit pixels in the current image frame and the lighting colors of the set of lit pixels based on the pixel category information corresponding to each of the traffic light pixels in the set of traffic light pixels. If the set of lit pixels exists, execute S211; otherwise, execute S212.

[0071] Specifically, since the illuminated area of ​​a traffic light must be within the pixel range corresponding to the traffic light, the pixel category information of each traffic light pixel in the traffic light pixel set can be distinguished to determine whether the traffic light has been completely illuminated, the illuminated position, and the color of the illuminated position. Since the pixel category information in the embodiment of the present application can include background, red light, and green light, when the traffic light is not illuminated, the pixel category information of each traffic light pixel should all be background category. At this time, it can be considered that there are no illuminated pixels in the traffic light pixels, and it can be considered that the illuminated pixel set in the current image frame does not exist. At this time, S212 is executed; when the traffic light is illuminated, a portion of the traffic light pixels must have corresponding pixel category information of the red light category or the green light category. At this time, each traffic light pixel with pixel category information of the red light category or the green light category is determined as a illuminated pixel, the set of each illuminated pixel is determined as a illuminated pixel set, and the color in the pixel category information corresponding to the illuminated pixel is determined as the illuminated color. At this time, S211 can be executed to restore the color actually displayed by the traffic light in the current image frame.

[0072] Optionally, the number of traffic light pixels in the traffic light pixel set can be counted, the number of lit pixels corresponding to different pixel category information in the lit pixel set can be determined, and the proportion of the number of lit pixels in the number of traffic light pixels can be determined. If the ratio exceeds a preset ratio threshold, the color of the corresponding pixel category information can be determined as the lit color of the lit pixel set.

[0073] S211 , performing color restoration on the current image frame according to the lighting color, the cumulative number of yellow light frames, the cumulative number of green light frames, the traffic light position information, and the image frame to be processed corresponding to the current image frame.

[0074] In this embodiment, the accumulated yellow light frames can be specifically understood as the number of frames in which the traffic light is continuously identified as yellow in multiple pending image frames adjacent to the current image frame. It is understood that when the traffic light is identified as red or green, the accumulated yellow light frames are reset to zero. That is, the accumulated yellow light frames corresponding to the current image frame can be used to indicate whether the traffic light was yellow at the time adjacent to the current image frame, as well as the duration of the yellow light. The accumulated green light frames can be specifically understood as the number of frames in which the traffic light is continuously identified as green in multiple pending image frames adjacent to the current image frame.

[0075] Specifically, it is possible to determine whether the lit pixel set in the current image frame is likely to display a yellow light based on the lit color. When it is determined that the current image frame is likely to display a yellow light, the corresponding lit pixel set of the current image frame is further distinguished as to whether it should display a red light or a yellow light based on the cumulative number of yellow light frames, the cumulative number of green light frames, the traffic light position information, and the traffic light lighting logic. The lit pixels are restored in color based on the determined traffic light color that the lit pixel set should display, and the pixels other than the lit pixel set in the current image frame are restored based on the image frame to be processed corresponding to the current image frame.

[0076] Optionally, the embodiment of the present application provides a specific implementation method for color restoration of the current image frame based on the lighting color, the cumulative number of yellow light frames, the cumulative number of green light frames, the traffic light position information, and the image frame to be processed corresponding to the current image frame, which can be divided into the following four cases:

[0077] 1) If the lit color is green, keep the color of the lit pixel set unchanged, restore the pixels in the current image frame except the lit pixel set to the color of the image frame to be processed corresponding to the current image frame, obtain the colored image frame corresponding to the current image frame, set the cumulative number of yellow light frames to zero, and increase the cumulative number of green light frames by one.

[0078] Specifically, if the illuminated color of the illuminated pixel set is identified as green, the illuminated traffic light in the current image frame can be directly considered to be a green light, and the illuminated range is the range where the illuminated pixel set is located. Since each pixel in the illuminated pixel set has been reassigned to a green pixel during processing, the color of the illuminated pixel set can be kept unchanged. At the same time, since each pixel in the current image frame other than the illuminated pixel set can be understood as a background pixel, that is, a pixel that does not require color modification, this part of the background pixels can be filled with the color of the corresponding pixel in the image frame to be processed corresponding to the current image frame, so that the expression of this part of the background pixels is the same as that in the image frame to be processed, and then the color-drawing image frame corresponding to the current image frame can be obtained. At this time, since the traffic light in the current image frame is green, the cumulative number of yellow light frames will be reset to zero, and the cumulative number of green light frames will be increased by one, so that the preliminary color-drawing image frame after the current image frame can restore the color of the traffic light.

[0079] 2) If the lit color is a preset non-green color and the cumulative number of green light frames is greater than zero, the color of the lit pixel set is restored to yellow, and the pixels in the current image frame except the lit pixel set are restored to the color of the image frame to be processed corresponding to the current image frame, to obtain the colored image frame corresponding to the current image frame, and the cumulative number of yellow light frames is increased by one, and the cumulative number of green light frames is set to zero.

[0080] Specifically, if the illuminated color of the illuminated pixel set is identified as a preset non-green color, and the cumulative number of green light frames is greater than zero, then it can be considered that the traffic light that was illuminated in the previous frame to be processed adjacent to the current image frame was a green light, and the current image frame is the first frame after the green light. Based on the color conversion logic of the traffic light, it can be known that the yellow light should be illuminated after the green light, so it can be considered that the illuminated traffic light in the current image frame is a yellow light, and the illuminated range is the range where the illuminated pixel set is located. At this time, each pixel in the illuminated pixel set can be reassigned to a yellow pixel. At the same time, since all pixels in the current image frame except the illuminated pixel set can be understood as background pixels, that is, pixels that do not need to be modified in color, this part of the background pixels can be filled with the color of the corresponding pixels in the image frame to be processed corresponding to the current image frame, so that the expression of this part of the background pixels is the same as that in the image frame to be processed, and then the color image frame corresponding to the current image frame can be obtained. At this time, since the traffic light in the current image frame is yellow, the current image frame can be considered as the first frame in which the yellow light is lit. Therefore, the cumulative number of green light frames can be set to zero, and the cumulative number of yellow light frames can be increased by one, so that the traffic light color can be restored in the preliminary color-drawing image frame after the current image frame.

[0081] 3) If the lit color is a preset non-green color and the cumulative number of yellow light frames is equal to one, the color of the lit pixel set is restored to yellow, and the pixels in the current image frame except the lit pixel set are restored to the color of the image frame to be processed corresponding to the current image frame, to obtain the colored image frame corresponding to the current image frame, and the cumulative number of yellow light frames is increased by one, and the initial yellow light position information is determined based on the previous image frame of the current image frame and the previous traffic light position information.

[0082] In this embodiment, the previous traffic light position information can be specifically understood as the position range of the traffic light in the previous colored image frame at the time corresponding to the current image frame. The initial yellow light position information can be specifically understood as the relative position information of the yellow light within the overall range of the traffic light in the first yellow light image frame of a traffic light lighting cycle.

[0083] Specifically, if the illuminated color of the illuminated pixel set is identified as a preset non-green color and the cumulative number of yellow light frames is equal to one, then the illuminated traffic light in the current image frame can be considered to be a yellow light, the illuminated range is the range within which the illuminated pixel set resides, and the previous color-decorated image frame at the corresponding moment of the current image frame is the first frame in which the yellow light is illuminated in this traffic light lighting cycle. At this time, each pixel in the illuminated pixel set can be reassigned to a yellow pixel. At the same time, since all pixels in the current image frame other than the illuminated pixel set can be understood as background pixels, that is, pixels that do not require color modification, these background pixels can be filled with the color of the corresponding pixels in the image frame to be processed corresponding to the current image frame, so that the representation of these background pixels is the same as in the image frame to be processed, thereby obtaining the color-decorated image frame corresponding to the current image frame. Since the cumulative number of green light frames must be zero at this time, there is no need to reset the cumulative number of green light frames to zero. Since the traffic light in the current image frame is a yellow light, the cumulative number of yellow light frames needs to be increased by one. Since the current image frame is the second frame with the yellow light on, the relative position relationship between the lit light and the entire traffic light in the previous color image frame can be determined based on the set of lit pixels in the previous color image frame and the previous traffic light position information, and this can be used as the initial yellow light position information to facilitate traffic light color restoration in the preliminary color image frame after the current image frame.

[0084] 4) If the lit color is a preset non-green color and the cumulative number of yellow light frames is greater than one, the current light position information is determined based on the lit pixel set and the traffic light position information, and the current image frame is restored in color based on the current light position information, the initial yellow light position information, and the image frame to be processed corresponding to the current image frame.

[0085] Specifically, if the illuminated color of the illuminated pixel set is identified as non-green and the cumulative number of yellow light frames is greater than one, it can be considered that the illuminated traffic light in the current image frame is likely a yellow light or a red light. In this case, the relative position relationship between the illuminated light and the entire traffic light in the current image frame can be determined based on the illuminated pixel set and the traffic light position information, and this can be used as the current light position information. Because the overall relative positions of yellow lights and red lights within the traffic light are different, there is generally a certain distance difference between the two. Therefore, the current light position information can be compared with the initial yellow light position information. A position offset ratio can be determined based on the current light position information and the initial yellow light position information. Then, based on the position offset ratio, it can be determined whether the illuminated light in the current image frame is at the position in the traffic light where the yellow light should be illuminated. Then, based on the judgment result and the image frame to be processed corresponding to the current image frame, the color of the current image frame can be restored.

[0086] Optionally, color restoration is performed on the current image frame based on the current light position information, the initial yellow light position information, and the image frame to be processed corresponding to the current image frame. Specifically, the color restoration can be divided into the following two cases:

[0087] A) If the position offset ratio is less than a preset offset ratio threshold, the color of the set of illuminated pixels is restored to yellow, and the pixels in the current image frame other than the set of illuminated pixels are restored to the color of the image frame to be processed corresponding to the current image frame, thereby obtaining a colored image frame corresponding to the current image frame, and the cumulative number of yellow light frames is increased by one.

[0088] In this embodiment, the preset offset ratio threshold may be specifically understood as a position offset ratio between a red light and a yellow light that is predetermined based on the distribution positions of lights of different colors in the traffic light.

[0089] Specifically, when the position offset ratio is less than a preset offset ratio threshold, it can be considered that the position of the lit light in the current image frame is not much different from the position of the yellow traffic light. In this case, the lit traffic light in the current image frame can be considered to be a yellow light, and the lighting range is the range within which the lit pixel set is located. At this time, each pixel in the lit pixel set can be reassigned to a yellow pixel. At the same time, since all pixels in the current image frame other than the lit pixel set can be understood as background pixels, that is, pixels that do not require color modification, these background pixels can be filled with the color of the corresponding pixels in the image frame to be processed corresponding to the current image frame, so that the expression of these background pixels is the same as that in the image frame to be processed, and thus a color-decorated image frame corresponding to the current image frame can be obtained. Since the traffic light in the current image frame is a yellow light, the cumulative number of yellow light frames needs to be increased by one to facilitate traffic light color restoration in the preliminary color-decorated image frame after the current image frame.

[0090] B) If the position offset ratio is greater than or equal to the preset offset ratio threshold, the illuminated color is restored to red, and the pixels in the current image frame except the set of illuminated pixels are restored to the color of the image frame to be processed corresponding to the current image frame, thereby obtaining the colored image frame corresponding to the current image frame, and the cumulative number of yellow light frames is set to zero.

[0091] Specifically, when the position offset ratio is greater than or equal to a preset offset ratio threshold, it can be considered that the position of the lit light in the current image frame deviates significantly from the position of the yellow light. In this case, the lit light in the current image frame is considered red, and the lit range is the range within the lit pixel set. In this case, each pixel in the lit pixel set can be reassigned to red. Furthermore, since all pixels in the current image frame other than the lit pixel set can be considered background pixels, i.e., pixels that do not require color modification, these background pixels can be filled with the color of the corresponding pixels in the pending image frame corresponding to the current image frame, making the representation of these background pixels identical to that in the pending image frame. This results in a color-coded image frame corresponding to the current image frame. Since the traffic light in the current image frame is red, the accumulated yellow light frames will be reset to zero. Since the accumulated green light frames are already zero, they do not need to be reset to zero again. This operation facilitates traffic light color restoration in the preliminary color-coded image frame following the current image frame.

[0092] S212: Restore each pixel in the current image frame to the color of the image frame to be processed corresponding to the current image frame to obtain a colored image frame corresponding to the current image frame, and increase the cumulative number of yellow light frames by one, and set the cumulative number of green light frames to zero.

[0093] Specifically, since the traffic light may not be lit as a whole during the flashing yellow portion of the traffic light during operation, when it is determined that there is no set of lit pixels in the current image frame, the current image frame is considered to be a flashing yellow light. At this time, each pixel in the current image frame can be understood as a background pixel, that is, a pixel that does not require color modification. Therefore, the pixels in the current image frame can be filled with the color of the corresponding pixel in the image frame to be processed corresponding to the current image frame, thereby obtaining a color-decorated image frame corresponding to the current image frame. At this time, since the current image frame corresponds to a scene where the traffic light is lit in yellow, the cumulative number of yellow light frames needs to be increased by one. Since it is uncertain whether the current image frame is the first frame after the green light, the cumulative number of green light frames can be set to zero to prevent an incorrect cumulative number of green light frames from appearing when the traffic light color is restored in the preliminary color-decorated image frame after the current image frame.

[0094] Furthermore, before obtaining the set of image frames to be processed corresponding to the video stream to be processed, the traffic light coloring model needs to be trained. The specific training may include the following steps:

[0095] 1) Acquire at least one historical image frame, and perform color recognition and conversion on the historical image frame so that the lit pixels in the historical image frame are reassigned to green pixels or non-green pixels, thereby generating an auxiliary historical image frame corresponding to the historical image frame.

[0096] In this embodiment, the historical image frame can be specifically understood as an image captured by a camera device during a historical period and obtained by a checkpoint electric police or a smart traffic platform. The auxiliary historical image frame can be specifically understood as a historical image frame after color conversion.

[0097] It is understandable that in order to ensure that the traffic light color model obtained through training achieves the same results as the training objectives in actual application, the processing of historical image frames when constructing training samples should be the same as the processing of image frames to be processed. Therefore, the process of color recognition and conversion of historical image frames will not be described in detail in this step.

[0098] 2) Obtain pixel category information of each pixel in the auxiliary historical image frame, and use the pixel category information as a label for the historical image frame and the auxiliary historical image frame to construct a training sample set.

[0099] The pixel category information includes red light category, green light category and background category.

[0100] Specifically, for the converted historical image frames, to ensure training accuracy, each pixel in the historical image frame can be manually labeled as belonging to the red light category, green light category, or background category. This can also obtain the pixel category information corresponding to each pixel in the auxiliary historical image frame. At the same time, since the trained traffic light colorization model is expected to output the pixel category information corresponding to each pixel in the input image, that is, the pixel category information can be understood as the label during the model training process, the pixel category information corresponding to each pixel in the historical image frame can be used as the label for the historical image frame and the auxiliary historical image frame. The historical image frame, the auxiliary historical image frame, and the label are combined to form a training sample.

[0101] It is understandable that in order to improve the training effect of the traffic light color model, multiple training samples can be constructed to train the model. The construction method of each training sample is the same as the above steps 1) and 2), and the set of multiple training samples generated is the training sample set.

[0102] In an embodiment of the present application, the labels in the training sample set only include red light category, green light category and background category, which means that the traffic light coloring model only needs to be trained for classification of the above three categories with obvious distinctions, thereby improving the accuracy of the trained model. During training, a group of corresponding historical image frames and auxiliary historical image frames are trained simultaneously, thereby improving the richness of feature extraction during the training process and making the model training results more suitable for pixel category division.

[0103] 3) Input the training sample set into the initial traffic light coloring model, and train the initial traffic light coloring model based on the constructed loss function until the preset convergence condition is met to obtain the traffic light coloring model.

[0104] In this embodiment, the initial traffic light color model can be specifically understood as a traffic light color model without weight adjustment, and its architecture is completely consistent with the traffic light color model. The preset convergence condition can be specifically understood as a condition set based on actual conditions for determining whether the traffic light color model has converged, that is, a condition for determining when to stop training the traffic light color model. Optionally, the preset convergence condition may include reaching a maximum number of iterations or the loss function value falling below a set loss function threshold, etc., which are not limited in this embodiment of the application.

[0105] Specifically, the historical image frames and auxiliary historical image frames in the training sample set are input into the initial traffic light coloring model as a set of input information. A loss function is constructed based on the error between the output of the initial traffic light coloring model and the labels in the training sample set. The weights of each neural network layer in the initial traffic light coloring model are adjusted based on the loss function until a preset convergence condition is met to obtain a traffic light coloring model that can be put into use.

[0106] The technical solution of this embodiment performs target recognition on the image frame to be processed to obtain the lit pixels therein. Based on the color recognition results of the lit pixels, the lit pixels are then reassigned. The pixels corresponding to the lit portion of the traffic light are converted into easily distinguishable non-green or green pixels to obtain the corresponding auxiliary image frame. The image frame to be processed and the corresponding auxiliary image frame form a set of image frames to be processed and input into the traffic light color model for processing. The result is an image with the same size as the images in the set of image frames to be processed, and each pixel in the output image has at least one candidate pixel category with a confidence level. By processing each pixel in the output image based on the confidence level, a preliminary color image frame with clear pixel category information for each pixel is obtained. Since the order of the frames in the preliminary color image frame set is the same as that in the set of image frames to be processed, it can be understood that the time sequence of the two is the same, and the lighting of the traffic light also has its corresponding time logic. Therefore, each preliminary color-decorated image frame can be processed as the current image frame in sequence. Based on the pixel category information corresponding to the set of traffic light pixels in the current image frame, it is determined whether the traffic light is on and the color of the light when it is on. If the traffic light is on, the color that the actual traffic light should display in the current image frame is determined based on the light color and the cumulative number of yellow light frames, the cumulative number of green light frames, and the traffic light position information before the current image frame, thereby achieving color restoration of the current image frame. By reassigning the color of the traffic light in the image frame to be processed to a preset non-green color or green, the pixel category division of each pixel in the preliminary color-decorated image frame is ensured to be clear. Then, based on the color change logic of the traffic light, the cumulative number of yellow light frames, the cumulative number of green light frames, and the traffic light position information, the color of each preliminary color-decorated image frame is restored, ensuring the accuracy of the yellow light restoration and avoiding the confusion of yellow and red lights caused by insufficient model templates or environmental factors when directly identifying the traffic light color.

[0107] Figure 3 A schematic diagram of the structure of a traffic light coloring device provided in an embodiment of the present application is shown in FIG. Figure 3 As shown, the traffic light coloring device includes an image frame acquisition module 31 , a preliminary coloring module 32 and a coloring image generation module 33 .

[0108] Among them, the image frame acquisition module 31 is used to obtain a set of image frames to be processed corresponding to the video stream to be processed, and perform color recognition conversion on each image frame to be processed in the set of image frames to be processed to generate a set of auxiliary image frames corresponding to the set of image frames to be processed; wherein the lit pixels in each auxiliary image frame in the set of auxiliary image frames are green pixels or preset non-green pixels; the preliminary coloring module 32 is used to determine each image frame to be processed and the corresponding auxiliary image frame as a set of image frames to be processed based on the corresponding relationship, and input each set of image frames to be processed into a pre-trained traffic light coloring model to determine a set of preliminary coloring image frames; the coloring image generation module 33 is used to perform color restoration on each preliminary coloring image frame based on the pixel category information in each preliminary coloring image frame and the traffic light position information in the image frame to be processed corresponding to each preliminary coloring image frame, to generate a set of coloring image frames corresponding to the set of image frames to be processed.

[0109] The technical solution of the embodiment of the present application pre-processes a set of image frames to be processed corresponding to a video stream to be processed, so that the illuminated portion of the traffic light in each set of image frames to be processed can be processed to a predetermined non-green color or green based on the actual detected color. In other words, the pixels corresponding to the easily confused yellow and red lights are all processed to red pixels. This ensures that the illuminated colors of the traffic lights displayed in the processed auxiliary image frames are easily distinguishable red and green, facilitating the subsequent correct recognition of the traffic light color by the model. The image frames to be processed and the corresponding auxiliary image frames are then grouped and input into a pre-trained traffic light colorization model for processing. The set of images are compared with each other to determine the pixel category information corresponding to each pixel therein. The images with the determined pixel category information are then determined as preliminary colorization image frames. Based on the pixel category information in each preliminary colorization image frame, the illuminated color of the traffic light can be determined. Then, the illuminated color of the traffic light in the preliminary colorization image frames can be restored according to the red, yellow, and green lighting logic of the traffic light, resulting in a final colorization image frame set corresponding to the set of image frames to be processed. Since the pre-trained traffic light coloring model only needs to distinguish between the red and green colors with obvious distinction, it is less affected by the scene in which the traffic light is located. Therefore, when training the traffic light coloring model, there is no need to provide different templates for different scenes to train it, which reduces the amount of data required for training and avoids traffic light coloring errors caused by insufficient scene templates. The amount of data calculation is reduced during use. After accurate category division, the lighting color of the traffic light in the preliminary coloring image frame is restored according to the lighting logic of the red, yellow and green lights in the traffic light, so that the yellow light in the traffic light can be accurately restored, thereby improving the accuracy of traffic light coloring.

[0110] Optionally, the image frame acquisition module 31 includes:

[0111] a lighted pixel determination unit, configured to obtain a set of image frames to be processed corresponding to the video stream to be processed, perform target recognition on each image frame to be processed in the set of image frames to be processed, and determine lighted pixels in the image frame to be processed;

[0112] A pixel re-assignment unit, configured to perform color recognition on the lit pixels and re-assign the lit pixels to green pixels or preset non-green pixels according to the color recognition result;

[0113] The auxiliary frame set determining unit is used to determine the image frame to be processed after pixel reassignment as the auxiliary image frame corresponding to the image frame to be processed, and determine the set composed of the auxiliary image frames as the auxiliary image frame set corresponding to the image frame set to be processed.

[0114] Optionally, the preliminary coloring module 32 includes:

[0115] an image frame group determining unit, configured to determine each to-be-processed image frame and the corresponding auxiliary image frame as an to-be-processed image frame group according to a corresponding relationship;

[0116] A feature extraction unit is configured to input a set of image frames to be processed into a feature extraction layer in a pre-trained traffic light colorization model to perform multi-scale feature extraction, and determine an intermediate feature map corresponding to the set of image frames to be processed and having the same size; wherein the residual structure of the feature extraction layer is a convolution residual structure and an identity mapping residual structure;

[0117] A classification unit is configured to input the intermediate feature map into the classification layer of the traffic light color model for classification, and determine at least one candidate pixel category corresponding to each pixel in the output image, as well as the category confidence of the candidate pixel category;

[0118] a preliminary coloring frame determining unit, configured to determine the candidate pixel category with the largest category confidence as the pixel category information of the pixel, and determine the output image after the pixel category information determination is completed as the preliminary coloring image frame corresponding to the image frame group to be processed;

[0119] The preliminary coloring set determining unit is configured to determine a set of preliminary coloring image frames corresponding to each to-be-processed image frame group as a preliminary coloring image frame set.

[0120] Optionally, the color image generation module 33 includes:

[0121] a traffic light pixel determination unit, configured to use each of the preliminary color-decorated image frames as a current image frame, and determine a set of traffic light pixels in the current image frame based on traffic light position information in the image frame to be processed corresponding to the current image frame;

[0122] a lighted pixel determination unit, configured to determine a lighted pixel set in a current image frame and a lighted color of the lighted pixel set according to pixel category information corresponding to each traffic light pixel in the traffic light pixel set;

[0123] The image color restoration unit is used to restore each pixel in the current image frame to the color of the image frame to be processed corresponding to the current image frame if the set of lit pixels does not exist, obtain the colored image frame corresponding to the current image frame, increase the cumulative number of yellow light frames by one, and set the cumulative number of green light frames to zero; otherwise, restore the color of the current image frame according to the lit color, the cumulative number of yellow light frames, the cumulative number of green light frames, the traffic light position information, and the image frame to be processed corresponding to the current image frame.

[0124] Optional image color restoration unit, specifically used for:

[0125] If the lit color is green, keep the color of the lit pixel set unchanged, restore the pixels in the current image frame except the lit pixel set to the color of the image frame to be processed corresponding to the current image frame, obtain the color-drawing image frame corresponding to the current image frame, and set the accumulated number of yellow light frames to zero, and increase the accumulated number of green light frames by one;

[0126] If the lit color is a preset non-green color and the cumulative number of green light frames is greater than zero, the color of the lit pixel set is restored to yellow, and the pixels in the current image frame except the lit pixel set are restored to the color of the image frame to be processed corresponding to the current image frame, to obtain the color-drawing image frame corresponding to the current image frame, and the cumulative number of yellow light frames is increased by one, and the cumulative number of green light frames is set to zero;

[0127] If the lit color is a preset non-green color and the cumulative number of yellow light frames is equal to one, the color of the lit pixel set is restored to yellow, and the pixels in the current image frame except the lit pixel set are restored to the color of the image frame to be processed corresponding to the current image frame, to obtain the color-drawing image frame corresponding to the current image frame, and the cumulative number of yellow light frames is increased by one, and the initial yellow light position information is determined based on the image frame before the current image frame and the previous traffic light position information;

[0128] If the lit color is a preset non-green color and the cumulative number of yellow light frames is greater than one, the current light position information is determined based on the lit pixel set and the traffic light position information, and the current image frame is restored in color based on the current light position information, the initial yellow light position information and the image frame to be processed corresponding to the current image frame.

[0129] Optionally, color restoration is performed on the current image frame according to the current light position information, the initial yellow light position information, and the image frame to be processed corresponding to the current image frame, including:

[0130] Determine a position offset ratio based on the current light position information and the initial yellow light position information;

[0131] If the position offset ratio is less than the preset offset ratio threshold, the color of the lit pixel set is restored to yellow, and the pixels in the current image frame except the lit pixel set are restored to the color of the image frame to be processed corresponding to the current image frame, obtaining the color-drawing image frame corresponding to the current image frame, and the cumulative number of yellow light frames is increased by one;

[0132] If the position offset ratio is greater than or equal to the preset offset ratio threshold, the lit color will be restored to red, and the pixels in the current image frame except the lit pixel set will be restored to the color of the image frame to be processed corresponding to the current image frame, so as to obtain the colored image frame corresponding to the current image frame, and the cumulative number of yellow light frames will be set to zero.

[0133] Optionally, before obtaining the set of image frames to be processed corresponding to the video stream to be processed, the method further includes:

[0134] Acquire at least one historical image frame, and perform color recognition and conversion on the historical image frame so that lit pixels in the historical image frame are reassigned to green pixels or preset non-green pixels, thereby generating an auxiliary historical image frame corresponding to the historical image frame;

[0135] Obtain pixel category information for each pixel in the auxiliary historical image frame, and use the pixel category information as a label for the historical image frame and the auxiliary historical image frame to construct a training sample set; wherein the pixel category information includes red light category, green light category, and background category;

[0136] The training sample set is input into the initial traffic light coloring model, and the initial traffic light coloring model is trained based on the constructed loss function until the preset convergence condition is met to obtain the traffic light coloring model.

[0137] The traffic light coloring device provided in the embodiment of the present application can execute the traffic light coloring method provided in any embodiment of the present application, and has the corresponding functional modules and beneficial effects of the execution method.

[0138] Figure 4 A schematic diagram of a traffic light coloring device provided in an embodiment of the present application. The traffic light coloring device 40 can be an electronic device, and is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or claimed herein.

[0139] like Figure 4As shown, the traffic light depiction device 40 includes at least one processor 41 and memory, such as a read-only memory (ROM) 42 and a random access memory (RAM) 43, communicatively connected to the at least one processor 41. The memory stores a computer program executable by the at least one processor. The processor 41 can perform various appropriate actions and processes based on the computer program stored in the ROM 42 or loaded from a storage unit 48 into the RAM 43. The RAM 43 can also store various programs and data required for the operation of the traffic light depiction device 40. The processor 41, ROM 42, and RAM 43 are interconnected via a bus 44. An input / output (I / O) interface 45 is also connected to the bus 44.

[0140] Multiple components in the traffic light painting device 40 are connected to an I / O interface 45, including an input unit 46, such as a keyboard, a mouse, etc.; an output unit 47, such as various types of displays, speakers, etc.; a storage unit 48, such as a magnetic disk, an optical disk, etc.; and a communication unit 49, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 49 allows the traffic light painting device 40 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0141] Processor 41 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of processor 41 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any other suitable processors, controllers, microcontrollers, etc. Processor 41 executes the various methods and processes described above, such as the traffic light coloring method.

[0142] In some embodiments, the traffic light coloring method can be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as storage unit 48. In some embodiments, part or all of the computer program can be loaded and / or installed on the traffic light coloring device 40 via ROM 42 and / or communication unit 49. When the computer program is loaded into RAM 43 and executed by processor 41, one or more steps of the traffic light coloring method described above can be performed. Alternatively, in other embodiments, processor 41 can be configured to execute the traffic light coloring method in any other suitable manner (e.g., via firmware).

[0143] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0144] Computer programs for implementing the methods of the present application may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0145] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. A computer-readable storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0146] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0147] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0148] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0149] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this application can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of this application can be achieved. This is not limited herein.

[0150] The above specific embodiments do not constitute a limitation on the scope of protection of this application. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application shall be included within the scope of protection of this application.

Claims

1. A traffic light coloring method, characterized in that: include: Obtaining a set of image frames to be processed corresponding to a video stream to be processed, and performing color recognition conversion on each of the image frames to be processed in the set to be processed to generate a set of auxiliary image frames corresponding to the set of image frames to be processed; wherein the lit pixels in each of the auxiliary image frames in the set of auxiliary image frames are green pixels or preset non-green pixels; Determining each of the to-be-processed image frames and the corresponding auxiliary image frames as an to-be-processed image frame group according to the corresponding relationship, and inputting each of the to-be-processed image frame groups into a pre-trained traffic light coloring model to determine a preliminary coloring image frame set; Performing color restoration on each of the preliminary colorized image frames based on pixel category information in each of the preliminary colorized image frames and traffic light position information in the image frames to be processed corresponding to each of the preliminary colorized image frames to generate a set of colorized image frames corresponding to the set of image frames to be processed; The color restoration of each of the preliminary colorized image frames according to the pixel category information in each of the preliminary colorized image frames and the traffic light position information in the image frames to be processed corresponding to each of the preliminary colorized image frames includes: Taking each of the preliminary colorized image frames as a current image frame, and determining a set of traffic light pixels in the current image frame according to traffic light position information in an image frame to be processed corresponding to the current image frame; determining a set of lit pixels in the current image frame and a lit color of the set of lit pixels according to pixel category information corresponding to each traffic light pixel in the set of traffic light pixels; When the lighting color of the current image frame is a preset non-green color, the lighting color of the traffic light in the current image frame is determined based on the restoration results of the preliminary colored image frames before the current image frame and the logical order of the lighting of the traffic lights, and the color restoration of the current image frame is completed based on the judgment result.

2. The method according to claim 1, characterized in that The step of performing color recognition conversion on each of the image frames to be processed in the image frame set to be processed to generate an auxiliary image frame set corresponding to the image frame set to be processed includes: For each image frame to be processed in the set of image frames to be processed, performing target recognition on the image frame to be processed, and determining lit pixels in the image frame to be processed; Performing color recognition on the lit pixel, and reassigning the lit pixel to a green pixel or a preset non-green pixel according to a color recognition result; The image frame to be processed after pixel reassignment is determined as the auxiliary image frame corresponding to the image frame to be processed, and the set consisting of the auxiliary image frames is determined as the auxiliary image frame set corresponding to the image frame set to be processed.

3. The method according to claim 1, characterized in that Inputting each of the to-be-processed image frame groups into a pre-trained traffic light coloring model to determine a preliminary coloring image frame set includes: For a group of image frames to be processed, input the group of image frames to be processed into a feature extraction layer in a pre-trained traffic light coloring model to perform multi-scale feature extraction, and determine an intermediate feature map corresponding to the group of image frames to be processed and having the same size; Inputting the intermediate feature map into the classification layer in the traffic light color model for category classification, determining at least one candidate pixel category corresponding to each pixel in the output image, and the category confidence of the candidate pixel category; Determining the candidate pixel category with the largest category confidence as the pixel category information of the pixel, and determining the output image after the pixel category information determination is completed as the preliminary coloring image frame corresponding to the image frame group to be processed; Determining a set of preliminary color-drawing image frames corresponding to each of the to-be-processed image frame groups as a preliminary color-drawing image frame set; Among them, the residual structure of the feature extraction layer is a convolution residual structure and an identity mapping residual structure.

4. The method according to claim 1, wherein The method further includes: performing color restoration on each of the preliminary colorized image frames based on pixel category information in each of the preliminary colorized image frames and traffic light position information in the image frames to be processed corresponding to each of the preliminary colorized image frames. If the set of lit pixels does not exist, each pixel in the current image frame is restored to the color of the image frame to be processed corresponding to the current image frame to obtain a colored image frame corresponding to the current image frame, and the cumulative number of yellow light frames is increased by one, and the cumulative number of green light frames is set to zero; otherwise, the color of the current image frame is restored according to the lit color, the cumulative number of yellow light frames, the cumulative number of green light frames, the traffic light position information and the image frame to be processed corresponding to the current image frame.

5. The method according to claim 4, characterized in that The performing color restoration on the current image frame according to the lighting color, the cumulative number of yellow light frames, the cumulative number of green light frames, the traffic light position information, and the image frame to be processed corresponding to the current image frame, includes: If the lit color is green, the color of the lit pixel set is maintained unchanged, and the pixels in the current image frame other than the lit pixel set are restored to the color of the image frame to be processed corresponding to the current image frame, thereby obtaining a colored image frame corresponding to the current image frame, and the accumulated number of yellow light frames is set to zero, while the accumulated number of green light frames is increased by one; If the lit color is a preset non-green color and the cumulative number of green light frames is greater than zero, the color of the lit pixel set is restored to yellow, and the pixels in the current image frame except the lit pixel set are restored to the color of the image frame to be processed corresponding to the current image frame, to obtain a colored image frame corresponding to the current image frame, and the cumulative number of yellow light frames is increased by one, and the cumulative number of green light frames is set to zero; If the lit color is a preset non-green color and the cumulative number of yellow light frames is equal to one, the color of the lit pixel set is restored to yellow, the pixels in the current image frame other than the lit pixel set are restored to the color of the image frame to be processed corresponding to the current image frame, to obtain a colored image frame corresponding to the current image frame, the cumulative number of yellow light frames is increased by one, and the initial yellow light position information is determined based on the image frame and the previous traffic light position information of the current image frame; If the lit color is a preset non-green color and the cumulative number of yellow light frames is greater than one, the current light position information is determined based on the lit pixel set and the traffic light position information, and the current image frame is restored in color based on the current light position information, the initial yellow light position information and the image frame to be processed corresponding to the current image frame.

6. The method according to claim 5, characterized in that The performing color restoration on the current image frame according to the current light position information, the initial yellow light position information, and the image frame to be processed corresponding to the current image frame includes: Determine a position offset ratio according to the current light position information and the initial yellow light position information; If the position offset ratio is less than a preset offset ratio threshold, the color of the set of lit pixels is restored to yellow, and the pixels in the current image frame other than the set of lit pixels are restored to the color of the image frame to be processed corresponding to the current image frame, thereby obtaining a colored image frame corresponding to the current image frame, and the accumulated number of yellow light frames is increased by one; If the position offset ratio is greater than or equal to the preset offset ratio threshold, the lit color is restored to red, and the pixels in the current image frame except the lit pixel set are restored to the color of the image frame to be processed corresponding to the current image frame, so as to obtain the colored image frame corresponding to the current image frame, and the cumulative number of yellow light frames is set to zero.

7. The method according to any one of claims 1 to 6, characterized in that Before obtaining the set of image frames to be processed corresponding to the video stream to be processed, the method further includes: Acquire at least one historical image frame, and perform color recognition and conversion on the historical image frame so that lit pixels in the historical image frame are reassigned to green pixels or preset non-green pixels, thereby generating an auxiliary historical image frame corresponding to the historical image frame; Obtaining pixel category information of each pixel in the auxiliary historical image frame, and using each pixel category information as a label for the historical image frame and the auxiliary historical image frame to construct a training sample set; wherein the pixel category information includes a red light category, a green light category, and a background category; The training sample set is input into an initial traffic light color depiction model, and the initial traffic light color depiction model is trained based on the constructed loss function until a preset convergence condition is met to obtain a traffic light color depiction model.

8. A traffic light coloring device, characterized in that: include: an image frame acquisition module, configured to acquire a set of image frames to be processed corresponding to a video stream to be processed, and perform color recognition conversion on each of the image frames to be processed in the set to be processed to generate a set of auxiliary image frames corresponding to the set of image frames to be processed; wherein the lit pixels in each of the auxiliary image frames in the set of auxiliary image frames are green pixels or preset non-green pixels; a preliminary coloring module, configured to determine, based on the correspondence, each of the to-be-processed image frames and the corresponding auxiliary image frames as a to-be-processed image frame group, and input each of the to-be-processed image frame groups into a pre-trained traffic light coloring model to determine a preliminary coloring image frame set; a color-drawing image generation module configured to perform color restoration on each of the preliminary color-drawing image frames based on pixel category information in each of the preliminary color-drawing image frames and traffic light position information in the image frames to be processed corresponding to each of the preliminary color-drawing image frames, thereby generating a set of color-drawing image frames corresponding to the set of image frames to be processed; The color image generation module is specifically used to: Taking each of the preliminary colorized image frames as a current image frame, and determining a set of traffic light pixels in the current image frame according to traffic light position information in an image frame to be processed corresponding to the current image frame; determining a set of lit pixels in the current image frame and a lit color of the set of lit pixels according to pixel category information corresponding to each traffic light pixel in the set of traffic light pixels; When the lighting color of the current image frame is a preset non-green color, the lighting color of the traffic light in the current image frame is determined based on the restoration results of the preliminary colored image frames before the current image frame and the logical order of the lighting of the traffic lights, and the color restoration of the current image frame is completed based on the judgment result.

9. A traffic light coloring device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the traffic light coloring method according to any one of claims 1 to 7.

10. A storage medium containing computer-executable instructions, characterized in that: When the computer executable instructions are executed by a computer processor, they are used to execute the traffic light coloring method according to any one of claims 1 to 7.

Images