Traffic light recognition method, device, equipment and storage medium

Abstract

The present application discloses a traffic light recognition method, apparatus, device, and storage medium, relating to the field of machine vision technology, including: determining the minimum bounding rectangle of an original traffic light image containing all light groups, and cropping the minimum bounding rectangle area to obtain an initial region of interest image; determining whether the size of the initial region of interest image meets a preset size condition; if not, re-segmenting the initial region of interest image to obtain a target region of interest image that meets the preset size condition; inputting the target region of interest image into a semantic segmentation network constructed based on a convolutional neural network algorithm to obtain a semantic segmentation image containing different pixel point categories; and realizing traffic light recognition and signal light eye enhancement based on the number of pixels and the position of the pixels corresponding to each pixel point category in the semantic segmentation image. The present application can improve the accuracy of traffic light recognition in complex scenarios and improve the processing efficiency of traffic light recognition.

Description

Technical Field

[0001] The present invention relates to the field of machine vision technology, and in particular to a traffic light recognition method, device, equipment and storage medium. Background Art

[0002] With the increasing prevalence of automobiles, installing electronic police systems at urban intersections to monitor violations such as red light running has become a trend in intelligent traffic management. Traffic light recognition plays a crucial role in electronic traffic police's adjudication of violations, with the accuracy of these decisions directly determining the accuracy of these decisions. Due to the inherent complexity and diversity of traffic lights, as well as their susceptibility to color cast and overexposure due to lighting and environmental factors, existing technologies for traffic light recognition in complex traffic scenarios struggle to achieve rapid and accurate results.

[0003] Therefore, how to improve the recognition efficiency and accuracy of traffic lights in complex scenarios is a problem that those skilled in the art currently need to solve. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a traffic light recognition method, device, equipment and storage medium, which can solve the problem of low traffic light recognition accuracy in complex scenarios and improve the processing efficiency of traffic light recognition. The specific solution is as follows:

[0005] In a first aspect, the present application discloses a traffic light recognition method, comprising:

[0006] Determine the minimum bounding rectangle of the original traffic light image that contains all the light groups, and crop the minimum bounding rectangle area to obtain an initial region of interest image;

[0007] Determining whether the size of the initial region of interest image meets a preset size condition;

[0008] If the preset size condition is not met, re-segmenting the initial region of interest image to obtain multiple segmented regions;

[0009] Cropping the plurality of segmented regions to obtain a plurality of light group images corresponding to the plurality of segmented regions, and splicing the plurality of light group images to obtain a target image of interest that meets the preset size condition;

[0010] Inputting the target image of interest into a semantic segmentation network constructed based on a convolutional neural network algorithm, and obtaining a semantic segmentation image containing different pixel categories output by the semantic segmentation network;

[0011] Based on the number of pixels corresponding to each pixel category in the semantic segmentation image, a signal light recognition result of the current traffic scene corresponding to the original signal light image is determined.

[0012] Optionally, determining the minimum bounding rectangle of all light groups in the original traffic light image and cropping the minimum bounding rectangle area to obtain an initial region of interest image includes:

[0013] Determine the location areas of all signal light groups in the original signal light image;

[0014] Determining, based on the location area, a minimum circumscribed rectangle of the original traffic light image that includes all of the traffic light groups;

[0015] The minimum circumscribed rectangular area is cropped according to a first preset cropping size to obtain an initial region of interest image.

[0016] Optionally, after determining whether the size of the initial ROI image meets a preset size condition, the method further includes:

[0017] If the size of the initial region of interest image meets the preset size condition, the initial region of interest image is determined as the target image of interest.

[0018] Optionally, re-segmenting the initial region-of-interest image includes:

[0019] Two adjacent light groups with the longest distance between them are selected from all any adjacent light groups on the initial region of interest image, and the initial region of interest image is re-segmented based on the target coordinates corresponding to the two adjacent light groups with the longest distance between them.

[0020] Optionally, determining the signal light recognition result of the current traffic scene corresponding to the original signal light image based on the number of pixels corresponding to each pixel category in the semantic segmentation image includes:

[0021] Counting the number of pixels corresponding to each pixel category corresponding to all light eyes in the semantic segmentation image;

[0022] Determine, from each lamp eye on the semantic segmentation image, the pixel category corresponding to the maximum number of pixels, and obtain a target pixel category corresponding to each lamp eye;

[0023] The color corresponding to the target pixel category is determined as the eye color of the light eye, and a signal light recognition result of the current traffic scene corresponding to the original signal light image is determined based on the eye color corresponding to each light eye.

[0024] Optionally, after determining the color corresponding to the target pixel category as the eye color of the eye, the method further includes:

[0025] Determining whether the eye color of the eye is consistent with a color in an original configuration color set corresponding to the eye;

[0026] If the eye color of the lamp eye is inconsistent with all the colors in the original configuration color set, the current eye color of the lamp eye is determined to be an abnormal identification color, and the eye color of the lamp eye is modified to the color corresponding to when the lamp eye is in the off state.

[0027] Optionally, determining the signal light recognition result of the current traffic scene corresponding to the original signal light image based on the light eye color corresponding to each light eye includes:

[0028] Selecting multiple light eyes with the same preset traffic indication direction from all the light eyes to obtain multiple target light eyes corresponding to the preset traffic indication directions;

[0029] Determining whether the eye colors of the plurality of target eyes in each of the preset traffic indication directions are consistent;

[0030] If they are consistent, the light eye color is determined as the signal light recognition result of the preset traffic indication direction;

[0031] If they are inconsistent, the eye colors corresponding to the multiple target eye lights in the preset traffic indication direction are modified to the colors corresponding to when the target eye lights are in the off state.

[0032] Optionally, after determining the signal light recognition result of the current traffic scene corresponding to the original signal light image based on the number of pixels corresponding to each pixel category in the semantic segmentation image, the method further includes:

[0033] Determine the target lamp eye location area from the semantic segmentation image;

[0034] Extracting a mask area corresponding to the light-emitting area of ​​the target lamp eye from the positioning area;

[0035] The image area corresponding to the mask area on the target image of interest is filled with color according to the target lamp eye color corresponding to the identified target lamp eye, so as to obtain the image area after color enhancement.

[0036] In a second aspect, the present application discloses a traffic light recognition device, comprising:

[0037] An initial image determination module is used to determine the minimum bounding rectangle of the original traffic light image containing all light groups, and to crop the minimum bounding rectangle area to obtain an initial region of interest image;

[0038] A condition judgment module, used to judge whether the size of the initial region of interest image meets a preset size condition;

[0039] an image re-segmentation module, configured to re-segment the initial ROI image to obtain a plurality of segmented regions when the size of the initial ROI image does not meet the preset size condition;

[0040] A region cropping module, configured to crop the plurality of segmented regions to obtain a plurality of light group images corresponding to the plurality of segmented regions;

[0041] An image stitching module, configured to stitch the plurality of light group images to obtain a target image of interest that meets the preset size condition;

[0042] An image semantic segmentation module is used to input the target image of interest into a semantic segmentation network constructed based on a convolutional neural network algorithm, and obtain a semantic segmentation image containing different pixel categories output by the semantic segmentation network;

[0043] The recognition result determination module is used to determine the signal light recognition result of the current traffic scene corresponding to the original signal light image based on the number of pixels corresponding to each pixel category in the semantic segmentation image.

[0044] In a third aspect, the present application discloses an electronic device, comprising:

[0045] Memory, used to store computer programs;

[0046] The processor is used to execute the computer program to implement the steps of the aforementioned traffic light recognition method.

[0047] In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the steps of the aforementioned traffic light recognition method are implemented.

[0048] It can be seen that the present application provides a method for traffic light recognition, including: determining the minimum circumscribed rectangle of the original traffic light image containing all light groups, and cropping the minimum circumscribed rectangle area to obtain an initial region of interest image; judging whether the size of the initial region of interest image meets a preset size condition; if it does not meet the preset size condition, re-segmenting the initial region of interest image to obtain multiple segmented regions; cropping the multiple segmented regions to obtain multiple light group images corresponding to the multiple segmented regions respectively; splicing the multiple light group images to obtain a target image of interest that meets the preset size condition; inputting the target image of interest into a semantic segmentation network constructed based on a convolutional neural network algorithm to obtain a semantic segmentation image containing different pixel point categories output by the semantic segmentation network; based on the number of pixels corresponding to each pixel point category in the semantic segmentation image, determining the traffic light recognition result of the current traffic scene corresponding to the original traffic light image. It can be seen that before the original traffic light image is input into the semantic segmentation network, the present application first crops the original traffic light image to obtain an initial region of interest image, and then re-segments the initial region of interest image that does not meet the preset conditions to determine the target image of interest, thereby improving the accuracy of the semantic segmentation network and ensuring the effect of subsequent traffic light recognition. The target image of interest is then input into the semantic segmentation network constructed based on the convolutional neural network algorithm to obtain a corresponding semantic segmentation image containing different pixel categories. By parsing the semantic segmentation image output by the semantic segmentation network, the traffic light state can be recognized based on the number of pixels corresponding to each pixel category in the semantic segmentation image obtained by analysis, thereby determining the traffic light recognition result of the current traffic scene corresponding to the original traffic light image.

[0049] In summary, the technical solution of this application using a semantic segmentation network to complete traffic light recognition has low performance overhead, strong robustness, is not affected by the environment, does not rely on prior knowledge, can solve the problem of low accuracy of traffic light recognition in complex scenarios, and improve the overall processing efficiency of traffic light recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0051] Figure 1 This is a flow chart of a traffic signal light recognition method disclosed in this application;

[0052] Figure 2aThis is a schematic diagram of an original signal light image disclosed in this application;

[0053] Figure 2b A schematic diagram of the minimum circumscribed rectangle corresponding to an original traffic light image disclosed in this application;

[0054] Figure 2c This is a schematic diagram of an initial region of interest image expanded α times as disclosed in this application;

[0055] Figure 3 This is a schematic diagram of an initial region of interest image disclosed in this application;

[0056] Figure 4 A flowchart of re-segmentation of an initial image of interest disclosed in this application;

[0057] Figure 5 This is a schematic diagram of the structure of a traffic light recognition device disclosed in this application;

[0058] Figure 6 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0059] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0060] Currently, the recognition status of traffic lights plays a vital role in the electronic traffic police's violation penalty determination process. The traffic light recognition results directly determine the accuracy of the relevant penalties. Due to the complexity and diversity of traffic lights themselves, and the susceptibility of traffic lights to color cast and overexposure under the influence of light and the environment, it is difficult for existing technologies to achieve fast and accurate recognition results. Therefore, this application provides a traffic light recognition method to address the problem of low traffic light recognition accuracy in complex scenarios and improve the overall processing efficiency of traffic light recognition.

[0061] The embodiment of the present invention discloses a method for identifying traffic lights. Figure 1 As shown, the method includes:

[0062] Step S11: determining the minimum bounding rectangle of the original traffic light image containing all the light groups, and cropping the minimum bounding rectangle area to obtain an initial region of interest image.

[0063] In this embodiment, the minimum bounding rectangle of all light groups in the original traffic light image is first determined, and the minimum bounding rectangle area is cropped to obtain an initial region of interest image. The initial region of interest needs to be obtained for the input of each semantic segmentation network.

[0064] In this embodiment, determining the minimum bounding rectangle of all light groups contained in the original traffic light image may include: determining the position area of ​​all traffic light groups in the original traffic light image; and determining the minimum bounding rectangle of all traffic light groups contained in the original traffic light image based on the position area. It is understandable that in each actual traffic scene, the position area of ​​each traffic light group is fixed. Therefore, before the entire project begins, the position areas of all traffic light groups are extracted from the original traffic light image using a pre-configured traffic light group area frame. That is, by drawing all traffic light groups in the original traffic light image according to the actual outer frames of the traffic light groups, a pre-configured traffic light group area frame is obtained. Therefore, the position areas of all traffic light groups can be extracted from the original traffic light image using the pre-configured traffic light group area frame. Finally, based on the position areas of all traffic light groups, the minimum bounding rectangle of all traffic light groups contained in the original traffic light image is determined.

[0065] Furthermore, when the minimum bounding rectangle is cropped to obtain the initial region of interest image, the minimum bounding rectangle may be cropped according to a first preset cropping size to obtain the initial region of interest image. That is, the initial region of interest image is an image expanded α times based on the minimum bounding rectangle, for example, Figure 2a is the original traffic light image, Figure 2b For the above Figure 2a The minimum bounding rectangle corresponding to the original traffic light image is then expanded by α times based on the minimum bounding rectangle to obtain the initial region of interest image for semantic segmentation. Figure 2c That is, in the above Figure 2b The initial region of interest image is expanded α times on the region of interest. The value of α can be 1.4.

[0066] Step S12: determining whether the size of the initial ROI image meets a preset size condition.

[0067] In this embodiment, after obtaining the initial region of interest image, it is first necessary to determine whether the size of the initial region of interest image meets the preset size conditions, for example, to determine whether the height or width of the image size corresponding to the initial region of interest image is not greater than γ times the image input size of the semantic segmentation network, where the value of γ can be 2.

[0068] Step S13: If the preset size condition is not met, the initial region of interest image is re-segmented to obtain a plurality of segmented regions.

[0069] It is understandable that there are many types of traffic lights in actual traffic scenes, including special scenes with multiple light groups, multiple light eyes, large distances between light groups, and special light groups. In actual traffic scenes, the number of light groups can be up to 8, and the number of light eyes in each light group can be up to 12. As a result, the image size of the randomly acquired original traffic light image of the current frame is too large and does not meet the input requirements of the semantic segmentation network. For example, when the number of light groups in the original traffic light image is large and the distances between the light groups are large, for example, two light groups are on both sides of the intersection, or when the number of light groups is large and the light groups include horizontal and vertical light groups, even after the original traffic light image is initially cropped to obtain the initial region of interest image, the initial region of interest image is still not large enough. The image size of the ROI image still far exceeds the specified image size for input to the semantic segmentation network. If the initial ROI image that does not meet the preset size conditions is directly sent to the semantic segmentation network, the initial ROI image needs to be compressed first. However, high-magnification compression will make the light eyes in the initial ROI image smaller, which directly affects the segmentation effect of the semantic segmentation network. Therefore, before the initial ROI image is input into the semantic segmentation network, the initial ROI image needs to be re-segmented, and the image obtained after processing contains all the light groups in the original traffic light image and meets the preset size conditions, thereby improving the accuracy of the network semantic segmentation and ensuring the subsequent recognition and enhancement effects.

[0070] In this embodiment, if the size of the initial region of interest image does not meet the preset size condition, the initial region of interest image is re-segmented based on the preset segmentation principle to obtain multiple segmented regions. It can be understood that if the size of the initial region of interest image does not meet the preset size condition, the initial region of interest image needs to be re-segmented, that is, if the height or width of the image size corresponding to the initial region of interest image is greater than γ times the image input size of the semantic segmentation network, the initial region of interest image needs to be re-segmented based on the preset segmentation principle, that is, the two adjacent light groups with the farthest distance between them are selected from all arbitrary adjacent light groups on the initial region of interest image, and the initial region of interest image is re-segmented based on the target coordinates corresponding to the two adjacent light groups with the farthest distance between them. After selecting the two adjacent light groups with the farthest distance between them, the initial region of interest image is re-segmented based on the target coordinates corresponding to the two adjacent light groups with the farthest distance between them. As Figure 3As shown, there are four light groups, among which the distance between adjacent light group 2 and light group 3 is the largest. At this time, we will perform segmentation between light group 2 and light group 3, and determine the larger horizontal coordinates of the upper right corner and the lower right corner of light group 2 as the target coordinates corresponding to light group 2, and the smaller horizontal coordinates of the upper left corner and the lower left corner of light group 3 as the target coordinates corresponding to light group 3. Then, based on the target coordinates corresponding to light group 2 and the target coordinates corresponding to light group 3, the initial region of interest image is re-segmented to obtain multiple segmented areas.

[0071] Step S14: cropping the plurality of segmented regions to obtain a plurality of light group images corresponding to the plurality of segmented regions, and splicing the plurality of light group images to obtain a target image of interest that meets the preset size condition.

[0072] In this embodiment, the plurality of segmented regions can be cropped according to a second preset cropping size to obtain a plurality of light group images corresponding to the plurality of segmented regions respectively; the plurality of light group images can be spliced ​​to obtain a target image of interest that meets the preset size condition. The plurality of light group images obtained by cropping the plurality of segmented regions according to the second preset cropping size are images expanded β times based on the cropped images. It should be noted that when cropping the segmented regions, the segmented regions containing the light groups are cropped, thereby obtaining light group images corresponding to the segmented regions containing the light groups respectively. The value of β can be 1.

[0073] In this embodiment, if the size of the initial region of interest image meets the preset size condition, that is, the height or width of the image size corresponding to the initial region of interest image is not greater than γ times the image input size of the semantic segmentation network, that is, the aspect ratio of the initial region of interest image is normal, then it means that it is not a complex intersection signal light recognition scene with a large distance between light groups, and the initial region of interest image can be directly determined as the target region of interest image. The above-mentioned target region of interest image meets the input size requirement for input to the semantic segmentation network.

[0074] For example, Figure 4As shown, the original traffic light image is obtained, the original traffic light image is initially cropped, and an initial region of interest image is obtained. It is determined whether the height or width corresponding to the image size of the initial region of interest image is greater than γ times the height or width of the image input specified by the semantic segmentation network. If the height or width corresponding to the image size of the initial region of interest image is greater than γ times the height or width of the image input specified by the semantic segmentation network, the two adjacent light groups with the greatest distance between them in the initial region of interest image are selected, and the initial region of interest image is re-segmented based on the target coordinates corresponding to the two adjacent light groups with the greatest distance between them, to obtain multiple cropped light group images, which are then spliced ​​to obtain a target image of interest. If the height or width corresponding to the image size of the initial region of interest image is not greater than γ times the height or width of the image input specified by the semantic segmentation network, the initial region of interest image is directly determined as the target image of interest.

[0075] Step S15: inputting the target image of interest into a semantic segmentation network constructed based on a convolutional neural network algorithm to obtain a semantic segmentation image containing different pixel categories output by the semantic segmentation network.

[0076] It should be pointed out that due to the wide variety of traffic lights and complex scenes, in order to ensure the effectiveness of traffic light segmentation, the training materials used to train the semantic segmentation network can include round lights, arrow lights, digital lights, and strip lights, etc., and cover multiple scenes such as daytime, nighttime, strong light, and backlight. It can also include training materials such as single light groups, multiple light groups, overexposed lights, and color cast lights. Use pixel "0" to mark the background point, pixel "1" to mark the red light eye, pixel "2" to mark the yellow light eye, and pixel "3" to mark the green light eye. The marked light eye image is used as the label sample, and then the processed training material and label sample are input into the semantic segmentation network constructed based on the convolutional neural network algorithm for training. For example, the above training material and the above label sample are input into the optimized Unet segmentation network for training.

[0077] In this embodiment, the target region of interest image is input into a trained semantic segmentation network to obtain a high-precision semantic segmentation image, that is, the semantic segmentation image output by the above semantic segmentation network contains different pixel categories, so the semantic segmentation image output by the semantic segmentation network can better distinguish between foreground points and background points. For example, the different pixel categories in the semantic segmentation image corresponding to the target image of interest output by the Unet semantic segmentation network can be 0, 1, 2, 3, and displayed as a completely black picture, so it is first necessary to convert the color labels of the pixels of different categories into different grayscale values ​​to distinguish the background and the light eyes of different states in the image, and then use the bilinear interpolation method to map the fixed-size semantic segmentation image output by the semantic segmentation network to the target image of interest in order to perform subsequent operations.

[0078] Step S16: Determine a signal light recognition result of a current traffic scene corresponding to the original signal light image based on the number of pixels corresponding to each pixel category in the semantic segmentation image.

[0079] It is understandable that the color state of each light eye can be determined according to the number of pixels corresponding to different pixel categories, thereby determining the signal light recognition result of the current traffic scene corresponding to the original signal light image.

[0080] Specifically, the number of pixels corresponding to each pixel category for all eyes in the semantically segmented image is counted; for each eye in the semantically segmented image, the pixel category corresponding to the largest number of pixels is determined, thereby obtaining a target pixel category corresponding to each eye; the color corresponding to the target pixel category is determined as the eye color of the eye, and based on the eye color corresponding to each eye, a signal light recognition result for the current traffic scene corresponding to the original signal light image is determined. For example, the starting point of a light group in a pre-configured configuration file in the semantically segmented image is obtained and mapped to the target image of interest. For a single digital light, processing is performed directly within the mapped image area, and the number of pixels corresponding to the pixel categories "0," "1," "2," and "3" is counted, thereby obtaining the number of background pixels, red pixels, yellow pixels, and green pixels. For a light group that does not have a single digital light, the location area corresponding to each eye in the semantically segmented image is first obtained, and each eye is processed separately to obtain the number of pixels corresponding to the pixel categories "0," "1," "2," and "3" at each eye. Furthermore, the credibility of the foreground point can also be determined based on the ratio of foreground points to background points in the current image. For example, for a certain type of light eye, when the number of pixels corresponding to the background point is greater than 10 times the number of foreground points, the foreground point is considered unreliable. The formula for determining the pixel category corresponding to the maximum number of pixels for the current light eye can be:

[0081] MaxPot=MAX(MAX(RPot,YPot),GPot));

[0082] Among them, RPot, YPot, and GPot represent the number of red, yellow, and green pixels respectively. Then the category with the largest number of pixels is used as the color of the lamp eye. If RPot = MaxPot, the color of the lamp eye is red. If YPot = MaxPot, the color of the lamp eye is yellow. If GPot = MaxPot, the color of the lamp eye is green.

[0083] In this embodiment, after determining the color of each eye in the semantically segmented image, it is further determined whether the eye color of the eye is consistent with a color in the original configuration color set corresponding to the eye. If the eye color of the eye is inconsistent with any color in the original configuration color set, the eye color of the eye is determined to be an abnormal identification color, and the eye color of the eye is modified to the color corresponding to the eye in the off state. It can be understood that the determined eye color is verified to determine whether the currently determined eye color is correct, that is, the final determined eye color is verified using the original configuration color set of each eye. If the currently determined eye color is not included in the original configuration color set, the currently determined eye color is considered to be incorrect, and the eye color of the current eye is modified to the color corresponding to the eye in the off state. If the currently determined eye color is included in the original configuration color set, the currently determined eye color is considered to be correct.

[0084] Furthermore, after determining the colors of all eyes and completing the color verification of all eyes, it is necessary to select multiple eyes with the same preset traffic direction from all eyes to obtain multiple target eyes corresponding to each preset traffic direction; determine whether the eyes colors of the multiple target eyes in each preset traffic direction are consistent; if they are consistent, determine the eyes colors as the traffic light recognition result for the preset traffic direction; if they are inconsistent, modify the eyes colors corresponding to the multiple target eyes in the preset traffic direction to the colors corresponding to the target eyes when they are in the off state. It is understood that by traversing all eyes and determining whether the eyes colors of the multiple eyes with the same preset traffic direction are consistent, if they are consistent, determine the eyes colors as the traffic light recognition result corresponding to the current preset traffic direction, thereby obtaining traffic light recognition results corresponding to different traffic directions, and then determine the traffic light recognition result for the current traffic scene corresponding to the original traffic light image based on the traffic light recognition results corresponding to the different traffic directions.

[0085] It should be pointed out that traffic lights in actual scenes are mainly divided into standard three-eye single-color lights, multi-eye multi-color lights, single-eye multi-color lights, multi-eye multi-color combined with digital lights, strip lights and five-eye lights. In addition, due to the influence of multiple factors such as light and environment, traffic lights may also have color cast, overexposure, low brightness, abnormal or blurred eye shape, large halo, blurred eye edges, etc. Therefore, traffic light recognition in complex scenes is relatively difficult. However, the semantic segmentation network is based on pixel-level processing, which can more accurately obtain the status and mask information of the eye, which is conducive to the status recognition of traffic lights in complex scenes.

[0086] In this embodiment, after determining the traffic light recognition result for the current traffic scene corresponding to the original traffic light image, the following steps may also be performed: determining the target light eye's location region from the semantically segmented image; extracting a mask region corresponding to the target light eye's luminous area from the location region; and color-filling the image region corresponding to the mask region on the target image of interest based on the identified target light eye's color to obtain a color-enhanced image region. It is understood that to enhance and colorize the light eye, first, an enhanced region is obtained, i.e., the target light eye's location region is determined from the semantically segmented image. Specifically, the starting point of the light group in a pre-configured configuration file in the semantically segmented image is obtained and mapped to the target image of interest. The location region of each light eye is expanded by δ times the light eye size, where δ can be 1 / 5. For digital light eyes, the location region of the light eye is used as the initial location region. For non-digital light eyes, the expanded location region is used as the initial location region.

[0087] Then, the pixels within the eye positioning area are traversed on the semantic segmentation image output by the semantic segmentation network, and the pixels within the eye positioning area are stored in the mask to be enhanced, and the pixels that are not 0 are converted to 255 to obtain a binary mask. In addition, due to factors such as environmental interference, the mask in the semantic segmentation image may contain certain noise points, so the obtained binary mask needs to be post-processed. For example, connected areas are extracted on the binary mask, the extracted connected areas are filled, and then the maximum connected area among all connected areas is obtained, and other smaller areas are deleted to retain the largest connected area, thereby reducing the impact of noise points. The binary mask is then projected horizontally and vertically, and the projection results are used to update the eye positioning area to obtain the minimum bounding rectangle of the mask foreground point, and the minimum bounding rectangle is determined as the precise positioning area for eye enhancement. Due to the influence of halo and other factors, the mask in the semantic segmentation image may have certain adhesion, especially for smaller arrow lights. Due to the special shape of the arrow light, the edge of the arrow light is processed. For example, the non-zero pixels that meet the conditions in the positioning area are forcibly set to 0 to avoid the adhesion of the light eye.

[0088] Finally, the obtained precise eye location region is mapped back to the target image of interest, resulting in the precise location region of the target eye in the target image of interest. The binary mask is processed and stored in the precise eye location region to obtain a mask region. A target color corresponding to the color category of the target eye is determined, and the image region corresponding to the mask region on the target image of interest is filled with the target color to obtain a color-enhanced image region. In other words, the eye is colored using the reported precise location region and mask value to obtain an enhanced eye image.

[0089] In this embodiment, a semantic segmentation network is used for traffic light recognition, improving the accuracy of traffic light recognition in complex scenarios such as small scale, color cast, and overexposure. Furthermore, the segmentation results output by the semantic segmentation network are used for foreground points of traffic light eye enhancement, improving the accuracy of enhanced foreground points of traffic light in scenarios with strong background interference, ensuring the validity of evidence for red light violation penalties and improving the customer experience. Furthermore, the segmentation results output by the semantic segmentation network are used to simultaneously perform traffic light color recognition and eye enhancement processing, simplifying the overall traffic light processing process, improving processing efficiency, and ensuring implementation in the project.

[0090] It can be seen that before the embodiment of the present application inputs the original traffic light image into the semantic segmentation network, the original traffic light image is first cropped to obtain an initial region of interest image, and then the initial region of interest image that does not meet the preset conditions is re-segmented to determine the target image of interest, that is, the initial region of interest image that does not meet the preset conditions is subjected to secondary segmentation processing, and the processed image is input as the target image of interest into the semantic segmentation network for subsequent recognition, thereby improving the accuracy of the semantic segmentation network and ensuring the effect of subsequent traffic light recognition, and then the target image of interest is input into the semantic segmentation network constructed based on the convolutional neural network algorithm to obtain a corresponding semantic segmentation image containing different pixel point categories. By parsing the semantic segmentation image output by the semantic segmentation network, the number of pixels and the position of the pixels corresponding to each pixel point category in the semantic segmentation image obtained by analysis can be used to realize traffic light recognition and signal light eye enhancement. It can be seen from this that the technical solution of the embodiment of the present application using a semantic segmentation network to complete traffic light recognition and traffic light eye enhancement is highly robust, is not affected by the environment, does not rely on prior knowledge, can solve the problem of low accuracy of traffic light recognition in complex scenarios, improves the overall processing efficiency of traffic light recognition, and ensures its implementation in actual projects. In addition, since the performance overhead of the technical solution of this application is low, it can well meet the needs of actual multi-task projects.

[0091] Correspondingly, the embodiment of the present application also discloses a traffic light recognition device, see Figure 5 As shown, the device includes:

[0092] An initial image determination module 11 is configured to determine a minimum bounding rectangle of the original traffic light image containing all light groups, and to crop the minimum bounding rectangle to obtain an initial region of interest image.

[0093] A condition judgment module 12 is used to judge whether the size of the initial ROI image meets a preset size condition;

[0094] An image re-segmentation module 13 is configured to re-segment the initial ROI image to obtain a plurality of segmented regions when the size of the initial ROI image does not meet the preset size condition;

[0095] A region cropping module 14 is configured to crop the plurality of segmented regions to obtain a plurality of light group images corresponding to the plurality of segmented regions;

[0096] An image stitching module 15 is configured to stitch the plurality of light group images to obtain a target image of interest that meets the preset size condition;

[0097] An image semantic segmentation module 16 is configured to input the target image of interest into a semantic segmentation network constructed based on a convolutional neural network algorithm, and obtain a semantic segmentation image containing different pixel categories output by the semantic segmentation network;

[0098] The recognition result determination module 17 is configured to determine a signal light recognition result of a current traffic scene corresponding to the original signal light image based on the number of pixels corresponding to each pixel category in the semantic segmentation image.

[0099] As can be seen from the above, in this embodiment, before the original traffic light image is input into the semantic segmentation network, the original traffic light image is first cropped to obtain an initial region of interest image, and then the initial region of interest image that does not meet the preset conditions is re-segmented to determine the target image of interest, thereby improving the accuracy of the semantic segmentation network and ensuring the effect of subsequent traffic light recognition. The target image of interest is then input into the semantic segmentation network constructed based on the convolutional neural network algorithm to obtain a corresponding semantic segmentation image containing different pixel categories. By parsing the semantic segmentation image output by the semantic segmentation network, the number of pixels and pixel positions corresponding to each pixel category in the semantic segmentation image can be obtained based on the analysis to realize the recognition of the traffic light state and the enhancement of the traffic light eye. It can be seen that the technical solution of the embodiment of the present application using a semantic segmentation network to complete traffic light recognition and traffic light eye enhancement is highly robust, is not affected by the environment, does not rely on prior knowledge, can solve the problem of low accuracy of traffic light recognition in complex scenes, improve the overall processing efficiency of traffic light recognition, ensure its implementation in actual engineering, and because the performance overhead of the technical solution of the present application is low, it can well meet the needs of actual multi-task projects.

[0100] In some specific embodiments, the initial image determination module 11 may specifically include:

[0101] a position area determination unit, configured to determine the position areas of all signal light groups in the original signal light image;

[0102] a bounding rectangle determining unit, configured to determine, based on the position area, a minimum bounding rectangle of the original traffic light image that includes all the traffic light groups;

[0103] The region cropping unit is configured to crop the minimum circumscribed rectangular region according to a first preset cropping size to obtain an initial region of interest image.

[0104] In some specific embodiments, the traffic light recognition device may include:

[0105] The target image determination module is configured to determine the initial region-of-interest image as a target image of interest when the size of the initial region-of-interest image meets the preset size condition.

[0106] In some specific embodiments, the image re-segmentation module 13 may specifically include:

[0107] an adjacent light group determining unit, configured to select two adjacent light groups with the greatest distance between them from all any adjacent light groups on the initial region of interest image;

[0108] The image re-segmentation unit is used to re-segment the initial region of interest image based on the target coordinates corresponding to the two adjacent light groups that are farthest apart.

[0109] In some specific embodiments, the recognition result determination module 17 may specifically include:

[0110] A pixel count counting unit, configured to count the number of pixels corresponding to each pixel category corresponding to all light eyes in the semantic segmentation image;

[0111] A category determination unit is configured to determine, from each lamp eye on the semantic segmentation image, the pixel category corresponding to the maximum number of pixels, and obtain a target pixel category corresponding to each lamp eye;

[0112] The recognition result determination unit is used to determine the color corresponding to the target pixel point category as the eye color of the light eye, and determine the traffic light recognition result of the current traffic scene corresponding to the original traffic light image based on the eye color corresponding to each light eye.

[0113] In some specific embodiments, after determining the color corresponding to the target pixel category as the eye color of the eye, the method may specifically include:

[0114] A color judgment unit, configured to judge whether the eye color of the eye is consistent with a color in an original configuration color set corresponding to the eye;

[0115] A lamp eye abnormality judgment unit is used to judge that the lamp eye color of the lamp eye is an abnormal identification color if the lamp eye color of the lamp eye is inconsistent with all the colors in the original configuration color set, and to modify the lamp eye color of the lamp eye to the color corresponding to when the lamp eye is in the off state.

[0116] In some specific embodiments, the recognition result determination unit may specifically include:

[0117] A light eye selection subunit is used to select multiple light eyes with the same preset traffic indication direction from all the light eyes, and obtain multiple target light eyes corresponding to each of the preset traffic indication directions;

[0118] The color judgment subunit is used to judge whether the eye colors of the multiple target eye lights in each of the preset traffic indication directions are consistent; if they are consistent, the eye light color is determined as the signal light identification result of the preset traffic indication direction; if they are inconsistent, the eye light color corresponding to the multiple target eye lights in the preset traffic indication direction is modified to the color corresponding to when the target eye light is in the off state.

[0119] In some specific embodiments, the traffic light recognition device may include:

[0120] A positioning area determination module, configured to determine the positioning area of ​​the target lamp eye from the semantic segmentation image;

[0121] A mask area extraction module, configured to extract a mask area corresponding to the light-emitting area of ​​the target lamp eye from the positioning area;

[0122] The color filling module is used to fill the image area corresponding to the mask area on the target image of interest with color according to the target eye color corresponding to the identified target eye, so as to obtain the image area after color enhancement.

[0123] Furthermore, an embodiment of the present application also provides an electronic device. Figure 6 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram should not be considered as any limitation to the scope of application of the present application.

[0124] Figure 6 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps of the traffic light recognition method disclosed in any of the aforementioned embodiments. Furthermore, the electronic device 20 in this embodiment may be a computer.

[0125] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device. The communication protocol it follows is any communication protocol that can be applied to the technical solution of this application and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world. Its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0126] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or CD, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0127] The operating system 221 is used to manage and control the hardware devices and computer program 222 on the electronic device 20, and can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of implementing the traffic light recognition method performed by the electronic device 20 as disclosed in any of the aforementioned embodiments, the computer program 222 may further include a computer program capable of implementing other specific tasks.

[0128] Furthermore, an embodiment of the present application also discloses a storage medium, in which a computer program is stored. When the computer program is loaded and executed by a processor, the steps of the traffic light recognition method disclosed in any of the aforementioned embodiments are implemented.

[0129] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. Reference can be made to the descriptions of the identical or similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and the relevant parts can be referred to the descriptions of the methods.

[0130] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0131] The above is a detailed introduction to the traffic light recognition method, device, equipment and storage medium provided by the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. At the same time, for those skilled in the art, according to the ideas of the present invention, there may be changes in the specific implementation methods and application scopes. In summary, the contents of this specification should not be understood as limiting the present invention.

Claims

1. A traffic light recognition method, characterized in that: include: Determine the minimum bounding rectangle of the original traffic light image that contains all the light groups, and crop the minimum bounding rectangle area to obtain an initial region of interest image; Determining whether the size of the initial region of interest image meets a preset size condition; If the preset size condition is not met, re-segmenting the initial region of interest image to obtain multiple segmented regions; Cropping the plurality of segmented regions to obtain a plurality of light group images corresponding to the plurality of segmented regions, and splicing the plurality of light group images to obtain a target image of interest that meets the preset size condition; Inputting the target image of interest into a semantic segmentation network constructed based on a convolutional neural network algorithm, and obtaining a semantic segmentation image containing different pixel categories output by the semantic segmentation network; Determining a signal light recognition result of a current traffic scene corresponding to the original signal light image based on the number of pixels corresponding to each pixel category in the semantic segmentation image; Determine the target lamp eye location area from the semantic segmentation image; Extracting a mask area corresponding to the light-emitting area of ​​the target lamp eye from the positioning area; The image area corresponding to the mask area on the target image of interest is filled with color according to the target lamp eye color corresponding to the identified target lamp eye, so as to obtain the image area after color enhancement.

2. The traffic light recognition method according to claim 1, characterized in that: The determining of the minimum bounding rectangle of all light groups in the original traffic light image and cropping the minimum bounding rectangle area to obtain an initial region of interest image includes: Determine the location areas of all signal light groups in the original signal light image; Determining, based on the location area, a minimum circumscribed rectangle of the original traffic light image that includes all of the traffic light groups; The minimum circumscribed rectangular area is cropped according to a first preset cropping size to obtain an initial region of interest image.

3. The traffic light recognition method according to claim 1, characterized in that: After determining whether the size of the initial ROI image meets a preset size condition, the method further includes: If the size of the initial region of interest image meets the preset size condition, the initial region of interest image is determined as the target image of interest.

4. The traffic light recognition method according to claim 1, characterized in that: The re-segmenting of the initial region of interest image comprises: Two adjacent light groups with the longest distance between them are selected from all any adjacent light groups on the initial region of interest image, and the initial region of interest image is re-segmented based on the target coordinates corresponding to the two adjacent light groups with the longest distance between them.

5. The traffic light recognition method according to claim 1, characterized in that: The determining, based on the number of pixels corresponding to each pixel category in the semantic segmentation image, a signal light recognition result of the current traffic scene corresponding to the original signal light image includes: Counting the number of pixels corresponding to each pixel category corresponding to all light eyes in the semantic segmentation image; Determine, from each lamp eye on the semantic segmentation image, the pixel category corresponding to the maximum number of pixels, and obtain a target pixel category corresponding to each lamp eye; The color corresponding to the target pixel category is determined as the eye color of the light eye, and a signal light recognition result of the current traffic scene corresponding to the original signal light image is determined based on the eye color corresponding to each light eye.

6. The traffic light recognition method according to claim 5, characterized in that: After determining the color corresponding to the target pixel category as the eye color of the eye, the method further includes: Determining whether the eye color of the eye is consistent with a color in an original configuration color set corresponding to the eye; If the eye color of the lamp eye is inconsistent with all the colors in the original configuration color set, the current eye color of the lamp eye is determined to be an abnormal identification color, and the eye color of the lamp eye is modified to the color corresponding to when the lamp eye is in the off state.

7. The traffic light recognition method according to claim 5, characterized in that: The determining of a signal light recognition result of a current traffic scene corresponding to the original signal light image based on the light eye color corresponding to each light eye includes: Selecting multiple light eyes with the same preset traffic indication direction from all the light eyes to obtain multiple target light eyes corresponding to the preset traffic indication directions; Determining whether the eye colors of the plurality of target eyes in each of the preset traffic indication directions are consistent; If they are consistent, the light eye color is determined as the signal light recognition result of the preset traffic indication direction; If they are inconsistent, the eye colors corresponding to the multiple target eye lights in the preset traffic indication direction are modified to the colors corresponding to when the target eye lights are in the off state.

8. A traffic light recognition device, characterized in that: include: An initial image determination module is used to determine the minimum bounding rectangle of the original traffic light image containing all light groups, and to crop the minimum bounding rectangle area to obtain an initial region of interest image; A condition judgment module, used to judge whether the size of the initial region of interest image meets a preset size condition; an image re-segmentation module, configured to re-segment the initial ROI image to obtain a plurality of segmented regions when the size of the initial ROI image does not meet the preset size condition; A region cropping module, configured to crop the plurality of segmented regions to obtain a plurality of light group images corresponding to the plurality of segmented regions; An image stitching module, configured to stitch the plurality of light group images to obtain a target image of interest that meets the preset size condition; An image semantic segmentation module is used to input the target image of interest into a semantic segmentation network constructed based on a convolutional neural network algorithm, and obtain a semantic segmentation image containing different pixel categories output by the semantic segmentation network; a recognition result determination module, configured to determine a signal light recognition result of a current traffic scene corresponding to the original signal light image based on the number of pixels corresponding to each pixel category in the semantic segmentation image; A positioning area determination module, configured to determine the positioning area of ​​the target lamp eye from the semantic segmentation image; A mask area extraction module, configured to extract a mask area corresponding to the light-emitting area of ​​the target lamp eye from the positioning area; The color filling module is used to fill the image area corresponding to the mask area on the target image of interest with color according to the target eye color corresponding to the identified target eye, so as to obtain the image area after color enhancement.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the steps of the traffic light recognition method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that Used to store a computer program; wherein, when the computer program is executed by a processor, the steps of the traffic light recognition method according to any one of claims 1 to 7 are implemented.

Images