Traffic light testing methods, computer equipment, and automobiles
By dividing the traffic light detection image into a grid and combining it with a countdown recognition model, the problems of low efficiency and low accuracy in existing traffic light detection technologies are solved, achieving wider and more accurate traffic light detection.
Patent Information
- Application Number
- CN202111555895.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-17
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2041-12-17
AI Technical Summary
Existing traffic light detection technologies suffer from low efficiency and low accuracy.
A pre-trained target traffic light detection model is used to divide the image to be detected into N grids for recognition. The model outputs the traffic light detection results at M scales for each grid, including the outer frame confidence, outer frame position and at least one inner frame attribute. When the image contains digit shapes, a countdown recognition model is used for further recognition.
It improves the accuracy and versatility of traffic light detection, ensuring that the output results have a wide coverage and are more accurate, and can effectively identify the outer and inner frame attributes of traffic lights as well as the countdown.
Smart Images

Figure CN116311193B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, and in particular to a traffic light detection method, computer equipment, and automobile. Background Technology
[0002] Traffic light detection is an indispensable part of autonomous and assisted driving technologies, and its recognition accuracy directly affects the safety of intelligent driving. Current traffic light detection technologies are mainly divided into traditional methods and deep learning methods, both of which suffer from low efficiency and low accuracy. Summary of the Invention
[0003] This invention provides a traffic light detection method, a computer device, and a vehicle to address the problems of low efficiency and low accuracy in existing traffic light detection methods.
[0004] This invention provides a traffic light detection method, comprising:
[0005] Acquire the image of the traffic light to be detected;
[0006] A pre-trained target traffic light detection model is used to divide the traffic light image to be detected into N grids for identification. The detection results of traffic lights at M scales corresponding to each grid are output. Each traffic light detection result includes outer frame confidence, outer frame position and at least one inner frame attribute. Each inner frame attribute includes inner frame confidence, inner frame position, inner frame color and inner frame shape. The outer frame position includes the center position and width and height offset of the traffic light outer frame relative to the grid. The inner frame position includes the center position and width and height offset of the traffic light inner frame relative to the grid. N≥2, M≥2.
[0007] Preferably, the traffic light detection method further includes:
[0008] Based on the traffic light detection results, determine whether the traffic light image to be detected contains a digital shape;
[0009] If the traffic light image to be detected contains a number shape, a pre-set countdown recognition model is used to recognize the traffic light image to obtain the traffic light countdown.
[0010] Preferably, the step of using a pre-set countdown recognition model to recognize the traffic light image to be detected and obtain the traffic light countdown includes:
[0011] The digital shape region in the traffic light image to be detected is cropped to obtain a valid traffic light image;
[0012] Based on the valid traffic light images, obtain at least one valid digital image;
[0013] A pre-set countdown recognition model is used to identify at least one of the valid digital images to obtain the traffic light countdown.
[0014] Preferably, obtaining at least one valid digital image based on the valid traffic light image includes:
[0015] Obtain the effective aspect ratio corresponding to the effective traffic light image, and compare the effective aspect ratio with the aspect ratio threshold;
[0016] If the effective aspect ratio is less than the aspect ratio threshold, then the effective traffic light image is determined as a valid digital image;
[0017] If the effective aspect ratio is not less than the aspect ratio threshold, then the effective traffic light image is segmented into at least two effective digital images.
[0018] Preferably, the step of using a pre-set countdown recognition model to recognize at least one of the valid digital images and obtain the traffic light countdown includes:
[0019] A countdown recognition model is used to identify each of the aforementioned valid digit images and obtain the recognition confidence of all standard digits;
[0020] Based on the recognition confidence of all the standard digits, obtain the valid digits corresponding to the valid digit image;
[0021] The traffic light countdown is obtained based on at least one of the valid digit images corresponding to the valid digits.
[0022] Preferably, obtaining the valid digits corresponding to the valid digit image based on the recognition confidence of all the standard digits includes:
[0023] From the recognition confidence of at least one of the standard digits, select the standard digit images corresponding to the top M standard digits with higher recognition confidence;
[0024] Structural similarity is determined between the valid digital image and the standard digital image corresponding to each standard digital image to obtain the structural similarity corresponding to the standard digital image;
[0025] Based on the recognition confidence and structural similarity corresponding to the standard number, the target recognition probability corresponding to the standard number is obtained;
[0026] The standard number with the highest target recognition probability is determined as the valid number corresponding to the valid number image.
[0027] Preferably, the traffic light detection method further includes:
[0028] Obtain model training samples and divide the model training samples into training set and test set; the model training samples include training traffic light images and training labels, the training labels include outer frame confidence, outer frame position and at least one inner frame attribute, each inner frame attribute includes inner frame confidence, inner frame position, inner frame color and inner frame shape;
[0029] The model training samples in the training set are input into the target neural network model for model training to obtain the original traffic light detection model;
[0030] The original traffic light detection model was tested using the model training samples in the test set to obtain the recognition accuracy of the original traffic light detection model.
[0031] If the recognition accuracy is greater than the accuracy threshold, then the original traffic light detection model is determined as the target traffic light detection model.
[0032] Preferably, the step of inputting the model training samples from the training set into the target neural network model for model training to obtain the original traffic light detection model includes:
[0033] The model training samples in the training set are input into the target neural network model for model training, and the recognition labels corresponding to the model training samples are obtained.
[0034] Based on the recognition labels and training labels corresponding to all model training samples, calculate the confidence loss value, position loss value, and category loss value. The confidence loss value is the loss value corresponding to the confidence of the outer frame and the confidence of the inner frame. The position loss value is the loss value corresponding to the position of the outer frame and the position of the inner frame. The category loss value is the loss value corresponding to the color of the inner frame and the shape of the inner frame.
[0035] Based on the confidence loss value, the location loss value, and the category loss value, the original traffic light detection model is obtained when the model convergence condition is met.
[0036] This invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the traffic light detection method described above.
[0037] This invention provides a vehicle, including a camera module and an on-board controller. The camera module is used to acquire images of traffic lights to be detected and send the images to the on-board controller. The on-board controller is used to execute the traffic light detection method described above.
[0038] The traffic light detection method, computer equipment, and vehicle provided in this invention employ a target traffic light detection model to identify the traffic light image to be detected. This allows the traffic light detection results to include the confidence level and position corresponding to the outer frame of the traffic light, as well as the inner frame attributes corresponding to at least one inner frame of the traffic light. These inner frame attributes include information such as confidence level, position, color, and shape, which helps improve the versatility and accuracy of traffic light detection. Furthermore, when using the target traffic light detection model to detect the traffic image to be detected, N grids are used for identification, and the detection results for each grid at M scales are output. This results in a wider coverage and more accurate output, helping to ensure the accuracy of the traffic light detection results. Attached Figure Description
[0039] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0040] Figure 1 This is a flowchart of a traffic light detection method according to an embodiment of the present invention;
[0041] Figure 2 This is a flowchart of a traffic light detection method according to an embodiment of the present invention;
[0042] Figure 3 This is another flowchart of a traffic light detection method according to one embodiment of the present invention;
[0043] Figure 4 This is another flowchart of a traffic light detection method according to one embodiment of the present invention;
[0044] Figure 5 This is another flowchart of a traffic light detection method according to one embodiment of the present invention;
[0045] Figure 6 This is another flowchart of a traffic light detection method according to one embodiment of the present invention;
[0046] Figure 7 This is another flowchart of a traffic light detection method according to one embodiment of the present invention;
[0047] Figure 8 This is another flowchart of a traffic light detection method in one embodiment of the present invention. Detailed Implementation
[0048] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0049] This invention provides a traffic light detection method that can be applied to a computer device. The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the traffic light detection method. In this example, the computer device can be an onboard controller installed in a vehicle, or it can be other devices.
[0050] In one embodiment, such as Figure 1 As shown, a traffic light detection method is provided. Taking the application of this traffic light detection method in a computer device as an example, the computer device can be an on-board controller installed in a vehicle. The traffic light detection method includes the following steps:
[0051] S101: Acquire the image of the traffic light to be detected;
[0052] S102: Using a pre-trained target traffic light detection model, the traffic light image to be detected is divided into N grids for recognition. The detection results of traffic lights at M scales corresponding to each grid are output. Each traffic light detection result includes the outer frame confidence, outer frame position and at least one inner frame attribute. Each inner frame attribute includes the inner frame confidence, inner frame position, inner frame color and inner frame shape. The outer frame position includes the center position and width and height offset of the traffic light outer frame relative to the grid. The inner frame position includes the center position and width and height offset of the traffic light inner frame relative to the grid. N≥2, M≥2.
[0053] Among them, the traffic light image to be detected refers to the traffic light image that needs to be detected.
[0054] As an example, in step S101, the computer device can acquire the image of the traffic light to be detected captured at the current moment, so as to detect and identify the traffic light detection result in the image of the traffic light to be detected. In this example, when the computer device is an on-board controller installed in the car, the on-board controller is connected to a camera and can receive the image of the traffic light to be detected captured by the camera in real time in front of the car, so as to identify the traffic light detection result in the image of the traffic light to be detected, so as to assist the intelligent driving of the vehicle based on the traffic light detection result.
[0055] The traffic light detection results include the following: Outer frame confidence score, which reflects whether the outer frame of the traffic light is illuminated; and Inner frame attributes. Outer frame confidence score reflects whether the outer frame of the traffic light is illuminated in the image of the traffic light being tested. Outer frame position reflects the center position and width / height offset of the outer frame relative to the preset detection frame; and Inner frame color reflects the color of the inner frame. The inner frame shape is information used to reflect the shape of the inner frame of the traffic light. The inner frame shape in the traffic light detection result can be understood as the true shape of the inner frame of the traffic light in the image of the traffic light to be tested.
[0056] As an example, in step S102, after acquiring the traffic light image to be detected, the computer device can call a pre-set target traffic light detection model to recognize the traffic light image and obtain the traffic light detection result. In this example, the target traffic light detection model is used to divide the traffic light image to be detected into N (N≥2) grids for recognition, and output the traffic light detection result corresponding to M (M≥2) scales for each grid. The traffic light detection result includes the outer frame confidence score, the outer frame position, and at least one inner frame attribute. Each inner frame attribute includes the inner frame confidence score, the inner frame position, the inner frame color, and the inner frame shape.
[0057] In a specific example, when using a YOLOv3-based traffic light detection model, the traffic light image to be detected can be divided into three grids of different scales. Each grid can output traffic light detection results relative to preset detection boxes at different scales. For example, each grid can output traffic light detection results at three different scales: 19*19, 38*38, and 76*76. Each traffic light detection result includes outer frame confidence, outer frame position, and at least one inner frame attribute. Each inner frame attribute includes inner frame confidence, inner frame position, inner frame color, and inner frame shape. The outer frame confidence and inner frame confidence can be represented by a single-digit number; for example, 1 indicates the light is on, and 0 indicates the light is off. The positions of the outer and inner frames can be represented by 4 digits, representing the center position of the traffic light's outer and inner frames relative to the preset detection frame, as well as their width and height offsets. The color of the inner frame can be represented by 4 digits, for example, 0 for red, 1 for yellow, and 2 for green. The shape of the inner frame can be represented by 13 digits, for example, 0 for circle, 1 for straight ahead, 2 for left turn, 3 for right turn, 4 for U-turn, 5 for straight ahead and left turn, 6 for straight ahead and right turn, 7 for permission to proceed (green light arrow pointing downwards), 8 for prohibition of passage, 9 for numbers, 10 for rectangular lights, 11 for bicycles, and 12 for pedestrians.
[0058] Generally, each traffic light includes a maximum of 3 inner frames. Therefore, the output detection result for each traffic light includes the outer frame confidence score (1 bit) and outer frame position (4 bits) corresponding to the outer frame, as well as the inner frame confidence score (1 bit), inner frame position (4 bits), inner frame color (3 bits), and inner frame shape (13 bits) corresponding to the 3 inner frames. That is, each output traffic light detection result includes a 68-bit value, meaning each grid outputs 204-bit values for the 3 traffic light detection results at different scales. The outer frame position and inner frame position here can be understood as the center position and width / height displacement of the traffic light's outer and inner frames relative to the preset detection frame. During traffic light detection, the 3 inner frames are arranged from top to bottom or left to right according to the illuminated area. If there are fewer than 3 illuminated areas, for example, only one inner frame, the inner frame confidence score of the first inner frame can be detected as 1, and the inner frame confidence scores of the subsequent 2 inner frames as 0. During the detection process of the target traffic light detection model, 3*3=9 preset detection boxes with different widths and heights are required. The traffic image to be detected can be divided into 3 grids of different scales, and each grid needs to output 3 traffic light detection results of different scales.
[0059] In this example, a target traffic light detection model is used. When detecting traffic lights in the image, if the deviation between the traffic light position in the image and the j-th preset detection box in the i-th grid is the smallest, then the j-th preset detection box in the i-th grid is used to detect that traffic light. The preset detection boxes here are selected from the nine cluster values chosen by clustering the width and height of all target detection boxes in the dataset before the current time step. These nine preset detection boxes should include three different size levels and three different grid sizes. For example, a 19*19 grid is larger and is responsible for detecting larger boxes, while a 76*76 grid is smaller and performs better in detecting smaller boxes, helping to ensure the accuracy of the traffic light detection results.
[0060] The traffic light detection method provided in this embodiment uses a target traffic light detection model to identify the traffic light image to be detected. This allows the traffic light detection results to include the confidence level and position of the traffic light's outer frame, as well as the inner frame attributes corresponding to at least one inner frame. These inner frame attributes include information such as confidence level, position, color, and shape, which helps improve the versatility and accuracy of traffic light detection. Furthermore, when using the target traffic light detection model to detect the traffic image, N grids are used for identification, and the model can output traffic light detection results at M scales for each grid. This results in a wider coverage and more accurate output, helping to ensure the accuracy of the traffic light detection results.
[0061] In one embodiment, such as Figure 2 As shown, traffic light detection methods also include:
[0062] S201: Based on the traffic light detection results, determine whether the traffic light image to be detected contains digital shapes;
[0063] S202: If the traffic light image to be detected contains a number shape, a pre-set countdown recognition model is used to recognize the traffic light image to obtain the traffic light countdown.
[0064] Among them, the shape of the number refers to the shape attribute of the number displayed on the traffic light. This shape of the number is generally used in scenarios with traffic light countdowns.
[0065] As an example, in step S201, after acquiring the traffic light detection results, the computer device can extract the recognition feature value corresponding to the position of the digital attribute from the traffic light detection results. If the recognition feature value is the target feature value, it is determined that the traffic light image to be detected contains a digital shape; if the recognition feature value is not the target feature value, it is determined that the traffic light image to be detected does not contain a digital shape. Here, the position of the digital attribute reflects the location of the digital attribute in the traffic light detection results. The recognition feature value refers to the feature value corresponding to the position of the digital attribute in the traffic light detection results, and can be 0 or 1. The target feature value is the feature value used to reflect the presence of a digital shape in the traffic light image to be detected; for example, 1 indicates the presence of a digital shape.
[0066] For example, in the traffic light detection results output by the target traffic light detection model, the outer frame confidence and inner frame confidence are represented by a single digit, for example, 1 indicates the light is on and 0 indicates the light is off. The outer frame position and inner frame position are represented by a 4-digit number, representing the center position of the traffic light's outer frame and inner frame relative to the preset detection frame, as well as their width and height offsets, respectively; the inner frame color is represented by a 4-digit number, for example, 0 indicates red, 1 indicates yellow, and 2 indicates green; its inner frame shape is represented by a 13-digit number, for example, 0 indicates a circle, 1 indicates straight ahead, 2 indicates left turn, 3 indicates right turn, 4 indicates U-turn, 5 indicates straight ahead and left turn, 6 indicates straight ahead and right turn, 7 indicates permission to proceed (green light arrow pointing downwards), 8 indicates prohibition to proceed, 9 indicates a number, 10 indicates a rectangular light, 11 indicates a bicycle, and 12 indicates a pedestrian. In this example, the traffic light detection result consists of 68 bits of 0 / 1 feature values. Specifically, the digital attribute position is the 10th shape under the inner frame shape in the three inner frame attributes, which is located at bits 23, 44, and 65 in the traffic light detection result. At this time, the computer device can determine the 0 / 1 feature values corresponding to the positions of bits 23, 44, and 65 in the traffic light detection result as the recognition feature values corresponding to the digital attribute positions. When the recognition feature value is the target feature value (such as 1), it can be determined that the traffic light image to be detected contains digital shapes. When the recognition feature value is not the target feature value (such as 0), it can be determined that the traffic light image to be detected does not contain digital shapes.
[0067] The countdown recognition model is a pre-trained model used to recognize traffic light countdowns. Understandably, since traffic light countdowns are numerical, a text recognition model can be used as the countdown recognition model, or a model determined by training on the MNIST digit set can be employed.
[0068] As an example, in step S202, the computer device can use a pre-set countdown recognition model to recognize the digital shapes in the traffic light image to be detected, so as to determine the traffic light countdown in the traffic light image to be detected, thereby realizing the recognition of the traffic light countdown in the traffic light image to be detected and ensuring the accuracy of traffic light detection.
[0069] In one embodiment, such as Figure 3 As shown, step S202, that is, if the traffic light image to be detected contains a digital shape, then a pre-set countdown recognition model is used to recognize the traffic light image to be detected and obtain the traffic light countdown, includes:
[0070] S301: Crop the digital shape region in the traffic light image to be detected to obtain a valid traffic light image;
[0071] S302: Based on valid traffic light images, obtain at least one valid digital image;
[0072] S303: Using a pre-set countdown recognition model, at least one valid digital image is recognized to obtain the traffic light countdown.
[0073] As an example, in step S301, during the countdown recognition process of the traffic light image to be detected, the computer device can first crop the numerical shape region in the traffic light image to be detected, so as to crop out only the numerical shape region corresponding to the traffic light countdown, thereby obtaining a valid traffic light image. Here, a valid traffic light image refers to an image that only contains the numerical shape region corresponding to the traffic light countdown.
[0074] In this example, the computer device can use binarization and edge cropping techniques to perform binarization and edge cropping on the traffic light image to be detected, so as to obtain a valid traffic light image that only contains the digital shape region corresponding to the traffic light countdown.
[0075] As an example, in step S302, after acquiring a valid traffic light image, the computer device can evaluate the countdown number corresponding to the traffic light countdown in the valid traffic light image. If the countdown number is a single digit, such as 0-9, the valid traffic light image can be directly determined as a valid number image. If the countdown number is not a single digit, that is, a combination of at least two single digits, such as 11, the valid traffic light image needs to be segmented to form a valid number image containing only a single digit. Here, a valid number image can be understood as an image containing only a single digit.
[0076] As an example, in step S303, after the computer device obtains at least one valid digital image based on the valid traffic light image, it can use a countdown recognition model to recognize the at least one valid digital image and obtain the valid number corresponding to each valid digital image; then, it splices the valid numbers corresponding to the at least one valid digital image to obtain the traffic light countdown.
[0077] Understandably, by determining the valid digit images containing only a single digit based on the valid traffic light images, and then using a countdown recognition model to identify the valid digit images, the valid digits corresponding to each valid digit image can be accurately identified, ensuring the accuracy and efficiency of all valid digit recognition. Finally, the valid digits corresponding to all valid digit images are spliced together to form a traffic light countdown, which can ensure the accuracy and efficiency of traffic light countdown recognition.
[0078] In one embodiment, such as Figure 4 As shown, step S302, which involves obtaining at least one valid digital image based on valid traffic light images, includes:
[0079] S401: Obtain the effective aspect ratio corresponding to the valid traffic light image, and compare the effective aspect ratio with the aspect ratio threshold;
[0080] S402: If the effective aspect ratio is less than the aspect ratio threshold, then the effective traffic light image is determined as a valid digital image;
[0081] S403: If the effective aspect ratio is not less than the aspect ratio threshold, then the effective traffic light image is segmented into at least two effective digital images.
[0082] The effective aspect ratio can be understood as the ratio of the width to the height of an effective traffic light image. The aspect ratio threshold is a pre-set threshold used to evaluate whether an image contains a single digit.
[0083] As an example, in step S401, the computer device can identify the effective aspect ratio corresponding to the valid traffic light image, that is, the ratio of the width to the height of the valid traffic light image is determined as the effective aspect ratio. Then, the computer device can compare the effective aspect ratio corresponding to the valid traffic light image with a preset aspect ratio threshold to assess whether the valid traffic light image contains only a single digit.
[0084] As an example, in step S402, the computer device compares the effective aspect ratio of the valid traffic light image with the aspect ratio threshold. When the effective aspect ratio is less than the aspect ratio threshold, it is determined that the effective aspect ratio of the valid traffic light image is small, and the valid traffic light image is very likely to contain only a single digit. Therefore, the valid traffic light image can be determined as a valid digit image containing only a single digit.
[0085] As an example, in step S403, the computer device compares the effective aspect ratio of the valid traffic light image with the aspect ratio threshold. When the effective aspect ratio is not less than the aspect ratio threshold, it is determined that the effective aspect ratio of the valid traffic light image is large, and the valid traffic light image cannot contain only a single digit. That is, it is determined that the valid traffic light image contains at least two single digits. Therefore, the valid traffic light image needs to be segmented to obtain at least two valid digit images.
[0086] Understandably, based on the comparison between the effective aspect ratio of the valid traffic light image and the pre-set aspect ratio threshold, it is evaluated whether the valid traffic light image contains at least two single digits to determine whether it is necessary to segment the valid traffic light image to obtain a valid digit image containing only a single digit, so as to ensure the accuracy and efficiency of subsequent countdown recognition using the valid digit image.
[0087] In one embodiment, such as Figure 5 As shown, step S303, which involves using a pre-set countdown recognition model to recognize at least one valid digital image and obtain the traffic light countdown, includes:
[0088] S501: Using a pre-set countdown recognition model, each valid digit image is recognized to obtain the recognition confidence of all standard digits;
[0089] S502: Based on the recognition confidence of all standard digits, obtain the valid digits corresponding to the valid digit image;
[0090] S501: Obtain the traffic light countdown based on the valid digits corresponding to at least one valid digit image.
[0091] In this context, "standard digits" refers to the digits pre-used in the countdown recognition model, such as the standard format digits 0, 1, 2, 3, 4, 5, 6, 7, 8, and 9. "Recognition confidence" refers to the confidence / probability with which the countdown recognition model identifies a valid digit image as a specific standard digit.
[0092] As an example, in step S501, the computer device can use a pre-set countdown recognition model to recognize each valid digital image and obtain the recognition confidence level corresponding to all standard numbers corresponding to the countdown recognition model, that is, obtain the recognition confidence level corresponding to standard numbers such as 0, 1, 2, 3, 4, 5, 6, 7, 8 and 9.
[0093] As an example, in step S502, after acquiring the recognition confidence scores corresponding to all standard digits, the computer device can employ a pre-configured target filtering strategy to filter the recognition confidence scores of all standard digits, determining the standard digits corresponding to recognition confidence scores that meet the filtering conditions as valid digits corresponding to the valid digit image. The target filtering strategy here is a pre-set strategy for filtering and determining valid digits. For example, when executing the target filtering strategy, the computer device can select the standard digit with the highest recognition confidence score from all the recognition confidence scores corresponding to all standard digits and determine it as a valid digit; or it can select the standard digit with a recognition confidence score greater than a confidence threshold from all the recognition confidence scores corresponding to all standard digits and determine it as a valid digit.
[0094] As an example, in step S503, after the computer device obtains the valid numbers corresponding to at least one valid digital image, it can splice at least one valid number according to the order of the at least one valid digital image to obtain the traffic light countdown.
[0095] Understandably, a countdown recognition model is used to identify the valid digit images, identify the recognition confidence level corresponding to all standard digits, and then select the valid digits corresponding to each valid digit image based on the recognition confidence level, ensuring the accuracy and efficiency of all valid digits; finally, the valid digits corresponding to all valid digit images are spliced together to form a traffic light countdown, which can ensure the accuracy and efficiency of traffic light countdown recognition.
[0096] In one embodiment, such as Figure 6 As shown, step S502, which involves obtaining the valid digits corresponding to the valid digit image based on the recognition confidence of all standard digits, includes:
[0097] S601: Select the standard digit images corresponding to the top M standard digits with higher recognition confidence from at least one standard digit;
[0098] S602: Perform structural similarity judgment on the valid digit image and the standard digit image corresponding to each standard digit, and obtain the structural similarity corresponding to the standard digit;
[0099] S603: Obtain the target recognition probability corresponding to the standard number based on the recognition confidence and structural similarity.
[0100] S604: The standard number with the highest target recognition probability is determined as the valid number corresponding to the valid number image.
[0101] Where M is a pre-set number of filters; for example, M can be set to 3. The standard digit image is an image containing standard digits used during the training of the countdown recognition model.
[0102] As an example, in step S601, the computer device obtains the standard digit images corresponding to the top M standard digits with higher recognition confidence from the recognition confidence of at least one standard digit. That is, from the recognition confidence of all standard digits corresponding to each valid digit image, the standard digit images corresponding to the top M standard digits with higher probability of determination are selected.
[0103] As an example, in step S602, the computer device can use a structural similarity (SSIM) algorithm to determine the structural similarity between the valid digit image and the standard digit image corresponding to each standard digit, obtaining the structural similarity corresponding to each standard digit. This structural similarity can reflect the structural similarity of the digits in the two images to a certain extent. For example, a structural similarity algorithm can be used. Where x and y refer to the effective digital image and the standard digital image, respectively, μ is the gray mean, σ is the gray standard deviation, and c1 and c2 are constants used to maintain stability.
[0104] As an example, in step S603, the computer device can perform weighted processing on the recognition confidence and structural similarity corresponding to each standard number to obtain the target recognition probability corresponding to each standard number, so that the target recognition probability is related to both the recognition confidence identified by the countdown recognition model and the shape and structure of the numbers in the image, which helps to ensure the accuracy of traffic light countdown recognition.
[0105] As an example, in step S604, the computer device can select the standard number with the highest target recognition probability based on the target recognition probabilities corresponding to the top M standard numbers with higher recognition confidence, and determine it as the valid number corresponding to the valid number image.
[0106] Understandably, since countdown recognition models are generally trained using handwritten digit images, and the forms of handwritten digit images differ from those of valid digit images displayed on LEDs, using a countdown recognition model to identify valid digit images for individual similar digits displayed on LEDs (such as 5 and 6) may result in false detections. That is, the recognition confidence levels of individual similar digits are not significantly different, which is not conducive to accurately identifying their valid digits. Therefore, it is advisable to select the top M standard digits with higher recognition confidence and use the standard digit images corresponding to these M standard digits for structural similarity verification. This helps to ensure the robustness of the algorithm and improve the accuracy of the recognition results.
[0107] In one embodiment, such as Figure 7 As shown, traffic light detection methods also include:
[0108] S701: Obtain model training samples and divide the model training samples into training set and test set; the model training samples include training traffic light images and training labels. The training labels include outer frame confidence, outer frame position and at least one inner frame attribute. Each inner frame attribute includes inner frame confidence, inner frame position, inner frame color and inner frame shape.
[0109] S702: Input the model training samples in the training set into the target neural network model to train the model and obtain the original traffic light detection model;
[0110] S703: Test the original traffic light detection model using model training samples in the test set to obtain the recognition accuracy of the original traffic light detection model;
[0111] S704: If the recognition accuracy is greater than the accuracy threshold, the original traffic light detection model will be determined as the target traffic light detection model.
[0112] Among them, the training traffic light image is an image containing traffic lights used to implement model training.
[0113] The training labels are labels that are used to annotate the traffic light images before model training. As an example, the training labels include outer frame confidence, outer frame position, and at least one inner frame attribute. Each inner frame attribute includes inner frame confidence, inner frame position, inner frame color, and inner frame shape.
[0114] Among them, the preset detection box is a detection box that is set in advance.
[0115] In this example, before model training, different detection boxes can be pre-set according to the different scale requirements corresponding to the target neural network model. The length and width of all detection boxes existing before the current time step can be clustered, and the detection boxes with the largest number of clusters can be selected as preset detection boxes. This allows for the selection of preset detection boxes with the smallest deviation from the traffic light image during subsequent model training and recognition, improving processing efficiency and accuracy. The number of preset detection boxes can be set by the user according to actual needs. Considering that traffic light detection is mainly used in scenarios that assist vehicle driving, the area occupied by traffic lights in the image varies as the vehicle moves from far to near. To cover multiple situations, the area of the preset detection boxes can be divided into three different size levels. Furthermore, the length-to-width ratio of traffic lights can be horizontal, vertical, or square, so to cover multiple situations, the length-to-width ratio should also have three different cases. Therefore, approximately 3*3 different sizes of preset detection boxes are preset. Since the amount of data in the training set is very large, it is difficult to summarize subjectively. Therefore, using a clustering method to select the scale corresponding to 9 cluster points to determine the preset detection boxes is faster and more effective than manually setting the size of the preset detection boxes. During the model design process, the 9 preset detection boxes include 3 different size levels, which ensures that the subsequent model output can have 3 different grid outputs. It can output the traffic light detection results corresponding to the 3 scales of each grid. For example, the 19*19 grid is larger and is responsible for detecting larger boxes, while the 76*76 grid is smaller and is better at detecting smaller boxes.
[0116] As an example, in step S701, the computer device can obtain training traffic light images for model training from a database, or obtain them through web crawling. Next, the computer device can receive training labels annotated by the user on the training traffic light images, and determine model training samples based on the training traffic light images and training labels. Finally, the computer device can divide all model training samples according to a specific partitioning rule to determine the training set and the test set; for example, it can partition all model training samples according to an 8:2 partitioning ratio to determine the training set and the test set.
[0117] In this example, the training labels for training traffic light images include outer frame confidence, outer frame position, and at least one inner frame attribute. Each inner frame attribute includes inner frame confidence, inner frame position, inner frame color, and inner frame shape.
[0118] Generally, the confidence levels of the outer and inner frames can be represented by a single digit, for example, 1 for a lit light and 0 for a non-lit light. The positions of the outer and inner frames can be represented by four digits, representing the center position of the traffic light's outer and inner frames relative to the preset detection frame and their width and height offsets, respectively. The color of the inner frame can be represented by four digits, for example, 0 for red, 1 for yellow, and 2 for green. The shape of the inner frame can be represented by 13 digits, for example, 0 for a circle, 1 for straight ahead, 2 for left turn, 3 for right turn, 4 for U-turn, 5 for straight ahead and left turn, 6 for straight ahead and right turn, 7 for permission to proceed (green light arrow pointing downwards), 8 for prohibition of passage, 9 for a number, 10 for a rectangular light, 11 for bicycles, and 12 for pedestrians. Each traffic light includes up to 3 inner frames. The confidence level of the outer frame, the position of the outer frame, and at least one inner frame attribute can be sorted in a specific order. The confidence level, position, color, and shape of each inner frame attribute are sorted to form a 39-bit training label. The true information can be determined based on the specific values in the training label.
[0119] In this example, the inner frame color and shape are labeled using a dual-label method, meaning each has a separate label. This helps reduce the dimensionality of the training labels and facilitates faster convergence of the model training. For example, the inner frame color can be represented by a 3-bit 0 / 1 vector, and the inner frame shape by a 13-bit 0 / 1 vector. For instance, green can be represented by 100, and a straight line can be represented by 1000000000000. This dual-label method allows for a 3+13 dimension representation of a green straight line. In contrast, using a single-label method for the combination of inner frame color and shape requires a 3*13 dimensional 0 / 1 vector (e.g., green straight line: 1000...000 (38 zeros)). This longer dimensionality of the training labels can negatively impact the convergence speed of subsequent model training and recognition.
[0120] The target neural network model is the neural network model used for training in this example. For instance, the target neural network model can be YOLOv3, which includes convolutional layers, normalization layers, and activation layers, and can output detection results at multiple different scales. For example, the target neural network model can output detection results at three different scales: 19*19, 38*38, and 76*76, representing the image being divided into grids of different scales. Each grid will output three detection results relative to the preset detection boxes at different scales. In this example, the number of layers and channels per layer in the target neural network model is moderate to ensure the accuracy and efficiency of the final target traffic light detection model.
[0121] The original traffic light detection model is the model used to implement traffic light detection, which was determined during model training and is an untested model.
[0122] As an example, in step S702, the computer device can input model training samples from the training set into the target neural network model for model training, thereby updating and adjusting the model parameters in the target neural network model. Specifically, it updates and adjusts the model parameters corresponding to the convolutional layers, normalization layers, and activation layers in the target neural network model to obtain an original traffic light detection model capable of recognizing traffic light detection results. Understandably, since the training labels of the model training samples can use a dual-label method for inner frame color and shape, the dimensionality of the training labels is shorter, which helps ensure rapid convergence of the original traffic light detection model training process and improves the model training speed. Furthermore, the training labels include outer frame confidence, outer frame position, and at least one inner frame attribute, and the inner frame attribute includes information such as inner frame confidence, inner frame position, inner frame color, and inner frame shape, which helps ensure the accuracy of subsequent traffic light detection.
[0123] As an example, in step S703, the computer device can input the model training samples from the test set into the original traffic light detection model for model testing, and obtain the test result corresponding to each model training sample output by the original traffic light detection model; if the test result corresponding to the model training sample is the same as the training label, then the test result of the model training sample is considered accurate; the number of accurate samples corresponding to the model training samples with accurate test results is counted and the total number of samples of all model training samples in the test set is compared, and the ratio of the number of accurate samples to the total number of samples is used to determine the recognition accuracy of the original traffic light detection model.
[0124] The accuracy threshold is a pre-set threshold used to evaluate whether the accuracy of the original traffic light detection model meets the standard.
[0125] As an example, in step S704, the computer device can compare the recognition accuracy of the original traffic light detection model with a preset accuracy threshold. If the recognition accuracy is greater than the accuracy threshold, it is determined that the recognition accuracy of the original traffic light detection model is high. At this time, the original traffic light detection model can be determined as the target traffic light detection model to ensure the recognition accuracy of the target traffic light detection model.
[0126] In the traffic light detection model training method provided in this embodiment, the training labels for the model training samples can use a dual-label method for inner frame color and inner frame shape, which shortens the dimension of the training labels and helps ensure rapid convergence of the target traffic light detection model training process, thus improving the model training speed. The training labels include outer frame confidence, outer frame position, and at least one inner frame attribute. The inner frame attributes include information such as inner frame confidence, inner frame position, inner frame color, and inner frame shape, ensuring that the target traffic light detection model can recognize the traffic light outer frame and its inner frame list, specifically including information such as illumination, position, color, and shape, which helps improve the universality and accuracy of traffic light detection. During model training, only the original traffic light detection models with an accuracy rate greater than the accuracy threshold are identified as the target traffic light detection models, which helps ensure the recognition accuracy of the target traffic light detection model.
[0127] In one embodiment, such as Figure 8 As shown, step S702 involves inputting the model training samples from the training set into the target neural network model for model training to obtain the original traffic light detection model, including:
[0128] S801: Input the model training samples in the training set into the target neural network model to train the model and obtain the recognition labels corresponding to the model training samples;
[0129] S802: Based on the recognition labels and training labels corresponding to all model training samples, calculate the confidence loss value, position loss value and category loss value. The confidence loss value is the loss value corresponding to the confidence of the outer frame and the confidence of the inner frame. The position loss value is the loss value corresponding to the position of the outer frame and the position of the inner frame. The category loss value is the loss value corresponding to the color of the inner frame and the shape of the inner frame.
[0130] S803: Based on the confidence loss value, location loss value, and category loss value, evaluate the model convergence conditions and obtain the original traffic light detection model.
[0131] Among them, the identification label is the label identified during the processing of the model training samples by the target neural network model, and it is the label for the identification and prediction of the model training samples.
[0132] As an example, in step S801, the computer device can input model training samples from the training set into the target neural network model for model training. It can obtain the recognition label corresponding to each model training sample. The recognition label has the same format as the training label and also includes information such as outer frame confidence, outer frame position, and at least one inner frame attribute. Each inner frame attribute also includes information such as inner frame confidence, inner frame position, inner frame color, and inner frame shape. Understandably, the training label of the model training sample reflects the actual information of the traffic lights in the model training sample, while the recognition label is the predicted information of the traffic lights in the model training sample.
[0133] As an example, in step S802, the computer device can calculate the model loss value based on the recognition labels and training labels of all model training samples in the test set. The model loss value here includes the confidence loss value calculated based on the confidence of the outer frame and the confidence of the inner frame, the position loss value calculated based on the position of the outer frame and the position of the inner frame, and the category loss value determined based on the color and shape of the inner frame.
[0134] In this example, a squared loss function can be used to calculate the confidence scores for the outer and inner bounding boxes in the identified labels, as well as the outer and inner bounding box confidence scores in the training labels, to obtain the confidence loss value. In this example, the confidence loss value is... L0 is the confidence loss value, out_conf * `out_conf` represents the confidence score of the bounding boxes in the training labels, which is the true value of the training samples; `out_conf` represents the confidence score of the bounding boxes in the recognition labels, which is the predicted value of the training samples; `in_conf` represents the confidence score of the bounding boxes in the recognition labels. * `in_conf` represents the confidence score of the inner bounding boxes in the training labels, which is the true value of the training samples; `in_conf` represents the confidence score of the inner bounding boxes in the recognition labels, which is the predicted value of the training samples; `N` represents the number of inner bounding boxes.
[0135] In this example, the squared loss function can be used to calculate the positional loss value for both the outer and inner bounding box positions in the identified labels and the outer and inner bounding box positions in the training labels. In this example, the positional loss value is... L1 is the location loss value, out_loc * `out_loc` represents the bounding box position in the training labels, which is the ground truth value of the training samples; `out_loc` represents the bounding box position in the recognition labels, which is the predicted value of the training samples; `in_loc` represents the bounding box position in the recognition labels. * `in_loc` represents the position of the inner frame in the training label, which is the true value of the training sample; `in_loc` represents the position of the inner frame in the recognition label, which is the predicted value of the training sample; `N` represents the number of inner frames.
[0136] In this example, the cross-entropy loss function can be used to calculate the category loss value by analyzing the inner frame color and shape in the identification labels and the inner frame color and shape in the training labels. In this example, the category loss value is... L2 is the class loss value; I k Indicates whether the k-th inner frame is lit; c * is the inner frame color in the training label, and is the true value of the training sample; c is the inner frame color in the recognition label, and is the predicted value of the training sample; s *is the inner frame shape in the training label, which is the true value of the training sample of the model, and s is the inner frame shape in the recognition label, which is the predicted value of the training sample of the model.
[0137] As an example, in step S803, the computer device can evaluate whether the target neural network model meets the model convergence condition based on the confidence loss value, location loss value, and category loss value. If the model convergence condition is met, the original traffic light detection model can be obtained. In this example, the computer device can perform weighted processing on the confidence loss value, location loss value, and category loss value to obtain the total loss value. When the total loss value meets the model convergence condition, the gradient descent method is used to optimize the model, thereby obtaining the original traffic light detection model.
[0138] Understandably, since the training labels and recognition labels of the model training samples both contain outer frame confidence, outer frame position, and at least one inner frame attribute, and the inner frame attribute includes information such as inner frame confidence, inner frame position, inner frame color, and inner frame shape, the confidence loss value, position loss value, and category loss value can be calculated based on the training labels and recognition labels respectively. Then, based on the confidence loss value, position loss value, and category loss value, it is evaluated whether the model convergence condition is met. Only when the model convergence condition is met can the trained original traffic light detection model be obtained, which can ensure that the original traffic light detection model can identify the traffic light outer frame and its inner frame list, specifically including information such as lighting, position, color, and shape, which helps to improve the universality and accuracy of traffic light detection.
[0139] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the traffic light detection method in the above embodiment. To avoid repetition, it will not be described in detail here.
[0140] In one embodiment, a vehicle is provided, including a camera module and an on-board controller. The camera module is used to acquire images of traffic lights to be detected and send the images to the on-board controller. The on-board controller is used to execute the traffic light detection method in the above embodiment. To avoid repetition, it will not be described in detail here.
[0141] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.
[0142] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.
[0143] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.
Claims
1. A traffic light detection method, characterized in that, include: Acquire the image of the traffic light to be detected; A pre-trained target traffic light detection model is used to divide the traffic light image to be detected into N grids for identification. A preset detection box in the grid with the smallest deviation from the traffic light position in the image is used to detect the traffic lights. The model outputs traffic light detection results for each grid at M scales. Each detection result includes an outer frame confidence score, an outer frame position, and at least one inner frame attribute. Each inner frame attribute includes an inner frame confidence score, an inner frame position, an inner frame color, and an inner frame shape. The inner frame color is the true color of the inner frame of the traffic light in the image; the inner frame shape is the true shape of the inner frame of the traffic light in the image; the outer frame position includes the center position and width / height offset of the outer frame relative to the grid; the inner frame position includes the center position and width / height offset of the inner frame relative to the grid. N M .
2. The traffic light detection method as described in claim 1, characterized in that, The traffic light detection method also includes: Based on the traffic light detection results, determine whether the traffic light image to be detected contains a digital shape; If the traffic light image to be detected contains a number shape, a pre-set countdown recognition model is used to recognize the traffic light image to obtain the traffic light countdown.
3. The traffic light detection method as described in claim 2, characterized in that, The step of using a pre-set countdown recognition model to recognize the traffic light image to be detected and obtain the traffic light countdown includes: The digital shape region in the traffic light image to be detected is cropped to obtain a valid traffic light image; Based on the valid traffic light images, obtain at least one valid digital image; A pre-set countdown recognition model is used to identify at least one of the valid digital images to obtain the traffic light countdown.
4. The traffic light detection method as described in claim 3, characterized in that, The step of obtaining at least one valid digital image based on the valid traffic light image includes: Obtain the effective aspect ratio corresponding to the effective traffic light image, and compare the effective aspect ratio with the aspect ratio threshold; If the effective aspect ratio is less than the aspect ratio threshold, then the effective traffic light image is determined as a valid digital image; If the effective aspect ratio is not less than the aspect ratio threshold, then the effective traffic light image is segmented into at least two effective digital images.
5. The traffic light detection method as described in claim 3, characterized in that, The step of using a pre-set countdown recognition model to recognize at least one of the valid digital images and obtain the traffic light countdown includes: A countdown recognition model is used to identify each of the aforementioned valid digit images and obtain the recognition confidence of all standard digits; Based on the recognition confidence of all the standard digits, obtain the valid digits corresponding to the valid digit image; The traffic light countdown is obtained based on at least one of the valid digit images corresponding to the valid digits.
6. The traffic light detection method as described in claim 5, characterized in that, The step of obtaining the valid digits corresponding to the valid digit image based on the recognition confidence of all the standard digits includes: From the recognition confidence of at least one of the standard digits, select the standard digit images corresponding to the top M standard digits with higher recognition confidence; Structural similarity is determined between the valid digital image and the standard digital image corresponding to each standard digital image to obtain the structural similarity corresponding to the standard digital image; Based on the recognition confidence and structural similarity corresponding to the standard number, the target recognition probability corresponding to the standard number is obtained; The standard number with the highest target recognition probability is determined as the valid number corresponding to the valid number image.
7. The traffic light detection method according to any one of claims 1-6, characterized in that, The traffic light detection method also includes: Obtain model training samples and divide the model training samples into training set and test set; the model training samples include training traffic light images and training labels, the training labels include outer frame confidence, outer frame position and at least one inner frame attribute, each inner frame attribute includes inner frame confidence, inner frame position, inner frame color and inner frame shape; The model training samples in the training set are input into the target neural network model for model training to obtain the original traffic light detection model; The original traffic light detection model was tested using the model training samples in the test set to obtain the recognition accuracy of the original traffic light detection model. If the recognition accuracy is greater than the accuracy threshold, then the original traffic light detection model is determined as the target traffic light detection model.
8. The traffic light detection method as described in claim 7, characterized in that, The step of inputting the model training samples from the training set into the target neural network model for model training to obtain the original traffic light detection model includes: The model training samples in the training set are input into the target neural network model for model training, and the recognition labels corresponding to the model training samples are obtained. Based on the recognition labels and training labels corresponding to all model training samples, calculate the confidence loss value, position loss value, and category loss value. The confidence loss value is the loss value corresponding to the confidence of the outer frame and the confidence of the inner frame. The position loss value is the loss value corresponding to the position of the outer frame and the position of the inner frame. The category loss value is the loss value corresponding to the color of the inner frame and the shape of the inner frame. Based on the confidence loss value, the location loss value, and the category loss value, the original traffic light detection model is obtained when the model convergence condition is met.
9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the traffic light detection method as described in any one of claims 1 to 8.
10. A vehicle, comprising a camera module and an on-board controller, wherein the camera module is used to acquire images of traffic lights to be detected and to send the images of traffic lights to be detected to the on-board controller, characterized in that, The vehicle controller is used to execute the traffic light detection method according to any one of claims 1-8.
Citation Information
Patent Citations
Method for identifying display state of traffic signal lamp
CN103177256A
A traffic signal lamp identification method
CN109389079A
Traffic signal lamp identification method and device, electronic equipment and storage medium
CN111079563A
