Traffic signal light state processing method, intelligent device, and storage medium

Abstract

This application relates to the field of autonomous driving technology, specifically providing a traffic light state processing method, intelligent device, and storage medium, aiming to solve the problem of accurately acquiring the state of traffic lights. To this end, the method provided by this application includes acquiring the perception results of multiple consecutive frames of images of a traffic light, the perception results including the state of a single light within the traffic light, the single light state including color and countdown numbers; acquiring the countdown numbers of a single light in multiple consecutive frames, determining the countdown activation state based on the deviation between the countdown numbers; smoothing the color of the single light in multiple consecutive frames to obtain the smoothed color of the current frame; determining the final color of the current frame based on the smoothed color and the countdown number of the single light in the current frame; if the final color of the current frame differs from the previous color, updating the countdown activation state to inactive. Through the above method, the color and countdown activation state of a single light can be accurately acquired.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, specifically to a traffic light state processing method, intelligent device, and storage medium. Background Technology

[0002] When controlling autonomous driving at intersections, it is necessary to accurately obtain the status of traffic lights (such as light color, countdown information, etc.) and then determine whether to control the vehicle to pass through the intersection based on the status of the traffic lights. Currently, the main method for obtaining the status of traffic lights is to use the camera on the vehicle to capture images of the traffic lights, and then use a perception model to identify the images to obtain the status of the traffic lights. The perception model is a neural network model.

[0003] However, the current perception models exhibit instability in the resulting state information. For example, the same traffic light might appear red, green, and then red sequentially across three consecutive frames. This color change does not conform to the typical color variation pattern of traffic lights; rather, it indicates an abnormal color jump between adjacent frames. Controlling the vehicle based on these unstable state results increases driving risks and compromises road safety.

[0004] Accordingly, a new technical solution is needed in this field to solve the above problems. Summary of the Invention

[0005] In order to overcome the above-mentioned deficiencies, this application is made to solve, or at least partially solve, the technical problem of how to obtain accurate and stable traffic signal status.

[0006] In a first aspect, a traffic light state processing method is provided, applied to a smart device, the method comprising:

[0007] The perception results of acquiring multiple consecutive frames of traffic light images are obtained. The perception results include the single light status of a traffic light. The single light status includes color and countdown numbers. The color is the lighting color when the single light is lit or unknown.

[0008] The countdown numbers of the single lamp in the consecutive frames of images are obtained, and the countdown activation state of the single lamp is determined according to the deviation between the countdown numbers. The countdown activation state is either active or inactive.

[0009] The color of the single lamp in the consecutive multi-frame images is smoothed to obtain the smooth color of the single lamp in the current frame, where the current frame is the frame with the latest timestamp in the consecutive multi-frame images;

[0010] The final color of the single light in the current frame is determined based on the smooth color and the countdown number of the single light in the current frame;

[0011] The countdown activation state is updated based on the final color of the current frame; if the final color is different from the previous color, the countdown activation state is updated to inactive, where the previous color is the final color of the single lamp in the image preceding the current frame.

[0012] In one technical solution of the above-mentioned traffic light state processing method, determining the final color of a single light in the current frame based on the smooth color and the countdown number of the single light in the current frame includes:

[0013] When the smooth color is the highlighted color, the final color is the smooth color;

[0014] When the smooth color is unknown, if the countdown activation state is active, the final color is determined based on the countdown number of the single light in the current frame and the previous color; if the countdown activation state is inactive, the final color of the current frame is unknown.

[0015] In one technical solution of the above-mentioned traffic light state processing method, determining the countdown activation state of a single light based on the deviation between the countdown digits includes:

[0016] The countdown deviation between every two adjacent images in the continuous multi-frame images is obtained, and the countdown deviation is the deviation of the countdown numbers of the single lamp between the two adjacent images;

[0017] If the countdown deviation between all adjacent images meets the preset activation condition, then the countdown activation state is activated;

[0018] If at least one of the countdown deviations does not meet the preset activation condition, then the countdown activation state is inactive;

[0019] The preset activation condition is that the countdown deviation is within a preset deviation range.

[0020] In one technical solution of the above-mentioned traffic signal light state processing method, the countdown deviation is within a preset deviation range when the representative value of the countdown deviation is less than a preset threshold.

[0021] The representative value of the countdown deviation is the absolute value of the difference between the time deviation and the countdown deviation, wherein the time deviation is the deviation between the timestamps of two images, and the two images are two adjacent images corresponding to the countdown deviation.

[0022] In one technical solution of the above-mentioned traffic light state processing method, the step of smoothing the color of the single light in the continuous multi-frame images to obtain the smoothed color of the single light in the current frame includes:

[0023] The high-frequency color is used as the smooth color of the single light in the current frame, and the high-frequency color is the single light color that appears most frequently in the consecutive multi-frame images.

[0024] In one technical solution of the above-mentioned traffic light state processing method, the method includes smoothing the color of the single light in the consecutive multi-frame images by means of the following method:

[0025] The color of the single lamp in the consecutive multi-frame images is matched with a preset color conversion template. The color conversion template includes M template colors arranged in sequence. The template color is the color of the single lamp in the two color conversion stages, and the converted color is the lighting color when the single lamp is lit. M≤N, where N is the number of consecutive multi-frame images.

[0026] If the match is successful, the color of the single light in the current frame will be used as the smooth color.

[0027] If the match fails, the high-frequency color will be used as the smooth color.

[0028] in,

[0029] A successful match is defined as follows: the color of the single light in the Nth to N-M+1th frames of a series of images is the same as the color of the Mth to 1st template in the color conversion template, respectively.

[0030] In one technical solution of the above-mentioned traffic signal light state processing method, the color conversion template includes a first color conversion template and a second color conversion template;

[0031] In the first color conversion template, the number of template colors is M = N = 5, and the first to third template colors are the same, the fourth to fifth template colors are the same lighting color, and the third and fourth template colors are different.

[0032] In the second color conversion module, the number of template colors is M = N-1 = 4, and the first to third template colors are the same, the fourth template color is a single lighting color, and the third and fourth template colors are different.

[0033] In one technical solution of the above-mentioned traffic light state processing method, the two color transition stages include:

[0034] The stage in which the color of a single lamp changes from one illuminated color to another; the stage in which the color of a single lamp changes from unknown to an illuminated color.

[0035] In one technical solution of the above-mentioned traffic light state processing method, determining the final color based on the countdown digit of the single light in the current frame and the previous color includes:

[0036] When the preceding color meets the preset maintenance condition, the final color is the preceding color;

[0037] When the preceding color does not meet the preset maintenance conditions

[0038] If the countdown activation state of the single light is active and the countdown number of the current frame is within the first preset countdown range, then the final color is the previous color;

[0039] If the countdown activation state of the single light is active and the countdown number of the current frame is not within the first preset countdown range, then the final color is unknown;

[0040] If the countdown activation state of the single lamp is inactive, then the final color is unknown;

[0041] The preset maintenance conditions include: the front color is the color of a single lamp when it is lit, and the first maintenance duration of the front color is less than the first preset duration.

[0042] In one technical solution of the above-mentioned traffic light state processing method, the method includes: if the final color of the current frame is different from the previous color, then the first retention duration of the previous color is cleared to zero.

[0043] In one technical solution of the above-mentioned traffic light state processing method, the single light state further includes a flashing state, wherein the flashing state is flashing, not flashing, or unknown, and the method includes:

[0044] The flickering state of the single lamp in the continuous multi-frame images is smoothed to obtain the smooth flickering state of the single lamp in the current frame;

[0045] Based on the smooth flashing state, determine the final flashing state of the single lamp in the current frame;

[0046] If the smooth flashing state is flashing or not flashing, then the final flashing state is the smooth flashing state;

[0047] If the smooth flashing state is unknown, the final flashing state of the current frame is determined based on the previous flashing state of the single lamp, where the previous flashing state is the final flashing state of the single lamp in the previous frame of the current frame.

[0048] In one technical solution of the above-mentioned traffic light state processing method, the step of smoothing the flashing state of the single light in the continuous multi-frame images to obtain the smooth flashing state of the single light in the current frame includes:

[0049] The high-frequency flickering state is taken as the smooth flickering state of the single lamp in the current frame. The high-frequency flickering state is the flickering state that the single lamp appears most frequently in the consecutive multi-frame images.

[0050] In one technical solution of the above-mentioned traffic light state processing method, the method includes smoothing the flashing state of the single light in the continuous multi-frame images by means of the following:

[0051] The flashing state of the single lamp in the continuous multi-frame image is matched with a preset flashing template. The flashing template includes a first flashing template and a second flashing template. The first flashing template includes a flashing state and the flashing state is flashing. The second flashing template includes two flashing states arranged in sequence and the flashing state is not flashing.

[0052] If the single lamp successfully matches the first flashing template or the second flashing template, the smooth flashing state is either flashing or not flashing;

[0053] If the single lamp fails to match both the first flashing template and the second flashing template, then the smooth flashing state is the high-frequency flashing state;

[0054] Wherein, the single lamp successfully matches the first flashing template as follows: the single lamp is flashing in one frame of the continuous multi-frame images;

[0055] The single lamp is successfully matched with the second flashing template when the single lamp is in a non-flickering state for two consecutive frames in the multi-frame image.

[0056] In one technical solution of the above-mentioned traffic light state processing method, the method includes: if the single light is successfully matched with both the first flashing template and the second flashing template, then the smooth flashing state is no flashing.

[0057] In one technical solution of the above-mentioned traffic light state processing method, the method includes smoothing the flashing state of the single light in the continuous multi-frame images by means of the following:

[0058] The flashing state of the single lamp in the continuous multi-frame images is matched with a preset first flashing template;

[0059] When the single lamp successfully matches the first flashing template, the flashing state of the single lamp in the continuous multi-frame image is matched with the preset second flashing template; if the single lamp successfully matches the second flashing template, the smooth flashing state is no flashing; if the single lamp fails to match the second flashing template, the smooth flashing state is flashing.

[0060] When the single lamp fails to match the first flashing template, the flashing state of the single lamp in the continuous multi-frame image is matched with the second flashing template; if the single lamp successfully matches the second flashing template, the smooth flashing state is no flashing; if the single lamp fails to match the second flashing template, the smooth flashing state is unknown.

[0061] Wherein, the first flashing template includes a flashing state and the flashing state is flashing, and the second flashing template includes two flashing states arranged in sequence and the flashing state is not flashing;

[0062] The single lamp is successfully matched with the first flashing template when the single lamp is in a flashing state in one of the consecutive multi-frame images.

[0063] The single lamp is successfully matched with the second flashing template when the single lamp is in a non-flickering state for two consecutive frames in the multi-frame image.

[0064] In one technical solution of the above-mentioned traffic light state processing method, determining the final flashing state of the current frame based on the previous flashing state of the single light includes:

[0065] When the preceding flashing state is no flashing, the final flashing state is no flashing;

[0066] When the pre-flashing state is flashing, if the final color of the single lamp in the current frame meets the preset flashing condition, then the final flashing state is flashing; if the final color of the single lamp in the current frame does not meet the preset flashing condition, then the final flashing state is unknown.

[0067] The preset flashing condition is either a first flashing condition or a second flashing condition;

[0068] The first blinking condition is: the final color of the current frame and the previous color are both the lighting colors when a single lamp is lit, and the final color of the current frame is different from the previous color;

[0069] The second flashing condition is: the final color of the current frame is the same as the color of a single lamp when it is lit, the second duration of the previous flashing state is less than the second preset duration, and the countdown number of the current frame is within the second preset countdown range.

[0070] In one technical solution of the above-mentioned traffic light state processing method, the method includes: if the final color of the current frame is different from the previous color, then the second duration of the previous flashing state is reset to zero.

[0071] In one technical solution of the above-mentioned traffic light state processing method, the single light state further includes a visible state, wherein the visible state is either visible or invisible, and the method includes:

[0072] The visibility state of the single lamp in the continuous multi-frame images is matched with a preset visibility template, the visibility template including a first visibility template and a second visibility template;

[0073] If a match is found with the first visible template, the occlusion state of the single light is activated.

[0074] If a match is found with the second visible template, the occlusion state of the single light is turned off.

[0075] The first visible template includes N sequentially arranged visible states, where the first to the (N-2)th visible states are visible, and the (N-1)th to the Nth visible states are invisible, and N is the number of consecutive multi-frame images;

[0076] The second visible template includes N visible states arranged in sequence, and the first to the (N-2)th visible states are invisible, while the (N-1)th to the Nth visible states are visible.

[0077] In one technical solution of the above-mentioned traffic light state processing method, the intelligent device is equipped with a perception model, and the perception result of acquiring multiple consecutive frames of traffic light images includes:

[0078] If the model result of the image is detected, the model result is used as the perception result of the image. The model result is the result obtained by the perception model in performing single-lamp state perception on the image.

[0079] If no model result of the image is detected, the virtual result is used as the perception result of the image, and all individual light states in the virtual result are unknown.

[0081] In a second aspect, a smart device is provided, the smart device comprising at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program that, when executed by the at least one processor, implements the method described in any of the technical solutions provided in the first aspect.

[0082] In a third aspect, a computer-readable storage medium is provided, wherein a plurality of program codes are stored therein, the program codes being adapted to be loaded and executed by a processor to perform the method described in any of the technical solutions provided in the first aspect.

[0083] Solution 1. A method for processing the state of a traffic signal light, characterized in that it is applied to a smart device, the method comprising:

[0084] The perception results of acquiring multiple consecutive frames of traffic light images are obtained. The perception results include the single light status of a traffic light. The single light status includes color and countdown numbers. The color is the lighting color when the single light is lit or unknown.

[0085] The countdown numbers of the single lamp in the consecutive frames of images are obtained, and the countdown activation state of the single lamp is determined according to the deviation between the countdown numbers. The countdown activation state is either active or inactive.

[0086] The color of the single lamp in the consecutive multi-frame images is smoothed to obtain the smooth color of the single lamp in the current frame, where the current frame is the frame with the latest timestamp in the consecutive multi-frame images;

[0087] The final color of the single light in the current frame is determined based on the smooth color and the countdown number of the single light in the current frame;

[0088] The countdown activation state is updated based on the final color of the current frame; if the final color is different from the previous color, the countdown activation state is updated to inactive, where the previous color is the final color of the single lamp in the image preceding the current frame.

[0089] Solution 2. The method according to Solution 1, characterized in that, determining the final color of the single lamp in the current frame based on the smooth color and the countdown number of the single lamp in the current frame includes:

[0090] When the smooth color is the highlighted color, the final color is the smooth color;

[0091] When the smooth color is unknown, if the countdown activation state is active, the final color is determined based on the countdown number of the single light in the current frame and the previous color; if the countdown activation state is inactive, the final color of the current frame is unknown.

[0092] Solution 3. The method according to Solution 1, characterized in that determining the countdown activation state of the single lamp based on the deviation between the countdown digits includes:

[0093] The countdown deviation between every two adjacent images in the continuous multi-frame images is obtained, and the countdown deviation is the deviation of the countdown numbers of the single lamp between the two adjacent images;

[0094] If the countdown deviation between all adjacent images meets the preset activation condition, then the countdown activation state is activated;

[0095] If at least one of the countdown deviations does not meet the preset activation condition, then the countdown activation state is inactive;

[0096] The preset activation condition is that the countdown deviation is within a preset deviation range.

[0097] Scheme 4. The method according to Scheme 3, characterized in that the countdown deviation is within a preset deviation range as follows: the representative value of the countdown deviation is less than a preset threshold; the representative value of the countdown deviation is the absolute value of the difference between the time deviation and the countdown deviation, wherein the time deviation is the deviation between two image timestamps, and the two images are two adjacent images corresponding to the countdown deviation.

[0098] Solution 5. The method according to Solution 1, characterized in that, the step of smoothing the color of the single lamp in the consecutive multi-frame images to obtain the smooth color of the single lamp in the current frame includes: using a high-frequency color as the smooth color of the single lamp in the current frame, wherein the high-frequency color is the single lamp color that appears most frequently in the consecutive multi-frame images.

[0099] Solution 6. The method according to Solution 5, characterized in that the method includes smoothing the color of the single lamp in the consecutive multi-frame images by:

[0100] The color of the single lamp in the consecutive multi-frame images is matched with a preset color conversion template. The color conversion template includes M template colors arranged in sequence. The template color is the color of the single lamp in the two color conversion stages, and the converted color is the lighting color when the single lamp is lit. M≤N, where N is the number of consecutive multi-frame images.

[0101] If the match is successful, the color of the single light in the current frame will be used as the smooth color.

[0102] If the match fails, the high-frequency color will be used as the smooth color.

[0103] Wherein, a successful match means that the color of the single light in the Nth to N-M+1th frames of a series of consecutive images is the same as the color of the Mth to 1st template in the color conversion template.

[0104] Solution 7. The method according to Solution 6, wherein the color conversion template includes a first color conversion template and a second color conversion template;

[0105] In the first color conversion template, the number of template colors is M = N = 5, and the first to third template colors are the same, the fourth to fifth template colors are the same lighting color, and the third and fourth template colors are different.

[0106] In the second color conversion module, the number of template colors is M = N-1 = 4, and the first to third template colors are the same, the fourth template color is a single lighting color, and the third and fourth template colors are different.

[0107] Solution 8. The method according to Solution 6, characterized in that the two color conversion stages include: a stage in which the color of the single lamp changes from one illumination color to another illumination color; and a stage in which the color of the single lamp changes from unknown to an illumination color.

[0108] Solution 9. The method according to Solution 2, characterized in that, determining the final color based on the countdown digit of the single light in the current frame and the preceding color includes:

[0109] When the preceding color meets the preset maintenance condition, the final color is the preceding color;

[0110] When the preceding color does not meet the preset maintenance conditions, if the countdown activation state of the single light is active and the countdown number of the current frame is within the first preset countdown range, then the final color is the preceding color; if the countdown activation state of the single light is active and the countdown number of the current frame is not within the first preset countdown range, then the final color is unknown; if the countdown activation state of the single light is inactive, then the final color is unknown.

[0111] The preset maintenance conditions include: the front color is the color of a single lamp when it is lit, and the first maintenance duration of the front color is less than the first preset duration.

[0112] Solution 10. The method according to Solution 9, characterized in that the method includes: if the final color of the current frame is different from the previous color, then the first retention duration of the previous color is cleared to zero.

[0113] Solution 11. The method according to Solution 1, characterized in that the single-lamp state further includes a flashing state, wherein the flashing state is flashing, not flashing, or unknown, and the method includes:

[0114] The flickering state of the single lamp in the continuous multi-frame images is smoothed to obtain the smooth flickering state of the single lamp in the current frame;

[0115] Based on the smooth flashing state, determine the final flashing state of the single lamp in the current frame;

[0116] If the smooth flashing state is flashing or not flashing, then the final flashing state is the smooth flashing state;

[0117] If the smooth flashing state is unknown, the final flashing state of the current frame is determined based on the previous flashing state of the single lamp, where the previous flashing state is the final flashing state of the single lamp in the previous frame of the current frame.

[0118] Solution 12. The method according to Solution 11, characterized in that, the step of smoothing the flickering state of the single lamp in the continuous multi-frame images to obtain the smooth flickering state of the single lamp in the current frame includes:

[0119] The high-frequency flickering state is taken as the smooth flickering state of the single lamp in the current frame. The high-frequency flickering state is the flickering state that the single lamp appears most frequently in the consecutive multi-frame images.

[0120] Solution 13. The method according to Solution 12, characterized in that the method includes smoothing the flickering state of the single lamp in the consecutive multi-frame images by:

[0121] The flashing state of the single lamp in the continuous multi-frame image is matched with a preset flashing template. The flashing template includes a first flashing template and a second flashing template. The first flashing template includes a flashing state and the flashing state is flashing. The second flashing template includes two flashing states arranged in sequence and the flashing state is not flashing.

[0122] If the single lamp successfully matches the first flashing template or the second flashing template, the smooth flashing state is either flashing or not flashing;

[0123] If the single lamp fails to match both the first flashing template and the second flashing template, then the smooth flashing state is the high-frequency flashing state;

[0124] Wherein, the single lamp successfully matches the first flashing template as follows: the single lamp is flashing in one frame of the continuous multi-frame images;

[0125] The single lamp is successfully matched with the second flashing template when the single lamp is in a non-flickering state for two consecutive frames in the multi-frame image.

[0126] Solution 14. The method according to Solution 13, characterized in that the method includes: if the single lamp is successfully matched with both the first flashing template and the second flashing template, then the smooth flashing state is no flashing.

[0127] Solution 15. The method according to Solution 12, characterized in that the method includes smoothing the flickering state of the single lamp in the consecutive multi-frame images by:

[0128] The flashing state of the single lamp in the continuous multi-frame images is matched with a preset first flashing template;

[0129] When the single lamp successfully matches the first flashing template, the flashing state of the single lamp in the continuous multi-frame image is matched with the preset second flashing template; if the single lamp successfully matches the second flashing template, the smooth flashing state is no flashing; if the single lamp fails to match the second flashing template, the smooth flashing state is flashing.

[0130] When the single lamp fails to match the first flashing template, the flashing state of the single lamp in the continuous multi-frame image is matched with the second flashing template; if the single lamp successfully matches the second flashing template, the smooth flashing state is no flashing; if the single lamp fails to match the second flashing template, the smooth flashing state is unknown.

[0131] Wherein, the first flashing template includes a flashing state and the flashing state is flashing, and the second flashing template includes two flashing states arranged in sequence and the flashing state is not flashing;

[0132] The single lamp is successfully matched with the first flashing template when the single lamp is in a flashing state in one of the consecutive multi-frame images.

[0133] The single lamp is successfully matched with the second flashing template when the single lamp is in a non-flickering state for two consecutive frames in the multi-frame image.

[0134] Solution 16. The method according to Solution 11, characterized in that, determining the final flashing state of the current frame based on the preceding flashing state of the single lamp includes:

[0135] When the preceding flashing state is no flashing, the final flashing state is no flashing;

[0136] When the pre-flashing state is flashing, if the final color of the single lamp in the current frame meets the preset flashing condition, then the final flashing state is flashing; if the final color of the single lamp in the current frame does not meet the preset flashing condition, then the final flashing state is unknown.

[0137] The preset flashing condition is either a first flashing condition or a second flashing condition. The first flashing condition is that the final color of the current frame and the previous color are both the colors when a single lamp is lit, and the final color of the current frame is different from the previous color. The second flashing condition is that the final color of the current frame is the color when a single lamp is lit, and the second duration of the previous flashing state is less than the second preset duration, and the countdown number of the current frame is within the second preset countdown range.

[0138] Solution 17. The method according to Solution 11, characterized in that the method includes: if the final color of the current frame is different from the previous color, then the second hold duration of the previous flashing state is cleared to zero.

[0139] Solution 18. The method according to Solution 1, characterized in that the single lamp state further includes a visible state, wherein the visible state is visible or invisible, and the method includes:

[0140] The visibility state of the single lamp in the continuous multi-frame images is matched with a preset visibility template, the visibility template including a first visibility template and a second visibility template;

[0141] If a match is found with the first visible template, the occlusion state of the single lamp is activated; if a match is found with the second visible template, the occlusion state of the single lamp is deactivated.

[0142] The first visible template includes N sequentially arranged visible states, where the first to the (N-2)th visible states are visible, and the (N-1)th to the Nth visible states are invisible, and N is the number of consecutive multi-frame images;

[0143] The second visible template includes N visible states arranged in sequence, and the first to the (N-2)th visible states are invisible, while the (N-1)th to the Nth visible states are visible.

[0144] Solution 19. The method according to Solution 1, characterized in that the intelligent device is equipped with a perception model, and the perception result of acquiring multiple consecutive frames of traffic light images includes:

[0145] If the model result of the image is detected, the model result is used as the perception result of the image. The model result is the result obtained by the perception model in performing single-lamp state perception on the image.

[0146] If no model result of the image is detected, the virtual result is used as the perception result of the image, and all individual light states in the virtual result are unknown.

[0147] Option 20. A smart device, characterized in that it comprises:

[0148] At least one processor;

[0149] And, a memory communicatively connected to the at least one processor;

[0150] The memory stores a computer program, which, when executed by the at least one processor, implements the traffic light state processing method as described in any one of schemes 1 to 19.

[0151] Scheme 21. A computer-readable storage medium storing a plurality of program codes, characterized in that the program codes are adapted to be loaded and run by a processor to perform the traffic signal light state processing method as described in any one of Schemes 1 to 19.

[0152] The above-described technical solutions of this application have at least one or more of the following beneficial effects:

[0153] In one technical solution for implementing the traffic light state processing method provided in this application, the method is applied to a smart device and includes the following steps:

[0154] The system acquires the perception results of multiple consecutive frames of traffic light images. These results include the individual light status of each light in the images, with each light status including its color and a countdown timer. The color is either the initial illuminated color or an unknown value. The system then acquires the countdown timer for each light across the multiple frames and determines its activation state based on the deviation between the countdown times. The activation state is either active or inactive. The system smooths the color of each light across the multiple frames to obtain a smoothed color for the current frame, which is the frame with the latest timestamp. Based on the smoothed color and the countdown timer in the current frame, the system determines the final color of the light in that frame. The system updates the countdown activation state based on the final color of the current frame. If the final color differs from the previous color, the countdown activation state is updated to inactive, with the previous color being the final color of the light in the frame preceding the current one.

[0155] If the countdown is active, it indicates that the countdown of a single light is reliable, and the countdown number of that single light in the image can be used normally. The remaining time the single light will remain lit can be determined based on the countdown number, and then the driving decision of the intelligent device can be based on the remaining time. The countdown number in the perception result is obtained from image perception. Due to factors such as perception accuracy and image quality, the countdown number perception may be inaccurate. If the countdown is deemed reliable simply because a countdown number is perceived, incorrect judgments are likely to occur. However, the above implementation scheme uses the changes in the countdown number of a single light in multiple consecutive frames of images to determine the countdown activation state. This is equivalent to performing temporal fusion of the countdown numbers in multiple consecutive frames of images, which can more accurately obtain the countdown activation state.

[0156] Similarly, the above implementation scheme utilizes the color of a single lamp in multiple consecutive frames of images to comprehensively determine the final color of the single lamp in the current frame. It also achieves temporal fusion of colors in multiple consecutive frames of images. Compared with determining the final color of a single lamp in the current frame based on the color in a single frame of images, the above implementation scheme can effectively improve the accuracy of the single lamp color.

[0157] Furthermore, the countdown timer of a single light is used as a constraint when obtaining the final color of the single light in the current frame, and the final color of the single light in the current frame is used as a constraint when determining the countdown activation state. That is, the final color and countdown activation state of a single light are mutually constrained when obtaining the final color and countdown activation state of a single light, which further improves the accuracy of the single light color and countdown activation state. Attached Figure Description

[0158] The disclosure of this application will become more readily understood with reference to the accompanying drawings. It will be readily understood by those skilled in the art that these drawings are for illustrative purposes only and are not intended to limit the scope of protection of this application. Wherein:

[0159] Figure 1 This is a schematic flowchart of the main steps of a traffic signal light state processing method according to an embodiment of this application;

[0160] Figure 2 This is a flowchart illustrating the main steps for determining the countdown activation state of a single lamp according to an embodiment of this application;

[0161] Figure 3 This is a flowchart illustrating the main steps for determining the final blinking state of a single lamp in the current frame according to an embodiment of this application.

[0162] Figure 4 This is a schematic flowchart illustrating the main steps of determining whether to activate the shading state of a single lamp according to an embodiment of this application;

[0163] Figure 5 This is a schematic flowchart of a traffic light state processing method according to an embodiment of this application;

[0164] Figure 6 This is a schematic diagram of a process for performing color temporal fusion on the perception results of multiple consecutive frames of images within a sliding window according to an embodiment of this application.

[0165] Figure 7 This is a schematic flowchart illustrating the flicker timing fusion of the perception results of multiple consecutive frames of images within a sliding window according to an embodiment of this application.

[0166] Figure 8 This is a schematic diagram of the main structure of a smart device according to an embodiment of this application.

[0167] Figure label:

[0168] 11: Memory; 12: Processor. Detailed Implementation

[0169] Some embodiments of this application are described below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of this application and are not intended to limit the scope of protection of this application.

[0170] In the description of this application, "processor" can include hardware, software, or a combination of both. A processor can be a central processing unit, microprocessor, graphics processor, digital signal processor, or any other suitable processor. A processor has data and / or signal processing capabilities. A processor can be implemented in software, in hardware, or a combination of both. Computer-readable storage media includes any suitable medium capable of storing program code, such as magnetic disks, hard disks, optical disks, flash memory, read-only memory, random access memory, etc.

[0171] The relevant user personal information that may be involved in the various embodiments of this application is processed in strict accordance with the requirements of laws and regulations, following the principles of legality, legitimacy, and necessity, based on the reasonable purpose of the business scenario, and includes personal information that users actively provide or that is generated as a result of using the product / service, as well as personal information obtained with user authorization.

[0172] The personal information processed in this application will vary depending on the specific product / service scenario and will be based on the specific scenario in which the user uses the product / service. This may involve the user's account information, device information, driving information, vehicle information, or other related information. This application will treat the user's personal information and its processing with the utmost diligence.

[0173] This application attaches great importance to the security of users' personal information and has taken reasonable and feasible security protection measures that comply with industry standards to protect users' information and prevent unauthorized access, disclosure, use, modification, damage or loss of personal information.

[0174] The following describes an embodiment of the traffic light state processing method provided in this application. This method is applied to intelligent devices, which may include driving equipment, intelligent vehicles, robots, etc. The intelligent devices are equipped with sensors and a perception model. The sensors may include cameras, which can capture images of the vehicle's environment. In this embodiment, the camera can capture images of traffic lights in the vehicle's environment. The perception model can be used to perceive the images of the traffic lights to obtain the state of a single light. The state of a single light is the state of a single light in the traffic light, which may include the color of the single light, countdown numbers, etc. In this embodiment, a conventional perception model can be used; the specific structure and working principle of the perception model will not be elaborated upon.

[0175] See appendix Figure 1 , Figure 1 This is a schematic flowchart illustrating the main steps of a traffic signal light state processing method according to an embodiment of this application. Figure 1 As shown, the traffic light state processing method in this embodiment mainly includes the following steps S101 to S105.

[0176] Step S101: Obtain the perception results of multiple consecutive frames of traffic light images. The perception results include the individual light status of each traffic light in the image. The individual light status includes color and countdown numbers. The color is the illuminated color of the individual light or is unknown. In some embodiments, the individual light status also includes the flashing status and occlusion status of the individual light. The flashing status indicates whether the individual light is flashing. The type of flashing status can be flashing, not flashing, or unknown. Unknown means it is impossible to determine whether the individual light is flashing or not. If the occlusion status is activated, it indicates that the individual light is occluded; if the occlusion status is deactivated, it indicates that the individual light is not occluded. In addition, the individual light status also includes the visibility status of the individual light. The visibility status indicates whether the individual light is visible in the image. The type of visibility status can be visible or invisible.

[0177] The image perception result is obtained by the perception model on the smart device through single-light state perception of the traffic light image. When it is necessary to obtain the perception result of multiple consecutive frames of images, the perception model can be used to obtain the perception result for each frame of the consecutive frames.

[0178] In some implementations, the perception results of multiple consecutive frames of traffic light images can be obtained through the following steps S1011 to S1013.

[0179] Step S1011: Determine whether the model result of the image has been detected. The model result is the result obtained by the perception model from the single-lamp state perception of the image. If detected, proceed to step S1012; if not detected, proceed to step S1013.

[0180] Step S1012: Use the model results as the perception results of the image.

[0181] Step S1013: If no model result is detected (e.g., a certain frame of image is missed, the perception model does not perceive this frame of image, and therefore, no model result for this frame of image is obtained), the state of a single lamp in the image cannot be determined at this time. Therefore, a virtual result can be set and used as the perception result of the image. All single lamp states in the virtual result are unknown.

[0182] Based on the method described in steps S1011 to S1013 above, a perception result can be obtained for each frame of a series of images, which is beneficial for subsequent steps S102 to S108 to perform temporal fusion of the perception results of the series of images and obtain an accurate single-lamp state.

[0183] The following explains the highlighted colors and unknown colors.

[0184] The illuminated color is a color used for traffic guidance. If a single light displays the illuminated color in multiple consecutive frames of images, then it can be determined that the single light is illuminated. In some implementations, the illuminated color may include green, red, and yellow, where green indicates permission to proceed, red indicates prohibition to proceed, and yellow indicates a warning.

[0185] If the color of a single lamp is not a lit color, then the color of that single lamp is determined to be unknown. For example, when a single lamp is off, it appears black in the image, but since black is not a lit color, even if the detected color of the single lamp is black, the color of the single lamp is determined to be unknown; similarly, if a single lamp is occluded in the image, its color cannot be detected, and in this case, the color of the single lamp is also determined to be unknown.

[0186] The countdown numbers are explained below.

[0187] Based on the semantic meaning of the shape of a single light head, a single light can include turn signals and countdown lights. The semantic meaning of the turn signal head shape is traffic direction indication. The shape of the turn signal head can include a disc, a straight arrow, a left turn arrow, a right turn arrow, and a U-turn arrow, etc. Different head shapes can indicate different traffic directions. For example, a disc indicates traffic directions including straight, left turn, and U-turn; a straight arrow indicates traffic direction of going straight.

[0188] The countdown timer light's bulb is shaped like a number, indicating the remaining time the turn signal will display the currently illuminated color. After the remaining time reaches 0, the turn signal will display another color. For example, if the current turn signal color is green and the countdown timer number is 15, it means that the green light has 15 seconds remaining. After 15 seconds, the turn signal will turn red. The countdown timer number can be understood as a countdown to the remaining time of the currently illuminated color; the changing countdown number reflects the turn signal's countdown process.

[0189] The single-lamp status in the perception result can be understood as the single-lamp status of the turn signal. The single-lamp status of the turn signal includes not only the status of the turn signal itself (such as color, flashing status, etc.), but also the countdown number used to indicate the remaining time of the turn signal for the currently illuminated color. This countdown number is obtained from the countdown light's number.

[0190] Step S102: Obtain the countdown numbers of a single lamp in multiple consecutive frames of images. Based on the deviation between the countdown numbers, determine the countdown activation state of the single lamp, which is either active or inactive.

[0191] The countdown numbers in the perception result are obtained by perceiving the image. Due to factors such as perception accuracy and image quality, the perception of the countdown numbers may be inaccurate. If the countdown is deemed reliable simply because it is perceived, incorrect judgments are likely to occur. To address this, this application uses the deviation between countdown numbers to determine the reliability of the countdown; if reliable, the countdown is activated (i.e., the countdown activation state is active); if not reliable, the countdown is deactivated (i.e., the countdown activation state is inactive).

[0192] During the countdown, the countdown numbers decrease sequentially, with the deviation between any two countdown numbers being the same. If the countdown numbers for a single light decrease sequentially across multiple consecutive frames, and the deviation between the countdown numbers in any two adjacent frames is within a preset deviation range, then the countdown is considered reliable and the countdown is active. Otherwise, the countdown is considered unreliable and the countdown is inactive. The countdown numbers decrease sequentially at equal time intervals, meaning they exhibit a clear pattern of change. The preset deviation range can be set based on the aforementioned time intervals, with a minimum value of 0 and a maximum value equal to the time interval value. For example, if the time interval is 1 second, the deviation range is 0 to 1.

[0193] When the countdown is active, the countdown timer displayed on the image for each individual light can be used normally. The remaining time for each light to remain lit is determined based on this countdown time, and the intelligent device's driving decision is then based on this remaining time. For example, when a vehicle approaches an intersection, its planned trajectory is to proceed straight through the intersection. If the detected illuminated light at the intersection is a green arrow pointing straight ahead, with a countdown of 10, then the remaining time allowed to proceed straight is 10 seconds. If it is predicted that the vehicle cannot proceed straight through the intersection within 10 seconds, the vehicle will be stopped before the stop line at the intersection, waiting for the light to turn green again before proceeding straight through. If it is predicted that the vehicle can proceed straight through the intersection within 10 seconds, the vehicle will proceed normally, proceeding straight through the intersection.

[0194] When the countdown is inactive, to ensure driving safety, the countdown numbers in the image of a single light can be omitted to determine the driving decision of the smart device.

[0195] Step S103: Smooth the color of a single lamp in multiple consecutive frames to obtain the smooth color of the single lamp in the current frame, where the current frame is the frame with the latest timestamp in the multiple consecutive frames.

[0196] Specifically, the result of the smoothing process is used as the smoothed color of a single light in the current frame. Due to factors such as perception accuracy and image quality, single-light color perception errors may occur, resulting in repeated color jumps. Smoothing can effectively reduce these repeated color jumps and maintain the stability of the single-light color. In some implementations, the high-frequency color of a single light in multiple consecutive frames can be used as the smoothed color of the single light in the current frame. The high-frequency color is the single-light color that appears most frequently in multiple consecutive frames. For example, if the colors of a single light in five consecutive frames are green, red, green, red, and green respectively, and the high-frequency color is green, then the smoothed color is green. Through the above implementation methods, the smoothing process of single-light colors can be completed conveniently and quickly.

[0197] Step S104: Determine the final color of the single light in the current frame based on the smooth color and the countdown number of the single light in the current frame.

[0198] When the smooth color is the illuminated color, it indicates that the color of the single lamp is clear and can be directly used as the final color. When the smooth color is unknown, it indicates that the color of the single lamp is not clear, and the countdown number of the single lamp in the current frame can be used to help obtain the final color of the current frame. Specifically, the countdown number of the current frame can be used to determine whether the countdown has ended. If it has not ended, the final color of the current frame is obtained based on the previous color, which is the final color of the single lamp in the previous frame. Otherwise, the smooth color is still used as the final color. In determining whether the countdown has ended, the countdown number indicating the end of the countdown can be obtained. If the countdown number of the current frame is the same as the countdown number indicating the end of the countdown, then it is determined that the countdown ended in the current frame.

[0199] In some implementations, the final color of a single light in the current frame can be determined based on the smoothed color and the countdown number of the single light in the current frame in the following way:

[0200] When the smooth color is the highlight color, the final color is the smooth color.

[0201] When the smooth color is unknown, the final color is determined based on the countdown activation state.

[0202] Specifically, if the countdown is active, the final color is determined based on the countdown number of the single light in the current frame and the previous color, where the previous color is the final color of the single light in the previous frame. If the countdown is inactive, the final color is unknown.

[0203] When the countdown is active, it indicates that the countdown for a single light is reliable, and the countdown number can be used to help determine the final color of the current frame. Specifically, the duration of the illuminated color in a traffic light is fixed, and the countdown range can be obtained based on the duration. For example, if the duration is 30 seconds, the countdown range can be from 30 to 1. If the countdown number for a single light in the current frame is still within the countdown range, it indicates that the single light is counting down normally for the preceding color, and the final color of the single light in the current frame is the same as the preceding color.

[0204] When the countdown activation state is inactive, it indicates that the countdown of a single light is not reliable and will not be used to help determine the final color of the current frame. Instead, the smooth color will still be used as the final color of the current frame. Since the smooth color is unknown at this time, the final color is also unknown.

[0205] Step S105: Update the countdown activation state according to the final color of the current frame; if the final color is different from the previous color, update the countdown activation state to inactive.

[0206] If the final color of the current frame differs from the previous color, it indicates that the color of the single lamp has changed, the countdown of the single lamp for the previous color has ended, and the countdown of the single lamp is no longer reliable. Therefore, the countdown activation state determined in step S102 needs to be updated to inactive. If the countdown activation state was originally inactive, it remains unchanged. It should be noted that if the countdown activation state is updated from active to inactive, step S104 will not be repeated based on the new countdown activation state, and the final color of the current frame will not change.

[0207] If the final color of the current frame is the same as the previous color, it indicates that the color of the single light has not changed. Regardless of whether the countdown activation state is active or inactive, it will not be updated, and the countdown activation state determined in the aforementioned step S102 will still be used.

[0208] In the methods described in steps S101 to S105 above, the countdown activation state is determined by utilizing the deviation between the countdown numbers of a single light in multiple consecutive frames of images. This is equivalent to performing a temporal fusion of the countdown numbers in multiple consecutive frames of images, which can more accurately obtain the countdown activation state. Similarly, the final color of a single light in the current frame is determined by comprehensively utilizing the color of a single light in multiple consecutive frames of images, which also achieves temporal fusion of colors in multiple consecutive frames of images. Compared with determining the final color of a single light in the current frame based on the color in a single frame of images, the above method can effectively improve the accuracy of the single light color. In addition, the countdown numbers of a single light are used as constraints when obtaining the final color of a single light in the current frame, and the final color of a single light in the current frame is also used as a constraint when determining the countdown activation state. That is, the final color of a single light and the countdown activation state are mutually constrained when obtaining them, which further improves the accuracy of the single light color and the countdown activation state.

[0209] The following description continues with an embodiment of the traffic light state processing method provided in this application, specifically describing steps S102, S103, and S104.

[0210] I. Explanation of step S102.

[0211] In some embodiments of step S102 above, when determining the countdown activation state of a single lamp based on the deviation between countdown digits, it can be done by... Figure 2 The following steps S1021 to S1024 are performed to determine this.

[0212] Step S1021: Obtain the countdown deviation between every two adjacent images in a series of consecutive frames. The countdown deviation is the difference between the countdown numbers of a single lamp in two adjacent images. For example, if the countdown numbers of a single lamp in two adjacent images are 16 and 15 respectively, then the countdown deviation is 1; if the countdown numbers of a single lamp in two adjacent images are both 16, then the countdown deviation is 0.

[0213] Step S1022: Determine whether the countdown deviation between all adjacent images meets the preset activation condition; if all meet the condition, proceed to step S1023; otherwise, proceed to step S1024.

[0214] The preset activation condition is: the countdown deviation is within the preset deviation range.

[0215] During the countdown, the countdown numbers decrease sequentially at equal time intervals, meaning the countdown numbers follow a clear pattern. A preset deviation range can be set based on these time intervals, with a minimum value of 0 and a maximum value equal to the time interval. For example, if the time interval is 1 second, the deviation range is 0 to 1.

[0216] If the countdown deviation between all adjacent images is within the deviation range, it indicates that the change of the countdown number of a single lamp in multiple consecutive frames conforms to the change pattern of the countdown number. In this case, the countdown of a single lamp is reliable, and therefore, we can proceed to step S1023 to activate the countdown. Otherwise, it indicates that the change of the countdown number does not conform to the change pattern of the countdown number. In this case, the countdown of a single lamp is not reliable, and therefore, we proceed to step S1024 to not activate the countdown.

[0217] Step S1023: The countdown activation status is active.

[0218] Step S1024: The countdown activation status is inactive.

[0219] Based on the method described in steps S1023 to S1024 above, it is possible to accurately analyze whether the countdown of a single lamp is reliable, and thus accurately determine the countdown activation state.

[0220] In some embodiments of step S1022 above, the countdown deviation is within a preset deviation range when the representative value of the countdown deviation is less than a preset threshold. The representative value of the countdown deviation is the absolute value of the difference between the time deviation and the countdown deviation. The time deviation is the deviation between two image timestamps. These two images are two adjacent images corresponding to the countdown deviation, that is, two adjacent images used to calculate the countdown deviation.

[0221] The representative value of the countdown deviation can be expressed as Δd=|(Cur_t-Pre_t)-(Cur_d-Pre_d)|, where Cur represents the frame with the later timestamp among two adjacent images, Pre represents the frame with the earlier timestamp among two adjacent images, Cur_t represents the timestamp of image Cur, Pre_t represents the timestamp of image Pre, and (Cur_t-Pre_t) represents the time deviation; Cur_d is the countdown number of a single lamp in image Cur, Pre_d is the countdown number of a single lamp in image Pre, and (Cur_d-Pre_d) represents the countdown deviation.

[0222] For multiple consecutive frames, the time interval between any two adjacent images is the same; that is, the time deviation (Cur_t - Pre_t) between any two adjacent images is identical. Furthermore, the camera captures images more frequently than the countdown numbers change frequently; therefore, the time interval between two adjacent images is less than the time interval between two adjacent countdown numbers. For example, the time interval between two adjacent images is 0.5 seconds, while the time interval between two adjacent countdown numbers is 1 second.

[0223] When setting the aforementioned preset threshold value, the maximum value of the countdown deviation (Cur_d - Pre_d) can be obtained. This maximum value and the value of (Cur_t - Pre_t) are substituted into |(Cur_t - Pre_t) - (Cur_d - Pre_d)| for calculation. Based on the calculation result and the maximum value of (Cur_d - Pre_d), the preset threshold value is set. The preset threshold value can be greater than the calculated result, and at the same time, the preset threshold value is less than or equal to the maximum value of (Cur_d - Pre_d).

[0224] If the representative value of the countdown deviation between all adjacent images is less than the preset threshold, it indicates not only that the change in the countdown numbers of a single lamp across multiple consecutive frames conforms to the pattern of countdown number changes, but also that the change in the countdown numbers conforms to the temporal change pattern of multiple consecutive frames, further increasing the reliability of the single lamp countdown. Therefore, setting preset activation conditions using the above implementation method can further improve the accuracy of the countdown activation state.

[0225] II. Explanation of step S103.

[0226] In some embodiments of step S103 above, the color of a single lamp in multiple consecutive frames can be smoothed through the following steps S1031 to S1033 to obtain the smooth color of the single lamp in the current frame.

[0227] Step S1031: Match the color of a single lamp in multiple consecutive frames with a preset color conversion template. The color conversion template includes M template colors arranged in sequence. The template colors are the colors of the single lamp in the two color conversion stages, and the converted color is the lighting color of the single lamp when it is lit. M≤N, where N is the number of consecutive frames.

[0228] Matching results include successful matches and failed matches.

[0229] A successful match is defined as follows: the color of a single LED in frames N through N-M+1 of a series of consecutive images matches the colors of the Mth through 1st templates in the color conversion template. For example, if M = N = 5, the colors of a single LED in a series of consecutive images are [N1, N2, N3, N4, N5], and the color conversion template is [M1, M2, M3, M4, M5]. If the colors of a single LED in frames N through N-M+1 of a series of consecutive images are N5, N4, N3, N2, N1, and the colors of the Mth through 1st templates in the color conversion template are M5, M4, M3, M2, M1, then a successful match is achieved if N5, N4, N3, N2, N1 match M5, M4, M3, M2, M1, respectively.

[0230] A match failure occurs when the color of a single light in the Nth to N-M+1th frames of the image does not match the color of the Mth to 1st template in the color conversion template, respectively.

[0231] In practical applications, different lighting colors alternate. If a match is successful, it indicates that the color of a single lamp in the current frame is the lighting color after normal alternation. Therefore, we can proceed to step S1032 and use the color of the single lamp in the current frame as the smoothing color. If the match fails, we proceed to step S1033 and use the high-frequency color as the smoothing color.

[0232] In some implementations, the two color conversion stages include the following two stages: (1) a stage where the color of a single lamp changes from one illumination color to another; and (2) a stage where the color of a single lamp changes from an unknown color to an illumination color. In this implementation, a color conversion template can be set for each stage. As long as it matches any one of the color conversion templates, the process can proceed to step S1032. If it fails to match any of the color conversion templates, the process proceeds to step S1033.

[0233] In some implementations, the color conversion template may include a first color conversion template and a second color conversion template, which will be described below.

[0234] (1) Explanation of the first color conversion template.

[0235] In the first color conversion template, the number of template colors is M=N=5, and the first to third template colors are the same, the fourth and fifth template colors are the same lighting color, and the third and fourth template colors are different. That is, the first three template colors are different from the last two template colors.

[0236] The following explanation uses the example of lighting up colors including red (R), green (G), and yellow (Y). U represents unknown, and the order of color switching is red, green, and yellow.

[0237] In the two color conversion stages, if the converted illuminated color is red, the following three first color conversion templates (i.e., red conversion templates) can be obtained:

[0238] "UUURR", "GGGRR", "YYYRR".

[0239] In the two color conversion stages, if the converted illuminated color is green, the following two first color conversion templates (i.e., greening templates) can be obtained:

[0240] "UUUGG" and "RRRGG".

[0241] In the two color conversion stages, if the converted illuminated color is yellow, the following two first color conversion templates (i.e., yellowing templates) can be obtained:

[0242] "UUUYY" and "GGGYY".

[0243] (2) Explanation of the second color conversion template.

[0244] In the second color conversion module, the number of template colors is M = N - 1 = 4. The first to third template colors are the same, the fourth template color is a single highlight color, and the third and fourth template colors are different. That is, the first three template colors are different from the last template color. The explanation will continue using the example above.

[0245] In the two color conversion stages, if the converted illuminated color is red, the following three second color conversion templates (i.e., red conversion templates) can be obtained:

[0246] "UUUR", "GGGR", "YYYR".

[0247] In the two color conversion stages, if the converted illuminated color is green, the following two second color conversion templates (i.e., greening templates) can be obtained:

[0248] "UUUG" and "RRRG".

[0249] In the two color conversion stages, if the converted illuminated color is yellow, the following two second color conversion templates (i.e., yellowing templates) can be obtained:

[0250] "UUUY" and "GGGY".

[0251] Using the second color conversion template, color conversion without delay can be achieved, while using the first color conversion template can prevent color jumps. The following explanation uses the first color conversion template "GGGRR" and the second color conversion template "GGGR" as examples.

[0252] If the color of a single lamp is "GGGGR" in 5 consecutive frames, meaning the lit color has just changed from green to red, then "GGGGR" will successfully match the second color conversion template "GGGR", and the color of the single lamp in the current frame will be determined to be red. This process achieves color conversion without delay.

[0253] Since a single LED needs to maintain its red color for a period of time after switching to red, its color in the next frame should still be red. That is, the color of the single LED in the next five consecutive frames will become "GGGRR". The high-frequency color in "GGGRR" is green (G). If this high-frequency color were used as the LED's color in the current frame, the current frame's color would become green, while the actual color is red, resulting in a color jump from red to green. However, in this application, "GGGRR" will successfully match the first color conversion template "GGGRR", and the current frame's color will remain red, avoiding the aforementioned color jump.

[0254] Step S1032: Use the color of a single light in the current frame as the smoothed color. That is, use the color in the perception result as the smoothed color.

[0255] Step S1033: Use the high-frequency color as the smoothing color. The high-frequency color is the single-lamp color that appears most frequently in multiple consecutive frames of images.

[0256] Based on the method described in steps S1031 to S1033 above, a preset color conversion template can be used to quickly and accurately obtain the smooth color of a single light in the current frame.

[0257] III. Explanation of step S104.

[0258] In some embodiments of step S104 above, the final color of the current frame can be determined by the following steps S1041 to S1045, based on the countdown number of a single lamp in the current frame and the previous color.

[0259] Step S1041: Determine whether the preceding color meets the preset maintenance conditions; if it does, proceed to step S1042; if it does not, proceed to step S1043.

[0260] The preset maintenance conditions include: the preceding color is the same as the color when a single light is on, and the first maintenance duration of the preceding color is less than the first preset duration. The maintenance time of the illuminated color in a traffic light is fixed, and the first preset duration is less than the maintenance time of the illuminated color. In some implementations, the first preset duration can be 1 second, meaning the illuminated color must be maintained for at least 1 second.

[0261] Step S1042: Use the preceding color as the final color of the current frame. Since the preceding color meets the preset maintenance condition, it indicates that the single lamp still displays the preceding color in the current frame, and the color of the single lamp has not changed. Therefore, the final color of the current frame is the preceding color.

[0262] Step S1043: Determine whether the countdown activation status of a single lamp is active; if it is active, proceed to step S1044; if it is not active, proceed to step S1046.

[0263] Step S1044: Determine whether the countdown number of the current frame is within the first preset countdown range; if it is, proceed to step S1045; if not, proceed to step S1046.

[0264] The first preset countdown range is obtained based on the duration of the illuminated color, and the length of the first preset countdown range is the same as the duration of the illumination.

[0265] It should be noted that if the countdown number of a single light in the current frame is unknown, then the countdown number in the current frame is not within the first preset countdown range.

[0266] Step S1045: Use the previous color as the final color of the current frame.

[0267] The following may occur when the front color does not meet the preset maintenance conditions:

[0268] (1) The preceding color is unknown, and the first holding time of the preceding color is less than the first preset duration. If the countdown number of the current frame is within the first preset countdown range, it indicates that the countdown has not ended. The unknown preceding color may be due to reasons such as a single light being blocked or the image being missed. In this case, the preceding color is still valid. Therefore, the preceding color is used as the final color of the current frame, that is, the final color of the current frame is still unknown. It should be noted that the holding time of different lit colors may be different, which leads to different first preset countdown ranges corresponding to different lit colors. If the preceding color is unknown, and the countdown number of the current frame is within the first preset countdown range corresponding to any lit color, the preceding color can be used as the final color of the current frame.

[0269] (2) The preceding color is the illuminated color, and the first hold duration of the preceding color is greater than or equal to the first preset duration. If the countdown number of the current frame is within the first preset countdown range, it indicates that the countdown has not ended. Although the first hold duration of the preceding color is greater than or equal to the first preset duration, the preceding color is still valid because the countdown has not ended. Therefore, the preceding color is used as the final color of the current frame. Since the first preset countdown range may be different for different illuminated colors, when the preceding color is the illuminated color, if the countdown number of the current frame is within the first preset countdown range corresponding to the preceding color, the preceding color is used as the final color of the current frame; otherwise, proceed to step S1046.

[0270] (3) The preceding color is unknown, and the first retention duration of the preceding color is greater than or equal to the first preset duration. This is similar to the cases (1) and (2) mentioned above, and will not be repeated here.

[0271] Step S1046: The final color of the current frame is unknown.

[0272] In some implementations, steps S1041 to S1046 may not be executed in the same order. Instead, it can be determined whether the previous color meets the preset maintenance condition and whether the countdown number of the current frame is within the first preset countdown range. The final color of the current frame is then determined based on the results of these two determinations. Specifically: when the previous color meets the preset maintenance condition, the final color is the previous color; when the previous color does not meet the preset maintenance condition, if the countdown activation state of a single light is active and the countdown number of the current frame is within the first preset countdown range, the final color is the previous color; if the countdown activation state of a single light is active and the countdown number of the current frame is not within the first preset countdown range, the final color is unknown; if the countdown activation state of a single light is inactive, the final color is unknown.

[0273] In some implementations, if the final color of the current frame is different from the previous color, it indicates that the color of a single light has changed, and the first hold duration of the previous color needs to be reset to zero.

[0274] Based on the above implementation method, the final color of a single light in the current frame can be accurately determined by utilizing the previous color and countdown information of the single light.

[0275] The embodiments of the traffic signal light state processing method provided in this application will be described below.

[0276] In some embodiments of this application, the single-lamp state also includes the flashing state of the single lamp in the image, where the flashing state is flashing, not flashing, or unknown. Flashing can be understood as the single lamp color repeatedly alternating between an illuminated color and black within a certain period of time. The duration of this period is the flashing duration. As long as the single lamp is at any moment within this period, regardless of whether the single lamp's color is an illuminated color or black, then the flashing state of the single lamp is confirmed as flashing. Taking green flashing as an example, the flashing duration is 3 seconds, and the single lamp color will repeatedly alternate between green flashing and black within these 3 seconds. The flashing state is obtained from the perception result, and the method for obtaining the perception result is described in the relevant method of step S101 in the aforementioned embodiments.

[0277] In this embodiment, it can be achieved through Figure 3 The following steps S201 to S204 are used to process the flashing state of a single lamp to obtain the final flashing state of the single lamp in the current frame.

[0278] Step S201: Smooth the flickering state of a single lamp in multiple consecutive frames of images to obtain the smooth flickering state of the single lamp in the current frame.

[0279] Specifically, the result of the smoothing process is taken as the smoothed flickering state of a single lamp in the current frame. Due to factors such as perception accuracy and image quality, errors may occur in preventing the flickering state of a single lamp, resulting in repeated jumps between flickering and non-flickering. Smoothing can effectively reduce this situation and maintain the stability of the single lamp's flickering state. In some implementations, the high-frequency flickering state of a single lamp in multiple consecutive frames can be taken as the smoothed flickering state of the single lamp in the current frame. The high-frequency flickering state is the flickering state that occurs most frequently in multiple consecutive frames. For example, if the high-frequency flickering state is non-flickering, then the smoothed flickering state is non-flickering. Through the above implementation methods, the smoothing process of the single lamp's flickering state can be completed conveniently and quickly.

[0280] Step S202: Determine the final flashing state of a single lamp in the current frame based on the smooth flashing state. If the smooth flashing state is flashing or not flashing, proceed to step S203; if the smooth flashing state is unknown, proceed to step S204.

[0281] Step S203: Set the smooth blinking state as the final blinking state of a single lamp in the current frame.

[0282] Step S204: Determine the final blink state of the current frame based on the previous blink state of the single lamp. The previous blink state is the final blink state of the single lamp in the previous frame of the image.

[0283] Specifically, if the preceding flashing state is not flashing, it indicates that the single lamp has stopped flashing, and the final flashing state of the current frame is not flashing. If the preceding flashing state is flashing, the single lamp may continue to flash or stop flashing in the current frame. Therefore, the smooth flashing state can still be regarded as the final flashing state of the single lamp in the current frame.

[0284] According to the method described in steps S201 to S204 above, the flickering state of a single lamp in multiple consecutive frames of images is used to comprehensively determine the final flickering state of a single lamp in the current frame, thereby realizing the temporal fusion of the flickering state in multiple consecutive frames of images. Compared with determining the final flickering state of a single lamp in the current frame based on the flickering state color in a single frame of image, the above method can effectively improve the accuracy of the flickering state of a single lamp.

[0285] The following description continues with an embodiment of the traffic light state processing method provided in this application, specifically describing steps S201 and S204 above.

[0286] I. Explanation of step S201.

[0287] In some embodiments of step S201 above, the flickering state of a single lamp in multiple consecutive frames can be smoothed through the following steps S2011 to S2013 to obtain the smooth flickering state of the single lamp in the current frame.

[0288] Step S2011: Match the flickering state of a single lamp in multiple consecutive frames of images with a preset flickering template. The flickering template includes a first flickering template and a second flickering template. The first flickering template includes a flickering state and the flickering state is flickering. The second flickering template includes two sequentially arranged flickering states and the flickering state is not flickering.

[0289] Matching results include successful matches and failed matches.

[0290] A successful match between a single LED and the first flashing template occurs when the LED flashes in at least one frame out of multiple consecutive images. Conversely, a failed match occurs when the LED remains stationary in all frames out of multiple consecutive images. If the LED successfully matches the first flashing template, it indicates that the LED is flashing; therefore, proceed to step S2012 to set the smooth flashing state to flashing.

[0291] A successful match between a single LED and the second flashing template occurs when the LED flashes for two consecutive frames out of a series of images, indicating that it is not flashing. Conversely, a failed match occurs when the LED does not flash for two consecutive frames out of a series of images. If the LED successfully matches the second flashing template, it indicates that the LED is not flashing. Therefore, proceed to step S2012 to set the smooth flashing state to non-flickering.

[0292] If a single light fails to match both the first and second flashing templates, proceed to step S2013.

[0293] Step S2012: If the match with the first flashing template is successful, the smooth flashing state is flashing; if the match with the second flashing template is successful, the smooth flashing state is not flashing.

[0294] In some implementations, if a single lamp successfully matches both the first and second flashing templates, an abnormal situation may occur where the flashing state is both flashing and not flashing. Since the flashing duration during the single lamp's illumination is usually relatively short, and it is not flashing most of the time, the smooth flashing state is set to not flashing in this case.

[0295] Step S2013: The smooth flashing state is a high-frequency flashing state.

[0296] Based on the method described in steps S2011 to S2013 above, the smooth flashing state of a single lamp in the current frame can be obtained quickly and accurately using a preset flashing template.

[0297] In some implementations, the first and second flashing templates can be matched sequentially, rather than in parallel. Specifically, in this implementation, the flashing state of a single lamp in multiple consecutive frames can be smoothed through the following steps S2014 to S2016 to obtain the smooth flashing state of the single lamp in the current frame.

[0298] Step S2014: Match the flashing state of a single light in multiple consecutive frames of images with the preset first flashing template; if the match is successful, proceed to step S2015; if the match is successful, proceed to step S2016.

[0299] Step S2015: Match the flickering state of a single lamp in multiple consecutive frames of images with a preset second flicker template; if the single lamp successfully matches the second flicker template, the smooth flickering state is no flickering; if the single lamp fails to match the second flicker template, the smooth flickering state is flickering.

[0300] Step S2016: Match the flickering state of a single lamp in multiple consecutive frames of images with the second flickering template; if the single lamp successfully matches the second flickering template, the smooth flickering state is no flickering; if the single lamp fails to match the second flickering template, the smooth flickering state is unknown.

[0301] The method described in steps S2014 to S2016 above can not only complete the matching of the first and second flashing templates, but also improve the efficiency of obtaining the smooth flashing state.

[0302] II. Explanation of step S204.

[0303] In some embodiments of step S204 above, the final flashing state of the current frame can be determined by following steps S2041 to S2045, based on the previous flashing state of a single lamp.

[0304] Step S2041: Determine whether the front blinking state is blinking or not blinking; if it is not blinking, proceed to step S2042; if it is blinking, proceed to step S2043.

[0305] Step S2042: The final blinking state is no blinking.

[0306] Step S2043: Determine whether the final color of a single light in the current frame meets the preset blinking conditions; if it does, proceed to step S2044; if it does not, proceed to step S2045.

[0307] The preset blinking condition is either the first blinking condition or the second blinking condition. The first and second blinking conditions are explained below.

[0308] (1) Explanation of the first flashing condition.

[0309] The first blinking condition is: the final color of the current frame and the previous color are both the colors when a single lamp is lit, and the final color of the current frame is different from the previous color.

[0310] If the first flashing condition is met when the preceding flashing state is flashing, it indicates that the single light has switched to a different illumination color while flashing. The transition between the two different illumination colors can be understood as a continuation of the previous flashing; therefore, the flashing state of the current frame is still determined as flashing. For example, if the final color of the current frame is a warning illumination color (e.g., yellow), and the preceding color is a color indicating permission to pass (e.g., red), and the first flashing condition is met when the preceding flashing state is flashing, it indicates that the single light is flashing green and then switches to yellow in the current frame.

[0311] (2) The second flashing condition is explained.

[0312] Second blinking condition: The final color of the current frame is the same as the color of a single lamp when it is lit, and the second duration of the previous blinking state is less than the second preset duration, and the countdown number of the current frame is within the second preset countdown range.

[0313] When a single LED is lit, it can consist of a constant-on phase and a blinking phase, arranged sequentially. During the constant-on phase, the LED continuously displays its illuminated color. During the blinking phase, the illuminated color alternates between constantly lit and black. The duration of both the constant-on and blinking phases is fixed, and the sum of their durations equals the duration of the illuminated color. For example, if the constant-on phase is 27 seconds, the blinking phase is 3 seconds, and the illuminated color lasts for 30 seconds.

[0314] The value of the second preset duration can be determined based on the duration of the flashing phase. For example, the second preset duration is equal to the duration of the flashing phase.

[0315] The second preset countdown range can be obtained based on the duration of the illuminated color. The length of the second preset countdown range is the same as the duration of the illumination. For example, if the illumination duration is 30 seconds, the second preset countdown range is from 30 to 1.

[0316] The second preset countdown range can also be obtained based on the countdown number range corresponding to the flashing phase. For example, if the constant light phase and the flashing phase are arranged from first to last, with the constant light phase being 27 seconds and the flashing phase being 3 seconds, the second preset countdown range is 3 to 1.

[0317] It should be noted that if the countdown number of a single lamp in the current frame is unknown, then the countdown number in the current frame is not within the range of the second preset countdown.

[0318] Step S2044: The final blinking state is blinking.

[0319] Step S2045: The final flashing state is unknown.

[0320] In some implementations, steps S2041 to S2045 may not be executed in the same order. Instead, it can be determined whether the preceding flash is flashing or not, and whether the final color of a single LED in the current frame meets the preset flashing condition. The final flashing state of the current frame is then determined based on the results of these two determinations. Specifically: if the preceding flashing state is not flashing, the final flashing state is not flashing; if the preceding flashing state is flashing, and the final color of a single LED in the current frame meets the preset flashing condition, the final flashing state is flashing; if the preceding flashing state is flashing, but the final color of a single LED in the current frame does not meet the preset flashing condition, the final flashing state is unknown.

[0321] In some implementations, if the final color of the current frame is different from the previous color, it indicates that the color of the single lamp has changed, and the second hold duration of the previous flashing state needs to be reset to zero.

[0322] Based on the above implementation method, the final blinking state of a single lamp in the current frame can be accurately determined by utilizing the pre-flickering state and countdown status of the single lamp.

[0323] The embodiments of the traffic signal light state processing method provided in this application will be described below.

[0324] In some embodiments of this application, the single-lamp state also includes the visibility state of the single lamp in the image, where the visibility state is either visible or invisible. A visible single lamp indicates that the single lamp is present in the image and cannot be detected from the image; a invisible single lamp indicates that the single lamp is not present in the image and cannot be detected from the image.

[0325] In this embodiment, it can be achieved through Figure 4 The following steps S301 to S303 are used to process the visibility state of a single lamp to determine whether to activate the occlusion state of the single lamp.

[0326] Step S301: Match the visibility state of a single light in multiple consecutive frames of images with a preset visibility template, which includes a first visibility template and a second visibility template.

[0327] The first visible template consists of N sequentially arranged visible states, where the 1st to the (N-2nd)th visible state is visible, and the (N-1th)th to the Nth visible state is invisible, where N is the number of consecutive frames. For example, if N = 5, visible is represented by Y, and invisible is represented by E, then the first visible template is [YYYEE].

[0328] The second visible template consists of N sequentially arranged visible states, where the first to the (N-2)th visible states are invisible, and the (N-1)th to the Nth visible states are visible. Referring to the example above, the second visible template is [EEEYY].

[0329] Matching results include successful matches and failed matches.

[0330] A successful match is defined as follows: the visible state of a single light in frames 1 to N of a series of consecutive images is identical to the first to Nth visible states in a preset visible template. If this condition cannot be met, the match is considered a failure.

[0331] In this embodiment, if a match is successfully made with the first visible template, the process proceeds to step S302. If a match is successfully made with the second visible template, the process proceeds to step S303.

[0332] Step S302: Activate the occlusion state of a single lamp.

[0333] If the occlusion state is activated, it indicates that the occlusion of a single lamp is reliable.

[0334] Step S303: Turn off the obstruction state of the single lamp.

[0335] If the occlusion state is not activated, it means that a single light being occluded is not reliable.

[0336] It should be noted that if a single light fails to match both the first and second visible templates, it will remain in the original active state of the occlusion state; if the original occlusion state was active, it will remain active until it successfully matches the second visible template and then turn off; if the original occlusion state was off, it will remain off until it successfully matches the first visible template and then turn on.

[0337] Based on the methods described in steps S301 to S303 above, the template can be used to quickly and accurately determine whether the occlusion state of a single light is reliable or unreliable. As can be seen from the aforementioned method embodiments, the color of a single light in an image may be unknown. If the occlusion state is active at this time, it can be determined that the unknown color is due to the occlusion of the single light. Furthermore, according to the aforementioned method embodiments, the final color of the single light in the current frame can be inferred when the color of the single light is unknown. If the occlusion state is active at this time, it is equivalent to achieving color inference when the single light is occluded. Based on this, if a single light is temporarily occluded by a dynamic obstacle, the color of the single light can also be inferred through the aforementioned method embodiments.

[0338] The following is in conjunction with the appendix Figure 5 To be continued Figure 7 The embodiments of the traffic signal light state processing method provided in this application will be described.

[0339] First, please refer to the appendix.Figure 5 , Figure 5 This example illustrates the overall flow of a traffic light state processing method. In this embodiment, the intelligent device is a vehicle, and the camera on the vehicle can capture images of traffic lights in the vehicle's environment. The perception model on the vehicle can perceive the traffic light images to obtain the state of each individual light. Figure 5 The model result in the text is the perception result output by the perception model. Figure 5 The size of the sliding window is determined based on the number of consecutive frames to be acquired in this application. In this embodiment, the number of images is 5. Therefore, the size of the sliding window is 5, meaning the sliding window is used to accommodate the perception results of 5 consecutive frames. Figure 5 As shown, the individual light status of traffic lights can be processed through the following steps S401 to S407 to obtain a more accurate individual light status.

[0340] Step S401: Determine whether the model results contain observation results of the current frame image. The observation results are the single-lamp states obtained by the perception model from the current frame image.

[0341] If it exists, proceed to step S402; if it does not exist, proceed to step S403.

[0342] Step S402: Add the observation results to the sliding window and proceed to step S404.

[0343] Step S403: Set a virtual result for the current frame image, add the virtual result to the sliding window, and proceed to step S404. All single-lamp states (including color, countdown timer, flashing state, and occlusion state) in the virtual result are unknown.

[0344] Step S404: When the perception results (including observation results or virtual results) of 5 consecutive frames of images are added to the sliding window, countdown tracking is performed on the perception results of the 5 consecutive frames of images. Countdown tracking is used to determine the countdown activation state of a single lamp, and the method of countdown tracking is the same as the method in step S102 of the aforementioned method embodiment.

[0345] Step S405: Perform occlusion temporal fusion on the perception results of 5 consecutive frames of images within the sliding window. Occlusion temporal fusion is used to determine whether the occlusion state of a single light is active or off. The method of occlusion temporal fusion is the same as that described in steps S301 to S303 in the aforementioned method embodiment.

[0346] Step S406: Perform color temporal fusion on the perception results of 5 consecutive frames of images within the sliding window. The method of color temporal fusion is the same as that described in steps S103 to S104 in the aforementioned method embodiment.

[0347] Step S407: Perform flicker timing fusion on the perception results of 5 consecutive frames of images within the sliding window. The flicker timing fusion method is the same as that described in steps S201 to S204 in the aforementioned method embodiment.

[0348] Based on the method described in steps S401 to S407 above, the countdown activation state, occlusion state, color, and flashing state of a single light in each frame of the image can be accurately obtained.

[0349] See appendix Figure 6 , Figure 6 Examples of color temporal fusion methods are illustrated in some embodiments. For example... Figure 6 As shown, color temporal fusion can be performed on the perception results of multiple consecutive frames of images within the sliding window through the following steps S501 to S511.

[0350] Step S501: Obtain the color of a single light in multiple consecutive frames within a sliding window.

[0351] Step S502: Match the color of a single light in multiple consecutive frames of images with a preset color conversion template; if the match is successful, proceed to step S503; if the match fails, proceed to step S504. The preset color conversion template and matching method are the same as those described in step S1031 of the aforementioned embodiment.

[0352] Step S503: Use the color of a single light in the current frame as the smoothed color, use this smoothed color as the final color of the current frame, and proceed to step S510. The current frame is the frame with the latest timestamp among multiple consecutive frames within the sliding window.

[0353] Step S504: Obtain the high-frequency color. The high-frequency color is the color that appears most frequently in multiple consecutive frames of images within a sliding window for a single light source.

[0354] Step S505: Determine whether the high-frequency color is an unknown color; if yes, proceed to step S506; if not, use the high-frequency color as the smoothed color, use the smoothed color as the final color of the current frame, and proceed to step S510.

[0355] Step S506: Determine whether the previous color meets the preset maintenance conditions; if it does, proceed to step S507; if it does not, proceed to step S508.

[0356] The preceding color is the final color of a single light in the previous frame of the image, and the preset maintenance condition is the same as the preset maintenance condition in step S1051 of the aforementioned method embodiment.

[0357] Step S507: Use the previous color as the final color of the current frame.

[0358] Step S508: Determine whether the countdown number of the current frame is within the first preset countdown range; if it is, proceed to step S507; if not, proceed to step S509.

[0359] Step S509: The final color of the current frame is unknown.

[0360] Step S510: Determine whether the final color of the current frame is the same as the previous color; if they are the same, proceed to step S512; if they are different, proceed to step S511.

[0361] Step S511: Update the state, then exit blending. Exiting blending ends the timing blending of the colors of individual lights within the current sliding window.

[0362] Update status includes:

[0363] Update the countdown activation status of a single light to deactivated.

[0364] Reset the first retention time of the front color to zero.

[0365] Clear the second duration of the pre-flash state to zero.

[0366] Step S512: Exit fusion.

[0367] Based on the methods described in steps S501 to S512 above, the final color of a single light in the current frame can be determined by comprehensively considering the colors of a single light across multiple consecutive frames, effectively improving the accuracy of the single light color. Furthermore, the countdown activation state and countdown number of the single light are used as constraints when obtaining its final color in the current frame, further enhancing the accuracy of the single light color.

[0368] See appendix Figure 7 , Figure 7 Examples of methods for flicker timing fusion are illustrated in some embodiments. For example... Figure 7 As shown, the perception results of multiple consecutive frames of images within the sliding window can be fused by the following steps S601 to S605.

[0369] Step S601: Obtain the blinking state of a single lamp in multiple consecutive frames within a sliding window.

[0370] Step S602: Match the flickering state of a single lamp in multiple consecutive frames of images with a preset flickering template. The flickering template includes a first flickering template and a second flickering template, which are the same as the flickering template in step S2011 of the aforementioned method embodiment.

[0371] If a single light successfully matches the first flashing template, the smooth flashing state is flashing; if it successfully matches the second flashing template, the smooth flashing state is not flashing; if a single light successfully matches both the first and second flashing templates, the smooth flashing state is not flashing.

[0372] If the match with both the first and second flashing templates fails, proceed to step S603.

[0373] Step S603: Obtain the high-frequency flickering state.

[0374] Step S604: Determine whether the high-frequency flickering state is unknown; if unknown, proceed to step S605; if flickering, the final flickering state of the current frame is flickering. If not flickering, the final flickering state of the current frame is not flickering.

[0375] Step S605: Obtain the previous blinking state and determine whether the previous blinking state is blinking; if it is not blinking, then the final blinking state of the current frame is not blinking.

[0376] When the current flashing state is flashing, the first and second flashing conditions are obtained to determine whether the final color of a single light in the current frame satisfies the first or second flashing conditions; if the first or second flashing conditions are satisfied, the final flashing state of the current frame is flashing; otherwise, the final flashing state of the current frame is not flashing.

[0377] According to the method described in steps S601 to S605 above, the flickering state of a single lamp in multiple consecutive frames of images is used to comprehensively determine the final flickering state of a single lamp in the current frame, which can effectively improve the accuracy of the flickering state of a single lamp.

[0378] It should be noted that although the steps in the above embodiments are described in a specific order, those skilled in the art will understand that in order to achieve the effect of this application, different steps do not necessarily have to be executed in such an order. They can be executed simultaneously (in parallel) or in other orders. These adjusted solutions are equivalent to the technical solutions described in this application and therefore will also fall within the protection scope of this application.

[0379] Those skilled in the art will understand that all or part of the processes in the method of the above-described embodiment can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable storage medium can include any entity or device capable of carrying the computer program code, a medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory, a random access memory, an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0380] Another aspect of this application provides a computer-readable storage medium.

[0381] In one embodiment of a computer-readable storage medium according to this application, the computer-readable storage medium may be configured to store a program for performing the traffic light state processing method of the above-described method embodiments. This program may be loaded and run by a processor to implement the traffic light state processing method. For ease of explanation, only the parts related to the embodiments of this application are shown; for specific technical details not disclosed, please refer to the method section of the embodiments of this application. The computer-readable storage medium may be a storage device comprising various electronic devices. Optionally, in the embodiments of this application, the computer-readable storage medium is a non-transitory computer-readable storage medium.

[0382] Another aspect of this application provides a smart device.

[0383] In one embodiment of a smart device according to this application, the smart device may include at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program, which, when executed by the at least one processor, implements the methods described in any of the above embodiments. See Appendix Figure 8 , Figure 8 The image exemplarily illustrates a communication connection between memory 11 and processor 12 via a bus.

[0384] In some embodiments of this application, the smart device may further include at least one sensor for sensing information. The sensor is communicatively connected to any type of processor mentioned in this application. Optionally, the smart device may further include an autonomous driving system for guiding the smart device to drive autonomously or assisting in driving. The processor communicates with the sensor and / or the autonomous driving system to perform the methods described in any of the above embodiments.

[0385] The technical solution of this application has been described above with reference to one embodiment shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of this application is obviously not limited to these specific embodiments. Without departing from the principles of this application, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of this application.

Claims

1. A method for processing the state of a traffic signal light, characterized in that, Applied to smart devices, the method includes: The perception results of acquiring multiple consecutive frames of traffic light images are obtained. The perception results include the single light status of a traffic light. The single light status includes color and countdown numbers. The color is the lighting color when the single light is lit or unknown. The countdown numbers of the single lamp in the consecutive frames of images are obtained, and the countdown activation state of the single lamp is determined according to the deviation between the countdown numbers. The countdown activation state is either active or inactive. The color of the single lamp in the consecutive multi-frame images is smoothed to obtain the smooth color of the single lamp in the current frame, where the current frame is the frame with the latest timestamp in the consecutive multi-frame images; The final color of the single light in the current frame is determined based on the smooth color and the countdown number of the single light in the current frame; The countdown activation state is updated based on the final color of the current frame; if the final color is different from the previous color, the countdown activation state is updated to inactive, where the previous color is the final color of the single lamp in the image preceding the current frame.

2. The method according to claim 1, characterized in that, Determining the final color of a single light in the current frame based on the smoothed color and the countdown number of the single light in the current frame includes: When the smooth color is the highlighted color, the final color is the smooth color; When the smooth color is unknown, if the countdown activation state is active, the final color is determined based on the countdown number of the single light in the current frame and the previous color; if the countdown activation state is inactive, the final color of the current frame is unknown.

3. The method according to claim 1, characterized in that, Determining the countdown activation state of a single lamp based on the deviation between the countdown digits includes: The countdown deviation between every two adjacent images in the continuous multi-frame images is obtained, and the countdown deviation is the deviation of the countdown numbers of the single lamp between the two adjacent images; If the countdown deviation between all adjacent images meets the preset activation condition, then the countdown activation state is activated; If at least one of the countdown deviations does not meet the preset activation condition, then the countdown activation state is inactive; The preset activation condition is that the countdown deviation is within a preset deviation range.

4. The method according to claim 3, characterized in that, The countdown deviation is within a preset deviation range when the representative value of the countdown deviation is less than a preset threshold. The representative value of the countdown deviation is the absolute value of the difference between the time deviation and the countdown deviation, wherein the time deviation is the deviation between the timestamps of two images, and the two images are two adjacent images corresponding to the countdown deviation.

5. The method according to claim 1, characterized in that, The step of smoothing the color of the single lamp in the consecutive frames of images to obtain the smoothed color of the single lamp in the current frame includes: The high-frequency color is used as the smooth color of the single light in the current frame, and the high-frequency color is the single light color that appears most frequently in the consecutive multi-frame images.

6. The method according to claim 5, characterized in that, The method includes smoothing the color of the single light in a series of consecutive frames of images by means of: The color of the single lamp in the consecutive multi-frame images is matched with a preset color conversion template. The color conversion template includes M template colors arranged in sequence. The template color is the color of the single lamp in the two color conversion stages, and the converted color is the lighting color when the single lamp is lit. M≤N, where N is the number of consecutive multi-frame images. If the match is successful, the color of the single light in the current frame will be used as the smooth color. If the match fails, the high-frequency color will be used as the smooth color. in, A successful match is defined as follows: the color of the single light in the Nth to N-M+1th frames of a series of images is the same as the color of the Mth to 1st template in the color conversion template, respectively.

7. The method according to claim 6, characterized in that, The color conversion template includes a first color conversion template and a second color conversion template; In the first color conversion template, the number of template colors is M = N = 5, and the first to third template colors are the same, the fourth to fifth template colors are the same lighting color, and the third and fourth template colors are different. In the second color conversion module, the number of template colors is M = N-1 = 4, and the first to third template colors are the same, the fourth template color is a single lighting color, and the third and fourth template colors are different.

8. The method according to claim 6, characterized in that, The two color conversion stages include: The stage in which the color of a single lamp changes from one illuminated color to another; The stage in which the color of a single lamp changes from unknown to lit.

9. The method according to claim 2, characterized in that, The step of determining the final color based on the countdown digit of the single light in the current frame and the preceding color includes: When the preceding color meets the preset maintenance condition, the final color is the preceding color; When the preceding color does not meet the preset maintenance conditions If the countdown activation state of the single light is active and the countdown number of the current frame is within the first preset countdown range, then the final color is the previous color; If the countdown activation state of the single light is active and the countdown number of the current frame is not within the first preset countdown range, then the final color is unknown; If the countdown activation state of the single lamp is inactive, then the final color is unknown; in, The preset maintenance conditions include: the front color is the color of a single lamp when it is lit, and the first maintenance duration of the front color is less than the first preset duration.

10. The method according to claim 9, characterized in that, The method includes: If the final color of the current frame is different from the previous color, then the first retention duration of the previous color is reset to zero.

Images