A signal light state detection method, device and equipment

Abstract

The application provides a signal lamp state detection method, device and equipment, the method comprises the following steps: obtaining a first sub-image from a to-be-detected image based on an initial lamp panel frame area, and determining a reference lighted area of a to-be-detected lamp panel frame based on the first sub-image; selecting one candidate offset detection result from a plurality of candidate offset detection results as a best offset detection result based on the reference lighted area and the initial lamp panel frame area; obtaining a target lamp panel frame area by offsetting the initial lamp panel frame area based on the best offset detection result, and obtaining a second sub-image corresponding to the target lamp panel frame area from the to-be-detected image; and detecting the signal lamp state of the to-be-detected lamp panel frame based on the second sub-image. Through the application scheme, the accuracy of signal lamp detection can be improved, the picture offset can be effectively detected, and the false detection and missed detection caused by picture jitter can be reduced.

Description

Technical Field

[0001] This application relates to the field of intelligent transportation, and in particular to a method, apparatus and equipment for detecting the status of traffic lights. Background Technology

[0002] Traffic lights are signals used to direct traffic. They generally consist of red, green, and yellow lights. A red light indicates "stop," a green light indicates "go," and a yellow light indicates "warning." Traffic lights include: motor vehicle traffic lights, non-motor vehicle traffic lights, pedestrian crossing traffic lights, directional indicator lights (arrow lights), lane traffic lights, flashing warning lights, and level crossing traffic lights, etc.

[0003] Traffic light status detection refers to acquiring and analyzing images of traffic lights to determine their status, such as the color of the light (i.e., whether the light is on). By detecting the status of traffic lights, it is possible to analyze whether the traffic lights are meeting expectations (e.g., whether the red light duration meets expectations) and issue alarms or notifications for abnormal states.

[0004] However, there is no effective detection method in the relevant technology for detecting the status of traffic lights, which makes it impossible to accurately detect the status of traffic lights, resulting in incorrect traffic light status. Summary of the Invention

[0005] This application provides a method for detecting the status of a traffic light, the method comprising:

[0006] Acquire a target scene image to be detected, the target scene image including a light panel frame to be detected;

[0007] Based on the configured initial light panel frame area, a first sub-image is obtained from the image to be detected, and a reference lighting area of ​​the light panel frame to be detected is determined based on the first sub-image.

[0008] Based on the reference lighting area and the initial lamp frame area, one candidate offset detection result is selected from multiple candidate offset detection results as the best offset detection result;

[0009] Based on the optimal offset detection result, the initial lamp frame region is offset to obtain the target lamp frame region, and the second sub-image corresponding to the target lamp frame region is obtained from the image to be detected;

[0010] The signal light status of the light panel frame to be detected is determined based on the second sub-image.

[0011] This application provides a method for detecting the status of a traffic light, the method comprising:

[0012] When it is determined that there is a traffic light malfunction event in the target scene based on the image to be detected of the target scene, the image to be detected is displayed, and a target rectangle is superimposed on the image to be detected;

[0013] The target rectangular frame is determined on the image to be detected based on the target lamp panel frame region, which is obtained by offsetting the configured initial lamp panel frame region based on the best offset detection result.

[0014] This application provides a traffic light status detection device, the device comprising:

[0015] The acquisition module is used to acquire a target scene image to be detected, the image to be detected including a light panel frame to be detected; based on the configured initial light panel frame area, a first sub-image is acquired from the image to be detected, and a reference lighting area of ​​the light panel frame to be detected is determined based on the first sub-image;

[0016] The processing module is configured to select one candidate offset detection result as the best offset detection result from multiple candidate offset detection results based on the reference lighting area and the initial lamp frame area; offset the initial lamp frame area based on the best offset detection result to obtain the target lamp frame area; and obtain the second sub-image corresponding to the target lamp frame area from the image to be detected.

[0017] The detection module is used to detect the status of the signal lights in the light panel frame to be detected based on the second sub-image.

[0018] This application provides an electronic device, including: a processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions to implement the traffic light status detection method of the example above in this application.

[0019] As can be seen from the above technical solutions, in this embodiment, the best offset detection result is selected from multiple candidate offset detection results. Based on the best offset detection result, the initial light panel frame area is offset to obtain the target light panel frame area. A second sub-image corresponding to the target light panel frame area is obtained from the image to be detected. The traffic light status is detected based on the second sub-image. The traffic light status is accurately detected based on the second sub-image, thus obtaining the correct traffic light status and improving the accuracy of traffic light detection. The offset detection method effectively detects image offset, reduces false detections and missed detections caused by image jitter, and solves problems such as false detections caused by camera pole shaking or false alarms caused by obstructions in the image, improving accuracy and reducing false detections. It also promptly detects abnormal traffic light states, improves the operational efficiency of the maintenance department, and reduces the possibility of traffic accidents. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating a traffic light status detection method according to one embodiment of this application;

[0021] Figure 2 This is a flowchart illustrating the initial configuration process in one embodiment of this application;

[0022] Figures 3A-3D This is a schematic diagram of a traffic light configuration in one embodiment of this application;

[0023] Figure 4 This is a flowchart illustrating the signal light status detection process in one embodiment of this application;

[0024] Figure 5 This is a schematic diagram of the screen offset detection process in one embodiment of this application;

[0025] Figure 6 This is a schematic diagram of the lamp panel detection algorithm in one embodiment of this application;

[0026] Figure 7 This is a flowchart illustrating the calculation of the reliability of the offset result in one embodiment of this application;

[0027] Figure 8A This is a schematic diagram of traffic light following and positioning in one embodiment of this application;

[0028] Figure 8B This is a schematic diagram of the initial segmentation of the traffic lights in one embodiment of this application;

[0029] Figure 9A This is a flowchart illustrating the process of determining the current state of a traffic light in one embodiment of this application.

[0030] Figure 9B This is a flowchart illustrating the determination of luminance and chromaticity information in one embodiment of this application.

[0031] Figure 9C This is a flowchart illustrating the calculation of the signal light's graphical state in one embodiment of this application.

[0032] Figure 9D This is a flowchart illustrating the comprehensive determination of traffic light status in one embodiment of this application;

[0033] Figure 10 This is an example diagram of single-channel polling in one embodiment of this application;

[0034] Figure 11 This is a schematic diagram of the structure of a signal light status detection device according to one embodiment of this application;

[0035] Figure 12This is a hardware structure diagram of an electronic device according to one embodiment of this application. Detailed Implementation

[0036] This application proposes a traffic light status detection method, which can be applied to electronic devices. See [link to relevant documentation]. Figure 1 The diagram shown is a flowchart of the traffic light status detection method, which may include:

[0037] Step 101: Obtain the image to be detected of the target scene, which includes the frame of the light panel to be detected.

[0038] Step 102: Based on the configured initial light panel frame area, obtain a first sub-image from the image to be detected, and determine the reference lighting area of ​​the light panel frame to be detected based on the first sub-image.

[0039] Step 103: Based on the reference lighting area and the initial lamp frame area, select one candidate offset detection result from multiple candidate offset detection results as the best offset detection result.

[0040] Step 104: Based on the best offset detection result, offset the initial lamp panel frame region to obtain the target lamp panel frame region, and obtain the second sub-image corresponding to the target lamp panel frame region from the image to be detected.

[0041] Step 105: Detect the signal light status of the light panel frame to be detected based on the second sub-image.

[0042] Based on the reference lighting area and the initial lamp frame area, a candidate offset detection result is selected as the best offset detection result from multiple candidate offset detection results. This may include: acquiring multiple candidate offset detection results; for each candidate offset detection result, correcting the initial lamp frame area based on the candidate offset detection result to obtain the corrected lamp frame area; and selecting the best offset detection result from multiple candidate offset detection results based on the intersection area between the reference lighting area and each corrected lamp frame area.

[0043] For example, multiple candidate offset detection results may include, but are not limited to, at least two of the following: lamp panel offset detection result, image offset detection result, the best offset detection result of the previous frame of the image to be detected, and fixed value offset detection result. Based on this, if the multiple candidate offset detection results include the lamp panel offset detection result, then obtaining the lamp panel offset detection result may include, but is not limited to: expanding the initial lamp panel frame region outward by a preset ratio; obtaining a third sub-image from the image to be detected based on the expanded lamp panel frame region; detecting the first lamp panel frame region based on the third sub-image; and obtaining the lamp panel offset detection result based on the offset value between the center position of the first lamp panel frame region and the center position of the initial lamp panel frame region.

[0044] For example, the initial lamp panel frame area may include the lower left corner lamp panel frame area and the upper right corner lamp panel frame area, and the first lamp panel frame area may include the lower left corner lamp panel frame area and the upper right corner lamp panel frame area. Obtaining a lamp panel offset detection result based on the offset value between the center position of the first lamp panel frame area and the center position of the initial lamp panel frame area may include: if the lower left corner lamp panel frame area meets a first preset condition and the upper right corner lamp panel frame area meets a second preset condition, then obtaining a lamp panel offset detection result based on the first offset value between the center position of the lower left corner lamp panel frame area and the center position of the lower left corner lamp panel frame area, and the second offset value between the center position of the upper right corner lamp panel frame area and the center position of the upper right corner lamp panel frame area. If the lower left corner lamp panel frame area meets the first preset condition but the upper right corner lamp panel frame area does not meet the second preset condition, then obtaining a lamp panel offset detection result based on the first offset value. If the first lower left corner light panel frame area does not meet the first preset condition, but the first upper right corner light panel frame area meets the second preset condition, then the light panel offset detection result is obtained based on the second offset value.

[0045] If the width difference ratio between the first lower left corner light panel frame area and the lower left corner light panel frame area is less than a threshold and the height difference ratio is less than a threshold, then the first preset condition is met; otherwise, the first preset condition is not met.

[0046] If the width difference between the upper right corner light panel area and the upper right corner light panel area is less than a threshold and the height difference is less than a threshold, then the second preset condition is met; otherwise, the second preset condition is not met.

[0047] For example, multiple candidate offset detection results may include, but are not limited to, at least two of the following: lamp panel offset detection result, image offset detection result, the best offset detection result of the previous frame of the image to be detected, and fixed value offset detection result. Based on this, if the multiple candidate offset detection results include the image offset detection result, then obtaining the image offset detection result may include, but is not limited to: obtaining the first frequency domain feature corresponding to the first sub-image and the second frequency domain feature corresponding to the reference sub-image, where the reference sub-image is a stored sub-image used to obtain the initial lamp panel frame region; determining the power spectrum based on the first and second frequency domain features, determining the feature matrix based on the power spectrum, and determining the point where the maximum value in the feature matrix is ​​located as the maximum correlation position; obtaining a sub-region within a preset range from the first sub-image with the maximum correlation position as the center, determining the weighted centroid of the sub-region, and obtaining the image offset detection result based on the weighted centroid.

[0048] For example, obtaining the first frequency domain feature corresponding to the first sub-image and the second frequency domain feature corresponding to the reference sub-image may include, but is not limited to: filtering the first sub-image to obtain a filtered first sub-image; padding the filtered first sub-image with zeros so that its width and height are powers of 2 to obtain a zero-padded first sub-image; multiplying a preset window with the zero-padded first sub-image to obtain a windowed first sub-image; performing a discrete Fourier transform on the windowed first sub-image and obtaining the real part to obtain the first frequency domain feature; filtering the reference sub-image to obtain a filtered reference sub-image; padding the filtered reference sub-image with zeros so that its width and height are powers of 2 to obtain a zero-padded reference sub-image; multiplying a preset window with the zero-padded reference sub-image to obtain a windowed reference sub-image; performing a discrete Fourier transform on the windowed reference sub-image and obtaining the real part to obtain the second frequency domain feature.

[0049] For example, selecting the best offset detection result from multiple candidate offset detection results based on the intersection area between the reference illuminated area and each corrected lamp panel frame area may include, but is not limited to: if multiple reference illuminated areas are obtained, for each reference illuminated area, a weight value can be determined based on the intersection area between the reference illuminated area and the corrected lamp panel frame area and the confidence level of the reference illuminated area. The confidence level can represent the probability that the reference illuminated area is illuminated. Then, the confidence level of the corrected lamp panel frame area can be determined based on the weight value of each reference illuminated area; the candidate offset detection result corresponding to the highest confidence level can be selected as the best offset detection result.

[0050] For example, detecting the signal light status of a light panel frame to be detected based on a second sub-image includes: dividing the second sub-image into multiple single-lamp sub-regions, each single-lamp sub-region corresponding to a signal light; for each single-lamp sub-region, determining a first detection result of the single-lamp sub-region based on the intersection area of ​​a reference lit area and the single-lamp sub-region; determining a second detection result of the single-lamp sub-region based on the brightness and chromaticity information of the single-lamp sub-region; determining a target detection result of the single-lamp sub-region based on the first and second detection results, wherein the target detection result indicates whether the signal light is lit or off; and determining the signal light status of the light panel frame to be detected based on the color of the signal light in each single-lamp sub-region and the target detection result of each single-lamp sub-region.

[0051] For example, dividing the second sub-image into multiple single-lamp sub-regions may include, but is not limited to: determining the position and orientation of the traffic lights based on the axis equation of the traffic lights, the position of the first traffic light, the interval between traffic lights, and the size of the traffic lights; for each traffic light, the corresponding single-lamp sub-region may be divided from the second sub-image based on the position and orientation of that traffic light.

[0052] For example, before determining the position and orientation of the traffic lights based on the axis equation of the traffic lights, the position of the first traffic light, the interval between traffic lights, and the size of the traffic lights, a first intermediate parameter is determined based on the first slope parameter and the first intercept parameter corresponding to the previous frame of the image to be detected, and the center position coordinates of the first traffic light in the second sub-image; a second slope parameter corresponding to the image to be detected is determined based on the first slope parameter, the configured following rate, the first intermediate parameter, and the center position coordinates; a second intercept parameter corresponding to the image to be detected is determined based on the first intercept parameter, the following rate, the first intermediate parameter, and the second slope parameter; a second intermediate parameter is determined based on the first initial distance, the first interval position, and the center position coordinates corresponding to the previous frame of the image to be detected; and a second intermediate parameter is determined based on the first initial distance... The following parameters are used to determine the second initial distance of the image to be detected, which represents the distance between the edge of the image and the center of the first traffic light in the second sub-image. Based on the first interval position, the following rate, and the second intermediate parameter, the second interval position of the image to be detected is determined, which represents the distance between two adjacent traffic lights. Based on the first size value of the image to be detected in the previous frame, the following rate, and the size of the traffic lights in the second sub-image, the second size value of the image to be detected is determined. Based on the second slope parameter and the second intercept parameter, the axis equation of the traffic lights is determined. Based on the second initial distance, the position of the first traffic light is determined. Based on the second interval position, the interval between traffic lights is determined. Based on the second size value, the size of the traffic lights is determined.

[0053] For example, determining the second detection result of a single-lamp sub-region based on its brightness and chromaticity information may include, but is not limited to: generating a brightness analysis image based on the RGB image corresponding to the single-lamp sub-region, where the pixel value of each pixel in the brightness analysis image is the maximum value of the R, G, and B components corresponding to that pixel in the RGB image; determining the average brightness value based on the pixel values ​​of all pixels in the brightness analysis image; calculating the average S component value of all pixels based on the HSV image corresponding to the single-lamp sub-region; and calculating the average H component value of all candidate pixels for candidate pixels whose S component value is greater than the average S component value. Based on this, if the average brightness value is greater than a threshold and the average H component value is within the color range, the second detection result indicates that the traffic light is on; if the average brightness value is greater than the threshold and the average H component value is not within the color range, the second detection result indicates that the traffic light is off; if the average brightness value is not greater than the threshold, the second detection result indicates that the traffic light is off; wherein, the color range is the color range of the traffic light corresponding to the single-lamp sub-region.

[0054] For example, determining the target detection result of the single-lamp sub-region based on the first detection result and the second detection result may include, but is not limited to: if the first detection result indicates that the signal light is on, and the second detection result indicates that the signal light is on, then the target detection result indicates that the signal light is on; if the first detection result indicates that the signal light is on, and the second detection result indicates that the signal light is off, and the light panel frame to be detected is a single-lamp frame, then the target detection result indicates that the signal light is on; if the first detection result indicates that the signal light is on, and the second detection result indicates that the signal light is off, and the light panel frame to be detected is not a single-lamp frame, and there are at least two signal lights with similar brightness and / or similar chromaticity, then the target detection result indicates that the signal light is off ... If the first detection result indicates the signal light is on, and the second detection result indicates the signal light is off, and the light panel frame to be tested is not a single light frame, and there are at least two signal lights with similar brightness and color, then the target detection result indicates the signal light is on. If the first detection result indicates the signal light is off, and the second detection result indicates the signal light is off, then the target detection result indicates the signal light is off. If the first detection result indicates the signal light is off, the second detection result indicates the signal light is on, and the reflection detection result indicates reflection, then the target detection result indicates the signal light is off. If the first detection result indicates the signal light is off, the second detection result indicates the signal light is on, and the reflection detection result indicates no reflection, then the target detection result indicates the signal light is on.

[0055] For example, after detecting the traffic light status of the light panel frame to be detected based on the second sub-image, the detection results of the traffic light status of multiple frames of images to be detected can be used to analyze whether there are traffic light abnormal events in the target scene; wherein, the traffic light abnormal events include at least one of the following: screen abnormal events, single light abnormal events, light group abnormal events, and intersection abnormal events; wherein, the configured abnormal events are divided into short polling abnormal events and long polling abnormal events, and the polling duration of long polling abnormal events is longer than that of short polling abnormal events. Based on the traffic light status detection results of multiple frames of images to be detected, analyze whether there are traffic light abnormal events in the target scene. This may include, but is not limited to: if the interval between the current time and the detection time of a short-polling abnormal event reaches the polling duration of the short-polling abnormal event, then based on the traffic light status detection results of multiple frames of images to be detected, analyze whether there is a traffic light abnormal event in the target scene corresponding to the short-polling abnormal event; if the interval between the current time and the detection time of a long-polling abnormal event reaches the polling duration of the long-polling abnormal event, then based on the traffic light status detection results of multiple frames of images to be detected, analyze whether there is a traffic light abnormal event in the target scene corresponding to the long-polling abnormal event.

[0056] As can be seen from the above technical solutions, in this embodiment, the best offset detection result is selected from multiple candidate offset detection results. Based on the best offset detection result, the initial light panel frame area is offset to obtain the target light panel frame area. A second sub-image corresponding to the target light panel frame area is obtained from the image to be detected. The traffic light status is detected based on the second sub-image. The traffic light status is accurately detected based on the second sub-image, thus obtaining the correct traffic light status and improving the accuracy of traffic light detection. The offset detection method effectively detects image offset, reduces false detections and missed detections caused by image jitter, and solves problems such as false detections caused by camera pole shaking or false alarms caused by obstructions in the image, improving accuracy and reducing false detections. It also promptly detects abnormal traffic light states, improves the operational efficiency of the maintenance department, and reduces the possibility of traffic accidents.

[0057] The technical solutions described above in the embodiments of this application will be explained below in conjunction with specific application scenarios.

[0058] Traffic light status detection refers to the process of acquiring and analyzing images of traffic lights to determine their status. Traffic lights can include: motor vehicle traffic lights, non-motor vehicle traffic lights, pedestrian crossing traffic lights, directional indicator lights (arrow lights), lane traffic lights, flashing warning lights, road and railway level crossing traffic lights, supplementary lighting, LED guidance screens, etc.

[0059] This application provides a traffic light status detection method that can achieve traffic light status recognition and monitoring based on artificial intelligence. It captures real-time video footage of the traffic lights by connecting to electronic police cameras (electronic police cameras are installed at intersections to monitor events and various violations on the road; they store recordings at intersection terminals and transmit them over the network for viewing and analysis). Deep learning technology is used to detect the status of the video footage. If the video footage meets the detection requirements, deep learning technology is used to detect various types of traffic lights, reflectors, streetlights, and other non-traffic lights. Based on user configuration or the natural logic of the intersection, it determines whether the traffic light's operating status is abnormal and reports this to the user platform via a data interface protocol.

[0060] The signal light status detection method in the embodiments of this application may involve the following process.

[0061] First, the initial configuration process.

[0062] This embodiment presents a configuration method that can automatically detect traffic light panels within a region and generate labels, reducing the user's configuration costs. During the initial configuration process, deep learning algorithms can be used to automatically learn the feature representations of traffic lights from a large number of traffic light images, improving the accuracy of traffic light detection.

[0063] See Figure 2 The diagram shown illustrates the initial configuration process, which may include:

[0064] Step 201: Obtain the video stream, which is the video stream of the traffic light to be detected.

[0065] For example, video streams can be obtained from traffic cameras at intersections or from other recording devices, as long as the video stream shows the traffic light to be detected. See also Figure 3A The diagram shown is a schematic of a traffic light. (See attached image.) Figure 3B As shown, the signal lights in the video stream need to ensure that the light panel area 1 and the lit light 2 are intact, so that subsequent processing can be performed based on the video stream.

[0066] Step 202: Decode a frame from the video stream. The data format of the frame can be YUV or other data formats. There are no restrictions on this. We will take YUV format as an example in the following steps.

[0067] Step 203: On the decoded frame, the user delineates the detection area.

[0068] For example, the detection area can be a rectangular frame detection area, or it can be a detection area of ​​other shapes; taking a rectangular frame detection area as an example, this detection area can include one or more light panels. See also... Figure 3C The diagram shown is a schematic of the detection area drawn by the user, that is, the user draws a rectangular detection area 3 on this screen.

[0069] Step 204: Obtain the sub-image of the detection region from the frame, input the sub-image of the detection region into the lamp panel detection model, and detect the lamp panel region through the lamp panel detection model.

[0070] For example, a light panel detection model can be pre-trained to detect light panel regions in an image. Therefore, after inputting a sub-image of the detection region into the light panel detection model, the light panel detection model can detect the light panel region where the signal light exists within the detection region and output the light panel region.

[0071] For example, the light panel detection model can also detect the category of traffic lights. Therefore, after inputting the sub-image of the detection area into the light panel detection model, the light panel detection model can also output the category of traffic lights. For example, the category may include traffic lights, countdown lights, supplementary lights, and LED light panels.

[0072] Step 205: If the traffic light type is a red-green traffic light, then the illuminated area within the light panel is detected using the traffic light detection model, and the size and shape of the illuminated traffic light are also detected. Based on the size and shape of the illuminated traffic light, the traffic light type is determined, such as a 3-light left-turn traffic light, a 3-light straight-ahead traffic light, or a single-light red-green traffic light.

[0073] For example, a traffic light detection model can be pre-trained to detect the illuminated areas within the light panel, as well as the size and shape of the illuminated traffic lights. Therefore, after inputting a sub-image of the detected area into the traffic light detection model, the model can output the illuminated areas within the light panel, and the size and shape of the illuminated traffic lights. Then, the traffic light type is determined based on the size and shape of the illuminated traffic lights.

[0074] Step 206: On a single frame, generate a draggable and adjustable light panel frame based on the detected light panel, and pre-fill the configuration of the corresponding light for each light panel according to the shape of the light.

[0075] For example, after detecting a traffic light panel, a panel frame that allows dragging and adjustment can be generated based on the traffic light panel, see [link to relevant documentation]. Figure 3D The diagram shown illustrates the light panel frame, allowing users to drag and adjust it. Furthermore, the configuration of each light panel corresponding to the traffic lights (such as the type and type of traffic light) can be pre-filled based on the shape of the traffic lights, greatly improving user configuration efficiency.

[0076] Step 207: The user adjusts the configuration of the light panel frame and the corresponding signal lights.

[0077] For example, step 207 is optional. Users can adjust the configuration of the light panel frame and the corresponding signal lights, or they can choose not to adjust these configurations. For instance, the user can adjust the detected light panel frames, delete falsely detected light panel frames, add missed detection light panel frames, and configure information such as the detection event type, time thresholds for each type, and detection time period for each light panel area; there are no restrictions on this. Clearly, during the initial configuration process, the signal light detection method and the monitoring alarm type lights can be configured.

[0078] For example, users can also assign each traffic light to a predefined intersection. For instance, several light groups can be configured to the left turn light on the south side, the straight light on the south side, the left turn light on the west side, the straight light on the west side, the left turn light on the north side, the straight light on the north side, the left turn light on the east side, and the straight light on the east side at intersection 1.

[0079] For example, the user can continue to define the next detection area and repeat the above process.

[0080] Step 208: Associate the user configuration with the traffic light and save the user configuration.

[0081] At this point, the initial configuration process is complete, yielding the light panel frame (the user-adjusted light panel frame) and the traffic light configuration (the user-adjusted traffic light configuration, i.e., the user configuration). Based on this, the following can be stored: the initial light panel frame area (for ease of distinction, the user-adjusted light panel frame is referred to as the initial light panel frame area, indicating that the area within the frame is the light panel frame), the reference sub-image (the reference sub-image is a sub-image of the initial light panel frame area obtained from the current frame), the current frame, and the user configuration. Of course, the initial configuration process can also store other information; there are no restrictions on this.

[0082] Second, the signal light status detection process, such as detecting the light panel and signal lights in the bitstream.

[0083] For example, during traffic light status detection, the position of the light panel and the position of the lit light can be detected, and multi-scale fusion analysis can be performed. For instance, due to detection problems caused by actual connected electronic police cameras such as imaging anomalies, abnormal arrow formation, reflections, and close spacing between light panels, the following method can be used to accurately determine the traffic light status of each light panel to be detected. Imaging anomalies refer to abnormal video image formation; the color imaging of electronic police cameras varies greatly, with some images showing severe mosaic, some displaying snowflake patterns, some exhibiting color casts, and some being blurry. Abnormal arrow formation means that the traffic light occupies a small area in the image, and its shape may be blurry, appearing only as a patch of color at the light's location. Reflections refer to situations where the traffic light reflects sunlight under specific lighting angles, forming white or yellow light spots. Close spacing between light panels may lead to unclear light panel boundaries, resulting in incorrect light panel detection and mismatching.

[0084] See Figure 4 The diagram shown illustrates the flow chart of the traffic light status detection process, which may include:

[0085] Step 401: Perform image preprocessing on the image to be detected.

[0086] For example, the bitstream can be decoded to obtain the image to be detected (i.e., the image to be detected of the target scene, and the image to be detected includes the frame of the light panel to be detected), and the image to be detected can be preprocessed (i.e., preprocessing). For instance, a white balance method can be used to preprocess the image to be detected, thereby using white balance to correct the color cast of the image and obtain the corrected image to be detected.

[0087] Step 402: Input the image to be detected into the trained super-resolution model (to improve image resolution). The super-resolution model processes the image to be detected to obtain a high-resolution image.

[0088] For example, a super-resolution model is used to convert an image at a first resolution into an image at a second resolution, where the second resolution is higher than the first, thereby improving the overall image resolution. Therefore, a super-resolution model can be used to process the image to be detected, resulting in a high-definition image and removing interference such as snow and blur. Reflections have strong texture features; detecting reflections can eliminate their interference.

[0089] Step 403: Perform video quality inspection on the image to be inspected.

[0090] For example, video quality analysis algorithms can be used to analyze whether the image to be detected has stripes, light panel occlusion, image freezing, or other factors that may affect detection. If any anomalies are found, the anomaly information is reported directly, and no further detection is performed. If no anomalies are found, step 404 is executed.

[0091] Step 404: Correct the image offset of the image to be detected.

[0092] For example, a first sub-image can be obtained from the image to be detected based on an initial lamp frame region, and a reference illuminated area of ​​the lamp frame to be detected can be determined based on the first sub-image. Multiple candidate offset detection results can be obtained, and for each candidate offset detection result, the initial lamp frame region is corrected to obtain a corrected lamp frame region. Based on the intersection area between the reference illuminated area and each corrected lamp frame region, the best offset detection result is selected from the multiple candidate offset detection results. The initial lamp frame region is offset based on the best offset detection result to obtain the target lamp frame region. A second sub-image corresponding to the target lamp frame region is obtained from the image to be detected, thereby achieving image offset correction.

[0093] For details on the implementation process of image offset correction, please refer to the following embodiments, which will not be repeated here.

[0094] Step 405: Detect the status of the traffic lights based on the image after image offset correction.

[0095] After image offset correction, the image can be a second sub-image, and the signal light status of the target light panel frame can be detected based on the second sub-image. The signal light status detection process may involve matching the detected signal light result to the light panel frame and determining the current state of the signal light, as described in subsequent embodiments.

[0096] Third, image offset correction.

[0097] Because cameras generally experience shaking, and the camera screws may slightly shift due to alternating wind resistance, and given that the distance between the camera and the traffic light is 50-100 meters, even slight angular changes can cause the traffic light in the image to deviate from its initial frame. In strong winds or when large vehicles pass by, the pole supporting the camera may sway, resulting in a shift of 1-2 traffic light sizes in the image. For these reasons, directly detecting the traffic light's status may lead to false matches. To address these findings, this embodiment designs an offset detection method that can effectively detect image offset, reducing false positives and false negatives caused by image shake.

[0098] During the image offset correction process, multiple candidate offset detection results can be obtained. These candidate offset detection results include at least two of the following: lamp offset detection result, image offset detection result, the best offset detection result of the previous frame of the image to be detected, and a fixed value offset detection result. The best offset detection result can be selected from all candidate offset detection results, and image offset correction can be performed based on the best offset detection result.

[0099] See Figure 5 The diagram illustrates the process of image offset detection, using multiple candidate offset detection results, including the lamp panel offset detection result, the image offset detection result, the best offset detection result of the previous frame's image to be detected, and the fixed-value offset detection result, as an example. Lamp panels can be detected in the configured nearby area (i.e., the initial lamp panel frame area). Offset detection is achieved based on the lamp panel offset detection algorithm, yielding the current lamp panel offset detection calculation result, which can be referred to as the lamp panel offset detection result. An optimized phase correlation method can be used to achieve image offset detection, yielding the current phase correlation method image offset detection calculation result, which can also be referred to as the image offset detection result.

[0100] The best offset detection result of the previous frame of the image to be detected (i.e., the previous offset detection result) refers to the best offset detection result selected from multiple candidate offset detection results corresponding to the previous frame of the image to be detected when processing the previous frame. A fixed-value offset detection result refers to using a fixed value as the offset detection result, such as (0,0), (0,1), (1,0), etc. Taking the fixed value (0,0) as the offset detection result as an example, the fixed-value offset detection result can also be called a non-offset detection result.

[0101] Traffic light detection can also be performed during the image offset correction process. Traffic light detection refers to: based on the initial light panel frame area, obtaining a first sub-image from the image to be detected, determining the reference lighting area of ​​the light panel frame to be detected based on the first sub-image, and the reference lighting area is the traffic light detection result.

[0102] During the image offset correction process, the reliability of each offset detection algorithm can be calculated. For example, based on the lamp offset detection result, the initial lamp frame region is corrected to obtain the corrected lamp frame region 1. The reliability of the offset detection algorithm for the lamp offset detection result is determined based on the intersection area between the reference lit area and the corrected lamp frame region 1. Based on the image offset detection result, the initial lamp frame region is corrected to obtain the corrected lamp frame region 2. The reliability of the offset detection algorithm for the image offset detection result is determined based on the intersection area between the reference lit area and the corrected lamp frame region 2. Based on the best offset detection result of the previous frame image to be detected, the initial lamp frame region is corrected to obtain the corrected lamp frame region 3. The reliability of the offset detection algorithm for the best offset detection result of the previous frame image to be detected is determined based on the intersection area between the reference lit area and the corrected lamp frame region 3. Based on the fixed value offset detection result, the initial lamp frame region is corrected to obtain the corrected lamp frame region 4. The reliability of the offset detection algorithm for the fixed value offset detection result is determined based on the intersection area between the reference lit area and the corrected lamp frame region 4.

[0103] Based on the confidence level of each offset detection algorithm, the offset detection result with the highest offset detection algorithm confidence level can be taken as the best offset detection result, which is the best offset value.

[0104] Fourth, the results of the lamp panel offset detection.

[0105] During the image offset correction process, the light panel offset detection results can be obtained. The light panel detection algorithm process can be found in [link to relevant documentation]. Figure 6 As shown, the lamp panel offset detection results are obtained using the following steps:

[0106] Step S11: Locate the light panel in the lower left and upper right corners of the user configuration.

[0107] For example, during the initial configuration process, an initial light panel frame area can be stored. Therefore, the user-configured lower left and upper right light panels can be searched for; that is, the lower left and upper right light panel frame areas can be searched within the user-configured initial light panel frame area. The left-right direction is prioritized, followed by the up-down direction. That is, the search is first conducted in the left-right direction, and then in the up-down direction. As long as the user has configured an initial light panel frame area, the lower left and upper right light panel frame areas will exist.

[0108] Step S12: Detect the light panel in the lower left outward expansion area. If the lower left and upper right are not the same light panel, then detect the light panel in the upper right outward expansion area. For example, detect the light panel by expanding a certain percentage outward from the lower left corner light panel frame area (center of the lower left corner light panel) (e.g., a magnification between 0.3x and 2x). Detect the light panel by expanding a certain percentage outward from the upper right corner light panel frame area (center of the upper right corner light panel).

[0109] For example, the area of ​​the bottom left light panel frame can be expanded outward by a preset ratio (taking a preset ratio of 2 times as an example, when expanded by 2 times, it means that the expanded area is twice the size of the bottom left light panel frame area). Based on the expanded light panel frame area (i.e., the bottom left light panel frame area + the expanded area), a third sub-image is obtained from the image to be detected. The light panel is detected in the third sub-image to obtain the first bottom left light panel frame area.

[0110] For example, if the upper right and lower left are the same light panel, it means there is only one initial light panel frame region. If the upper right and lower left are not the same light panel, it means there are at least two initial light panel frame regions, i.e., there is an upper right corner light panel frame region. The upper right corner light panel frame region can be expanded outward by a preset ratio. Based on the expanded light panel frame region (i.e., the upper right corner light panel frame region + the expanded region), a third sub-image is obtained from the image to be detected. The light panel is detected within the third sub-image to obtain the first upper right corner light panel frame region.

[0111] Step S13: After detecting the light panels in the lower left outer area, find the lower leftmost light panel in the results, and ensure that the difference in width and height between it and the configured light panel is less than a threshold. After detecting the light panels in the upper right outer area, find the upper rightmost light panel in the results, and ensure that the difference in width and height between it and the configured light panel is less than a threshold.

[0112] For example, when detecting the light panel in the lower left outer region to obtain a first lower left corner light panel frame area (which can be multiple), it is determined whether there is a first lower left corner light panel frame area that meets the first preset condition (i.e., searching for the lower leftmost light panel in the results). For instance, if the ratio of the width difference between the first lower left corner light panel frame area and the lower leftmost light panel frame area (the initial light panel frame area configured by the user includes the lower leftmost light panel frame area) is less than a threshold, and the ratio of the height difference is less than a threshold, then the first lower left corner light panel frame area meets the first preset condition; otherwise, the first lower left corner light panel frame area does not meet the first preset condition.

[0113] For example, when detecting the light panel in the upper right outward expansion area to obtain a first upper right corner light panel frame area (which can be multiple), it is determined whether there is a first upper right corner light panel frame area that meets the second preset condition (i.e., searching for the upper rightmost light panel in the results). For instance, if the width difference ratio between the first upper right corner light panel frame area and the upper rightmost light panel frame area (the user-configured initial light panel frame area includes the upper rightmost light panel frame area) is less than a threshold, and the height difference ratio is less than a threshold, then the first upper right corner light panel frame area meets the second preset condition; otherwise, the first upper right corner light panel frame area does not meet the second preset condition.

[0114] Step S14: Calculate the offset value of the bottom left corner light panel. For example, calculate the first offset value between the center position of the first bottom left corner light panel frame area and the center position of the bottom left corner light panel frame area.

[0115] Step S15: Calculate the offset value of the top right corner light panel. For example, calculate the second offset value between the center position of the first top right corner light panel frame area and the center position of the top right corner light panel frame area.

[0116] Step S16: If both detections are successful and the deviation between the two results is less than the threshold, then take the average value and output the result.

[0117] For example, if the lower left corner of the first light panel frame meets the first preset condition and the upper right corner of the first light panel frame meets the second preset condition, then both are successfully detected. Furthermore, if the difference between the first offset value and the second offset value is less than a threshold, then the light panel offset detection result is obtained based on the first and second offset values. For example, the light panel offset detection result can be the average of the first and second offset values.

[0118] Step S17: If the lower left detection is successful, then take the lower left offset result.

[0119] For example, if the lower left corner of the first light panel frame meets the first preset condition, but the upper right corner of the first light panel frame does not meet the second preset condition, then the first offset value is used as the light panel offset detection result.

[0120] Step S18: If the upper right detection is successful, then take the upper right offset result.

[0121] For example, if the first lower left corner light panel frame area does not meet the first preset condition, but the first upper right corner light panel frame area meets the second preset condition, then the second offset value is used as the light panel offset detection result.

[0122] Step S19: If the bottom left detection fails or the top right detection fails, output "Matching Failed".

[0123] Fifth, image offset detection results.

[0124] During the image offset correction process, image offset detection results can be obtained, that is, image offset detection results can be obtained using an optimized phase correlation method. For example, the image offset detection results can be obtained using the following steps:

[0125] Step S21: Assume the configured alignment region image is srcImgOri, and the current alignment region image is srcImgNow. srcImgOri and srcImgNow have the same size and shape, and both are YUV images. The three-channel YUV image of srcImgOri is used for calculation with only the Y component retained, and the three-channel YUV image of srcImgNow is used for calculation with only the Y component retained.

[0126] For example, during the initial configuration process, an initial lamp panel frame region and a reference sub-image (i.e., a sub-image used to obtain the initial lamp panel frame region) are already stored. The reference sub-image can be used as the comparison region image, srcImgOri. Furthermore, after obtaining the image to be detected, a first sub-image can be obtained from the image to be detected based on the initial lamp panel frame region. This first sub-image can be used as the comparison region image, srcImgNow.

[0127] Step S22: Filter srcImgNow (first sub-image) to obtain the filtered first sub-image, and filter srcImgOri (reference sub-image) to obtain the filtered reference sub-image.

[0128] For example, a 3x3 Gaussian low-pass filter can be used to filter srcImgOri and srcImgNow. The Gaussian low-pass filter removes noise while maintaining the smoothness of the image region, without introducing too much edge distortion, which is especially important for displacement estimation. Furthermore, the Gaussian low-pass filter has a Gaussian shape in the frequency domain, effectively suppressing high-frequency noise while preserving low-frequency displacement information, which is helpful for removing camera-introduced noise in srcImgOri and srcImgNow.

[0129] Step S23: Pad the first sub-image after filtering with zeros so that the width and height are powers of 2, to obtain the first sub-image srcImgNow_Pad with zeros; pad the reference sub-image after filtering with zeros so that the width and height are powers of 2, to obtain the reference sub-image srcImgOri_Pad with zeros.

[0130] For example, we can take the filtered images srcImgOri and srcImgNow as the center, and pad them with zeros to make the image width and height satisfy powers of 2, thus obtaining srcImgOri_Pad and srcImgNow_Pad.

[0131] Step S24: Multiply the preset window with the first sub-image after zero padding to obtain the first sub-image after window operation; multiply the preset window with the reference sub-image after zero padding to obtain the reference sub-image after window operation.

[0132] For example, a Hamming window win with the same size as srcImgOri_Pad can be generated, i.e., a preset window. Multiplying the preset window by the first sub-image srcImgNow_Pad after zero-padding yields the first sub-image srcImgNow_Wined after windowing, i.e., srcImgNow_Wined = srcImgNow_Pad * win. Using this method can suppress the boundary effects generated during Fourier transform truncation and reduce spectral leakage.

[0133] In addition, the preset window is multiplied by the zero-padding reference sub-image srcImgOri_Pad to obtain the window-operated reference sub-image srcImgOri_Wined, that is, srcImgOri_Wined = srcImgOri_Pad * win.

[0134] Step S25: Perform a Discrete Fourier Transform on the first sub-image after the windowing operation and obtain the real-valued part to obtain the first frequency domain feature corresponding to the first sub-image. Perform a Discrete Fourier Transform on the reference sub-image after the windowing operation and obtain the real-valued part to obtain the second frequency domain feature corresponding to the reference sub-image.

[0135] For example, a Discrete Fourier Transform can be applied to the first sub-image srcImgNow_Wined after windowing operations to obtain the real part result FFT_Now, which can be used as the first frequency domain feature.

[0136] For example, a Discrete Fourier Transform can be applied to the reference sub-image srcImgOri_Wined after windowing operations to obtain the real part result FFT_Ori, which serves as the second frequency domain feature.

[0137] Step S26: Determine the power spectrum based on the first frequency domain characteristics and the second frequency domain characteristics.

[0138] For example, based on the first frequency domain feature FFT_Now and the second frequency domain feature FFT_Ori, the power spectrum R can be determined using the following formula. Of course, the following formula is just an example and is not intended to be limiting.

[0139]

[0140] Step S27: Determine the characteristic matrix based on the power spectrum.

[0141] For example, an inverse Fourier transform can be performed on the power spectrum R to obtain the r matrix. The inverse Fourier transform can be implemented using the following formula: r = DFT -1 (R). Then, the r matrix is ​​divided into four sub-matrices with a width of / 2x and a height of / 2y along the center row and center column. The positions of the top-left and bottom-right matrix corners are then swapped, along with the top-right and bottom-left matrix corners, to obtain the characteristic matrix. This operation moves the zero-frequency component to the center of the spectrum, facilitating subsequent calculations. Of course, the above is just an example of obtaining the characteristic matrix; other operations can be performed on the r matrix to obtain the characteristic matrix, or the r matrix can be used as the characteristic matrix.

[0142] Step S28: Determine the point where the maximum value is located in the feature matrix as the location of the maximum correlation.

[0143] For example, the point where the maximum value (feature point) is located can be found from the feature matrix (i.e. the matrix obtained by inverse transformation). This point represents the location of the maximum correlation between two images in the spatial domain.

[0144] Step S29: Using the location with the highest correlation as the center, obtain a sub-region within a preset range from the first sub-image, determine the weighted centroid of the sub-region, and obtain the image offset detection result based on the weighted centroid.

[0145] For example, in traffic scenarios, additional noise interference, such as moving cars on open ground or swaying leaves, can interfere with the phase correlation algorithm and even cause significant deviations. Therefore, to balance reducing noise impact and increasing estimation accuracy, the weighted centroid is calculated within the range of the maximum point center (3, 3). That is, using the location of maximum correlation as the center, a sub-region within a preset range is obtained from the first sub-image. The preset range can be (3, 3), and the weighted centroid of the sub-region is determined. For example, the weighted centroid of the sub-region can be determined using the following formula, although this formula is just an example.

[0146]

[0147]

[0148] In the above formula, x i and y i These are pixel coordinates, specifically the pixel coordinates w within a range of (3, 3) centered on the location of maximum relevance. i It is the eigenvalue of the pixel coordinate in the feature matrix.

[0149] After obtaining the weighted centroid of the sub-region, the image offset detection result is obtained based on the weighted centroid. For example, the center coordinates (x_c, y_c) of srcImgOri_Pad (the reference sub-image after zero padding) or other images are found, and the offset value x_c-x in the x-direction is obtained. centroid The offset value y_c-y in the y direction centroid The offset values ​​in the x-direction and y-direction together constitute the image offset detection result.

[0150] Sixth, traffic light inspection.

[0151] During the image offset correction process, traffic light detection can be performed to obtain a reference illuminated area. For example, based on the initial light panel frame area, a first sub-image is obtained from the image to be detected. Based on the first sub-image, the reference illuminated area of ​​the light panel frame to be detected is determined, and the reference illuminated area is the traffic light detection result.

[0152] For example, a traffic light detection model can be used to detect traffic lights. That is, the first sub-image can be input into the traffic light detection model, and the traffic light detection model can output a reference lit area.

[0153] For example, during real-time bitstream analysis, the traffic light detection model can be labeled as traffic light on, indicator light, supplementary light, reflector, etc. Traffic light on is not labeled with color or category; all traffic lights, pedestrian lights, and non-motorized vehicle lights are categorized as traffic lights. Indicator lights are all long or rectangular lights displaying one or more lines of text. Supplementary lights are illumination lights that illuminate the environment. Reflectors are areas of the traffic light that are highlighted by natural light shining on the light panel. Based on the above traffic light detection model, the model can output only the reference illuminated area, indicating that this reference illuminated area is lit.

[0154] For example, the traffic light detection results also have a confidence threshold. Traffic light detection results below the confidence threshold will be filtered out, while traffic light detection results above the confidence threshold will be output.

[0155] Using the above method, the traffic light detection model does not need to detect the color of the traffic lights. Therefore, it eliminates the problem of missed detection and false detection in certain cases of abnormal traffic light color imaging.

[0156] Seventh, calculate the reliability of the offset results.

[0157] During the image offset correction process, the reliability of each offset detection algorithm can be calculated. See also Figure 7 The diagram shown illustrates the process for calculating the reliability of the offset results. This process may include:

[0158] Step S31: Obtain the calculation result of the lamp panel offset detection. Based on the calculation result, obtain the corrected lamp panel frame position. The calculation result of the lamp panel offset detection is the lamp panel offset detection result. Based on the lamp panel offset detection result, the initial lamp panel frame area is corrected to obtain the corrected lamp panel frame area 1.

[0159] Step S32: Obtain the image offset detection calculation result of this phase correlation method, and obtain the corrected lamp panel frame position based on the image offset detection calculation result of this phase correlation method. For example, the image offset detection calculation result of this phase correlation method can be the image offset detection result. Therefore, the initial lamp panel frame region can be corrected based on the image offset detection result to obtain the corrected lamp panel frame region 2.

[0160] Step S33: Obtain the previous offset detection result, and obtain the corrected lamp frame position based on the previous offset detection result. The previous offset detection result is the best offset detection result of the previous frame of the image to be detected. Based on this best offset detection result, the initial lamp frame region is corrected to obtain the corrected lamp frame region 3.

[0161] Step S34: Obtain the non-offset detection result, and obtain the corrected lamp frame position based on the non-offset detection result. For example, if the non-offset detection result is a fixed-value offset detection result, the initial lamp frame area is corrected based on the fixed-value offset detection result to obtain the corrected lamp frame area 4. For example, if the fixed-value offset detection result is 0, the corrected lamp frame area 4 is the same as the initial lamp frame area.

[0162] Step S35: Obtain the traffic light detection result. For example, the first sub-image can be input into the traffic light detection model, and the traffic light detection model can output the reference lit area (i.e., the traffic light detection result).

[0163] Step S36: Accumulate the product of the maximum intersection area between each detected traffic light and the corrected light panel detection frame and the traffic light confidence level, and determine the confidence level of the offset result based on this product value.

[0164] For example, the results of the light panel offset detection, image offset detection, the previous offset detection, and no offset detection are compared with the traffic light detection results. For each traffic light detection result, the process iterates through the data to determine whether the traffic light is on the offset-corrected light panel. The area of ​​the intersection between each detected traffic light and the corrected light panel detection frame is calculated. The detection confidence of the light is then combined with the intersection area using a certain ratio and mathematical relationship to obtain the confidence level of the offset scheme.

[0165] For example, the reliability of the light panel offset detection result is determined based on the intersection area between the reference illuminated area and the corrected light panel frame area 1. During the reliability determination process, if multiple reference illuminated areas are obtained, for each reference illuminated area, a weight value can be determined based on the intersection area between the reference illuminated area and the corrected light panel frame area 1, and the confidence level of the reference illuminated area. This confidence level represents the probability that the reference illuminated area is illuminated (output by the traffic light detection model). Then, the reliability of the corrected light panel frame area 1 can be determined based on the weight value of each reference illuminated area.

[0166] For example, the confidence level of the corrected lamp panel frame region 1, i.e., the confidence level of the lamp panel offset detection result, can be determined using the following formula: (Intersection area of ​​traffic light i and light panel j × confidence level of traffic light i). In the above formula, the confidence level is the sum of the product of the intersection area of ​​each light and the light panel and the confidence level of the traffic light.

[0167] For example, the confidence level of the image offset detection result can be determined based on the intersection area between the reference illuminated area and the corrected lamp disk frame area 2. In the confidence level determination process, if multiple reference illuminated areas are obtained, for each reference illuminated area, a weight value can be determined based on the intersection area between the reference illuminated area and the corrected lamp disk frame area 2, and the confidence level of the reference illuminated area. Then, the confidence level of the corrected lamp disk frame area 2 can be determined based on the weight value of each reference illuminated area.

[0168] For example, the reliability of the optimal offset detection result for the previous frame image is determined based on the intersection area between the reference illuminated area and the corrected lamp disk frame area 3. The reliability of the fixed-value offset detection result is determined based on the intersection area between the reference illuminated area and the corrected lamp disk frame area 4.

[0169] Step S37: Based on the confidence level of each offset detection result, the offset detection result corresponding to the highest confidence level can be taken as the best offset detection result, which is the best offset value.

[0170] After obtaining the optimal offset detection result, the initial light panel frame region can be offset based on the optimal offset detection result to obtain the target light panel frame region; that is, the initial light panel frame region is offset using the optimal offset value. Based on this, a second sub-image corresponding to the target light panel frame region can be obtained from the image to be detected. Then, the traffic light status can be detected based on the second sub-image.

[0171] Eighth, the traffic lights match the light frames.

[0172] During traffic light status detection, the status of the traffic light within the target light panel frame can be detected based on the second sub-image. This process involves matching the detected traffic light result to the light panel frame and determining the current status of the traffic light. In the process of matching the detected traffic light result to the light panel frame (i.e., traffic light matching the light panel frame), it is necessary to determine which traffic light the detected status result belongs to.

[0173] For example, the second sub-image can be divided into multiple single-lamp sub-regions, each corresponding to a traffic light. After obtaining the reference illuminated area, the IOU (Intersection over Union) between the reference illuminated area and each single-lamp sub-region can be calculated. The traffic light corresponding to the single-lamp sub-region with the largest IOU is the traffic light corresponding to the reference illuminated area, meaning the detected traffic light matches the light frame.

[0174] To divide the second sub-image into multiple single-lamp sub-regions, the position of the traffic lights can be followed, and their pose can be detected simultaneously. Therefore, it is necessary to model the traffic lights. See [link / reference] Figure 8A The diagram illustrates traffic light following and positioning. The position and attitude of the traffic lights are described using four parameters: the axis equation of the traffic light, the position of the first traffic light, the spacing between the traffic light sub-lights, and the size of the traffic light sub-lights. Initially, each traffic light within the traffic light panel is configured such that the panel frame is divided into N equal parts, where N is the number of traffic lights within the panel for that type of traffic light. See also... Figure 8B The diagram shown illustrates the initial segmentation of the traffic lights. After each detected traffic light illuminates and matches its corresponding frame, the actual position of the traffic light is corrected using the method described below.

[0175] For example, the second sub-image is divided into multiple single-lamp sub-regions, each corresponding to a traffic light. This division may include, but is not limited to, determining the position and orientation of the traffic lights based on their axis equation, the position of the first traffic light, the spacing between traffic lights, and their dimensions. For each traffic light, the corresponding single-lamp sub-region can be divided from the second sub-image based on its position and orientation. For instance, the single-lamp sub-region can be divided from the second sub-image using the following steps:

[0176] Step S41: Determine the first intermediate parameter based on the first slope parameter and the first intercept parameter corresponding to the previous frame of the image to be detected, and the center position coordinates of the first traffic light in the second sub-image.

[0177] For example, for the first frame of the image to be detected, see [link to relevant documentation]. Figure 8B As shown, in the subsequent process, for each frame of the image to be detected, the second slope parameter and the second intercept parameter corresponding to the image to be detected are determined based on the first slope parameter and the first intercept parameter corresponding to the previous frame of the image to be detected.

[0178] For example, the first intermediate parameter e can be determined using the following formula: e = ax i +by i , a represents the first slope parameter, b represents the first intercept parameter, x i y i This indicates the coordinates of the center position of the first traffic light.

[0179] Step S42: Based on the first slope parameter, the configured following rate, the first intermediate parameter, and the center position coordinates of the first traffic light, determine the second slope parameter corresponding to the image to be detected.

[0180] For example, the second slope parameter can be determined using the following formula: The 'a' before the equals sign represents the second slope parameter, and the 'a' after the equals sign represents the first slope parameter. R represents the following rate, which can be configured empirically; for example, it can be set between 0.01 and 0.5. i The vertical coordinate of the center position of the first traffic light is represented by 'e', ​​where 'e' represents the first intermediate parameter.

[0181] Step S43: Based on the first intercept parameter, the configured follow rate, the first intermediate parameter, and the acquired second slope parameter, determine the second intercept parameter corresponding to the image to be detected.

[0182] For example, the second intercept parameter can be determined using the following formula: The b before the equals sign can represent the second intercept parameter, the b after the equals sign can represent the first intercept parameter, R can represent the following rate, e can represent the first intermediate parameter, and a can represent the second slope parameter.

[0183] Step S44: Determine the second intermediate parameter based on the first initial distance and first interval position corresponding to the previous frame of the image to be detected, and the center position coordinates of the first traffic light in the second sub-image. The first initial distance can represent the distance between the image edge and the center of the first traffic light in the second sub-image of the previous frame of the image to be detected, and the first interval position can represent the distance between two adjacent traffic lights.

[0184] For example, the second intermediate parameter can be determined using the following formula: Δ p =pos + i*gap - xi, Δ p The second intermediate parameter is represented by pos, which represents the first initial distance corresponding to the previous frame of the image to be detected, and gap, which represents the first interval position corresponding to the previous frame of the image to be detected. i The x-coordinate of the center position of the first traffic light is represented by , and i represents the i-th traffic light detected so far. The value of i for the first traffic light is 1.

[0185] Step S45: Based on the first initial distance corresponding to the previous frame's image to be detected, the configured following rate, and the second intermediate parameters, determine the second initial distance corresponding to the image to be detected. The second initial distance represents the distance between the image edge and the center of the first traffic light in the second sub-image of the current frame's image to be detected.

[0186] For example, the second initial distance can be determined using the following formula: pos = pos - RΔ p The pos before the equals sign represents the second initial distance corresponding to the image to be detected, and the pos after the equals sign represents the first initial distance corresponding to the previous frame image to be detected. R can represent the following rate, Δ p This indicates the second intermediate parameter.

[0187] Step S46: Based on the first interval position, the following rate, and the second intermediate parameter, determine the second interval position corresponding to the image to be detected. The second interval position represents the distance between two adjacent traffic lights.

[0188] For example, the second interval position can be determined using the following formula: gap = gap - R * i * Δ p The gap before the equals sign represents the second interval position in the image to be detected, and the gap after the equals sign represents the first interval position in the previous frame of the image to be detected. R can represent the following rate, and Δ p This indicates the second intermediate parameter.

[0189] Step S47: Based on the first size value corresponding to the image to be detected in the previous frame, the following rate, and the size of the traffic light in the second sub-image, determine the second size value corresponding to the image to be detected.

[0190] For example, the second size value can be determined using the following formula: size = size - R * (size - s) i The size before the equals sign represents the second size value corresponding to the image to be detected, and the size before the equals sign represents the first size value corresponding to the image to be detected in the previous frame. R can represent the following rate, s i This represents the size of the traffic light in the second sub-image, specifically the size of the currently detected i-th traffic light sub-light, s. i .

[0191] Step S48: After obtaining the above parameters, the axis equation of the traffic light can be determined based on the second slope parameter and the second intercept parameter, the position of the first traffic light can be determined based on the second initial distance, the interval between traffic lights can be determined based on the second interval position, and the size of the traffic light can be determined based on the second size value.

[0192] Since the axis equation of a traffic light, the position of the first traffic light, the interval between traffic lights, and the size of the traffic lights can reflect the position and orientation of the traffic lights, the position and orientation of each traffic light can be determined based on these factors. Furthermore, for each traffic light, a corresponding single-light sub-region can be divided from the second sub-image based on its position and orientation.

[0193] In summary, after each traffic light is detected, before matching the traffic light to its socket, the coordinate frames of the previously followed single lights are first superimposed with the overall offset of the current frame for matching. The matching process is as follows: for each detected traffic light, all traffic light following frames within each light panel are traversed, and the Interchange of Union (IOU) between the traffic light and its following frame is calculated. The signal light with the largest IOU is considered to be matched with that frame.

[0194] Ninth, determine the current status of the traffic lights.

[0195] During traffic light status detection, the status of the traffic light within the target light panel frame can be detected based on the second sub-image. This detection process involves matching the detected traffic light to the target light panel frame and determining the current status of the traffic light. The current status determination requires identifying the traffic light status within the target light panel frame. For example, whether the traffic light is on or off.

[0196] For example, after matching the light frame, the current status (color, shape) of the traffic light can be obtained based on the configuration information of the traffic light, such as the red light for left turn being on, and the red and green lights for going straight being on.

[0197] During the process of determining the current state of the traffic lights, see [link / reference]. Figure 9A The diagram shown is a flowchart illustrating the process of determining the current state of a traffic light. This process may include: matching the traffic light to the light frame, rechecking the color and brightness of the traffic light, and marking the current status of the traffic light group. The following will explain this process.

[0198] Based on the traffic light matching frame results, after dividing the second sub-image into multiple single-light sub-regions, for each single-light sub-region, the first detection result is determined based on the intersection area between the reference lit area and the single-light sub-region. For example, the first detection result of the single-light sub-region corresponding to the largest intersection area indicates that the traffic light corresponding to that single-light sub-region is lit. The first detection result of the single-light sub-region corresponding to a non-maximum intersection area indicates that the traffic light corresponding to that single-light sub-region is off.

[0199] For the re-inspection of traffic light chromaticity and luminance, after dividing the second sub-image into multiple single-light sub-regions, for each single-light sub-region, the second detection result of the single-light sub-region is determined based on the luminance and chromaticity information of the single-light sub-region. The second detection result indicates whether the traffic light corresponding to the single-light sub-region is on or off.

[0200] To mark the current signal light status of the light group, for each individual light sub-area, the target detection result for that sub-area is determined based on the first and second detection results. The target detection result indicates whether the corresponding signal light in that sub-area is on or off. Based on this, the signal light status of the light panel to be detected can be determined according to the color of the signal light in each individual light sub-area and the target detection result for each individual light sub-area; that is, the signal light status includes color and on / off information.

[0201] For the re-inspection of traffic light chromaticity and luminance, in addition to using deep learning networks to detect the on / off state of traffic lights, we can also comprehensively analyze the luminance, chromaticity, saturation, and other parameters of each traffic light position on the light panel, and then re-inspect the results of the deep learning network detection. The re-inspection process may include:

[0202] Step S51: Calculate the average brightness and chromaticity of each signal light frame, that is, for each single light sub-area (taking a single light sub-area as an example), determine the brightness and chromaticity information of that single light sub-area.

[0203] For example, see Figure 9B The diagram shown illustrates the process for determining luminance and chromaticity information.

[0204] Convert YUV to RGB. For example, convert the YUV image corresponding to a single lamp sub-region to the corresponding RGB image, and then determine the brightness information of that single lamp sub-region based on the RGB image.

[0205] Obtain the maximum values ​​of the R, G, and B components for each pixel within the light frame area. For example, generate a luminance analysis image based on the RGB image corresponding to the single light sub-area. The pixel value of each pixel in this luminance analysis image is the maximum value of the R, G, and B components corresponding to that pixel in the RGB image.

[0206] The average value is calculated for the maximum value of each pixel within the region. For example, the average brightness value is determined based on the pixel values ​​of all pixels in the brightness analysis image, which is the average of the pixel values ​​of all pixels.

[0207] At this point, we can obtain the brightness information, which can be the average brightness value (average brightness).

[0208] Convert YUV to HSV. For example, convert the YUV image corresponding to a single lamp sub-region to the HSV image corresponding to that single lamp sub-region, and then determine the brightness information of that single lamp sub-region based on the HSV image.

[0209] Obtain the average value of the S-component of each pixel within the light frame area. For example, based on the HSV image corresponding to this single light sub-region, the S-component of each pixel can be determined, and the average value of the S-component of all pixels can be calculated.

[0210] For each pixel whose S-component is greater than the average component, calculate the mean of the H-component. For example, based on the average component (mean of the S-component), identify candidate pixels in the HSV image whose S-component is greater than the mean of the S-component. For these candidate pixels whose S-component is greater than the mean of the S-component, calculate the mean of the H-component for all candidate pixels.

[0211] At this point, chromaticity information can be obtained, which can be represented by the mean of the H component. In subsequent comprehensive analysis, based on the mean of the H component, the color of the lamp can be determined according to the color recognition range of the H component.

[0212] Step S52: Determine the second detection result of the single lamp sub-region based on the brightness and chromaticity information of the single lamp sub-region. That is, comprehensively analyze the image statistical state of the lamp according to the signal lamp type and statistical information. The image statistical state is the second detection result, which indicates whether the lamp is on or off.

[0213] For example, if the average brightness value is greater than the threshold and the average H component value is within the color range, the second detection result indicates that the signal light is on. Alternatively, if the average brightness value is greater than the threshold and the average H component value is not within the color range, the second detection result indicates that the signal light is off. Alternatively, if the average brightness value is not greater than the threshold, the second detection result indicates that the signal light is off; wherein, the color range is the color range of the signal light corresponding to the single-lamp sub-region.

[0214] For example, see Figure 9C The diagram shown is a flowchart illustrating the calculation of the image state of a traffic light.

[0215] For a three-light signal light, if the average brightness at a certain light position is greater than a threshold and the average H value at that light position is within the color chromaticity range, then that light is on. If the average brightness at a certain light position is greater than the threshold but the average H value at that light position is not within the color chromaticity range, then that light is off. If the average brightness at a certain light position is greater than the threshold but not within the color chromaticity range, then that light is off.

[0216] For non-three-lamp signal lights, if the average brightness at a given position exceeds a threshold and the average H value at that position falls within the color chromaticity range, the light at that position is on. If the average brightness at a given position exceeds the threshold but the average H value at that position does not fall within the color chromaticity range, the light at that position is off. If the average brightness at a given position does not exceed the threshold, the light at that position is off.

[0217] For example, regarding color chromaticity range, in actual imaging, the color chromaticity variation range of the red traffic light is relatively wide. Three-light traffic lights provide a wider judgment threshold, reducing the probability of missed detection. Therefore, different chromaticity matching schemes can be used for three-light and single-light traffic lights.

[0218] For example, for a three-light traffic light: Red light: H values ​​between 275° and 360° and between 0° and 40° are considered red. Yellow light: H values ​​between 20° and 45°. Green light: H values ​​between 40° and 140°. For a single-light traffic light: H values ​​are between 0° and 125° and between 275° and 360°. Red light: H values ​​between 275° and 360° and between 0° and 25° are considered red. Yellow light: H values ​​between 25° and 45°. Green light: H values ​​between 45° and 125°. Other lights: Indicator lights and supplementary lights do not consider chromaticity information; only average brightness that meets threshold requirements is considered.

[0219] To mark the current signal light status of a traffic light group, the signal light status can be comprehensively determined and marked in the traffic light group information, updating the current signal light status. By combining the output results of the signal light detection model with the results of the re-inspection process, the signal light status of each signal light following the light frame is comprehensively determined, that is, the target detection result of a single light sub-region is determined based on the first and second detection results. Since the signal light detection model is trained with reflection categories, it can accurately identify the reflective areas falling on the signal light panel. Therefore, the reflection results can eliminate image-based false detections. At the same time, some reflections are incorrectly classified into the signal light category in the signal light detection model because reflections usually reflect more than one light frame. Therefore, the chromaticity and brightness of multiple light panels can be judged. If there are similar chromaticity and brightness, and the image detection shows the light is off, it can be identified as reflection.

[0220] For example, see Figure 9D The diagram shown is a flowchart illustrating the process of comprehensively determining the status of traffic lights.

[0221] Iterate through each frame of each light panel. For example, for each light panel, you can iterate through each frame of that light panel in turn. Each frame is a single light sub-region. We will take a single light sub-region as an example from now on.

[0222] If the traffic light detection is successful (i.e., the first detection result indicates that the traffic light is on), and the image detection is successful (i.e., the second detection result indicates that the traffic light is on), then the light at that location is on. In other words, the target detection result indicates that the traffic light is on, meaning the target detection result indicates that the traffic light in the single-light sub-region is on.

[0223] Alternatively, if the signal light detection is successful but the image detection is unsuccessful (i.e., the second detection result indicates that the signal light is off), and the detection light panel frame is a single light frame (i.e., a single light is detected), then the light at that position is on, meaning the target detection result indicates that the signal light in the single light sub-region is on.

[0224] Alternatively, if the signal light detection is successful but the image detection is unsuccessful, the detection light panel frame is not a single light frame (single light is not successful), and there are multiple lights with similar chromaticity and brightness (at least two signal lights with similar brightness and / or similar chromaticity), then it is determined to be reflective, and the light at that position is off. That is, the target detection result indicates that the signal light in the single light sub-area is off, and the signal light in the single light sub-area is reflective.

[0225] Alternatively, if the signal light detection is successful but the image detection is unsuccessful, the light panel frame to be detected is not a single light frame (single light is not successful), and there are multiple lights with similar chromaticity and brightness (there are no at least two signal lights with similar brightness and similar chromaticity), then the light at that position is on, and the target detection result indicates that the light is on.

[0226] Alternatively, if the signal light detection fails (i.e., the first detection result indicates that the signal light is off), and the image detection fails to confirm that the signal light is on, then the light at that location is off, meaning the target detection result indicates that the signal light is off.

[0227] Alternatively, if the signal light detection fails to detect the light being on, but the image detection confirms the light is on, and the reflection detection result indicates that the reflection is confirmed, then the light at that location is off; that is, the target detection result indicates that the signal light is off.

[0228] Alternatively, if the signal light detection fails to detect the light being on, but the image detection confirms the light is on, and the reflection detection result indicates that the reflection is not on, then the light at that location is on; that is, the target detection result indicates that the signal light is on.

[0229] By marking the current state of the traffic lights, and using a cross-validation method combining deep learning and image processing, the accuracy of traffic light detection has been improved, reducing false detections caused by relying on a single method.

[0230] Tenth, judging abnormal status of traffic lights.

[0231] After detecting the traffic light status of the target light panel frame based on the second sub-image (i.e., the traffic light status detection result of each frame of the target image), it is also possible to analyze whether there are traffic light abnormal events in the target scene based on the traffic light status detection results of multiple frames of the target image. Traffic light abnormal events include, but are not limited to, at least one of the following: image abnormal events, single light abnormal events, light group abnormal events, and intersection abnormal events.

[0232] For example, in addition to detecting the status of traffic light panels, traffic lights, and reflections based on single-frame images, a traffic light status recognition method based on time series analysis is adopted. That is, a traffic light status recognition method based on time series analysis is designed to accurately identify the status of traffic lights of different colors such as red, green, and yellow.

[0233] For example, single-frame recognition might suffer from localized blurring or pixelation due to network fluctuations, or traffic lights might be temporarily obscured by raindrops or insects in front of the camera. Using single-frame results as the basis for alarms is insufficient for abnormal state detection in this scenario and is prone to false positives. Therefore, this embodiment proposes a traffic light state recognition method based on time series analysis. Furthermore, if intersection light groups are configured during initial setup, the abnormality of the traffic light's operating state can be determined based on natural logic.

[0234] For example, during the analysis, the abnormal state of the traffic lights can be judged according to four steps and dimensions: abnormal event analysis of the screen, single light event analysis, light group event analysis, and intersection event analysis.

[0235] Based on the traffic light detection results, determine if any abnormal conditions exist in the image. For example, if an abnormal condition exists (e.g., image obstruction, excessively blurry image, image stripes, no signal, image freeze, etc.), the time accumulation is performed. When the accumulated time exceeds the alarm threshold, an image abnormality event alarm is generated. If the image recovers, the time accumulation is reset.

[0236] Under normal analysis conditions, the state (on, off) time of each traffic light is statistically analyzed (accumulated). Each light's state change has a configurable transition time (e.g., in the case of three traffic lights, the transition time of the red light is set to 1 second. When the red light is off, the off state is accumulated for more than 1 second before it is considered that the red light is off. At this time, the on state time is cleared. The transition time can prevent accidental events from disrupting the abnormal judgment logic).

[0237] Determine if any alarm events exist within the traffic light group. Analyze the status of each light in each group (e.g., a three-light straight-ahead light group consists of a red straight-ahead light, a green straight-ahead light, and a yellow straight-ahead light) to identify any abnormal events (e.g., the red and green straight-ahead lights are on simultaneously, or the red straight-ahead light is off for an extended period). For each type of abnormal event, a timer is used. When the accumulated time of the abnormal state exceeds a set threshold, an alarm for an abnormal traffic light event is generated. When the abnormal state is resolved within the threshold, the timer for that event is reset.

[0238] Intersection event analysis. Based on the light groups assigned to each location at the intersection during user configuration, the system determines whether any abnormal events exist at the intersection (e.g., if all lights at the intersection are off, a power outage event is considered; if the green lights for straight traffic on the south and east sides are on simultaneously, a traffic light state conflict event is considered; no restrictions are imposed on this type of abnormal event). Each type of abnormal event has a time counter. When the accumulated time of the abnormal state exceeds a set threshold, an intersection traffic light abnormal event alarm is generated. When the abnormal state is resolved within the threshold, the time counter for that event is reset.

[0239] When there are no abnormal events at the intersection and the time accumulated by the time counter exceeds the set threshold under normal conditions, a normal event is generated, meaning that no alarm for abnormal events needs to be generated.

[0240] For example, for event capture, the analysis frame rate (frequency) can be more than twice the expected frequency of the event. For instance, if a yellow light flashes once every 1 second, the analysis frequency should be higher than 2 frames per second.

[0241] The above analysis methods can reduce sporadic anomalies caused by single-frame traffic light detection. A comprehensive assessment of intersection conditions is conducted, analyzing traffic light anomalies from the perspective of traffic participants. Factors affecting detection (network anomalies, dirty lenses, foliage obstructing the lens, etc.) are identified promptly, providing a basis for user equipment maintenance.

[0242] The following table 1 provides a detailed explanation of several common types of abnormal events, without imposing any restrictions.

[0243] Table 1

[0244]

[0245] Eleventh, traffic light polling scheme.

[0246] For example, abnormal events are categorized into short-polling abnormal events and long-polling abnormal events, with the polling duration of long-polling abnormal events being longer than that of short-polling abnormal events. Based on this, if the interval between the current time and the detection time of a short-polling abnormal event reaches the polling duration of the short-polling abnormal event, then based on the traffic light status detection results of multiple frames of the images to be detected, it is analyzed whether there is a traffic light abnormal event in the target scene corresponding to the short-polling abnormal event; conversely, if the interval between the current time and the detection time of a long-polling abnormal event reaches the polling duration of the long-polling abnormal event, then based on the traffic light status detection results of multiple frames of the images to be detected, it is analyzed whether there is a traffic light abnormal event in the target scene corresponding to the long-polling abnormal event.

[0247] For example, in the process of analyzing abnormal events, real-time analysis can be performed when the number of streams is small, while polling logic can be used when the number of streams is large to check for abnormal events. For instance, graphics cards or other hardware supporting deep learning can be added, with each graphics card or equivalent hardware supporting the same number of streams (e.g., a single piece of hardware can support 200 video streams, and four pieces of hardware can support 800 streetlights). Based on this, the working principle of the abnormal event analysis process is as follows:

[0248] 1. Analyze the reasoning ability of the signal light detection model of the analysis equipment. For example, if a graphics card is inserted, the graphics card can run 16 signal light detection models at the same time.

[0249] 2. If a user connects to fewer than 16 video streams, perform full frame rate analysis on all 16 streams. If more than 16 video streams connect, perform polling analysis. During polling analysis, reduce the analysis frame rate to at least the necessary analysis frame rate (the analysis frame rate (frequency) must be greater than twice the expected frequency of the event).

[0250] 3. Analyze the video streams for each intersection together.

[0251] 4. During polling analysis, restore the state of the channel from the hard drive as previously analyzed. If abnormal events exist, determine if the abnormal events have been mitigated. If no abnormal events exist, continue analyzing whether the events are normal. After analysis is complete, save the current state to the hard drive and connect to the next channel for analysis.

[0252] 5. Each channel is divided into long polling and short polling. Short polling is used to detect anomalies that can be detected within a short time, such as screen abnormalities, lights going out, simultaneous traffic lights on, flashing yellow lights, indicator lights going out, auxiliary lights going out, traffic lights tilting, power outages at intersections, and traffic light state conflicts. The short polling time is relatively short, determined by the maximum time threshold configured by the user for these events, usually around 10 seconds. A long poll is performed after every 10 short polls, with each long poll lasting 3 minutes of analysis. In addition to detecting the short polling time, it can also detect anomalies that require longer statistical time, such as a single traffic light remaining constantly on or a single traffic light remaining constantly off.

[0253] For example, suppose the detection performance is 200 channels / GPU, but at the same time, a single GPU can only support decoding 16 channels of video, which is far from meeting the performance requirements. From the perspective of traffic light fault detection needs, as long as the fault exists, it can be detected in a short time. There is no need to conduct detection for a long time. Most defects can be found by analyzing for a period of time, and a few faults can be found by conducting more detection cycles.

[0254] Therefore, for polling analysis of more than 16 channels, such as when accessing 32 bitstreams, an example diagram of polling a single algorithm channel can be found in [reference needed]. Figure 10 As shown. For a single algorithm channel, it supports switching between different external channels for processing. Each external channel analyzes for a period of time and then stops analyzing, and then switches to the next external channel for analysis. After a complete polling cycle, all external channels have been processed.

[0255] As can be seen from the above technical solutions, the offset detection method designed in this application effectively detects image offset, reduces false detections and missed detections caused by image jitter, and can solve problems such as false detections caused by the shaking of the camera pole and false alarms caused by obstructions in the image, thereby improving accuracy and reducing false detections. It can promptly detect abnormal traffic light states, improve the operational efficiency of the maintenance department, and reduce the possibility of traffic accidents. It can monitor and alarm the operational status of traffic lights at each intersection at a higher level. A traffic light state recognition method based on time series analysis is designed to accurately identify the states of different colors of traffic lights such as red, green, and yellow. An offset detection method is adopted to effectively solve the jitter problem in traffic light detection. It can detect traffic lights that are off, have insufficient brightness, are damaged, have opposite logic traffic lights on simultaneously, have yellow flashing traffic lights, have a traffic light that is off for a long time, have a traffic light that is on for a long time, have a traffic light group obstructed, have a traffic light offset, have a traffic light tilted, have a traffic light power outage (all traffic lights at the intersection are not lit), have traffic light conflicts at the intersection, and have abnormal events related to normal traffic light operation. It can detect the status of individual lights, light groups, and traffic light logic at intersections. It can analyze and process interference in the image, reducing false alarms not caused by traffic lights. It effectively solves common problems encountered in applications such as glare. It effectively addresses the blurriness caused by small light panels in large-screen scenes. It can handle color cast anomalies. It supports high-volume video analysis and dynamic expansion, reducing equipment costs.

[0256] Based on the same concept as the above-described method, this application proposes a traffic light status detection method. When it is determined that an abnormal traffic light event exists in the target scene based on the image to be detected of the target scene, the image to be detected can be displayed, and a target rectangle can be superimposed on the image to be detected. The target rectangle is determined on the image to be detected based on the target light panel frame region, and the target light panel frame region can be obtained by offsetting the configured initial light panel frame region based on the optimal offset detection result.

[0257] For example, after displaying the image to be detected, an initial rectangular frame can also be overlaid on the image to be detected. The initial rectangular frame is determined on the image to be detected based on the initial light panel frame area.

[0258] The implementation method for "determining the existence of traffic light abnormal events based on the image to be detected" can be found in the "Traffic Light Abnormal State Judgment" process, and will not be repeated here. The implementation method for "offsetting the initial light panel frame region based on the optimal offset detection result to obtain the target light panel frame region, and determining the target rectangular box on the image to be detected based on the target light panel frame region" can be found in the "Initial Configuration" process for the initial light panel frame region, and in the "Image Offset Correction" process for the optimal offset detection result, and will not be repeated here.

[0259] Based on the same concept as the above method, this application proposes a traffic light status detection device, see [link]. Figure 11 The diagram shown is a structural schematic of the device, which may include:

[0260] The acquisition module 1101 is used to acquire a target scene image to be detected, the image to be detected including a light panel frame to be detected; based on a configured initial light panel frame region, a first sub-image is acquired from the image to be detected, and a reference lighting region of the light panel frame to be detected is determined based on the first sub-image; the processing module 1102 is used to select a candidate offset detection result as the best offset detection result from multiple candidate offset detection results based on the reference lighting region and the initial light panel frame region; offset the initial light panel frame region based on the best offset detection result to obtain a target light panel frame region, and acquire a second sub-image corresponding to the target light panel frame region from the image to be detected; the detection module 1103 is used to detect the signal light status of the light panel frame to be detected based on the second sub-image.

[0261] For example, when the processing module 1102 selects a candidate offset detection result as the best offset detection result from multiple candidate offset detection results based on the reference lighting area and the initial lamp frame area, it specifically performs the following steps: obtaining multiple candidate offset detection results; for each candidate offset detection result, correcting the initial lamp frame area based on the candidate offset detection result to obtain a corrected lamp frame area; and selecting the best offset detection result from the multiple candidate offset detection results based on the intersection area between the reference lighting area and each corrected lamp frame area.

[0262] For example, the plurality of candidate offset detection results include at least two of the following: lamp panel offset detection result, image offset detection result, best offset detection result of the previous frame image to be detected, and fixed value offset detection result; if the plurality of candidate offset detection results include the lamp panel offset detection result, the processing module 1102 specifically performs the following when obtaining the lamp panel offset detection result: expands the initial lamp panel frame region outward by a preset ratio, obtains a third sub-image from the image to be detected based on the expanded lamp panel frame region, detects the first lamp panel frame region based on the third sub-image, and obtains the lamp panel offset detection result based on the offset value between the center position of the first lamp panel frame region and the center position of the initial lamp panel frame region.

[0263] For example, the initial lamp panel frame area includes a bottom left lamp panel frame area and a top right lamp panel frame area, and the first lamp panel frame area includes a first bottom left lamp panel frame area and a first top right lamp panel frame area; when the processing module 1102 obtains the lamp panel offset detection result based on the offset value between the center position of the first lamp panel frame area and the center position of the initial lamp panel frame area, it is specifically used to: if the first bottom left lamp panel frame area meets a first preset condition and the first top right lamp panel frame area meets a second preset condition, then based on the first offset value between the center position of the first bottom left lamp panel frame area and the center position of the bottom left lamp panel frame area, and the second offset value between the center position of the first top right lamp panel frame area and the center position of the top right lamp panel frame area, obtain the lamp panel offset detection result. If the first lower left corner light panel frame area meets the first preset condition, and the first upper right corner light panel frame area does not meet the second preset condition, then the light panel offset detection result is obtained based on the first offset value; if the first lower left corner light panel frame area does not meet the first preset condition, and the first upper right corner light panel frame area meets the second preset condition, then the light panel offset detection result is obtained based on the second offset value; wherein, if the width difference ratio between the first lower left corner light panel frame area and the lower left corner light panel frame area is less than a threshold and the height difference ratio is less than a threshold, then the first preset condition is met; otherwise, the first preset condition is not met; if the width difference ratio between the first upper right corner light panel frame area and the upper right corner light panel frame area is less than a threshold and the height difference ratio is less than a threshold, then the second preset condition is met; otherwise, the second preset condition is not met.

[0264] For example, the multiple candidate offset detection results include at least two of the following: lamp disk offset detection result, image offset detection result, the best offset detection result of the previous frame image to be detected, and fixed value offset detection result; if the multiple candidate offset detection results include the image offset detection result, the processing module 1102 specifically obtains the image offset detection result by: obtaining the first frequency domain feature corresponding to the first sub-image and the second frequency domain feature corresponding to the reference sub-image, wherein the reference sub-image is a stored sub-image used to obtain the initial lamp disk frame region; determining the power spectrum based on the first frequency domain feature and the second frequency domain feature, determining the feature matrix based on the power spectrum, and determining the point where the maximum value in the feature matrix is ​​located as the maximum correlation position; obtaining a sub-region within a preset range from the first sub-image with the maximum correlation position as the center, determining the weighted centroid of the sub-region, and obtaining the image offset detection result based on the weighted centroid.

[0265] For example, when the processing module 1102 obtains the first frequency domain feature corresponding to the first sub-image and the second frequency domain feature corresponding to the reference sub-image, it specifically performs the following steps: filtering the first sub-image to obtain a filtered first sub-image; padding the filtered first sub-image with zeros so that its width and height are powers of 2 to obtain a zero-padded first sub-image; multiplying a preset window with the zero-padded first sub-image to obtain a windowed first sub-image; performing a discrete Fourier transform on the windowed first sub-image and obtaining the real number part to obtain the first frequency domain feature; filtering the reference sub-image to obtain a filtered reference sub-image; padding the filtered reference sub-image with zeros so that its width and height are powers of 2 to obtain a zero-padded reference sub-image; multiplying the preset window with the zero-padded reference sub-image to obtain a windowed reference sub-image; performing a discrete Fourier transform on the windowed reference sub-image and obtaining the real number part to obtain the second frequency domain feature.

[0266] For example, when the processing module 1102 selects the best offset detection result from the multiple candidate offset detection results based on the intersection area between the reference lighting area and each corrected lamp disk frame area, it specifically performs the following: if multiple reference lighting areas are obtained, for each reference lighting area, based on the intersection area between the reference lighting area and the corrected lamp disk frame area and the confidence level of the reference lighting area, a weight value for the reference lighting area is determined, wherein the confidence level represents the probability that the reference lighting area is lit; the confidence level of the corrected lamp disk frame area is determined based on the weight value of each reference lighting area; and the candidate offset detection result corresponding to the highest confidence level is selected as the best offset detection result.

[0267] For example, when the detection module 1103 detects the signal light status of the light panel frame to be detected based on the second sub-image, it is specifically used to: divide the second sub-image into multiple single-lamp sub-regions, each single-lamp sub-region corresponding to a signal light; for each single-lamp sub-region, determine a first detection result of the single-lamp sub-region based on the intersection area of ​​the reference lit area and the single-lamp sub-region; determine a second detection result of the single-lamp sub-region based on the brightness information and chromaticity information of the single-lamp sub-region; determine a target detection result of the single-lamp sub-region based on the first detection result and the second detection result, wherein the target detection result indicates whether the signal light is lit or not; and determine the signal light status of the light panel frame to be detected based on the color of the signal light corresponding to each single-lamp sub-region and the target detection result of each single-lamp sub-region.

[0268] For example, when the detection module 1103 divides the second sub-image into multiple single-lamp sub-regions, it is specifically used to: determine the position and orientation of the traffic lights based on the axis equation of the traffic lights, the position of the first traffic light, the interval between traffic lights, and the size of the traffic lights; and for each traffic light, divide the corresponding single-lamp sub-region from the second sub-image based on the position and orientation of the traffic light.

[0269] For example, the detection module 1103 is further configured to, before determining the position and orientation of the traffic lights based on the axis equation of the traffic lights, the position of the first traffic light, the interval between traffic lights, and the size of the traffic lights, determine a first intermediate parameter based on the first slope parameter and the first intercept parameter corresponding to the previous frame of the image to be detected, and the center position coordinates of the first traffic light in the second sub-image; determine a second slope parameter corresponding to the image to be detected based on the first slope parameter, the following rate, the first intermediate parameter, and the center position coordinates; determine a second intercept parameter corresponding to the image to be detected based on the first intercept parameter, the following rate, the first intermediate parameter, and the second slope parameter; determine a second intermediate parameter based on the first initial distance and the first interval position corresponding to the previous frame of the image to be detected, and the center position coordinates; and determine a second intermediate parameter based on the first initial distance, the following rate, the first intermediate parameter, and the center position coordinates; and determine a second intercept parameter based on the first initial distance, the following rate, the first intermediate parameter, and the second slope parameter; and determine a second intermediate parameter based on the first initial distance, the following rate, the first intermediate parameter, and the center position coordinates. Based on the following rate and the second intermediate parameter, a second initial distance is determined for the image to be detected, where the second initial distance represents the distance between the image edge and the center of the first traffic light in the second sub-image; based on the first interval position, the following rate, and the second intermediate parameter, a second interval position is determined for the image to be detected, where the second interval position represents the distance between two adjacent traffic lights; based on the first size value corresponding to the image to be detected in the previous frame, the following rate, and the size of the traffic lights in the second sub-image, a second size value is determined for the image to be detected; based on the second slope parameter and the second intercept parameter, the axis equation of the traffic light is determined; based on the second initial distance, the position of the first traffic light is determined; based on the second interval position, the interval between the traffic lights is determined; and based on the second size value, the size of the traffic light is determined.

[0270] For example, when the detection module 1103 determines the second detection result of the single-lamp sub-region based on the brightness and chromaticity information of the single-lamp sub-region, it specifically performs the following steps: generating a brightness analysis image based on the RGB image corresponding to the single-lamp sub-region, wherein the pixel value of each pixel in the brightness analysis image is the maximum value of the R component, G component, and B component corresponding to that pixel in the RGB image; determining the average brightness value based on the pixel values ​​of all pixels in the brightness analysis image; calculating the average S component value of all pixels based on the HSV image corresponding to the single-lamp sub-region; calculating the average H component value of all candidate pixels for candidate pixels whose S component value is greater than the average S component value; if the average brightness value is greater than a threshold and the average H component value is within the color range, then the second detection result indicates that the traffic light is on; if the average brightness value is greater than a threshold and the average H component value is not within the color range, then the second detection result indicates that the traffic light is off; if the average brightness value is not greater than a threshold, then the second detection result indicates that the traffic light is off; the color range is the color range of the traffic light corresponding to the single-lamp sub-region.

[0271] For example, when the detection module 1103 determines the target detection result of the single-lamp sub-region based on the first detection result and the second detection result, it is specifically used for: if the first detection result indicates that the signal light is on and the second detection result indicates that the signal light is on, then the target detection result indicates that the signal light is on; if the first detection result indicates that the signal light is on and the second detection result indicates that the signal light is off, and the light panel frame to be detected is a single-lamp frame, then the target detection result indicates that the signal light is on; if the first detection result indicates that the signal light is on and the second detection result indicates that the signal light is off, and the light panel frame to be detected is not a single-lamp frame, and there are at least two signal lights with similar brightness and / or similar chromaticity, then the target detection result indicates that the signal light is off ... then the target detection result indicates that the signal light is off; if the first detection result indicates that the signal light is on and the second detection result indicates that the signal light is off, then the target detection result indicates that the signal light is off; if the first detection result indicates that the signal light is on and the second detection result indicates that the signal light is off, then the target detection result indicates that the signal light is off. If the first detection result indicates the signal light is on, and the second detection result indicates the signal light is off, and the light panel frame to be tested is not a single light frame, and there are at least two signal lights with similar brightness and color, then the target detection result indicates the signal light is on. If the first detection result indicates the signal light is off, and the second detection result indicates the signal light is off, then the target detection result indicates the signal light is off. If the first detection result indicates the signal light is off, the second detection result indicates the signal light is on, and the reflection detection result indicates reflection, then the target detection result indicates the signal light is off. If the first detection result indicates the signal light is off, the second detection result indicates the signal light is on, and the reflection detection result indicates no reflection, then the target detection result indicates the signal light is on.

[0272] For example, the processing module 1102 is further configured to, after detecting the traffic light status of the light panel frame to be detected based on the second sub-image, analyze whether there is a traffic light abnormal event in the target scene based on the traffic light status detection results of multiple frames of images to be detected; wherein, the traffic light abnormal event includes at least one of the following: screen abnormal event, single light abnormal event, light group abnormal event, intersection abnormal event;

[0273] The configured abnormal events are divided into short-polling abnormal events and long-polling abnormal events, with the polling duration of long-polling abnormal events being longer than that of short-polling abnormal events. Specifically, when the processing module 1102 analyzes whether a traffic light abnormal event exists in the target scene based on the traffic light status detection results of multiple frames of images to be detected, it performs the following: if the interval between the current time and the detection time of a short-polling abnormal event reaches the polling duration of the short-polling abnormal event, it analyzes whether a traffic light abnormal event corresponding to the short-polling abnormal event exists in the target scene based on the traffic light status detection results of multiple frames of images to be detected; if the interval between the current time and the detection time of a long-polling abnormal event reaches the polling duration of the long-polling abnormal event, it analyzes whether a traffic light abnormal event corresponding to the long-polling abnormal event exists in the target scene based on the traffic light status detection results of multiple frames of images to be detected.

[0274] Based on the same concept as the above method, this application proposes an electronic device, see [link to previous application]. Figure 12 As shown, it includes: a processor 1201 and a machine-readable storage medium 1202, the machine-readable storage medium 1202 storing machine-executable instructions that can be executed by the processor 1201; the processor 1201 is used to execute the machine-executable instructions to implement the traffic light status detection method disclosed in the above example of this application.

[0275] Based on the same concept as the above method, this application also provides a machine-readable storage medium storing a plurality of computer instructions, which, when executed by a processor, can implement the traffic light status detection method disclosed in the above example of this application.

[0276] The aforementioned machine-readable storage medium can be any electronic, magnetic, optical, or other physical storage device that can contain or store information, such as executable instructions, data, etc. For example, machine-readable storage media can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drives (such as hard disk drives), solid-state drives, any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or combinations thereof.

[0277] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, embodiments of this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0278] The above description is merely an embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of this application should be included within the scope of the claims of this application.

Claims

1. A method for detecting the status of a traffic light, characterized in that, The method includes: Acquire a target scene image to be detected, the target scene image including a light panel frame to be detected; Based on the configured initial light panel frame area, a first sub-image is obtained from the image to be detected, and a reference lighting area of ​​the light panel frame to be detected is determined based on the first sub-image. Based on the reference illuminated area and the initial lamp frame area, a candidate offset detection result is selected as the optimal offset detection result from multiple candidate offset detection results; wherein, the candidate offset detection result represents the offset of the actual position of the signal light in the image to be detected relative to the initial lamp frame area; wherein, multiple candidate offset detection results are obtained, and for each candidate offset detection result, the initial lamp frame area is corrected based on the candidate offset detection result to obtain a corrected lamp frame area; based on the intersection area between the reference illuminated area and each corrected lamp frame area, the optimal offset detection result is selected from the multiple candidate offset detection results; Based on the optimal offset detection result, the initial lamp frame region is offset to obtain the target lamp frame region, and the second sub-image corresponding to the target lamp frame region is obtained from the image to be detected; The signal light status of the light panel frame to be detected is determined based on the second sub-image.

2. The method according to claim 1, characterized in that, The multiple candidate offset detection results include at least two of the following: lamp offset detection result, image offset detection result, the best offset detection result of the previous frame of the image to be detected, and fixed value offset detection result; If the multiple candidate offset detection results include the lamp panel offset detection result, obtaining the lamp panel offset detection result includes: The initial lamp frame area is expanded outward by a preset ratio, and a third sub-image is obtained from the image to be detected based on the expanded lamp frame area. The first lamp frame area is detected based on the third sub-image. The lamp offset detection result is obtained based on the offset value between the center position of the first lamp frame area and the center position of the initial lamp frame area.

3. The method according to claim 2, characterized in that, The initial lamp frame area includes the lower left corner lamp frame area and the upper right corner lamp frame area, and the first lamp frame area includes the lower left corner lamp frame area and the upper right corner lamp frame area. The step of obtaining the lamp offset detection result based on the offset value between the center position of the first lamp frame region and the center position of the initial lamp frame region includes: If the first lower left corner lamp panel frame area meets the first preset condition and the first upper right corner lamp panel frame area meets the second preset condition, then based on the first offset value between the center position of the first lower left corner lamp panel frame area and the center position of the leftmost lower left corner lamp panel frame area, and the second offset value between the center position of the first upper right corner lamp panel frame area and the center position of the rightmost upper right corner lamp panel frame area, the lamp panel offset detection result is obtained. If the first lower left corner lamp panel frame area meets the first preset condition, and the first upper right corner lamp panel frame area does not meet the second preset condition, then the lamp panel offset detection result is obtained based on the first offset value. If the first lower left corner lamp panel frame area does not meet the first preset condition, and the first upper right corner lamp panel frame area meets the second preset condition, then the lamp panel offset detection result is obtained based on the second offset value. If the width difference ratio between the first lower left corner light panel frame area and the lower left corner light panel frame area is less than a threshold and the height difference ratio is less than a threshold, then the first preset condition is met; otherwise, the first preset condition is not met. If the width difference between the upper right corner light panel area and the upper right corner light panel area is less than a threshold and the height difference is less than a threshold, then the second preset condition is met; otherwise, the second preset condition is not met.

4. The method according to claim 1, characterized in that, The multiple candidate offset detection results include at least two of the following: lamp offset detection result, image offset detection result, the best offset detection result of the previous frame of the image to be detected, and fixed value offset detection result; If the multiple candidate offset detection results include image offset detection results, obtaining the image offset detection results includes: Obtain the first frequency domain feature corresponding to the first sub-image and the second frequency domain feature corresponding to the reference sub-image, wherein the reference sub-image is a stored sub-image used to obtain the initial lamp frame region; The power spectrum is determined based on the first frequency domain feature and the second frequency domain feature, and the feature matrix is ​​determined based on the power spectrum. The point where the maximum value is located in the feature matrix is ​​determined as the position of maximum correlation. Centered on the location with the highest correlation, a sub-region within a preset range is obtained from the first sub-image, the weighted centroid of the sub-region is determined, and the image offset detection result is obtained based on the weighted centroid.

5. The method according to claim 4, characterized in that, The step of obtaining the first frequency domain feature corresponding to the first sub-image and the second frequency domain feature corresponding to the reference sub-image includes: The first sub-image is filtered to obtain a filtered first sub-image; zeros are padded around the filtered first sub-image so that the width and height are powers of 2 to obtain a zero-padded first sub-image; a preset window is multiplied by the zero-padded first sub-image to obtain a windowed first sub-image; a discrete Fourier transform is performed on the windowed first sub-image and the real part is obtained to obtain the first frequency domain feature. The reference sub-image is filtered to obtain a filtered reference sub-image; zeros are padded around the filtered reference sub-image so that the width and height are powers of 2 to obtain a zero-padded reference sub-image; the preset window is multiplied by the zero-padded reference sub-image to obtain a windowed reference sub-image; a discrete Fourier transform is performed on the windowed reference sub-image and the real part is obtained to obtain the second frequency domain feature.

6. The method according to any one of claims 1-5, characterized in that, The step of selecting the best offset detection result from the plurality of candidate offset detection results based on the intersection area between the reference lighting area and each corrected lamp frame area includes: If multiple reference lighting areas are obtained, for each reference lighting area, a weight value is determined based on the intersection area between the reference lighting area and the corrected lamp frame area and the confidence level of the reference lighting area. The confidence level represents the probability that the reference lighting area is lit. The reliability of the corrected lamp panel frame region is determined based on the weight value of each reference lighting region. The candidate offset detection result corresponding to the highest confidence level is selected as the best offset detection result.

7. The method according to any one of claims 1-5, characterized in that, The step of detecting the signal light status of the light panel frame to be detected based on the second sub-image includes: The second sub-image is divided into multiple single-lamp sub-regions, and each single-lamp sub-region corresponds to one traffic light. For each single-lamp sub-region, a first detection result for the single-lamp sub-region is determined based on the intersection area between the reference lit area and the single-lamp sub-region; a second detection result for the single-lamp sub-region is determined based on the brightness and chromaticity information of the single-lamp sub-region; and a target detection result for the single-lamp sub-region is determined based on the first and second detection results, wherein the target detection result indicates whether the signal light is lit or off. Based on the color of the signal light corresponding to each individual light sub-region and the target detection result of each individual light sub-region, the signal light status of the light panel frame to be detected is determined.

8. The method according to claim 7, characterized in that, The step of dividing the second sub-image into multiple single-light sub-regions includes: Based on the axis equation of the traffic lights, the position of the first traffic light, the interval between traffic lights, and the size of the traffic lights, the position and orientation of the traffic lights are determined; for each traffic light, based on the position and orientation of the traffic light, the corresponding single-light sub-region is divided from the second sub-image.

9. The method according to claim 8, characterized in that, Before determining the position and orientation of the traffic lights based on the axis equation of the traffic lights, the position of the first traffic light, the interval between traffic lights, and the size of the traffic lights, the method further includes: Based on the first slope parameter and first intercept parameter corresponding to the previous frame of the image to be detected, and the center position coordinates of the first traffic light in the second sub-image, a first intermediate parameter is determined; based on the first slope parameter, the configured following rate, the first intermediate parameter, and the center position coordinates, a second slope parameter corresponding to the image to be detected is determined; based on the first intercept parameter, the following rate, the first intermediate parameter, and the second slope parameter, a second intercept parameter corresponding to the image to be detected is determined. Based on the first initial distance and first interval position corresponding to the previous frame of the image to be detected, and the center position coordinates, a second intermediate parameter is determined; based on the first initial distance, the following rate, and the second intermediate parameter, a second initial distance corresponding to the image to be detected is determined, and the second initial distance represents the distance between the image edge and the center of the first traffic light in the second sub-image; based on the first interval position, the following rate, and the second intermediate parameter, a second interval position corresponding to the image to be detected is determined, and the second interval position represents the distance between two adjacent traffic lights; Based on the first size value corresponding to the image to be detected in the previous frame, the following rate, and the size of the traffic light in the second sub-image, the second size value corresponding to the image to be detected is determined. The axis equation of the traffic light is determined based on the second slope parameter and the second intercept parameter. The position of the first traffic light is determined based on the second initial distance. The interval between the traffic lights is determined based on the second interval position. The size of the traffic light is determined based on the second size value.

10. The method according to claim 7, characterized in that, The determination of the second detection result of the single-lamp sub-region based on the brightness and chromaticity information of the single-lamp sub-region includes: A brightness analysis image is generated based on the RGB image corresponding to the single lamp sub-region. The pixel value of each pixel in the brightness analysis image is the maximum value of the R component, G component, and B component corresponding to that pixel in the RGB image. The average brightness value is determined based on the pixel values ​​of all pixels in the brightness analysis image. Based on the HSV image corresponding to the single-lamp sub-region, the mean value of the S component of all pixels is calculated; for candidate pixels whose S component is greater than the mean value of the S component, the mean value of the H component of all candidate pixels is calculated. If the average brightness value is greater than the threshold and the average value of the H component is within the color range, then the second detection result indicates that the signal light is on; if the average brightness value is greater than the threshold and the average value of the H component is not within the color range, then the second detection result indicates that the signal light is off; if the average brightness value is not greater than the threshold, then the second detection result indicates that the signal light is off. The color range refers to the color range of the signal light corresponding to the single-lamp sub-area.

11. The method according to any one of claims 1-5, characterized in that, After detecting the signal light status of the light panel frame to be detected based on the second sub-image, the method further includes: Based on the traffic light status detection results of multiple frames of images to be detected, the system analyzes whether there are any abnormal traffic light events in the target scene; wherein, the abnormal traffic light events include at least one of the following: Abnormal events in the display screen, abnormal events in a single light, abnormal events in a group of lights, and abnormal events at an intersection; The configured abnormal events are divided into short-polling abnormal events and long-polling abnormal events, with the polling duration of long-polling abnormal events being longer than that of short-polling abnormal events. Based on the traffic light status detection results of multiple frames of images to be detected, the system analyzes whether there are traffic light abnormal events in the target scene, including: If the interval between the current time and the detection time of the short polling anomaly event reaches the polling duration of the short polling anomaly event, then based on the traffic light status detection results of multiple frames of images to be detected, it is analyzed whether there is a traffic light anomaly event in the target scene corresponding to the short polling anomaly event. If the interval between the current time and the detection time of the long-polling anomaly event reaches the polling duration of the long-polling anomaly event, then based on the traffic light status detection results of multiple frames of images to be detected, it is analyzed whether there is a traffic light anomaly event in the target scene corresponding to the long-polling anomaly event.

12. A method for detecting the status of a traffic light, characterized in that, The method includes: When it is determined that there is a traffic light malfunction event in the target scene based on the image to be detected of the target scene, the image to be detected is displayed, and a target rectangle is superimposed on the image to be detected; The target rectangular frame is determined on the image to be detected based on the target light panel frame area, which is obtained by offsetting the configured initial light panel frame area based on the optimal offset detection result; wherein, the optimal offset detection result represents the offset of the actual position of the signal light in the image to be detected relative to the configured initial light panel frame area. The optimal offset detection result is obtained based on the method described in claim 1.

13. The method according to claim 12, characterized in that, After displaying the image to be detected, the method further includes: An initial rectangular frame is superimposed and displayed on the image to be detected; The initial rectangular frame is determined on the image to be detected based on the initial light panel frame region.

14. A signal light status detection device, characterized in that, The device includes: The acquisition module is used to acquire a target scene image to be detected, the image to be detected including a light panel frame to be detected; based on the configured initial light panel frame area, a first sub-image is acquired from the image to be detected, and a reference lighting area of ​​the light panel frame to be detected is determined based on the first sub-image; The processing module is configured to: select a candidate offset detection result as the optimal offset detection result from multiple candidate offset detection results based on the reference illuminated area and the initial lamp frame area; offset the initial lamp frame area based on the optimal offset detection result to obtain a target lamp frame area; and obtain a second sub-image corresponding to the target lamp frame area from the image to be detected. The candidate offset detection result represents the offset of the actual position of the signal light in the image to be detected relative to the initial lamp frame area. Specifically, when the processing module selects a candidate offset detection result as the optimal offset detection result from multiple candidate offset detection results based on the reference illuminated area and the initial lamp frame area, it is configured to: obtain multiple candidate offset detection results; for each candidate offset detection result, correct the initial lamp frame area based on the candidate offset detection result to obtain a corrected lamp frame area; and select the optimal offset detection result from the multiple candidate offset detection results based on the intersection area between the reference illuminated area and each corrected lamp frame area. The detection module is used to detect the status of the signal lights in the light panel frame to be detected based on the second sub-image.

15. The apparatus according to claim 14, Its features are, in, The multiple candidate offset detection results include at least two of the following: lamp offset detection result, image offset detection result, the best offset detection result of the previous frame of the image to be detected, and fixed value offset detection result; If the multiple candidate offset detection results include the lamp panel offset detection result, the processing module obtains the lamp panel offset detection result by: expanding the initial lamp panel frame region outward by a preset ratio, obtaining a third sub-image from the image to be detected based on the expanded lamp panel frame region, detecting the first lamp panel frame region based on the third sub-image, and obtaining the lamp panel offset detection result based on the offset value between the center position of the first lamp panel frame region and the center position of the initial lamp panel frame region. The initial lamp panel frame area includes a bottom left lamp panel frame area and a top right lamp panel frame area. The first lamp panel frame area includes a bottom left lamp panel frame area and a top right lamp panel frame area. When the processing module obtains the lamp panel offset detection result based on the offset value between the center position of the first lamp panel frame area and the center position of the initial lamp panel frame area, it is specifically used as follows: if the bottom left lamp panel frame area meets a first preset condition and the top right lamp panel frame area meets a second preset condition, then based on the first offset value between the center position of the bottom left lamp panel frame area and the center position of the bottom left lamp panel frame area, and the second offset value between the center position of the top right lamp panel frame area and the center position of the top right lamp panel frame area, the lamp panel offset detection result is obtained; if the... If the first lower left corner light panel frame area meets the first preset condition, and the first upper right corner light panel frame area does not meet the second preset condition, then the light panel offset detection result is obtained based on the first offset value; if the first lower left corner light panel frame area does not meet the first preset condition, and the first upper right corner light panel frame area meets the second preset condition, then the light panel offset detection result is obtained based on the second offset value; wherein, if the width difference ratio between the first lower left corner light panel frame area and the lower left corner light panel frame area is less than a threshold and the height difference ratio is less than a threshold, then the first preset condition is met; otherwise, the first preset condition is not met; if the width difference ratio between the first upper right corner light panel frame area and the upper right corner light panel frame area is less than a threshold and the height difference ratio is less than a threshold, then the second preset condition is met; otherwise, the second preset condition is not met. The plurality of candidate offset detection results include at least two of the following: lamp disk offset detection result, image offset detection result, best offset detection result of the previous frame image to be detected, and fixed value offset detection result; if the plurality of candidate offset detection results include image offset detection result, the processing module obtains the image offset detection result by: obtaining a first frequency domain feature corresponding to a first sub-image and a second frequency domain feature corresponding to a reference sub-image, wherein the reference sub-image is a stored sub-image used to obtain the initial lamp disk frame region; determining a power spectrum based on the first frequency domain feature and the second frequency domain feature, determining a feature matrix based on the power spectrum, and determining the point where the maximum value in the feature matrix is ​​located as the maximum correlation position; obtaining a sub-region within a preset range from the first sub-image with the maximum correlation position as the center, determining the weighted centroid of the sub-region, and obtaining the image offset detection result based on the weighted centroid; Specifically, when the processing module obtains the first frequency domain feature corresponding to the first sub-image and the second frequency domain feature corresponding to the reference sub-image, it performs the following steps: filtering the first sub-image to obtain a filtered first sub-image; padding the filtered first sub-image with zeros to ensure that its width and height are powers of 2, resulting in a zero-padded first sub-image; multiplying a preset window with the zero-padded first sub-image to obtain a windowed first sub-image; performing a discrete Fourier transform on the windowed first sub-image and obtaining the real number part to obtain the first frequency domain feature; filtering the reference sub-image to obtain a filtered reference sub-image; padding the filtered reference sub-image with zeros to ensure that its width and height are powers of 2, resulting in a zero-padded reference sub-image; multiplying the preset window with the zero-padded reference sub-image to obtain a windowed reference sub-image; and performing a discrete Fourier transform on the windowed reference sub-image and obtaining the real number part to obtain the second frequency domain feature. Specifically, when the processing module selects the optimal offset detection result from the multiple candidate offset detection results based on the intersection area between the reference illuminated area and each corrected lamp frame area, it performs the following steps: If multiple reference illuminated areas are obtained, for each reference illuminated area, based on the intersection area between the reference illuminated area and the corrected lamp frame area and the confidence level of the reference illuminated area, a weight value for the reference illuminated area is determined, where the confidence level represents the probability that the reference illuminated area is illuminated; the confidence level of the corrected lamp frame area is determined based on the weight value of each reference illuminated area; and the candidate offset detection result corresponding to the highest confidence level is selected as the optimal offset detection result. Specifically, when the detection module detects the signal light status of the light panel frame to be detected based on the second sub-image, it is used to: divide the second sub-image into multiple single-lamp sub-regions, each single-lamp sub-region corresponding to a signal light; for each single-lamp sub-region, determine a first detection result of the single-lamp sub-region based on the intersection area of ​​the reference lit area and the single-lamp sub-region; determine a second detection result of the single-lamp sub-region based on the brightness and chromaticity information of the single-lamp sub-region; determine a target detection result of the single-lamp sub-region based on the first and second detection results, wherein the target detection result indicates whether the signal light is lit or off; and determine the signal light status of the light panel frame to be detected based on the color of the signal light corresponding to each single-lamp sub-region and the target detection result of each single-lamp sub-region. Specifically, when the detection module divides the second sub-image into multiple single-lamp sub-regions, it is used to: determine the position and orientation of the traffic lights based on the axis equation of the traffic lights, the position of the first traffic light, the interval between traffic lights, and the size of the traffic lights; and for each traffic light, divide the corresponding single-lamp sub-region from the second sub-image based on the position and orientation of the traffic light. The detection module is further configured to, before determining the position and orientation of the traffic lights based on the traffic light's axis equation, the position of the first traffic light, the interval between traffic lights, and the size of the traffic lights, determine a first intermediate parameter based on the first slope parameter and the first intercept parameter corresponding to the previous frame of the image to be detected, and the center position coordinates of the first traffic light in the second sub-image; determine a second slope parameter corresponding to the image to be detected based on the first slope parameter, the configured following rate, the first intermediate parameter, and the center position coordinates; determine a second intercept parameter corresponding to the image to be detected based on the first intercept parameter, the following rate, the first intermediate parameter, and the second slope parameter; determine a second intermediate parameter based on the first initial distance and the first interval position corresponding to the previous frame of the image to be detected, and the center position coordinates; and determine a second intermediate parameter based on the first initial distance, the configured following rate, the configured following rate, and the center position coordinates; and determine a second intermediate parameter based on the first initial distance, the configured following rate, the configured following rate, and the center position coordinates. Based on the following rate and the second intermediate parameter, a second initial distance is determined for the image to be detected, where the second initial distance represents the distance between the image edge and the center of the first traffic light in the second sub-image; based on the first interval position, the following rate, and the second intermediate parameter, a second interval position is determined for the image to be detected, where the second interval position represents the distance between two adjacent traffic lights; based on the first size value corresponding to the image to be detected in the previous frame, the following rate, and the size of the traffic lights in the second sub-image, a second size value is determined for the image to be detected; based on the second slope parameter and the second intercept parameter, the axis equation of the traffic light is determined; based on the second initial distance, the position of the first traffic light is determined; based on the second interval position, the interval between the traffic lights is determined; and based on the second size value, the size of the traffic light is determined. Specifically, when the detection module determines the second detection result of a single-lamp sub-region based on its brightness and chromaticity information, it performs the following steps: It generates a brightness analysis image based on the RGB image corresponding to the single-lamp sub-region, where the pixel value of each pixel in the brightness analysis image is the maximum value of the R, G, and B components corresponding to that pixel in the RGB image; it determines an average brightness value based on the pixel values ​​of all pixels in the brightness analysis image; it calculates the average S component value of all pixels based on the HSV image corresponding to the single-lamp sub-region; for candidate pixels whose S component value is greater than the average S component value, it calculates the average H component value of all candidate pixels; if the average brightness value is greater than a threshold and the average H component value is within the color range, the second detection result indicates that the traffic light is on; if the average brightness value is greater than the threshold and the average H component value is not within the color range, the second detection result indicates that the traffic light is off; if the average brightness value is not greater than the threshold, the second detection result indicates that the traffic light is off; the color range is the color range of the traffic light corresponding to the single-lamp sub-region. The processing module is further configured to, after detecting the traffic light status of the light panel frame to be detected based on the second sub-image, analyze whether there are any traffic light abnormal events in the target scene based on the traffic light status detection results of multiple frames of images to be detected; wherein, the traffic light abnormal events include at least one of the following: screen abnormal events, single light abnormal events, light group abnormal events, and intersection abnormal events. The configured abnormal events are divided into short-polling abnormal events and long-polling abnormal events, with the polling duration of long-polling abnormal events being longer than that of short-polling abnormal events. When the processing module analyzes whether a traffic light abnormal event exists in the target scene based on the traffic light status detection results of multiple frames of images to be detected, it specifically performs the following: if the interval between the current time and the detection time of a short-polling abnormal event reaches the polling duration of the short-polling abnormal event, then based on the traffic light status detection results of multiple frames of images to be detected, it analyzes whether a traffic light abnormal event corresponding to the short-polling abnormal event exists in the target scene; if the interval between the current time and the detection time of a long-polling abnormal event reaches the polling duration of the long-polling abnormal event, then based on the traffic light status detection results of multiple frames of images to be detected, it analyzes whether a traffic light abnormal event corresponding to the long-polling abnormal event exists in the target scene.

16. An electronic device, characterized in that, include: A processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions that can be executed by the processor; The processor is configured to execute machine-executable instructions to implement the method of any one of claims 1-13.

Images