A method and system for non-motor vehicle red light running management based on multi-view camera

Abstract

The application discloses a non-motor vehicle red light running management method and system based on a multi-camera, relates to the field of intelligent illegal parking management, and comprises the following steps: for the non-motor vehicle red light running illegal management scene of the intersection scene, since the non-motor vehicle target is small, the license plate is small, the illegal behavior judgment is inaccurate, the capture rate is low, and the non-motor vehicle license plate recognition is inaccurate, the advantages of multi-camera cooperation, target time and space alignment, edge calculation and real-time signal lamp state acquisition are fully utilized, so that a non-motor vehicle red light running management method based on multiple dimensions is realized, and high accuracy and high capture rate are realized.

Description

Technical Field

[0001] This invention relates to the field of intelligent illegal parking management, and in particular to a method and system for managing non-motorized vehicles running red lights based on multi-camera systems. Background Technology

[0002] Running red lights by non-motorized vehicles is a violation of traffic rules that seriously impacts urban traffic order, road safety, and the safety of citizens' travel. Therefore, strengthening the management of non-motorized vehicle red-light running is essential. By utilizing artificial intelligence video image recognition technology combined with intelligent traffic management systems, voice prompt systems, and signal detection systems, a complete solution for managing non-motorized vehicle red-light running can be formed. This solution strengthens the monitoring and management of non-motorized vehicle red-light running at intersections, improves the efficiency of supervising non-motorized vehicle violations, effectively reduces the phenomenon of non-motorized vehicle red-light running, enhances road safety, protects the safety of citizens' travel, and creates an orderly, safe, and civilized road traffic environment.

[0003] However, due to the small size of non-motorized vehicles and their license plates, the effective pixel count of the license plates in video footage is low, making identification difficult. Obtaining evidence of violations requires capturing the entire process of a non-motorized vehicle running a red light, including image evidence of the red light, while also accurately identifying the license plate. Traditional monocular cameras struggle to balance the relationship between a large field of view and a large effective pixel count for the license plate; both are difficult to satisfy simultaneously. Therefore, the accuracy of judging non-motorized vehicles running red lights and recognizing license plates is too low. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a method and system for managing non-motorized vehicles running red lights based on multi-view cameras, which can solve the problem of low accuracy in judging non-motorized vehicles running red lights and recognizing license plates.

[0005] To achieve the above objectives, on the one hand, the present invention provides a method for managing non-motorized vehicles running red lights based on multi-view cameras, the method comprising: a traffic signal detector acquiring real-time status information of traffic lights from a traffic signal controller at an intersection;

[0006] The edge computing main controller decodes the video stream in real time and performs non-motorized vehicle target detection and recognition based on the decoded video stream. Based on the non-motorized vehicle target detection and recognition results and the real-time status information of the traffic lights, it determines whether there is any non-motorized vehicle violation.

[0007] If present, the cloud computing server performs secondary identification and verification on the non-motorized vehicle license plates identified by the edge computing main control unit, and sends the non-motorized vehicle red-light violation data after secondary verification to the traffic police department's traffic violation business system through the data security boundary.

[0008] Furthermore, before the step of the traffic signal detector acquiring the real-time status information of the traffic lights from the intersection traffic signal controller, the method further includes:

[0009] Adjust the short-focus and long-focus cameras in the binocular camera. The short-focus camera's view can cover the stop line on the side of the non-motorized vehicle lane and the stop line on the opposite side of the intersection. At the same time, the view should also capture the traffic lights on the opposite side of the intersection. The long-focus camera's view covers a close-up of the middle part of the intersection.

[0010] Draw lines to mark the positions of the stop line on the local side and the stop line on the opposite side within the field of view of the binocular camera. The positions of the stop line on the opposite side are marked and recorded as line segment L1. The positions of the linked capture lines L3 and L4 are located in the middle of the intersection. Draw linked capture lines L33 and L44, where L33 is in the same position as L3 of the short-focus camera and L44 is in the same position as L4 of the short-focus camera.

[0011] The video frame times of the long and short focal length cameras in the binocular camera are synchronized with the time of the edge computing host computer.

[0012] Furthermore, the steps of the edge computing main control unit decoding the video stream in real time and performing non-motorized vehicle target detection and recognition based on the decoded video stream include:

[0013] The edge computing host computer acquires the main bitstream data in real time from the long and short focal length cameras of the binocular camera via the RTSP protocol;

[0014] The edge computing host performs real-time hardware decoding on the acquired two-channel bitstream data to obtain the decoded image frame and SEI information frame SEI-Frame, and obtains the millisecond-level precision timestamp of the current PIC-Frame from the SEI-Frame;

[0015] The edge computing host compares the image frames PIC-Frame after decoding the two bitstream data with the timestamp information extracted from SEI-Frame, and selects the frame with the smallest timestamp error for pairing and caching.

[0016] Furthermore, the step of determining whether a non-motorized vehicle violation exists based on the non-motorized vehicle target detection and identification results and the real-time status information of the traffic light includes:

[0017] When the edge computing main controller detects that the red light signal is on, it decodes the video stream data PIC-Frame from the short-focus camera in the binocular camera and uses a multi-target detection and tracking algorithm model to perform real-time detection and tracking. When a non-motorized vehicle crosses the stop line while the red light is on, the violation capture is initiated, recording the short-focus lens scene image evidence before the non-motorized vehicle crosses the line, and continuously detecting and tracking the non-motorized vehicle.

[0018] When a non-motorized vehicle crosses the linkage capture line, the linkage capture is activated. The edge computing main controller obtains the image from the telephoto camera at the same time from the cached pairing frame. At this time, the illegal non-motorized vehicle should be between lines L33 and L44 in the telephoto camera. If it is not between lines L33 and L44, the next pairing frame is obtained.

[0019] The edge computing main controller uses a non-motorized vehicle license plate recognition algorithm to identify the telephoto camera images in the acquired paired frames, selects the character with the most recognition results, and saves one telephoto camera image and one short-focus camera image when the illegal non-motorized vehicle is driving between L3 and L4.

[0020] The edge computing main control unit detects and tracks the non-motorized vehicle. When the illegal non-motorized vehicle crosses the L4 line, it stops acquiring paired frames. When the illegal non-motorized vehicle crosses the opposite stop line L2, it saves a short-focus lens scene image as evidence after the non-motorized vehicle crosses the L2 line.

[0021] The edge computing main controller sends the generated non-motorized vehicle violation data to the cloud computing server.

[0022] Furthermore, if the aforementioned situation exists, the steps of the cloud computing server performing secondary identification and verification of the non-motorized vehicle license plates identified by the edge computing main control unit, and sending the secondary verified non-motorized vehicle red-light violation data to the traffic police department's traffic violation business system through the data security boundary include:

[0023] The cloud computing server performs secondary recognition and verification of license plate numbers on the received non-motorized vehicle images;

[0024] The cloud computing server sends the verified violation data to the traffic police business system through the security boundary.

[0025] On the other hand, the present invention provides a non-motorized vehicle red light violation management system based on a multi-view camera. The system includes: a traffic signal detector, used to obtain real-time status information of traffic lights from the traffic signal controller at the intersection;

[0026] The edge computing main control unit is used to decode the video stream in real time and perform non-motorized vehicle target detection and recognition based on the decoded video stream. Based on the non-motorized vehicle target detection and recognition results and the real-time status information of the traffic lights, it determines whether there is any non-motorized vehicle violation.

[0027] The cloud computing server, if present, is used to perform secondary identification and verification of the license plates of non-motorized vehicles that violate traffic rules, as identified by the edge computing main control unit. The data of non-motorized vehicles running red lights after secondary verification is then sent to the traffic violation business system of the traffic police department through the data security boundary.

[0028] Furthermore, the system also includes: a binocular camera;

[0029] The binocular camera is specifically used to adjust the short-focus camera and the long-focus camera in the binocular camera. The short-focus camera can cover the stop line on the side of the non-motorized vehicle lane and the stop line on the opposite side of the intersection. At the same time, the image should also be able to capture the traffic lights on the opposite side of the intersection. The long-focus camera's field of view covers a close-up of the middle part of the intersection.

[0030] Draw lines to mark the positions of the stop line on the local side and the stop line on the opposite side within the field of view of the binocular camera. The positions of the stop line on the opposite side are marked and recorded as line segment L1. The positions of the linked capture lines L3 and L4 are located in the middle of the intersection. Draw linked capture lines L33 and L44, where L33 is in the same position as L3 of the short-focus camera and L44 is in the same position as L4 of the short-focus camera.

[0031] The video frame times of the long and short focal length cameras in the binocular camera are synchronized with the time of the edge computing host computer.

[0032] Furthermore, the edge computing main controller is specifically used to acquire main stream data in real time from the long and short focal length cameras of the binocular camera via the RTSP protocol; to perform real-time hardware decoding on the acquired two stream data to obtain decoded image frames and SEI information frames (SEI-Frame), and to obtain the millisecond-precision timestamp of the current PIC-Frame from the SEI-Frame; to compare the image frames (PIC-Frame) after decoding the two stream data with the timestamp information extracted from the SEI-Frame, and to select the frame with the smallest timestamp error for pairing and caching.

[0033] Furthermore, the edge computing main controller is specifically used to, when the red light signal is on, decode the video stream data PIC-Frame from the short-focus camera in the binocular camera and perform real-time detection and tracking using a multi-target detection and tracking algorithm model; when a non-motorized vehicle crosses the stop line while the light is on, violation capture is initiated, recording the short-focus lens scene image evidence before the non-motorized vehicle crosses the line, and continuously detecting and tracking the non-motorized vehicle; when the non-motorized vehicle crosses the linkage capture line, linkage capture is initiated, and the edge computing main controller obtains the image from the long-focus camera at the same time from the cached paired frame. At this time, the violating non-motorized vehicle should be between lines L33 and L44 in the long-focus camera; if it is not in line L33... Between L4 and L4, the next paired frame is obtained; the edge computing main control unit uses a non-motorized vehicle license plate recognition algorithm to identify the telephoto camera image in the obtained paired frame, selects the character with the most recognition results, and saves one telephoto camera image and one short-focus camera image when the illegal non-motorized vehicle is traveling between L3 and L4; the edge computing main control unit detects and tracks the non-motorized vehicle, and stops obtaining paired frames when the illegal non-motorized vehicle crosses the L4 line; when the illegal non-motorized vehicle crosses the opposite stop line L2, a short-focus lens scene image is saved as evidence of the non-motorized vehicle crossing the L2 line; the edge computing main control unit sends the generated non-motorized vehicle violation data to the cloud computing server.

[0034] Furthermore, the cloud computing server is specifically used to perform secondary recognition and verification of license plate numbers on the received non-motorized vehicle images; the cloud computing server then sends the verified violation data to the traffic police business system through a security boundary.

[0035] This invention provides a method and system for managing non-motorized vehicles running red lights based on multi-camera systems. Addressing the issues of small non-motorized vehicle targets, small license plates, inaccurate violation judgment, low capture rate, and inaccurate license plate recognition at intersections, this invention fully utilizes the advantages of multi-camera collaboration, target temporal and spatial alignment, edge computing, and real-time traffic light status acquisition to achieve a multi-dimensional method for managing non-motorized vehicles running red lights, resulting in high accuracy and high capture rate. Attached Figure Description

[0036] Figure 1 This is a flowchart of a non-motorized vehicle red-light violation management method based on multi-view cameras provided by the present invention;

[0037] Figure 2 This is a schematic diagram of a non-motorized vehicle red-light violation management system based on multi-view cameras provided by the present invention. Detailed Implementation

[0038] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0039] like Figure 1 As shown in the figure, an embodiment of the present invention provides a method for managing non-motorized vehicles running red lights based on multi-view cameras, which includes the following steps:

[0040] 101. Traffic signal detectors obtain real-time status information of traffic lights from traffic signal controllers at intersections.

[0041] In this embodiment of the invention, before step 101, the system further includes: a binocular camera A, which has two built-in independent network video cameras, one telephoto and one short-focus. Typically, a standard crossroads requires four binocular cameras A. The short-focus and telephoto cameras in the binocular camera are adjusted so that the short-focus camera's view covers the stop line on its side of the non-motorized vehicle lane and the stop line on the opposite side of the intersection, while also capturing the traffic lights on the opposite side of the intersection. The telephoto camera's view covers a close-up of the middle section of the intersection.

[0042] Line marking is performed within the field of view of the short-focus and long-focus cameras in the binocular camera system. The position of the stop line on this side is marked on the screen and recorded as line segment L1, the position of the stop line on the opposite side is recorded as L2, and the linked capture lines L3 and L4 are drawn, with L3 and L4 located in the middle of the intersection. Linked capture lines L33 and L44 are also drawn, where L33 corresponds to the position of L3 on the short-focus camera, and L44 corresponds to the position of L4 on the short-focus camera. The video frame times of the long-focus and short-focus cameras in the binocular camera system are synchronized with the edge computing main control unit.

[0043] 102. The edge computing main controller decodes the video stream in real time and performs non-motorized vehicle target detection and recognition based on the decoded video stream. Based on the non-motorized vehicle target detection and recognition results and the real-time status information of the traffic indicator, it determines whether there is any non-motorized vehicle violation.

[0044] Specifically, the edge computing master controller acquires main stream data in real time from the long-focus and short-focus cameras of the binocular camera via the RTSP protocol; the edge computing master controller performs real-time hardware decoding on the acquired two streams of data to obtain decoded image frames and SEI information frames (SEI-Frame), and obtains the millisecond-precision timestamp of the current PIC-Frame from the SEI-Frame; the edge computing master controller compares the decoded image frames (PIC-Frame) with the timestamp information extracted from the SEI-Frame, and selects the frame with the smallest timestamp error for pairing and caching.

[0045] Furthermore, the step of determining whether a non-motorized vehicle violation exists based on the non-motorized vehicle target detection and identification results and the real-time status information of the traffic light includes:

[0046] When the edge computing main controller detects a red light, it decodes the PIC-Frame video stream from the short-focus camera in the binocular camera and uses a multi-target detection and tracking algorithm model for real-time detection and tracking. When a non-motorized vehicle crosses the stop line while the light is red, violation capture is initiated, recording the short-focus image evidence of the non-motorized vehicle before it crosses the line, and continuously detecting and tracking the non-motorized vehicle. When the non-motorized vehicle crosses the linkage capture line, linkage capture is initiated. The edge computing main controller retrieves the image from the long-focus camera at the same time from the cached paired frame. At this point, the violating non-motorized vehicle should be between lines L33 and L44 in the long-focus camera image. If it is not between lines L33 and L44... Between these points, the edge computing main control unit acquires the next paired frame; it then uses a non-motorized vehicle license plate recognition algorithm to identify the telephoto camera image in the acquired paired frame, selects the character with the most recognition results, and saves one telephoto camera image and one short-focus camera image of the illegal non-motorized vehicle traveling between L3 and L4; the edge computing main control unit detects and tracks the non-motorized vehicle, and stops acquiring paired frames when the illegal non-motorized vehicle crosses the L4 line; when the illegal non-motorized vehicle crosses the opposite stop line L2, it saves a short-focus lens scene image as evidence of the non-motorized vehicle crossing the L2 line; the edge computing main control unit sends the generated non-motorized vehicle violation data to the cloud computing server.

[0047] 103. If present, the cloud computing server will perform secondary identification and verification of the non-motorized vehicle license plates identified by the edge computing main control unit, and send the non-motorized vehicle red-light violation data after secondary verification to the traffic violation business system of the traffic police department through the data security boundary.

[0048] For embodiments of the present invention, specific application scenarios may be as follows, but are not limited thereto, including: (1) a binocular camera A, which has two built-in independent network video cameras, one telephoto and one short-focus. Typically, a standard crossroads requires four binocular cameras A.

[0049] (2) Traffic signal detector B is used to obtain the real-time status of traffic lights from the traffic signal controller at the intersection. Typically, one traffic signal detector B is required for a standard crossroads.

[0050] (3) A high-performance edge computing main controller C, with built-in AI computing unit and video decoding hardware, is used for real-time decoding of video streams and detection, tracking, and license plate recognition of non-motorized vehicles. Typically, one high-performance edge computing main controller C is required for a standard intersection.

[0051] (4) Voice playback device D, used to provide voice reminders when non-motorized vehicle violations are detected. Typically, four voice playback devices D are required for a standard intersection.

[0052] (5) Cloud computing server E is used to perform cloud-edge collaboration with the main control machine C, perform secondary calculations, and perform secondary identification and verification of the license plates of non-motorized vehicles captured by the main control machine C to further improve the accuracy of license plate recognition; at the same time, the non-motorized vehicles running red lights after secondary verification are sent to the traffic violation business system of the traffic police department through the data security boundary.

[0053] The specific business processing flow is as follows:

[0054] (1) Install binocular camera A on the traffic enforcement pole at the intersection and adjust the field of view. First, adjust the short-focus camera in binocular camera A. By adjusting the focal length, the short-focus camera's image should cover the stop line on the non-motorized vehicle lane and the stop line on the opposite side of the intersection. At the same time, the image should also capture the traffic lights on the opposite side of the intersection for complete evidence collection. Second, adjust the telephoto lens in binocular camera A. By adjusting the focal length, the field of view should only cover a close-up of the middle part of the intersection. The purpose is to magnify the non-motorized vehicle target and obtain a clear license plate. Complete the field of view adjustment for the binocular cameras in all four directions of the intersection.

[0055] (2) Draw lines within the field of view of the short-focus camera in binocular camera A. Mark the position of the stop line on this side as line segment L1, the position of the stop line on the opposite side as L2, and the linked capture lines L3 and L4, which are located in the middle of the intersection. Draw lines for the long-focus camera in binocular camera A, drawing linked capture lines L33 and L44. L33 corresponds to the position of L3 on the short-focus camera, and L44 corresponds to the position of L4 on the short-focus camera. Complete the line drawing for the binocular cameras in all four directions of the intersection in sequence.

[0056] (3) In order to ensure that the video frame time synchronization source of the long and short focal length cameras in the binocular camera A is consistent, the NTP time synchronization method is adopted, and they are all timed to synchronize with the edge computing host C.

[0057] (4) To ensure that the long-focus and short-focus cameras within the binocular camera A can achieve millisecond-level time alignment of video frames, this solution employs the method of inserting SEI frames into the RTSP video stream. An SEI information frame is inserted after each video frame, and the payload of the SEI frame records the millisecond-precision time of the current video frame. For example, if the video stream's frame rate is 30 frames per second, then an SEI information frame is inserted after each of these 30 video data frames, recording the millisecond timestamp of the current video frame.

[0058] (5) The edge computing main controller C obtains the current traffic light signal status of the intersection from the traffic signal detector B, and starts to capture violations when the light is red.

[0059] (6) Edge computing host C acquires H.264 main bitstream data in real time from the long and short focal length cameras of binocular camera A via the RTSP protocol; Edge computing host C performs real-time hardware decoding on the acquired two H.264 bitstream data to obtain the decoded image frame PIC-Frame and SEI information frame SEI-Frame, and obtains the millisecond-level precision timestamp of the current PIC-Frame from the SEI-Frame; Edge computing host C compares the image frame PIC-Frame after decoding the two H.264 bitstream data with the timestamp information extracted from the SEI-Frame, selects the frame with the smallest timestamp error as the matching time-consistent image frame, and performs pairing and caching.

[0060] (7) When the edge computing main controller C obtains the red light signal, it uses a multi-target detection and tracking algorithm model to perform real-time detection and tracking of the PIC-Frame video stream decoded by the short-focus camera in the binocular camera A. When the non-motorized vehicle crosses the stop line L1 in the red light state, the violation capture is started, and the short-focus lens scene image evidence before the non-motorized vehicle crosses the L1 line is recorded. The non-motorized vehicle is continuously detected and tracked. At this time, the voice player is notified to send a red light violation prompt voice.

[0061] (8) When a non-motorized vehicle crosses the linkage capture line L3, the linkage capture is activated. The edge computing main controller C obtains the image from the telephoto camera at the same time from the cached paired frames. At this time, the violating non-motorized vehicle should be between lines L33 and L44 in the telephoto camera. If it is not between lines L33 and L44, the paired frame is discarded, and the next paired frame is obtained. In this way, paired frames when the non-motorized vehicle is traveling between L3 and L4 are continuously obtained.

[0062] (9) The edge computing main controller C uses a non-motorized vehicle license plate recognition algorithm to identify the telephoto camera images in a series of paired frames, and performs character voting to select the character with the most recognition results. It also saves one telephoto camera image and one short-focus camera image when the illegal non-motorized vehicle is traveling between L3 and L4.

[0063] (10) The main control unit C continuously detects and tracks the non-motorized vehicle. When the illegal non-motorized vehicle crosses the L4 line, it stops acquiring paired frames. When the illegal non-motorized vehicle crosses the opposite stop line L2, it saves a short-focus lens scene image as evidence of the non-motorized vehicle crossing the L2 line.

[0064] (11) The main control computer C sends the generated non-motor vehicle violation data to the cloud computing server E. The data includes license plate number, long-focus panoramic camera images of the non-motor vehicle before crossing the L1 line, between the L3 and L4 lines, and after crossing the L2 line, as well as short-focus close-up camera images of the non-motor vehicle between the L33 and L44 lines.

[0065] (12) The cloud computing server E performs secondary recognition and verification of the license plate number of the received non-motorized vehicle image; the cloud computing server E sends the verified violation data to the traffic police business system through the security boundary to complete the evidence collection of non-motorized vehicle running red light violation.

[0066] This invention provides a multi-camera-based method for managing non-motorized vehicles running red lights. Addressing the challenges of managing non-motorized vehicles running red lights at intersections, which suffer from small vehicle size, small license plates, inaccurate violation determination, low capture rate, and inaccurate license plate recognition, this method fully leverages the advantages of multi-camera collaboration, target temporal and spatial alignment, edge computing, and real-time traffic light status acquisition. This results in a multi-dimensional method for managing non-motorized vehicles running red lights, achieving high accuracy and high capture rate.

[0067] To implement the method provided in the embodiments of the present invention, the embodiments of the present invention provide a non-motorized vehicle red-light violation management system based on multi-view cameras, such as... Figure 2 As shown, the system includes: a traffic signal detector 21, an edge computing main controller 22, a cloud computing server 23, and a binocular camera 24.

[0068] Traffic signal detector 21 is used to obtain real-time status information of traffic lights from the traffic signal controller at the intersection;

[0069] The edge computing main controller 22 is used to decode the video stream in real time and perform non-motorized vehicle target detection and recognition based on the decoded video stream. Based on the non-motorized vehicle target detection and recognition results and the real-time status information of the traffic indicator, it is determined whether there is any non-motorized vehicle violation.

[0070] The cloud computing server 23 is used, if present, to perform secondary identification and verification of the non-motorized vehicle license plates identified by the edge computing main control unit for illegal violations, and to send the non-motorized vehicle red-light violation data after secondary verification to the traffic violation business system of the traffic police department through the data security boundary.

[0071] Furthermore, the system also includes: a binocular camera 24;

[0072] The binocular camera 24 is specifically used to adjust the short-focus camera and the long-focus camera in the binocular camera. The short-focus camera can cover the stop line on the side of the non-motorized vehicle lane and the stop line on the opposite side of the intersection. At the same time, the image should also be able to capture the traffic lights on the opposite side of the intersection. The long-focus camera's field of view covers a close-up of the middle part of the intersection.

[0073] Draw lines to mark the positions of the stop line on the local side and the stop line on the opposite side within the field of view of the binocular camera. The positions of the stop line on the opposite side are marked and recorded as line segment L1. The positions of the linked capture lines L3 and L4 are located in the middle of the intersection. Draw linked capture lines L33 and L44, where L33 is in the same position as L3 of the short-focus camera and L44 is in the same position as L4 of the short-focus camera.

[0074] The video frame times of the long and short focal length cameras in the binocular camera are synchronized with the time of the edge computing host computer.

[0075] Furthermore, the edge computing main controller 22 is specifically used to acquire main stream data in real time from the long and short focal length cameras of the binocular camera via the RTSP protocol; to perform real-time hardware decoding on the acquired two stream data to obtain decoded image frames and SEI information frames SEI-Frame, and to obtain the millisecond-level precision timestamp of the current PIC-Frame from the SEI-Frame; to compare the image frame PIC-Frame after decoding the two stream data with the timestamp information extracted from the SEI-Frame, and to select the frame with the smallest timestamp error for pairing and caching.

[0076] Furthermore, the edge computing main controller 22 is specifically used to obtain the video stream decoding data PIC-Frame from the short-focus camera in the binocular camera when the red light signal is on, and to perform real-time detection and tracking using a multi-target detection and tracking algorithm model; when a non-motorized vehicle crosses the stop line while the red light is on, the violation capture is initiated, recording the short-focus lens scene image evidence before the non-motorized vehicle crosses the line, and continuously detecting and tracking the non-motorized vehicle; when the non-motorized vehicle crosses the linkage capture line, the linkage capture is initiated, and the edge computing main controller obtains the image from the long-focus camera at the same time from the cached paired frame. At this time, the violating non-motorized vehicle should be between lines L33 and L44 in the long-focus camera. If it is not in L3... Between lines 3 and L4, the next paired frame is acquired. The edge computing main control unit uses a non-motorized vehicle license plate recognition algorithm to identify the telephoto camera image in the acquired paired frame, selects the character with the most recognition results, and saves one telephoto camera image and one short-focus camera image of the illegal non-motorized vehicle traveling between L3 and L4. The edge computing main control unit detects and tracks the non-motorized vehicle. When the illegal non-motorized vehicle crosses the L4 line, the acquisition of paired frames stops. When the illegal non-motorized vehicle crosses the opposite stop line L2, a short-focus lens scene image is saved as evidence of the non-motorized vehicle crossing the L2 line. The edge computing main control unit sends the generated non-motorized vehicle violation data to the cloud computing server.

[0077] Furthermore, the cloud computing server 23 is specifically used to perform secondary recognition and verification of license plate numbers on the received non-motorized vehicle images; the cloud computing server sends the verified violation data to the traffic police business system through the security boundary.

[0078] This invention provides a non-motorized vehicle red-light violation management system based on multi-cameras. Addressing the issues of small non-motorized vehicle targets, small license plates, inaccurate violation judgment, low capture rate, and inaccurate license plate recognition at intersections, this invention fully utilizes the advantages of multi-camera collaboration, target temporal and spatial alignment, edge computing, and real-time traffic light status acquisition to achieve a multi-dimensional non-motorized vehicle red-light violation management method with high accuracy and high capture rate.

[0079] It should be understood that the specific order or hierarchy of steps in the disclosed process is an example of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the process may be rearranged without departing from the scope of this disclosure. The appended method claims provide elements of various steps in an exemplary order and are not intended to limit the scope to the specific order or hierarchy described.

[0080] In the above detailed description, various features are combined together in a single embodiment to simplify this disclosure. This approach to disclosure should not be construed as reflecting an intention that embodiments of the claimed subject matter require more features than are explicitly stated in each claim. Rather, as reflected in the appended claims, the invention is presented with fewer features than all of the features of the single disclosed embodiment. Therefore, the appended claims are hereby explicitly incorporated into the detailed description, wherein each claim stands alone as a preferred embodiment of the invention.

[0081] The disclosed embodiments have been described above to enable any person skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments without departing from the spirit and scope of this disclosure. Therefore, this disclosure is not limited to the embodiments given herein, but is consistent with the broadest scope of the principles and novel features disclosed in this application.

[0082] The foregoing description includes examples of one or more embodiments. It is certainly impossible to describe all possible combinations of components or methods in order to describe the above embodiments, but those skilled in the art will recognize that further combinations and arrangements of the various embodiments are possible. Therefore, the embodiments described herein are intended to cover all such changes, modifications, and variations that fall within the scope of the appended claims. Furthermore, the term "comprising" as used in the specification or claims is interpreted in a manner similar to the term "including," as interpreted when used as a conjunction in the claims. Additionally, the use of any term "or" in the specification of the claims is intended to mean "non-exclusive or."

[0083] Those skilled in the art will also understand that the various illustrative logical blocks, units, and steps listed in the embodiments of the present invention can be implemented by electronic hardware, computer software, or a combination of both. To clearly demonstrate the interchangeability of hardware and software, the functions of the various illustrative components, units, and steps described above have been generally described. Whether such functionality is implemented through hardware or software depends on the specific application and the overall system design requirements. Those skilled in the art can implement the described functions using various methods for each specific application, but such implementation should not be construed as exceeding the scope of protection of the embodiments of the present invention.

[0084] The various illustrative logic blocks or units described in the embodiments of this invention can be implemented or operate the described functions using a general-purpose processor, digital signal processor, application-specific integrated circuit (ASIC), field-programmable gate array or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof. The general-purpose processor can be a microprocessor; alternatively, it can be any conventional processor, controller, microcontroller, or state machine. The processor can also be implemented using a combination of computing devices, such as a digital signal processor and a microprocessor, multiple microprocessors, one or more microprocessors combined with a digital signal processor core, or any other similar configuration.

[0085] The steps of the methods or algorithms described in the embodiments of this invention can be directly embedded in hardware, a software module executed by a processor, or a combination of both. The software module can be stored in RAM, flash memory, ROM, EPROM, EEPROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium in the art. Exemplarily, the storage medium can be connected to the processor so that the processor can read information from and write information to the storage medium. Optionally, the storage medium can also be integrated into the processor. The processor and storage medium can be housed in an ASIC, which can be housed in a user terminal. Optionally, the processor and storage medium can also be housed in different components of the user terminal.

[0086] In one or more exemplary designs, the functions described in the embodiments of the present invention can be implemented in hardware, software, firmware, or any combination of these three. If implemented in software, these functions can be stored on a computer-readable medium or transmitted on a computer-readable medium in the form of one or more instructions or code. Computer-readable media include computer storage media and communication media that facilitate the transfer of computer programs from one place to another. Storage media can be any available media that can be accessed by a general-purpose or special-purpose computer. For example, such computer-readable media can include, but is not limited to, RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store program code in the form of instructions or data structures and other forms that can be read by a general-purpose or special-purpose computer, or a general-purpose or special-purpose processor. Furthermore, any connection can be suitably defined as a computer-readable medium, for example, if the software is transmitted from a website, server or other remote resource via a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) or wirelessly, such as infrared, wireless and microwave, it is also included in the defined computer-readable medium. The disks and discs mentioned include compressed disks, laser discs, optical discs, DVDs, floppy disks, and Blu-ray discs. Disks typically copy data magnetically, while discs typically copy data optically using lasers. Combinations of the above can also be contained in computer-readable media.

[0087] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for managing non-motorized vehicles running red lights based on multi-camera systems, characterized in that, The method includes: Traffic signal detectors obtain real-time status information of traffic lights from traffic signal controllers at intersections; The edge computing main controller decodes the video stream in real time and performs non-motorized vehicle target detection and recognition based on the decoded video stream. Based on the non-motorized vehicle target detection and recognition results and the real-time status information of the traffic lights, it determines whether there is any non-motorized vehicle violation. If present, the cloud computing server will perform secondary identification and verification of the non-motorized vehicle license plates identified by the edge computing main control unit, and send the non-motorized vehicle red light violation data after secondary verification to the traffic violation business system of the traffic police department through the data security boundary; Before the step of the traffic signal detector obtaining the real-time status information of the traffic lights from the intersection traffic signal controller, the method further includes: Adjust the short-focus and long-focus cameras in the binocular camera. The short-focus camera's view can cover the stop line on the side of the non-motorized vehicle lane and the stop line on the opposite side of the intersection. At the same time, the view should also capture the traffic lights on the opposite side of the intersection. The long-focus camera's view covers a close-up of the middle part of the intersection. Draw lines to mark the positions of the stop line on the local side and the stop line on the opposite side within the field of view of the binocular camera. The positions of the stop line on the opposite side are marked and recorded as line segment L1. The positions of the linked capture lines L3 and L4 are located in the middle of the intersection. Draw linked capture lines L33 and L44, where L33 is in the same position as L3 of the short-focus camera and L44 is in the same position as L4 of the short-focus camera. The video frame times of the long and short focal length cameras in the binocular camera are synchronized with the time of the edge computing main control unit. The steps of the edge computing main control unit decoding the video stream in real time and performing non-motorized vehicle target detection and recognition based on the decoded video stream include: The edge computing host computer acquires the main bitstream data in real time from the long and short focal length cameras of the binocular camera via the RTSP protocol; The edge computing main controller performs real-time hardware decoding on the acquired two-channel bitstream data to obtain the decoded image frame and SEI information frame SE I-Frame, and obtains the millisecond-level precision timestamp of the current PIC-Frame from the SE I-Frame; The edge computing main controller compares the image frames PIC-Frame after decoding the two bitstream data with the timestamp information extracted from the SE I-Frame, and selects the frame with the smallest timestamp error for pairing and caching. The step of determining whether there is a non-motorized vehicle violation based on the non-motorized vehicle target detection and recognition results and the real-time status information of the traffic lights includes: When the edge computing main controller detects that the red light signal is on, it decodes the video stream data PIC-Frame from the short-focus camera in the binocular camera and uses a multi-target detection and tracking algorithm model to perform real-time detection and tracking. When a non-motorized vehicle crosses the stop line while the red light is on, the violation capture is initiated, recording the short-focus lens scene image evidence before the non-motorized vehicle crosses the line, and continuously detecting and tracking the non-motorized vehicle. When a non-motorized vehicle crosses the linkage capture line, the linkage capture is activated. The edge computing main controller obtains the image from the telephoto camera at the same time from the cached pairing frame. At this time, the illegal non-motorized vehicle should be between lines L33 and L44 in the telephoto camera. If it is not between lines L33 and L44, the next pairing frame is obtained. The edge computing main controller uses a non-motorized vehicle license plate recognition algorithm to identify the telephoto camera images in the acquired paired frames, selects the character with the most recognition results, and saves one telephoto camera image and one short-focus camera image when the illegal non-motorized vehicle is driving between L3 and L4. The edge computing main control unit detects and tracks the non-motorized vehicle. When the illegal non-motorized vehicle crosses the L4 line, it stops acquiring paired frames. When the illegal non-motorized vehicle crosses the opposite stop line L2, it saves a short-focus lens scene image as evidence after the non-motorized vehicle crosses the L2 line. The edge computing main controller sends the generated non-motorized vehicle violation data to the cloud computing server.

2. The method for managing non-motorized vehicles running red lights based on multi-cameras according to claim 1, characterized in that, If such a situation exists, the steps of the cloud computing server performing secondary recognition and verification of the non-motorized vehicle license plates identified by the edge computing main control unit, and sending the secondary verified non-motorized vehicle red-light violation data to the traffic police department's traffic violation business system through the data security boundary include: The cloud computing server performs secondary recognition and verification of license plate numbers on the received non-motorized vehicle images; The cloud computing server sends the verified violation data to the traffic police business system through the security boundary.

3. A non-motorized vehicle red-light violation management system based on multi-cameras, characterized in that, The system includes: Traffic signal detectors are used to obtain real-time status information of traffic lights from traffic signal controllers at intersections; The edge computing main control unit is used to decode the video stream in real time and perform non-motorized vehicle target detection and recognition based on the decoded video stream. Based on the non-motorized vehicle target detection and recognition results and the real-time status information of the traffic lights, it determines whether there is any non-motorized vehicle violation. The cloud computing server, if present, is used to perform secondary identification and verification of the non-motorized vehicle license plates identified by the edge computing main control unit for illegal violations, and to send the non-motorized vehicle red-light violation data after secondary verification to the traffic violation business system of the traffic police department through the data security boundary; The system also includes: a binocular camera; The binocular camera is specifically used to adjust the short-focus camera and the long-focus camera in the binocular camera. The short-focus camera can cover the stop line on the side of the non-motorized vehicle lane and the stop line on the opposite side of the intersection. At the same time, the image should also be able to capture the traffic lights on the opposite side of the intersection. The long-focus camera's field of view covers a close-up of the middle part of the intersection. Draw lines to mark the positions of the stop line on the local side and the stop line on the opposite side within the field of view of the binocular camera. The positions of the stop line on the opposite side are marked and recorded as line segment L1. The positions of the linked capture lines L3 and L4 are located in the middle of the intersection. Draw linked capture lines L33 and L44, where L33 is in the same position as L3 of the short-focus camera and L44 is in the same position as L4 of the short-focus camera. The video frame times of the long and short focal length cameras in the binocular camera are synchronized with the time of the edge computing main control unit. The edge computing main controller is specifically used to acquire main stream data in real time from the long and short focal length cameras of a stereo camera via the RTSP protocol; to perform real-time hardware decoding on the acquired two stream data to obtain decoded image frames and SE I information frames (SE I-Frame), and to obtain the millisecond-precision timestamp of the current PIC-Frame from the SE I-Frame; to compare the image frames (PIC-Frame) after decoding the two stream data with the timestamp information extracted from the SE I-Frame, and to select the frame with the smallest timestamp error for pairing and caching; The edge computing main controller is specifically used to decode the PIC-Frame video stream from the short-focus camera in the binocular camera system when the red light signal is on, and then use a multi-target detection and tracking algorithm model for real-time detection and tracking. When a non-motorized vehicle crosses the stop line while the light is on, the violation capture is initiated, recording the short-focus image evidence of the non-motorized vehicle before it crosses the line, and continuously detecting and tracking the non-motorized vehicle. When the non-motorized vehicle crosses the linkage capture line, the linkage capture is initiated, and the edge computing main controller retrieves the image from the long-focus camera at the same time from the cached paired frame. At this point, the violating non-motorized vehicle should be between lines L33 and L44 in the long-focus camera image. If it is not between lines L33 and L4... Between the four lines, the next paired frame is obtained; the edge computing main control unit uses a non-motorized vehicle license plate recognition algorithm to identify the telephoto camera image in the obtained paired frame, selects the character with the most recognition results, and saves one telephoto camera image and one short-focus camera image when the illegal non-motorized vehicle is traveling between L3 and L4; the edge computing main control unit detects and tracks the non-motorized vehicle, and stops obtaining paired frames when the illegal non-motorized vehicle crosses the L4 line; when the illegal non-motorized vehicle crosses the opposite stop line L2, a short-focus lens scene image is saved as evidence of the non-motorized vehicle crossing the L2 line; the edge computing main control unit sends the generated non-motorized vehicle violation data to the cloud computing server.

4. A non-motorized vehicle red-light violation management system based on multi-cameras according to claim 3, characterized in that, The cloud computing server is specifically used to perform secondary recognition and verification of license plate numbers on received non-motorized vehicle images. The cloud computing server sends the verified violation data to the traffic police business system through the security boundary.

Images