System and method for detecting a traffic event

Abstract

A system for detecting a traffic event, comprising: a traffic event detection processor for receiving a video stream from one or more camera units and a plurality of environmental data from one or more environmental sensors; the plurality of environmental data and the video stream are processed by a plurality of traffic event detection modules, each module for processing the environmental data or video stream to detect, track, or monitor a type of traffic event, and processing the detected, tracked, or monitored traffic event to generate one or more event alerts.

Description

Technical Field

[0001] The present disclosure relates to a system and method for detecting traffic events, and in particular, although not exclusively, to a system and method for detecting traffic violations using machine vision and / or environmental sensors. Background Art

[0002] Traffic rules and regulations are usually set by governments to control the movement of people and vehicles. However, despite clear rules on the flow and direction of traffic, road users, drivers or pedestrians often ignore these rules for convenience or cost savings.

[0003] Policeing such rules is also difficult given the amount of enforcement resources required to police all streets and roads in a city. Furthermore, since vehicles and pedestrians are constantly moving, such traffic violations may also occur at different times and locations due to previous enforcement actions. Therefore, policing traffic violations is a difficult task that requires a lot of resources.

[0004] The task becomes more complex when there are large incidents, as these incidents may involve a large number of people and vehicles in a small area. The police's work in these incidents becomes more difficult due to the large number of people and vehicles and the large space in which these people and vehicles may move. It is desirable to solve or overcome these challenges to use police resources efficiently. Summary of the invention

[0005] According to a first aspect of the present disclosure, a system for detecting a traffic event is provided, the system comprising:

[0006] A media gateway for receiving a traffic video stream of a traffic area;

[0007] a traffic incident detection processor, which is used to detect traffic incidents in the traffic area;

[0008] An alert generation routine is used to generate an alert when the traffic event satisfies one or more traffic event conditions.

[0009] In an embodiment of the first aspect, the traffic event includes the presence of an object in the traffic area.

[0010] In an embodiment of the first aspect, the detected object present in the traffic area is analyzed to determine its identity.

[0011] In an embodiment of the first aspect, the detected object present in the traffic area is analyzed to determine its object type, driving direction, moving condition, speed, or any one or a combination thereof.

[0012] In an embodiment of the first aspect, the traffic area is analyzed to determine a number of objects detected in the traffic area.

[0013] In an embodiment of the first aspect, the one or more traffic event conditions include a type of traffic area.

[0014] In an embodiment of the first aspect, the traffic event condition includes a duration of existence of the object in the traffic area.

[0015] In an embodiment of the first aspect, the object is a vehicle.

[0016] In an embodiment of the first aspect, the alert is an alert of a traffic violation, and the alert of the traffic violation is further processed to generate a traffic violation prosecution procedure.

[0017] In an embodiment of the first aspect, the image stream of the traffic violation is stored and further associated with the traffic violation prosecution program.

[0018] According to a second aspect of the present disclosure, a system for detecting traffic events is provided. The system comprises:

[0019] a traffic event detection processor for receiving a video stream from one or more camera units and a plurality of environmental data from one or more environmental sensors;

[0020] Processing the plurality of environmental data and the video streams by a plurality of traffic event detection modules, each module being configured to process the environmental data or the video stream to detect, track or monitor the type of traffic events, and

[0021] The detected, tracked or monitored traffic events are processed to generate one or more event alerts.

[0022] In an embodiment of the second aspect, the system further comprises: a live feed channel for directing the video stream or the plurality of environmental data to a user interface for user access.

[0023] In one embodiment of the second aspect, the one or more camera units or the one or more environmental sensors communicate with the traffic event detection processor through one or more edge devices, and the edge device is used to at least partially process the video stream or the multiple environmental data before transmitting the video stream or the multiple environmental data to the traffic event detection processor.

[0024] In an embodiment of the second aspect, the one or more edge devices are further used to at least partially control the one or more cameras or the one or more environmental sensors.

[0025] In an embodiment of the second aspect, the live feed channel is configured to be segmented into a plurality of channels, each of the channels being configured to contain the video stream or environmental data from a user-selected camera or environmental sensor.

[0026] In an embodiment of the second aspect, outputs from the one or more of the plurality of traffic event detection modules are input into one or more other traffic event detection modules to detect traffic events.

[0027] In an embodiment of the second aspect, the traffic detection processor is used to select one or more traffic event detection modules to detect traffic events.

[0028] In an embodiment of the second aspect, the traffic event detection module includes one or more of the following:

[0029] Object detection module;

[0030] Object counting module;

[0031] Motion tracking module;

[0032] Dangerous or suspicious object detection module;

[0033] accident / incident detection or prediction modules; and

[0034] Behavior detection module.

[0035] In an embodiment of the second aspect, the processor and the module are deployed on a cloud-based architecture.

[0036] In an embodiment of the second aspect, the detected events, video streams and environment data are stored on a database and are used to be retrieved to generate one or more virtual models, each of which is used to model a traffic event within the environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Embodiments of the present disclosure will be described below by way of examples with reference to the accompanying drawings.

[0038] Figure 1 is a schematic block diagram of a computer system according to one embodiment of the present disclosure, the computer system being arranged to be implemented to operate as a system and method for detecting traffic events;

[0039] Figure 2 is a block diagram of an exemplary embodiment of a system for detecting traffic incidents;

[0040] Figure 3 To show Figure 2 A block diagram of an exemplary embodiment of a traffic event detection processor of a system for detecting traffic events;

[0041] Figure 4 for Figure 2 An example screenshot of an interface of a system for detecting traffic incidents;

[0042] Figure 5 is a block diagram of another embodiment of a system and method for detecting a traffic event; and

[0043] Figure 6 for Figure 5 A block diagram of an example of a traffic incident detection processor in a system. DETAILED DESCRIPTION

[0044] refer to Figure 1 , shows an embodiment of a computer server, which is used to operate with or be implemented as an exemplary embodiment of the present disclosure. This embodiment provides a system and method for detecting traffic events. The system and method include the following steps:

[0045] A media gateway for receiving a traffic video stream of a traffic area;

[0046] a traffic incident detection processor, which is used to detect traffic incidents in the traffic area;

[0047] An alert generation routine is used to generate an alert when the traffic event satisfies one or more traffic event conditions.

[0048] In this exemplary embodiment, a computer server, which may be a computer or a computer system, may include an interface and a processor, each of which is implemented by a computer with an appropriate user interface. The computer may be implemented by any computing architecture, including: a portable computer, a tablet computer, a stand-alone personal computer (PC), a smart device, an Internet of Things (IoT) device, an edge computing device, a client / server architecture, a "dumb" terminal / host architecture, a cloud computing-based architecture, or any other appropriate architecture. The computing device may be appropriately programmed to implement the present disclosure.

[0049] In this embodiment, the system and method are used to receive a video stream of a working area that is under surveillance to detect traffic violations or violations. The system and method will process the video stream to determine whether a traffic violation has occurred and identify the responsible vehicle or person (party) responsible for the violation. Preferably, the system can receive input from a user (who can mark specific preset areas where violations or violations may occur) and input from a user (who can define preset rules or condition sets about what constitutes a violation or violation). This can be expressed as which objects can be detected in the preset area, and when these objects are detected, what conditions must be met to determine that a violation has occurred.

[0050] Once these inputs are received from the user, the system may process a video stream, or a stream of images captured from one or more connected cameras. The cameras are used to continuously capture a video stream of a pre-set area where a violation may occur. The system includes a traffic event processor that is designed to process the video stream and determine whether an event, such as a traffic event that may include a violation or illegality, has occurred or been performed in the video stream. The traffic event processor may first process the image to determine whether an object, such as a vehicle or pedestrian, has entered the pre-set area; if the object is detected in the pre-set area, it may determine whether the event constitutes a violation or illegality by comparing the event to a pre-set violation rule or condition. In some cases, the video stream may be further processed to identify other objects, other events, or repeated occurrences of objects over time, which may be determining factors in determining whether a violation or illegality has occurred.

[0051] When the traffic detection processor determines that a violation or irregularity has occurred, an alert may be generated so that further action can be taken against the parties involved in the violation or irregularity. These measures may include, but are not limited to, issuing a violation notice or dispatching law enforcement officers to the location where the violation occurred. These embodiments may have advantages because the system can automatically monitor and detect violations by processing video streams, thereby reducing the workload of law enforcement officers.

[0052] like Figure 1As shown, a schematic diagram of a computer system or computer server 100 is shown. The computer system or computer server 100 is used to implement an exemplary embodiment of a system for detecting traffic events. In this embodiment, the computer system is a server 100, which includes appropriate components necessary for receiving, storing and executing appropriate computer instructions. These components may include: a processing unit 102, which includes a central processing unit (CPU), a math co-processing unit (Math Co-processing Unit, Math Processor), a graphics processing unit (Graphics Processing Unit, GPU) or a tensor processing unit (Tensor Processing Unit, TPU) for tensor or multi-dimensional array calculation or manipulation operations; a read-only memory (Read-Only Memory, ROM) 104; a random access memory (Random-Access Memory, RAM) 106; an input / output (Input / Output, I / O) device (e.g., a disk drive 108); an input device 110 (e.g., an Ethernet port, a USB port, etc.); a display 112, for example, a liquid crystal display, a light-emitting display or any other suitable display; and a communication link 114. The server 100 may include instructions that may be stored in ROM 104, RAM 106, or disk drive 108 and executed by the processing unit 102. A plurality of communication links 114 may be provided that may be variously connected to one or more computing devices, such as servers, personal computers, terminals, wireless or handheld computing devices, IoT devices, smart devices, and edge computing devices. At least one of the plurality of communication links may be connected to an external computing network via a telephone line or other type of communication link.

[0053] The server 100 may include a storage device such as a physical disk drive 108, which may include a solid-state drive, a hard disk drive, an optical drive, a tape drive, or a remote or cloud-based storage device. The server 100 may use a single disk drive or multiple disk drives, or a remote storage service. The server 100 may also have a suitable operating system 116 resident on the disk drive or in the ROM of the server 100.

[0054] The computer or computing device may also provide the necessary computing power to operate or interface with the machine learning network of the object neural network to provide various functions and outputs. The neural network may be implemented locally, or may be accessed or partially accessed through a server or cloud-based service. The machine learning network may also be untrained, partially trained, or fully trained, and / or may also be retrained, modified, or updated over time.

[0055] refer to Figure 2 , a block diagram of an exemplary embodiment of a system for detecting traffic events is shown. The system includes:

[0056] A media gateway for receiving a traffic video stream of a traffic area;

[0057] A traffic incident detection processor, which is used to detect traffic incidents within a traffic area;

[0058] An alert generation routine is used to generate an alert when a traffic event satisfies one or more traffic event conditions.

[0059] In this embodiment, the system 200 for detecting traffic events includes a traffic event detection processor 202 for receiving an image stream 201 captured by a camera. The cameras are directed at a space, location, or area being monitored. Such a space, location, or area may include an area where traffic exists, such as a road, a lane, a tunnel, an intersection, a parking area, or any other area where objects such as vehicles or pedestrians may travel through, stand, park, or gather, and thus may be referred to as a traffic space or traffic area.

[0060] Preferably, the camera can capture a space where an object such as a vehicle or person could enter and potentially cause a violation, or where a violation or infraction could be generated. Examples of this could include areas that could attract unauthorized parking, stopping, or loading vehicles, as well as no parking areas, no entry signs, or one-way areas, or areas where only certain types of vehicles are allowed. For example, a loading zone for trucks, vans, or station wagons only, not for passenger cars, hatchbacks, or sedans; or a bike lane for scooters, bicycles, unicycles, or pedestrians, not for cars, motorcycles, or large vehicles.

[0061] like Figure 2 As shown, the image stream 201 can be received by a media gateway (not shown) of the traffic event detection processor 202. These media streams can be transmitted to the media gateway via a communication network, which connects each of the cameras or storage devices to the traffic event detection processor 202. Once the gateway receives the media stream 201, the media stream 201 can be processed by the traffic event detection processor 202 to identify any violation alerts or any detected traffic information. The traffic event detection processor 202 can also store violation events in the form of media 210, such as fragments or portions of the media stream 201, and save them when violations are detected. The processor 202 can also generate a violation alert 208 indicating a violation or illegality, and transmit the alert to a user or other computer system, such as a computer system for issuing violation notifications, for further processing.

[0062] Preferably, in order to operate so as to detect violations or offenses, the traffic event detection processor 202 may first receive user input marking specific areas for detecting objects such as vehicles or pedestrians. In this regard, the interface provided by the system may first display an image of the area that has been captured by the camera, and the image is part of the picture of the media stream, which will then be processed by the processor 202 for detecting traffic events. The user can then mark these areas by drawing a picture around the image of the violation areas 204 or by inputting specific coordinates into the system 202. Once these areas 204 are marked, these violation areas are known to the system 202, and the system 202 can then proceed to observe objects that can be detected within these areas 204. As examples, such areas may include areas around driveways, street curbs, property fronts, access entrances, areas outside fire hydrants, or any other area where entry, parking, or lingering may occur and it is desirable to detect such activities.

[0063] Preferably, the traffic event detection processor 202 may also receive preset violation conditions 206, which may include a set of conditions or rules that need to be violated before a violation or violation is detected. Such preset violation conditions may include, but are not limited to, the time period during which a vehicle is detected in a specific area, the direction of travel of a specific vehicle through a specific area, or the speed of a vehicle passing through the area, or the type of vehicle passing through the area. Conditions 206 may also involve multiple objects, multiple areas, or may also trigger further processing for further identifying objects. As an example, for the detection of a no-parking zone, when a vehicle is detected in the zone, the image is further processed to determine whether the vehicle is completely stopped in the zone, or whether the vehicle is just moving slowly through without violating the law. Therefore, the duration of the detected object is a condition for determining a violation.

[0064] In another example, such as monitoring a loading zone, a vehicle may be detected within the zone, but a pre-set condition may dictate that the vehicle may use the loading zone if it is a truck or van. Thus, the processor 202 may then proceed to consider the details of the detected vehicle and determine whether the vehicle is a truck or van, or a passenger car before deciding whether a violation has occurred.

[0065] Preferably, the traffic event detection processor 202 uses an image processing system to detect such object violations or violations. Preferably, the artificial intelligence system that processes the image can first identify objects such as vehicles or pedestrians, and then proceed to determine the details of the object. These details can then be compared with preset violation conditions to identify whether a violation has occurred. In the event that a violation occurs and a violation alert 208 is generated and sent, such an alert can also be processed to automatically issue a violation notification to the relevant department. The media stream in which the violation has been captured can also be stored as necessary evidence of the violation. In addition, information related to the traffic that the traffic event detection processor 202 may have detected, including the number of objects detected, the type of objects detected, the speed of the objects detected, and the information of the events in which these objects have been detected, can also be stored as traffic information 212 for further analysis so that data for any relevant area can be provided to planners and operators.

[0066] refer to Figure 3 , an exemplary embodiment of a traffic event detection processor 202 is shown, which is shown in more detail and referred to as reference numeral 300. In this example, a media stream 201 is first processed by an object detection processor 302 to detect objects, such as vehicles or people, within a marked surveillance area. Preferably, an image of an area under surveillance may first be marked by a user 204, such that if an object is detected within that particular area, the object is considered to have been detected. An example of this may include, but is not limited to, allowing a user to mark an area on an image, such as a frame around a portion of an image representing a particular area. The area may be an area of ​​the image where an object may enter the frame, such as a vehicle or a person, and therefore may trigger a violation if the presence of the vehicle or person within the frame meets a particular preset condition or rule 206. As an example, the area 204 may be a stationary parking area captured by a camera pointed at a road, passage, or lane area that has been specifically marked by a user.

[0067] Once an object such as a vehicle is detected from the image stream entering the area, the vehicle will be detected by the object image processing system 302. The object image processing system 302 can use a machine learning system of an object neural network classifier to identify the existence of the object and what the object is. Once an object in the picture is detected, other trained neural networks or image processing systems 304 can be used to further analyze and determine what the object may be and obtain any specific details of the object, including shape, size, brand, color or any other details that are considered important. The object and any details determined in the image processing 304 of the media stream can then be recorded. In the case of a vehicle being detected, its license plate or vehicle ID can also be detected by a license plate or sign detector, which can also use a system based on machine learning (convolutional neural network). The type of vehicle can also be detected by a classifier. Therefore, the size of the vehicle can be used to estimate its type, and the shape or brand of the vehicle can be used to detect the type of vehicle, thereby detecting whether the vehicle is eligible to travel in a specific area.

[0068] In addition, when a vehicle is considered to be moving, the various frames of the media stream can be compared to determine the speed or determine whether the vehicle is stationary, and can also be used to detect the direction of movement of the vehicle, and determine whether the vehicle enters or exits the frame of the media stream. Such information can then be recorded and compared 306 with a set of preset violation conditions 206 to determine whether the vehicle has committed a violation or violation. As an example, in the case where the marked area is a "no entry zone", if the vehicle enters the area within a preset amount of time, then the violation condition may be met, in which case a violation alert may be generated 308. Detailed information about the vehicle and its violation obtained from the media stream will be recorded for further storage or processing 310 as evidence or proof of the violation.

[0069] The user may update or add other predefined violation conditions at any time to define various types of violation conditions 206. Other types of violation conditions may also include: any length of time the vehicle stays in a no-parking zone, whether the vehicle speed exceeds a certain speed limit, the number of vehicles passing through, or whether the direction of travel is correct, such as whether the vehicle is traveling in the correct direction on a one-way street. The vehicle type may also be related to the implementation of, for example, bicycle lanes, public transportation lanes, bus or truck zones. In a more advanced example, the vehicle license plate may also be detected to determine whether the vehicle is authorized to use the particular road during a curfew or traffic control. Such preset violation conditions 206 may be specifically entered by the user, and each set of conditions may be associated with a specific area that has been marked by the user.

[0070] The advantage of the exemplary embodiments of the present disclosure is that the detection of vehicles and violation conditions can be automatically performed by a computer system that operates as a system for monitoring traffic events. With the necessary area markers 204 set and a set of preset violation conditions 206 defined, a convolutional neural network (CNN) or other neural network-based learning system can be used to detect vehicles and identify detailed information about the vehicles to determine whether a violation has occurred. Similarly, other objects such as people, pedestrians, or even animals can also be detected to determine whether there are people or animals in a particular area. The direction, speed, and count of people or animals moving can also be performed by the system for detecting events.

[0071] refer to Figure 4 , shows an example interface of a system for detecting events. In this example, interfaces A, B, and C show a lane entrance, and the user can first mark the design area of ​​the lane entrance with a rectangle. Then, the user can submit a rule that stipulates that no vehicle is allowed to park at the lane entrance.

[0072] In screenshot B, a vehicle is detected approaching the lane entrance, in which case the vehicle details including license plate and vehicle type and size may be detected and recorded. As shown in screenshot C, the vehicle is detected entering the marked area, so assuming the vehicle stays in that particular area for more than a predetermined time, a violation will be automatically generated. Once a violation is detected, a violation alert will be generated with the vehicle details so that the relevant road management department can issue a violation notice.

[0073] refer to Figure 5 , another exemplary implementation of a system and method for detecting a traffic event is shown, comprising:

[0074] a traffic event detection processor for receiving a video stream from one or more camera units and a plurality of environmental data from one or more environmental sensors;

[0075] Processing the plurality of environmental data and the video streams by a plurality of traffic event detection modules, each module being configured to process the environmental data or the video stream to detect, track or monitor the type of traffic events, and

[0076] The detected, tracked or monitored traffic events are processed to generate one or more event alerts.

[0077] In this embodiment, the traffic event detection processor 502 is connected or communicated with one or more cameras 508, each of which is used to capture the position, area, space or scene where a traffic event or a traffic-related event monitoring, tracking or surveillance is required. In these areas, spaces or positions, objects may appear in traffic in a static or passing form. These objects may include, but are not limited to: personnel, vehicles, animals, articles, packages, products or any other objects that may be static or flow over time in this area. The traffic event detection processor 502 is also used to connect or communicate with one or more environmental sensors 506, each of which is used to detect environmental data related to the state or condition of the environment. Environmental data, which may include independent or correlated variables, is used to measure or represent specific attributes or characteristics of the environment separately over time.

[0078] Preferably, the traffic event detection processor 502 is implemented in the form of software or hardware or a combination of both, and can be implemented in a similar Figure 1 The processor 502 can also be implemented in whole or in part to operate on a cloud-based server or provided as a cloud-based service, so that the user can use the processor 502 from any location with a data connection. The processor 502 can also communicate with other devices, including cameras 508, sensors 506, edge devices 504, or communicate with users or other user-commanded computing devices through interfaces or gateways. If necessary, the processor 502 can also be reprogrammed based on the user's preset conditions to perform additional functions.

[0079] like Figure 5 As shown, the traffic event detection processor 502 is used to process image streams from one or more cameras 508 and environmental data received from sensors 506 to determine or detect one or more traffic events. These events may be related to various events related to traffic areas or places, such as roads, sidewalks, paths, tracks, paths, trails, crowds, spaces, parks, areas, rooms, halls, entrances, doorways, exits, facilities, stations, traffic signal areas, parking lots, buildings, etc., and can be used for crowd management, traffic management, law enforcement, accident or incident detection and response, safety or security assessment, or for preventing or convicting crimes, overcrowding, safety or security risks, etc. As a non-exclusive list of some practical examples, the traffic event detection processor 502 can be used to:

[0080] 1. Perform intelligent traffic management: In an intelligent traffic management system, the traffic detection processor 502 can use one or more of its traffic event detection modules 510 to detect and track cars, bicycles, unicycles, scooters, buses, trucks, motorcycles, or other vehicles and estimate their speed and trajectory. These estimates can also trigger alarms or violations, such as violations about incorrect driving directions or failure to comply with speed limits. Edge computing can also be used to process this data in real time, providing insights into traffic patterns and enabling traffic lights to be optimized to reduce congestion.

[0081] 2. Perform retail analytics: In a retail store, the traffic detection processor 502 can use one or more of its traffic event detection modules 510 to count the number of people in the store, estimate their individual postures, and track their movements. Edge computing can also be used to process this data in real time, providing insights into customer behavior and enabling retailers to optimize store layouts and product placement.

[0082] 3. Perform smart parking management: In a smart parking management system, the traffic detection processor 502 can detect and track cars and identify license plates using one or more of its traffic event detection modules 510. Edge computing can also be used to process this data in real time, providing an understanding of parking patterns and enabling parking spaces to be optimized to reduce congestion.

[0083] 4. Perform industrial automation: In an industrial automation system, the traffic detection processor 502 can use one or more of its traffic event detection modules 510 to detect and track objects, estimate human posture, and monitor their movement. Edge computing can also be used to process this data in real time, provide insights into the production process and enable the production line to be optimized to improve efficiency.

[0084] 5. Perform security supervision: In a security monitoring system, the traffic detection processor 502 can use one or more of its traffic event detection modules 510 to detect and track people and cars, identify license plates, and estimate the posture of any detected person. Edge computing can also be used to process this data in real time, providing insights into security threats and enabling security measures to be optimized to improve safety.

[0085] In this embodiment, the traffic event detection processor 502 is used to receive input from one or more camera devices 508, each of which is used to capture one or more image streams representing specific areas where people, animals, vehicles (including all types of vehicles, bicycles, motorcycles, scooters, electric scooters, "hover boards", unicycles, gyroscope-enabled transportation devices, autonomous vehicles, autonomous aerial vehicles of objects UAVs, items, products, or other objects may be placed or flow or travel through. In these areas, traffic events may occur from time to time or regularly over a period of time. These events can be predictable events or unpredictable events. Therefore, in some areas, it is preferred to continuously monitor or track events and issue an alarm when an event or incident is detected, or generate an alarm for a certain result or observation as needed (e.g., the total number of people, vehicles or other objects, the speed of vehicles, the direction of travel of vehicles, elevation, etc.).

[0086] In addition to one or more cameras for capturing the image streams of these traffic areas, environmental sensors may also be provided in the environment to determine and record environmental data or attributes. Such sensors may include temperature sensors, light sensors, humidity sensors, moisture sensors, wind sensors, sound sensors, or detectors. Each of these sensors is used to detect specific attributes, characteristics, or variables of the environment, and any attributes, characteristics, or variables that can be changed within the environment may also be provided to the traffic event detection processor 502 as environmental data to assist in detecting any traffic events.

[0087] In these embodiments, environmental data is also used to judge traffic incidents or accidents, which has advantages, because environmental conditions may affect the behavior of people in the environment or the behavior of vehicles (operators thereof). In the example of vehicle traffic management, adverse weather events, such as rainy days or haze weather, may reduce visibility and cause poor road conditions. Such an environment may affect the driving mode of vehicles on specific roads. By accessing these environmental data, the traffic incident detection processor 502 can generate an alarm of adverse weather conditions so that system users, such as accident response teams, can receive the alarm of potential traffic accidents in time. The traffic incident detection processor 502 can also be used to adjust its operation, especially sensitive to traffic speed, to monitor or predict traffic accidents or events. By starting other traffic incident detection modules 510, the processor can monitor potential accidents or track specific events. Similarly, in crowd control applications, due to weather conditions (such as extreme high temperatures) may change the behavior of the crowd, causing the crowd to become more excited or aggressive. In this case, the traffic event detection processor 512 can generate an alarm indicating that the crowd may become more aggressive and alert human supervisors to this possibility, while activating other traffic event detection modules 510 to detect aggressive or criminal behavior.

[0088] Preferably, the environmental sensors 506 and camera units 508 are connected and communicate with the traffic event detection processor 502 through the edge device 504, although in some examples, the sensors 506 and camera units 508 may also be selectively connected directly to the traffic event detection processor 502. In these example embodiments, there may be multiple edge devices 504 in the environment that will monitor the environment, and each device 504 may be used to provide processing capabilities for processing video data or environmental data from sensors, or control cameras 508 and sensors 506. This is advantageous because the edge devices 504 can distribute the processing of information and the control of the camera units 508 and environmental sensors 506, thereby reducing the use of network bandwidth when transmitting such information to the traffic event detection processor 502. The edge device 504 can also provide autonomous processing and control of the camera or sensor unit, including turning the camera or sensor on or off and controlling the camera or sensor to listen for or detect or capture specific events, so that when the connection with the traffic event detection processor 502 is interrupted or disconnected due to unforeseen events, the network of cameras 508 and sensors 506 can still maintain normal operation.

[0089] like Figure 5 As shown, the camera 508 is used to capture and transmit an image stream, and can be arranged as a single camera, a camera array, or a combination of a single camera and a camera array. In some examples, the camera 508 can include any pre-existing network camera system that provides a real-time streaming protocol (RTSP) video stream. The video stream can be used as a data source for the edge device 504, or in some examples, as redundancy, the video stream can be transmitted to the traffic event detection processor 502 for processing.

[0090] In some examples, different resolutions and frame rates (FPS) may be required based on user needs. Preferably, the camera is capable of providing at least a stable 1080P resolution and 20 frames per second (FPS) camera video stream, although higher resolutions (such as 2K, 4K) and higher frame rates may also be used to obtain more accurate results and better image quality depending on the processing power of the edge device 504 used.

[0091] In some examples, the image stream of the camera system 308 can be previewed by the user in the cloud web application, whereby the user can connect and view the image stream. The cloud web application can transmit the video stream from the edge device 504 or the traffic event detection processor 502 to the user through the cloud web application, thereby allowing the user to view the image stream as needed.

[0092] In this embodiment, the environmental sensor 506 may include any sensor that detects or acquires properties or characteristics in the environment, including sensors such as light, humidity, temperature, wind speed, sound, air quality, chemical composition, etc. These sensors can support common industrial communication protocols such as Modbus, NB-IoT, and LoRaWAN to help achieve compatibility and connectivity. The sensor data obtained from the environmental sensor 506 can be used as a data source consumed by the edge device 504. The edge device 504 can act as a gateway and master device to manage different sensor devices under its local area network (LAN). These sensors can be connected to the edge device 504 via a wired or wireless connection as needed.

[0093] In some example embodiments, edge device 504 may be a computing device with communication capabilities designed to connect and interface with each camera 508 and sensor 506 and connect these cameras 508 and sensors 506 to traffic event detection processor 502. Edge device 504 may provide control or data processing capabilities, or may provide only minimal processing and control capabilities as needed.

[0094] In some examples, edge device 504 can be used to process video stream data from camera 508. Such processing can include compression of video stream transmission and computer vision processing to perform object detection or object classification. In these configurations, the edge device can access an object detection or classification system, which can include a traditional object detection system or a learning network type classification system, such as a convolutional neural network (CNN). Preferably, any such network can be used to provide a lightweight implementation so that edge devices with lower computing power can use these networks accordingly.

[0095] Once the image is processed, the edge device 504 can send different kinds of metadata, such as object counts, timestamps, to the remote cloud server or traffic incident detection processor 502. In this embodiment, environmental data obtained from environmental sensors 506 can be transmitted through the edge device 504, so that the edge device 504 can act as a gateway to aggregate data from different environmental sensors 506 and send it to the remote cloud server or to the traffic incident detection processor 502.

[0096] In an exemplary deployment, the inventors selected the Nvidia Jetson & Orin series, which provide onboard computers with GPUs, for use as implementations of the example edge device 405. These hardware selections are suitable for processing video streaming data. Nvidia's Deepstream library can also be used to perform certain video processing and object detection tasks. On the edge device 504, a trained AI model running on an Nvidia GPU can also be implemented, which is accelerated by TensorRT. In these examples, the environmental data collected by the edge device 504 can also be sent to a cloud server via the MQ Telemetry Transport (MQTT) protocol, which is a message protocol for lightweight data exchange on the Internet.

[0097] like Figure 5 As shown, once the image stream from the camera 508 and the environmental data from the environmental sensor 506 have been acquired and processed by the edge device 504 (in the example where the edge device 504 performs the processing), the data is then transmitted to the traffic event detection processor 502, which is used to process the image stream from the camera 508 and the environmental data to determine one or more events. In this example, an embodiment of the traffic event detection processor 502 includes a plurality of traffic event detection modules 510, each of which is used to detect a specific type of traffic event or accident, or the presence, number, behavior, or condition of a specific object. As an example, Figure 5 As shown, multiple traffic event detection modules 510 include but are not limited to:

[0098] Object Detection Module - used to determine whether there are any specific objects, such as people, vehicles, objects, animals, etc., in a specific area within a specific time, and generate event or accident alerts for the existence of the object.

[0099] Object Classification Module - used to determine what an object is once it is detected and to obtain any information associated with the object. This can be used to identify if an object is a car, truck, van or other type of vehicle, or to determine if a person is an adult or a child. Multiple object classification modules can be implemented to detect specific objects or details of interest. These can include another object classification module that is used to determine the registration number of a vehicle, or the presence of people or cargo within the vehicle, etc.

[0100] Motion Tracking Module - Used to determine the speed and direction of objects moving within a monitored space. This module can track objects and determine the path or direction they move over time. This information can then be used to determine if an object (such as a vehicle or person) enters or exits a specific monitored space.

[0101] Dangerous / suspicious object detection module - used to detect dangerous or suspicious objects in the monitored space. It can use the object classification module to determine whether a dangerous object is present, or other criteria or logical rules to determine whether an object is suspicious, such as by detecting objects placed where they should not be (for example, luggage on a seat, or an obstacle placed in the middle of a road or passage).

[0102] Incident Detection Model - used to detect specific incidents or events and to identify if a specific incident has occurred. This can be done by first using the Object Detection module to determine if an object is present and if that object overlaps with another detected object, then whether a collision has occurred. Other objects that collide together, such as people, may indicate that a crowd is gathering, which could trigger an alarm for a crowd control application.

[0103] Behavior Detection Module - used to detect the behavior of monitored people or objects. This can include tracking the movement of objects or the movement of people using posture estimation methods. Specific postures or actions may indicate normal or irregular movement in people or objects, thereby determining that activities are occurring, and these irregularities may be alerted to the user.

[0104] Additional object traffic event detection modules can be implemented to detect additional events based on the image stream and sensor data expected by future users. Different modules can also be used by the traffic event detection processor 502 or other traffic event detection modules 510 to detect or track specific objects, accidents or events. In addition, each traffic event detection module can also use the image data of the camera 508 and / or the environmental data of the sensor 506 to determine the object, accident or event.

[0105] Preferably, each of these traffic event detection modules 510 is used to process environmental data as well as image streams to achieve its specific detection goals. In addition, in some advanced embodiments, the traffic event detection processor 502 can also utilize individual traffic event detection modules 510 and, based on the output of each module 510, pass the input to other traffic event detection modules 510. Thus, for example, an object detection module can be utilized to detect the presence of motor vehicles at a particular location over time, and any output from the particular motor vehicle detection module will be received by the traffic event detection processor 502 and further input into an object counting module to determine the number of vehicles passing through or present in a particular area. Each traffic event detection module can also use environmental data such as sound to determine whether these events have occurred.

[0106] Similarly, the traffic event detection processor 502 can use a behavior detection module to detect the behavior of individuals in a specific traffic area, and further process based on their behavior, and determine whether any potential danger or accident events have occurred in the specific monitoring area through an accident detection module or a dangerous object detection module.

[0107] Preferably, the implementation of the traffic event detection module 510 is personalized so that the traffic event detection processor 502 can call each individual module in any specific order as needed. Therefore, the traffic event detection processor 502 can be implemented through specific logic, utilizing various traffic event detection modules, and more modules can be integrated in the future to provide additional functionality. In these examples, in order for the processor 502 to use multiple modules 510 to determine the occurrence of an event, the image stream and environmental data must be synchronized so that each module 510 can take into account the time point when the data occurred. In order to achieve this synchronization, the timestamps of the image stream and environmental data must be consistent when processed. Even if the processing of the image stream and environmental data may be distributed over different time ranges in the process of calling different modules to perform analysis, the consistency of timestamps is still the key to ensure synchronization and accurate event detection.

[0108] As shown in this example, the traffic event detection processor 502 is also used to operate when one or more specific traffic events are detected. These detected events can be provided as real-time alerts 512 to users in monitoring positions to monitor or track individual targets, behaviors, or events in a specific area so that appropriate measures can be taken for prevention or control. Users can also view these events through the cloud server using live feeds 516 from various cameras or sensors, which can also be provided to users through the cloud server. Users can also select specific sensors 506 or cameras 508 and control these cameras or sensors to obtain specific data from each device.

[0109] Additionally, the Traffic Incident Detection Processor allows the user to group sensors or cameras based on their location or sensors or cameras of interest so that the user can continue to monitor the groupings of these cameras or sensors. This is achieved through the Traffic Incident Detection Processor, which is able to communicate with the edge device, enabling the user to specify and control specific cameras or sensors.

[0110] In addition, the traffic incident detection processor is also used to store and record all data from the cameras and sensors, as well as the input and output of each traffic incident detection module 510, into a database 518, and provide these data as reports 514 for subsequent reference or research. This is beneficial for traffic management or safety audits, whereby existing records can be checked to improve and further enhance safety, as well as to handle or determine liability for insurance claims.

[0111] refer to Figure 6 , a block diagram of an example embodiment of a traffic event detection processor is shown, and its communication with a set of traffic event detection modules. As shown in this exemplary embodiment, the traffic event detection processor 502 is configured to operate on a cloud-based device 602, so that each event detection module 510 can also exist in the cloud or as a service of a remote server connected to the cloud 602. This is advantageous because each module 510 can be deployed in a different geographical location from the traffic event detection processor 502, while allowing the traffic event detection processor 502 to call each detection module 510 at any time as needed.

[0112] Preferably, additional traffic event detection modules 510 may be added to the list in the future, and specific logic or methods may be further implemented in the traffic event detection processor 502 to determine which modules 510 to use in different situations. In this particular exemplary embodiment, a non-exhaustive list of traffic event detection modules 510 may be implemented now or in the future based on the user's preferences using a specific architecture of the system for detecting traffic events. The traffic event detection modules 510 include object detection modules, object counting modules, motion tracking modules, dangerous or suspicious object detection modules, accident / incident detection or prediction modules, and behavior detection modules or other additional modules to allow the traffic event detection processor to implement and utilize the extension of additional traffic event detection modules in order to detect any new accidents or events desired by the user.

[0113] In one example, the object detection module 604 is used to detect objects such as cars, people, or animals in a specific space. This can utilize machine vision provided by a trained convolutional neural network (CNN) to identify specific objects by classifying the objects detected in the image stream. Once these objects are detected, the specific object detection results based on their location and time will be returned to the traffic event detection processor 502. In turn, when the traffic event detection processor 502 is used to identify whether certain objects should appear in a specific area, the processor 502 may generate an alarm flag for potential violations or alarms related to whether the object should appear in the location. This type of object detection can be applied to vehicles in prohibited areas, and people in prohibited areas or gates to detect whether objects are approved or authorized to pass through gates or areas.

[0114] like Figure 6As shown, an object counting module 606 is also provided, which is used to count specific objects that have passed through a specific space or in a specific place. One such example includes a people counting function, which is used for crowd control by counting the number of people who have passed through a specific place or area. This function can be performed by first calling the object object measurement module to determine the number of people in a specific space or the number of specific people passing through a specific space. Similarly, the object counting function can be used to count the number of vehicles passing through a specific doorway or area or parked in a specific area. This is an example in which the traffic event detection processor 502 can be programmed to utilize each traffic event detection module in a logical manner to achieve a specific function required by the user.

[0115] The motion tracking module 608 as shown in the figure is used to track the motion of a specific object. For example, an object such as a vehicle or a person may travel in a specific direction or against a specific direction, so it is desirable to identify the direction and speed of the object. Module 608 can use an object detection system to subsequently track the object through a specific space, determine the path of the object, identify the movement or path of the object, and notify the traffic event detection processor of the speed and path taken by the object being tracked. This is useful because different objects, such as vehicles or people, can move from different spaces to another space, so when a vehicle or a person moves, the traffic event detection processor 502 can continuously track the object when the object moves around the monitoring area of ​​definition, thereby allowing identification of its last known position.

[0116] In this embodiment, the dangerous / suspicious object detection module 610 can be used to determine specific areas or objects that are considered dangerous or suspicious. For example, the monitored or monitored area may include a doorway to a specific event, in which case objects that are considered dangerous may be prohibited from entering the specific area. The dangerous / suspicious object detection module 610 is used to detect such objects so that an alarm can be generated for the response team to deal with these objects. Other objects, such as explosives and contraband, can also be detected and dealt with.

[0117] The accident detection module 614 is used to detect accidents or events that can be detected or predicted in a specific space. The accident detection module 614 can be implemented by implementing specific rules for object detection, which are used to detect interacting objects, thereby indicating that an accident or event may have occurred. This may include two objects colliding in a space, or approaching each other. Environmental sensor data, such as sound or wind speed, can also be used to determine whether a specific event has occurred. For example, it may include a traffic accident in which a vehicle and a pedestrian collide, or a falling object hits other objects.

[0118] like Figure 6As shown, the behavior detection module 612 is a detector of the behavior of individuals that have been detected in a particular area under surveillance. The module can use posture estimation methods to identify the posture of individual users detected in a particular space. When the posture may indicate some type of confrontation, such as a fighting posture or interpersonal interaction, may indicate the presence of a conflict or some situation, the behavior detection module can flag these detections to the traffic incident detection processor 502 for further processing or to alert the response team.

[0119] Embodiments of the system and method for detecting traffic events may be advantageous because they may provide, without limitation, the following functionality:

[0120] ● Provide customers with a variety of ready-made computer vision solutions, such as traffic analysis of vehicles / people / general objects, such as calculating the number of objects existing in a user-predefined area, the number of objects crossing a user-predefined line, and the length of time objects stay in an area.

[0121] Dangerous / suspicious object detection.

[0122] Incident detection based on preset criteria.

[0123] Pose estimation for action detection.

[0124] ● Collect environmental data from different types of sensors, such as temperature sensors, light sensors, and humidity sensors, for monitoring or recording.

[0125] Learning systems using object neural networks analyze collected environmental data to identify defect / anomaly detection or data pattern analysis.

[0126] ●Provide a user interface for visualizing and analyzing data collected in web applications hosted in cloud servers.

[0127] Group data by providing date time, smart filtering of device name, location name, etc.

[0128] Visualize data in different types of charts (pie charts, line graphs).

[0129] Generate reports based on preset templates in PDF format.

[0130] ●Provide a convenient user interface for users to manage the status of edge devices, cameras, and sensors.

[0131] Visualize and track device status in real time.

[0132] Get instant notification of device failures.

[0133] Group instruments based on location.

[0134] Users can preview the real-time or historical camera video streams.

[0135] In addition, embodiments of the system for detecting traffic incidents are also capable of generating "digital twins" for urban planning and other studies or analyses. Environmental data and events detected by the system can be stored to generate a virtual model of an area that has been captured by multiple cameras and sensors. The data can then be used to generate a virtual model of a specific area or space. In cases where the network of cameras and sensors extends over a large geographic area (e.g., an entire suburb, residential area, or geographic area of ​​a city), the captured data can be used to generate a virtual model for modeling, analysis, or simulation purposes. Such a virtual model of a physical system may be referred to as a "digital twin."

[0136] Digital twin technology has revolutionized the way we operate and manage complex systems. Digital twin models can be used to simulate, monitor and control systems in real time. One of the key benefits of digital twins is that they enable users, who may be consultants and planners, to understand the behavior, performance and maintenance requirements of a system without the need for physical intervention.

[0137] The integration of digital twin technology with computer vision and edge computing can open up new possibilities in a wide range of applications including smart cities, transportation systems, and retail stores. By using computer vision algorithms, edge computing, and digital twin technology, new systems can be built to automatically detect and track objects, recognize license plates, count people, and estimate their postures. Here are five examples of how the generation of a digital twin model can be used and presented in the system through an exemplary embodiment of a system for detecting traffic incidents:

[0138] 1. Intelligent traffic management: In intelligent traffic management systems, digital twin models can be used to create virtual copies of road networks, which can be used to simulate traffic flows and identify potential bottlenecks. Computer vision algorithms can be used to detect and track cars and estimate their speeds and trajectories. Edge computing can be used to process this data in real time, providing insights into traffic patterns and enabling traffic lights to be optimized to reduce congestion.

[0139] 2. Retail Analytics: In retail stores, digital twin models can be used to create a virtual copy of the store, which can be used to simulate customer behavior and optimize store layout. Computer vision algorithms can be used to count the number of people in the store, estimate their posture, and track their movement. Edge computing can be used to process this data in real time, providing insights into customer behavior and enabling retailers to optimize store layout and product placement.

[0140] 3. Smart parking management: In a smart parking management system, digital twins can be used to create a virtual copy of a parking lot, which can be used to simulate parking patterns and identify potential bottlenecks. Computer vision algorithms can be used to detect and track cars and identify license plates. Edge computing can be used to process this data in real time, providing an understanding of parking patterns and enabling parking spaces to be optimized to reduce congestion.

[0141] 4. Industrial Automation: In industrial automation systems, digital twin models can be used to create virtual copies of production lines, which can be used to simulate production processes and identify potential bottlenecks. Computer vision algorithms can be used to detect and track objects, estimate human postures, and monitor their movements. Edge computing can be used to process this data in real time, providing insight into the production process and enabling production lines to be optimized for improved efficiency.

[0142] 5. Security supervision: In security monitoring systems, digital twin models can be used to create virtual copies of monitored areas, which can be used to simulate security scenarios and identify potential threats. Computer vision algorithms can be used to detect and track people and cars, identify license plates, and estimate their postures. Edge computing can be used to process this data in real time, providing insights into security threats and enabling security measures to be optimized to improve safety.

[0143] Digital twins can also be integrated with computer vision and edge computing, which are powerful tools that can be used to build intelligent systems in a wide range of applications. By creating a virtual replica of a physical system and using computer vision algorithms and edge computing to process data in real time, users can understand the behavior, performance, and maintenance requirements of the system, allowing us to optimize the system and improve its efficiency and safety. The use of digital twin technology in a variety of applications is growing rapidly and is expected to become an important part of intelligent systems in the future.

[0144] Although not required, the embodiments described with reference to the accompanying drawings may be implemented as an application programming interface (API) or a series of libraries used by developers, or may be included in another software application, such as a terminal or personal computer operating system or a portable computing device operating system. Generally, since program modules include routines, programs, objects, components, and data files that help perform specific functions, those skilled in the art will understand that the functionality of a software application may be distributed among multiple routines, objects, or components to achieve the same functionality required herein.

[0145] It should also be understood that in the case where the methods and systems of the present disclosure are implemented in whole or in part by a computing system, any suitable computing system architecture may be used. This would include stand-alone computers, network computers, and dedicated hardware devices. Where the terms "computing system" and "computing device" are used, these terms are intended to include any suitable configuration of computer hardware capable of implementing the described functionality.

[0146] It will be appreciated by those skilled in the art that various changes and / or modifications may be made to the present disclosure as shown in the specific embodiments without departing from the spirit or scope of the present disclosure as broadly described. Therefore, the present embodiments are to be considered in all aspects as illustrative and not restrictive.

[0147] Unless otherwise indicated, any reference to prior art contained herein should not be taken as an admission that the information is common general knowledge.

Claims

1. A system for detecting traffic incidents, characterized in that: include: A media gateway for receiving a traffic video stream of a traffic area; a traffic incident detection processor, which is used to detect traffic incidents in the traffic area; An alert generation routine is used to generate an alert when the traffic event satisfies one or more traffic event conditions.

2. The system for detecting traffic incidents according to claim 1, characterized in that: in, The traffic event includes the existence of an object in the traffic area.

3. The system for detecting traffic incidents according to claim 2, characterized in that: in, The detected objects present in the traffic area are analyzed to determine their identities.

4. The system for detecting traffic incidents according to claim 3, characterized in that: in, The detected objects existing in the traffic area are analyzed to determine their object type, travel direction, movement status, speed, or any one or a combination thereof.

5. The system for detecting traffic events according to claim 4, characterized in that: in, The traffic area is analyzed to determine a number of objects detected within the traffic area.

6. The system for detecting traffic incidents according to claim 5, characterized in that: in, The one or more traffic event conditions include a type of traffic area.

7. The system for detecting traffic incidents according to claim 6, characterized in that: in, The traffic event condition includes a duration of existence of the object in the traffic area.

8. The system for detecting traffic events according to claim 7, characterized in that: in, The object is a vehicle.

9. The system for detecting traffic incidents according to claim 8, characterized in that: in, The alert is an alert of a traffic violation, and the alert of the traffic violation is further processed to generate a traffic violation prosecution procedure.

10. The system for detecting traffic incidents according to claim 9, characterized in that: in, The image stream of the traffic violation is stored and further associated with the traffic violation prosecution program.

11. A system for detecting traffic incidents, characterized in that: include: a traffic event detection processor for receiving a video stream from one or more camera units and a plurality of environmental data from one or more environmental sensors; Processing the plurality of environmental data and the video stream by a plurality of traffic event detection modules, each module being configured to process the environmental data or the video stream to detect, track or monitor the type of traffic events; as well as The detected, tracked or monitored traffic events are processed to generate one or more event alerts.

12. The system for detecting traffic events according to claim 11, characterized in that: Also includes: A live feed channel is used to direct the video stream or the plurality of environmental data to a user interface for user access.

13. The system for detecting traffic events according to claim 12, characterized in that: in, The one or more camera units or the one or more environmental sensors communicate with the traffic event detection processor via one or more edge devices, and the edge device is used to at least partially process the video stream or the multiple environmental data before transmitting the video stream or the multiple environmental data to the traffic event detection processor.

14. The system for detecting traffic events according to claim 13, characterized in that: in, The one or more edge devices are also used to at least partially control the one or more cameras or the one or more environmental sensors.

15. The system for detecting traffic events according to claim 14, characterized in that: in, The live feed channel is segmented into a plurality of channels, each of the channels being configured to contain the video stream or environmental data from a user-selected camera or environmental sensor.

16. The system for detecting traffic incidents according to claim 15, characterized in that: in, Outputs from the one or more of the plurality of traffic event detection modules are input into one or more other traffic event detection modules to detect traffic events.

17. The system for detecting traffic events according to claim 16, characterized in that: in, The traffic detection processor is used to select one or more traffic event detection modules to detect traffic events.

18. The system for detecting traffic events according to claim 17, characterized in that: in, The traffic incident detection module includes one or more of the following: Object detection module; Object counting module; Motion tracking module; Dangerous or suspicious object detection module; Accident / incident detection or prediction module; as well as Behavior detection module.

19. The system for detecting traffic events according to claim 18, characterized in that: in, The processor and the module are deployed on a cloud-based architecture.

20. The system for detecting traffic events according to claim 19, characterized in that: in, The detected events, video streams and environment data are stored on a database and are used to be retrieved to generate one or more virtual models, each of which is used to model a traffic event within the environment.

Images