Artificial Intelligence (AI)-based mobile real-time 360-degree traffic data and video recording and tracking system and method

Abstract

An artificial intelligence (AI)-based mobile real-time 360-degree traffic data, video recording, and tracking system and method is disclosed. More specifically, a system of video cameras and other data sensors is mounted on a vehicle to capture information from all four sides (360 degrees) of the vehicle. The captured information is input into a computer programmed with artificial intelligence, which analyzes the information for potential traffic violations and reports the information to authorities.

Description

Technical Field

[0001] The present invention relates to a mobile real-time 360-degree traffic data and video recording and tracking system and method based on artificial intelligence (AI). More specifically, the present invention relates to a system of video cameras and other data sensors mounted on a vehicle that captures information in all directions (360 degrees) from the vehicle. The present invention further relates to inputting the detected information into a computer including a computer system programmed using AI, analyzing information about possible traffic violations, and reporting that information to the authorities.

Background Art

[0002] Traffic violation detection systems known today utilize cameras, lasers, and radars to detect, in addition to license plate recognition, speed violations, stop sign violations, red light violations, bus lane violations, wrong-way driving, left turn violations, and parking violations. Since such systems are attached to specific locations, the scope of information is limited to the vicinity of the attachment location and is usually stationary. Some systems, such as the LogiPix® system (<www.logipix.com>), further include computer programming that analyzes information about specific violations, which can be exported to the authorities.

[0003] There are also violations that can be seen by checking the video, such as ignoring red lights and inappropriate right or left turns, but other violations require additional data analysis. For example, tailgating and drunk driving require analysis of other factors such as sudden steering, speed violations, and slow driving over a certain period of time. Also, dangers on the road such as potholes and floods may not be easily detected by stationary cameras.

Summary of the Invention

[0004] The system and method of the present invention comprises multiple cameras and other data sensors mounted on a vehicle to collect information about other vehicles and road conditions in the vicinity of the vehicle. This information is fed into a computer system programmed using artificial intelligence (AI) to analyze traffic violation information and report it to the authorities along with background information. The cameras are installed around the vehicle and can record 360 degrees of surrounding vehicles. The cameras can store information in memory and transmit it to a remote computer in real time or when a Wi-Fi® signal is available. Both audio and video are recorded by the cameras. In addition, sensors such as radar, LIDAR, and lasers detect speed and transmit this information to the remote computer as well. The timing of the audio and video from the cameras and the data sensed from the other sensors are time-synchronized.

[0005] The programmed computer may be local to the vehicle, or it may be located in a remote computer. The computer is programmed to analyze the received information regarding traffic violations. The conclusion that a traffic violation has occurred, along with the underlying information, is transmitted from the programmed computer to the authorities or to other individuals or entities designated by the system user.

[0006] Furthermore, road hazards such as potholes and flooding can be detected by cameras inside the vehicle and reported to road safety authorities. [Brief explanation of the drawing]

[0007] The present invention will be described with reference to the following drawings.

[0008] [Figure 1] Figure 1 is a schematic diagram of a system in which a programmable computer is installed in a vehicle. [Figure 2] Figure 2 is a car orthogonal projection of a vehicle showing the arrangement of the camera and module according to one embodiment of the present invention. [Modes for carrying out the invention]

[0009] The system and method of the present invention comprises a plurality of cameras and other data sensors mounted on a vehicle to collect information about other vehicles and road conditions in the vicinity of the vehicle. In one embodiment, cameras are mounted on the vehicle facing outwards on the bow (front), driver's side (port), rear (stern), and passenger side (starboard). Additional cameras may be mounted on the vehicle from other locations and may include cameras for recording the interior of the vehicle. In one embodiment, cameras recording the interior of the vehicle record vehicle data such as speed and direction. In one embodiment, cameras record only video. In one embodiment, cameras record audio and video. In one embodiment, cameras capture vehicle license plates, and in one embodiment, cameras capture street names. In one embodiment, cameras capture images of vehicles in the vicinity of the vehicle on which the system is installed. In one embodiment, GPS data of the vehicle on which the system is installed may be recorded.

[0010] In one embodiment, information from the camera and sensors is transmitted wirelessly to the system for storage in a database. In one embodiment, one or more of the camera and sensors are hardwired to the system.

[0011] In addition to cameras, other sensors can also be mounted on the vehicle. Such sensors include lasers, radar, and / or LiDAR. In one embodiment, the laser, radar, and / or LiDAR detect the vehicle's speed at multiple time data points.

[0012] In one embodiment, the system becomes active when the vehicle on which it is installed starts up. In another embodiment, the system needs to be started before it becomes usable.

[0013] The information may be enhanced with information from other sources, such as weather forecasts, data acquired from stationary cameras and sensors, and data acquired from aerial sources. In one embodiment, the aerial source may be a drone, and in one embodiment, the information is synchronized with publicly available information, such as mapping software like Google® Maps.

[0014] Information detected and recorded by cameras and sensors is supplied to a programmed computer. Information from various cameras and sensors is time-stamped to synchronize the time the information was detected. The computer is programmed to analyze traffic violation information, which can be reported to authorities along with background information. The computer may be programmed using machine learning (ML) algorithms or artificial intelligence (AI). The computer may also be programmed using relevant standards and laws of the geographical area where the information is recorded. Such relevant standards and laws may include speed limits, laws regarding helmet use on motorcycles, and parking restrictions in various locations. The computer can be programmed using any programming language currently known or to be developed in the future. The system may reside on any type of computer device, including desktop computers, mainframe computers, mobile applications on smartphones, and mobile applications on smart tablets and notebooks. The system may run on a web-based application designed using, for example, HTML, CSS, jQuery, Javascript, or PHP. Information may be stored in a backend database using, for example, MySQL.

[0015] In one embodiment, the programmed computer may include a system-on-a-chip ("SOC"). In one embodiment, the programmed computer may include a computer programmed to emulate the SOC.

[0016] The computer will be programmed using AI that provides multiple, varied, conditional datasets of data considered to be regular driving patterns and baseline data points. These baseline data points provide the programmed computer with the legal conditions for specific rules, such as driving at an appropriate speed along a highway. The datasets will consist of examples that are shown as "negative events" or non-violations. The datasets will further consist of examples that are shown as "positive events." Based on the datasets, the programmed computer will "learn" to distinguish between negative and positive events.

[0017] This "learning" process will be repeated for each individual violation and in accordance with the rules and laws applicable in various geographical jurisdictions. Furthermore, if rules or laws change, the programmed computer may be reprogrammed in a similar manner to reflect those changes.

[0018] As the programmed computer "learns," the results are ultimately only randomly reviewed by humans to ensure they are operating within programmed parameters and to minimize false positive events.

[0019] When the system determines, according to its programming, that a "positive event" has occurred, the programmed computer will review the pre- and post-event timeframes of the positive event and blend them with the relevant cameras and sensors to create a video of the positive event. The video may also include further timeframes before and after the positive event. In addition, other information from the sensors may be associated with the video as files, showing information such as license plate information of surrounding vehicles.

[0020] Initially, in one embodiment, a human operator of the system would label positive and negative events and manually match videos of these events. The manually matched videos would then be provided to a programmed computer as examples of "positive events" and "negative events" to further enhance the system's ability to distinguish between them.

[0021] The system then assigns a number to each violation and sends it as a link to the assigned authority, which can then review it and issue a citation. The data may be stored for a predetermined period of time on a server requested and approved by the authority in its geographical jurisdiction. The data may be made accessible only to the authority and the registered owner of the vehicle in the video. The links provided to the authority may be encrypted.

[0022] In one embodiment, neither the data reviewer, the AI ​​programmer, nor any of the individuals involved in data collection and matching have access to the personal information of those represented in the collected information. In one embodiment, the recorded information is considered to be in the public domain, and the various tools used for data collection may be available to the general public.

[0023] Violations that can be detected may be easily identified by reviewing video and / or laser / radar / LIDAR information, such as improper lane changes, improper U-turns, illegal left and right turns, ignoring red lights and stop signs, improper parking, riding without a helmet for motorcyclists, speeding, and failing to yield to pedestrians. Other violations can be detected by combining and analyzing information from various cameras and sensors. For example, drunk driving can be analyzed based on various factors such as whether the vehicle is going too fast or too slow, whether it crosses the center line, whether it moves into an adjacent lane, and whether it makes sudden turns. Stalking can be detected by detecting the relative speed of vehicles and the distance over a period of time.

[0024] The camera and sensors are attached around the vehicle and can record the surrounding vehicles in 360 degrees. The camera stores the information in the memory mounted on the vehicle. This information can be transmitted to a remote computer in real time or when a Wi-Fi (registered trademark) signal is available. In one embodiment, information that may be subject to privacy laws such as GDPR can be transferred in real time when the camera and sensors are physically communicating with a database and / or a programmed computer.

[0025] The conclusion made by the programmed computer that a traffic violation (“positive event”) has occurred is transmitted, together with the basic information, from the programmed computer to other persons or entities designated by the authorities or the users of the system. In one embodiment, the authorities are local police or state police. In one embodiment, the information is transmitted to other agencies such as insurance companies or the National Highway Traffic Safety Administration (NHTSA).

[0026] Furthermore, the information can be stored for later investigation and research. For example, hazards on the road such as potholes and floods can be detected by the cameras in the vehicle and reported to the road safety authorities. Also, by searching the recorded information taken around the area when a crime or an incident has occurred, eyewitnesses can be found.

[0027] Furthermore, the recorded information can be used in the prosecution of traffic violations and other violations if measures are taken to prove the authenticity of the information, including the chain of custody required by the authorities using the information in this way.

[0028] The information can be encrypted using currently known and later standardized encryption protocols.

[0029] Violations and occurrences that can be observed and / or detected by the analysis of the stored information include the following. - Tracking of license plates; - Driving with an expired license; - Face recognition / tracking; - Traffic flow data; - Detection of drunk drivers; - Traffic violations / crimes; - Poor / illegal maintenance; - Road accidents and / or road hazards; - Public safety hazard; - Littering; - Using a mobile phone while driving a car; - Illegal lane changes, including illegal overtaking of a moving vehicle; - Domestic violence; - Riots on the streets; - Chasing other vehicles at a dangerous distance; - Speeding; - Reckless driving, reckless and dangerous behavior; - In addition, it is possible to detect specific use case scenarios upon request.

[0030] Recorded information and analysis of such information by programmed computers may be transmitted to local authorities on an ongoing basis or upon request. Recorded information and analysis of such information by programmed computers may be provided to local authorities in batches. Authorities may also receive signals indicating that certain information, such as traffic accidents or dangers to public safety, requires immediate attention.

[0031] Looking at Figure 1, a schematic diagram of a system-programmed computer inside a vehicle is shown. This system includes a case 1 enclosing a CPU 2, a power supply 3, RAM 4, memory 5, a system fan 10, and a power connector 11. In the embodiment shown in Figure 1, the system further comprises a cellular modem 6, a wireless network module 7, multiple camera connectors 8 to which multiple cameras 12 are mounted, multiple antennas 9, a Bluetooth® module 13, a GPS module 14, an accelerometer / gyroscope module 15, a GPS antenna 16, a radar 17 and radar connector 21, a laser 18 and laser connector 22, and a LiDAR 19 and LiDAR connector 23. The system may further include hardwired connections 20 for mobile communication devices.

[0032] In one embodiment, the camera, sensors, and memory are permanently located in the vehicle. In this embodiment, the programmed computer is located remotely from the vehicle.

[0033] In one embodiment, the memory 5 is comprised of a hard disk drive. In another embodiment, the memory 5 is comprised of a solid-state drive. In another embodiment, one or more of the multiple cameras 12 are comprised of high-resolution cameras.

[0034] Various elements of the system communicate with each other according to standard protocols, and information is stored and received according to standard data file types.

[0035] Figure 2 is a orthogonal projection of a vehicle showing the arrangement of cameras and modules according to one embodiment of the present invention. The vehicle 200 is shown in a top view, right side view, left side view, rear view, and front view. The front camera 205, rear camera 210, side camera (driver's side) 215, side camera (passenger's side) 220, laser 225, LIDAR 230, radar 235, GPS antenna 240, and cellular antenna 245 are mounted on the vehicle 200 as shown in this embodiment. Multiple cameras and sensors can be used as desired. In one embodiment, an infrared lamp module may be integrated with one or more of the front camera 205, rear camera 210, side camera (driver's side) 215, and side camera (passenger's side) 220.

[0036] Examples

[0037] A vehicle equipped with the system described may be stopped at a red light in lane 2 of a six-lane intersection. A vehicle driven by a third party may approach from behind in lane 1 and proceed through the red light while remaining stopped. This system can record events from cameras installed at the rear, sides, and front. Video recordings of events with timestamps can be stored in memory. A programmed computer can analyze the events by combining the video recordings according to the timestamps to obtain a sequence of events detailing the third party's running of the red light. The applicable agency can receive a notification of the violation in a video and data format showing the approaching third party vehicle from the rear camera view. Footage from the side cameras can show the third party vehicle passing the vehicle equipped with the system. Next, the footage from the front camera can show that the third party vehicle is committing a violation by running a red light. Footage from the various cameras can be combined according to the timestamps to create a single, unified video. The agency can then decide whether to pursue legal action against the owner of the third party vehicle for the traffic violation.

[0038] Vehicles with the system installed can collect information that may indicate a parking violation. For example, a programmed computer may determine a first line within the frame, which represents the nominal direction of the parking space in question. The programmed computer can then detect the presence of a vehicle in the parking space. The programmed computer may further determine a second line within the frame, which may represent the orientation of the detected vehicle. The programmed computer may calculate the angle between the first and second lines. Based on this calculation, the programmed computer can determine whether the detected vehicle is in violation of parking regulations based on the calculated angle. The video and computational analysis can be provided to the vehicle owner and local authorities to decide whether to pursue the parking violation.

[0039] Although the present invention has been described with reference to specific embodiments and applications, numerous variations and modifications can be made by those skilled in the art without departing from the spirit and scope of the invention as claimed. Therefore, the scope of the invention should be determined by reference to the claims.

Claims

1. A computer system for detecting traffic violations using artificial intelligence, A first vehicle (200) including the interior and exterior, Multiple cameras mounted on the first vehicle, facing away from the interior of the first vehicle, recording 360° of third-party vehicles and road conditions surrounding the outside of the first vehicle, and recording one or more streams of video information in digital format, (12, 205, 210, 215, 220) Multiple sensors mounted on the first vehicle, each of which records one or more streams of sensor information in digital format, (17, 18, 19, 225, 230, 235) A medium for storing digital format data, which is located inside the first vehicle, and which communicates with the plurality of cameras and the plurality of sensors, A non-temporary memory device that embodies one or more routines capable of operating to detect objects using an artificial neural network, the non-temporary memory device comprising a receiver module, a detector module, and a logic module, A CPU that communicates with the non-temporary storage device, and is capable of operating to execute one or more routines embodied in the non-temporary storage device, and Equipped with, One or more streams of video information include the activities of third-party vehicles and persons located outside the first vehicle, and / or road conditions outside the first vehicle. The digital format data stored in the data storage medium is transmitted to the receiver module. The receiving module detects one or more images in the received data in a digital format that includes video information. The detector module selects one or more events of interest from the received data in a digital format that includes video information. The logic module determines whether the one or more events of interest selected by the detector module include one or more traffic violations committed by the third party's vehicle and / or person. If the logical module determines that one or more events of interest include one or more traffic violations, the logical module identifies a predetermined time frame before and after the one or more traffic violations and combines it with one or more streams of video information to create a video of the one or more traffic violations. Computer system.

2. One or more of the aforementioned cameras include a high-resolution video camera. The system according to claim 1.

3. The one or more sensors include radar, laser, LIDAR, or a combination thereof. The system according to claim 1.

4. The aforementioned one or more traffic violations include, but are not limited to, driving under the influence of alcohol, driving under the influence of alcohol, committing one or more crimes, having a poorly maintained vehicle, illegal maintenance, causing a traffic accident, endangering public safety, littering, using a mobile phone while driving, illegal lane changes, illegal overtaking of a moving vehicle, domestic violence, road rioting, pursuing another vehicle at a dangerous distance, speeding, reckless driving, reckless endangerment, and combinations thereof. The system according to claim 1.

5. The video information identified as containing one or more traffic violations is transmitted to the local authorities. The system according to claim 1.

6. The video information and sensor information are timestamped, and the sensor information corresponding to the timestamp of the video information identified as containing one or more traffic violations is transmitted to the local authority. The system according to claim 5.

7. The aforementioned traffic violations include license plate tracking, facial recognition, facial tracking, traffic flow data, and combinations thereof. The system according to claim 1.

8. The aforementioned non-temporary storage device further comprises a training module, The training module trains the artificial neural network to detect traffic violations based on previously manually classified images or a series of images. The system according to claim 1.

9. Images or sets of images previously classified manually will be classified manually in accordance with laws, regulations, and combinations thereof. The system according to claim 8.

10. The images or series of images that were previously manually classified were manually classified to include traffic violations. The system according to claim 9.

11. The non-temporary storage device and the CPU are located inside the first vehicle. The system according to claim 1.

12. The logic module determines whether one or more events of interest selected by the detector module include a road hazard, and if one or more events of interest include a road hazard, video information identified as including a road hazard is transmitted to the local authority. The system according to claim 1.

Images