Road event notification system using sensor devices

Abstract

The present invention relates to a road event notification system using a sensor device, comprising: an electronic device constituting an infrastructure of sensor devices that communicate with each other via a communication protocol; and a user terminal, wherein the electronic device constituting the infrastructure of the sensor device comprises a sensor device for capturing an image of a road; a processor that directly processes an image processing-based artificial intelligence model and determines an event occurring on the road using the image of the road captured by the sensor device as an input of the image processing-based artificial intelligence model; and a memory, and the user terminal comprises a user terminal processor unit for receiving data related to the road event determined via the communication protocol; and an output unit for outputting the received data.

Description

[Technical Field]

[0001] The present invention relates to a road event notification system using a sensor device. [Background technology]

[0002] With the increase in the number of automobiles, the incidence of traffic accidents is increasing significantly. Therefore, it is becoming increasingly important to clarify who is responsible for an accident through an objective investigation of the facts related to the accident. Generally, when a traffic accident occurs, the parties involved agree on whether or not there is fault and the degree of fault, or the police are dispatched to inspect and record the accident situation, determine the circumstances at the time of the accident, and determine whether or not there is fault and the degree of fault. However, this method has the problem of relying on the subjective judgment of the parties involved in the accident and police officers, without providing an accurate analysis of the circumstances of the traffic accident.

[0003] To address this issue, an accident video analysis system and an analysis method using the same that determines the degree of fault by comparing black box video with similar accident video are disclosed in Patent Document 1. However, the invention disclosed in Patent Document 1 still has the problem that the black box video uses spatially limited video and cannot utilize standard traffic accident information.

[0004] Furthermore, while the video surveillance equipment that has already been installed and is currently in operation is making progress in providing traffic information through the analysis of real-time road footage, it is not capable of analyzing the current situation of traffic accidents in real time.

[0005] Existing driver assistance systems have mainly adopted a method of providing information necessary for driving a vehicle by detecting the distance and objects around the vehicle and monitoring the surrounding environment in real time based on sensor technology such as radar and ultrasonic sensors.

[0006] However, some sensors have to send collected data to a central server for processing, which can cause delays in data transmission and processing, and there are problems such as a large amount of bandwidth being consumed in the process of transmitting large amounts of sensor data to the central server.In addition, there are problems such as privacy issues arising from the transmission of personal information to the central server.

[0007] Therefore, to solve this problem, a driver assistance system using edge AI is required, and vehicle-to-vehicle communication and communication technology between vehicles and road infrastructure using edge AI can improve the performance of driver assistance systems through the exchange of information on road conditions.

[0008] Edge AI can process data and make decisions locally, enabling real-time responses, reducing bandwidth consumption, and protecting privacy, contributing to improved reliability and stability. The present invention provides a driver assistance system that works in cooperation with a central server-based system to deliver optimal performance. Summary of the Invention [Problem to be solved by the invention]

[0009] In order to solve the above-mentioned problems, an object of the present invention is to provide a traffic accident analysis system and method for estimating the similarity between actual traffic accident information and standard traffic accident information.

[0010] In order to solve the above-mentioned problems, the present invention provides a system for assisting drivers in real time by utilizing electronic devices and AR devices that constitute an infrastructure of multiple sensor devices that communicate with each other via a communication protocol. [Means for solving the problem]

[0011] In order to achieve the above-mentioned object, a traffic accident analysis system according to an embodiment of the present disclosure may include: a dataset construction unit that constructs each standard traffic accident dataset from a plurality of standard traffic accident information; an omniscient representation unit that maps video data of actual traffic accident information taken on a road onto a two-dimensional planar map; and a traffic accident estimation unit that compares the actual traffic accident information mapped by the omniscient representation unit with each standard traffic accident dataset constructed by the dataset construction unit, and estimates standard traffic accident information similar to the actual traffic accident information.

[0012] The data set construction unit may classify the types of traffic accidents from each standard traffic accident information and perform labeling according to the classified traffic accident types to construct the respective standard traffic accident data sets.

[0013] The types of traffic accidents may include vehicle-to-vehicle, vehicle-to-person, vehicle-to-motorcycle, and vehicle-to-bicycle.

[0014] The dataset constructed by the dataset construction unit may be a dataset constructed to include not only the video data of the standard traffic accident information but also text data that is explanatory material explaining the traffic accident.

[0015] The omniscient representation unit can identify the frame of an object from the actual traffic accident video and display a bounding box to identify the direction and angle of the object.

[0016] The omniscient representation unit can display circular distance lines at predetermined distance intervals on the actual traffic accident video, and perform mapping on the two-dimensional planar map taking into account distance and angle using the displayed circular distance lines.

[0017] The actual traffic accident information may include information such as traffic signals at the time of the traffic accident video.

[0018] The traffic accident estimation unit may include a unit for extracting image features and a unit for classifying images in a manner that effectively recognizes and emphasizes features between adjacent images while maintaining spatial information of the image.

[0019] The traffic accident analysis system may include a video monitoring device and a management server, the video monitoring device may include the omniscient representation unit and the traffic accident estimation unit, and the management server may include the dataset construction unit.

[0020] The traffic accident estimation unit of the video monitoring device can provide standard traffic accident information of a situation similar to actual traffic accident information and a similarity to the actual traffic accident information to the user terminal.

[0021] A traffic accident analysis method in a traffic accident analysis system according to another embodiment of the present disclosure may include the steps of constructing respective standard traffic accident datasets from a large number of standard traffic accident information pieces; mapping video data of actual traffic accident information captured on a road onto a two-dimensional planar map; and comparing the actual traffic accident information mapped in the mapping step with each standard traffic accident dataset constructed in the constructing step, and estimating standard traffic accident information similar to the actual traffic accident information.

[0022] The present invention discloses a driver assistance system using an AR device, which includes an electronic device that forms an infrastructure of multiple sensor devices that communicate with each other via a communication protocol, and an AR device, wherein the electronic device that forms the infrastructure of the multiple sensor devices includes a sensor device for capturing road images, a processor that directly processes an image processing-based artificial intelligence model and performs a situation assessment operation related to the road using the road image captured by the sensor device as an input to the image processing-based artificial intelligence model, and a memory, and the AR device includes an AR device processor unit that receives data related to the situation assessment via the communication protocol, and a display unit that outputs the received data.

[0023] The present invention discloses a driver assistance system using an AR device, in which the AR device is in a form that can be worn by the driver.

[0024] The present invention discloses a driver assistance system using an AR device, in which the display unit outputs data related to the road conditions as an augmented reality image.

[0025] The present invention discloses a driver assistance system using AR equipment, wherein the road conditions are at least one of a traffic accident on the road, congestion on the road, construction work on the road, failure of facilities on the road, restrictions on the road, illegal parking or stopping on the road, and violation of traffic rules on the road.

[0026] The present invention discloses a driver assistance system using an AR device, in which the display unit outputs data related to the location of a pedestrian or object in a blind spot of a moving vehicle as an augmented reality image.

[0027] The present invention discloses a driver assistance system using an AR device, in which the display unit outputs data related to road lanes or road guide markings as an augmented reality image.

[0028] The present invention discloses a driver assistance system using an AR device, in which the display unit outputs data related to maintaining a safe distance from other vehicles located within a critical distance while driving as an augmented reality image.

[0029] The present invention discloses a driver assistance system using an AR device, in which the display unit outputs data related to a parking location with the shortest distance between the destination of the vehicle driver and the parking location as an augmented reality image.

[0030] The present invention discloses a driver assistance system using an AR device, which includes an AR device and an electronic device that forms an infrastructure of multiple sensor devices that communicate with each other via a communication protocol, wherein the electronic device that forms the infrastructure of the multiple sensor devices includes a sensor device for capturing road images, a processor, and a memory, and the AR device includes a display unit, an AR device processor unit that directly processes an image processing-based artificial intelligence model, receives the road image captured by the sensor device via the communication protocol, and performs a situation assessment operation related to the road using the received road image as an input to the image processing-based artificial intelligence model, and a display unit that outputs data related to the performed situation assessment.

[0031] The present invention discloses a driver assistance system using AR equipment, comprising: an electronic device that constitutes an infrastructure of numerous sensor devices that communicate with each other via a communication protocol; an external terminal; and an AR device, wherein the electronic device that constitutes the infrastructure of the numerous sensor devices comprises a sensor device for capturing images of a road, a processor, and a memory, wherein the external terminal comprises an external terminal processor that processes a general-purpose artificial intelligence model, receives the images of the road captured by the sensor device via the communication protocol, and performs a situation assessment operation related to the road using the received road images as input to the general-purpose artificial intelligence model, and the AR device comprises an AR device processor unit that receives data related to the performed situation assessment via the communication protocol, and a display unit that outputs the received data.

[0032] The present invention relates to a driver assistance system using an AR device, which includes an electronic device constituting an infrastructure of a number of sensor devices that communicate with each other through a communication protocol, an external terminal, and an AR device, and the electronic device constituting the infrastructure of the number of sensor devices includes a sensor device for capturing an image of a road, a processor that directly processes an image processing-based artificial intelligence model and performs a part or all of a situation judgment operation related to the road using the image of the road captured by the sensor device as an input of the image processing-based artificial intelligence model, and a memory. and an external terminal processor that communicates at least one of the data related to the situation judgment performed in part or in whole by the processor via the communication protocol, and performs a part or all of the situation judgment operation related to the road using the received road image and at least one of the data related to the situation judgment performed in part or in whole by the processor as inputs to the general-purpose artificial intelligence model, wherein the AR device includes an AR device processor unit that receives the data related to the performed part or all of the situation judgment from the processor and the external terminal processor, and a display unit that outputs the received data.

[0033] The present invention relates to a driver assistance system using an AR device, which includes an electronic device constituting an infrastructure of a large number of sensor devices that communicate with each other via a communication protocol, an external terminal, and an AR device, wherein the electronic device constituting the infrastructure of the large number of sensor devices includes a sensor device for capturing an image of a road, a processor, and a memory, and the external terminal includes an external terminal processor that processes a general-purpose artificial intelligence model, communicates the image of the road captured by the sensor device via the communication protocol, and performs a partial or entire situation assessment operation related to the road using the received image of the road as an input to the general-purpose artificial intelligence model, and the AR device discloses a driver assistance system using an AR device, including: an AR device processor unit that directly processes an image processing-based artificial intelligence model, communicates at least one of the image of the road captured by the sensor device and data related to situation judgment partially or fully performed by the external terminal processor via the communication protocol, and performs a partial or full situation judgment operation related to the road using at least one of the received image of the road and data related to situation judgment partially or fully performed by the processor as inputs to the image processing-based artificial intelligence model; and a display unit that outputs the data related to the performed situation judgment.

[0034] The present invention relates to a driver assistance system using an AR device, which includes an electronic device constituting an infrastructure of a number of sensor devices communicating with each other via a communication protocol, an external terminal, and an AR device, wherein the electronic device constituting the infrastructure of the number of sensor devices includes a sensor device for capturing an image of a road, a processor that directly processes an image processing-based artificial intelligence model and performs a partial or entire situation judgment operation related to the road using data related to the image of the road captured by the sensor device as an input of the image processing-based artificial intelligence model, and a memory, wherein the external terminal processes a general purpose artificial intelligence model and communicates at least one of the image of the road captured by the sensor device and data related to the situation judgment performed in part or in whole by the processor via the communication protocol, and transmits at least one of the received road image and data related to the situation judgment performed in part or in whole by the processor to the general purpose artificial intelligence model. and an external terminal processor that performs a part or all of the situation judgment operations related to the road as an input of an artificial intelligence model, wherein the AR device directly processes an image processing-based artificial intelligence model, and communicates at least one of an image of the road captured by the sensor device, data related to the situation judgment performed partly or fully by the external terminal processor, and data related to the situation judgment performed partly or fully by the processor via the communication protocol, and performs a part or all of the situation judgment operations related to the road as an input of the image processing-based artificial intelligence model, and a display unit that outputs the data related to the performed situation judgment.

[0035] The present invention discloses a road event notification system using a sensor device, comprising: an electronic device constituting an infrastructure of sensor devices that communicate with each other via a communication protocol; and a user terminal, wherein the electronic device constituting the infrastructure of the sensor device comprises a sensor device for capturing images of a road; a processor that directly processes an image processing-based artificial intelligence model and determines an event occurring on the road using the image of the road captured by the sensor device as an input of the image processing-based artificial intelligence model; and a memory, wherein the user terminal comprises a user terminal processor unit for receiving data related to the road event determined via the communication protocol; and an output unit for outputting the received data.

[0036] The present invention discloses a road event notification system using a sensor device, in which the user terminal processor directly processes an image processing-based artificial intelligence model, receives the road image captured from the sensor device, and determines some or all of the road events as input to the image processing-based artificial intelligence model.

[0037] The present invention discloses a road event notification system using a sensor device, wherein the road event is at least one of a traffic accident on the road, congestion on the road, road construction, failure of facilities on the road, road restrictions, illegal parking or stopping on the road, and the occurrence of a vehicle violating traffic rules on the road.

[0038] The present invention discloses a road event notification system using a sensor device, wherein the output unit outputs an augmented reality image related to the road event.

[0039] The present invention discloses a road event notification system using a sensor device, in which the processor classifies road events based on a classification model trained using actual road events as label data.

[0040] The present invention relates to a road event notification system using a sensor device, which includes an electronic device constituting an infrastructure of sensor devices that communicate with each other through a communication protocol, a user terminal, and an external terminal, and the electronic device constituting the infrastructure of the sensor device includes a sensor device for capturing road images, a processor that directly processes an image processing-based artificial intelligence model and determines some or all of the road events using the road images captured by the sensor device as input to the image processing-based artificial intelligence model, and a memory, and the external terminal processes a general artificial intelligence model and determines the road images captured by the sensor device, data related to the road events determined by the processor, and the road events. and an external terminal processor that receives at least one of the received data of the road image taken by the sensor device and the data on road occurrence events determined by the processor as inputs to the general-purpose artificial intelligence model to determine some or all of the road occurrence events, wherein the user terminal includes a user terminal processor unit that receives at least one of the data on road occurrence events determined by the processor via the communication protocol and the data on road occurrence events determined by the external terminal processor, and an output unit that outputs the received data. [Effects of the Invention]

[0041] With the above-described configuration, the present invention can objectively determine whether or not there is liability for negligence and the degree of negligence by analyzing actual traffic accident information by estimating the similarity based on standard traffic accident information.

[0042] The present invention also enables traffic accidents to be analyzed in a two-dimensional manner by mapping video data of actual traffic accident information onto a two-dimensional map, thereby enabling more accurate estimation of the type of traffic accident.

[0043] The present invention also enables safety diagnosis and prevention of traffic safety solutions by constructing a data set based on the type of traffic accident as basic data for preventing, diagnosing, responding to, and predicting traffic accidents.

[0044] Edge AI can process data and make decisions locally, enabling real-time responses, reducing bandwidth consumption, and protecting privacy, contributing to improved reliability and stability. The present invention provides a driver assistance system that works in cooperation with a central server-based system to deliver optimal performance.

[0045] The present invention has the effect of assisting the driver in driving in real time by allowing the electronic device and external terminal that constitute the infrastructure of the AR device and multiple sensor devices to communicate with each other using a communication protocol, and outputting data related to situation judgment related to road conditions in real time from the display unit of the AR device. [Brief explanation of the drawings]

[0046] [Figure 1a] FIG. 1 is a block diagram for explaining a driver assistance system using an edge AI according to the present invention. [Figure 1b] 1 is a diagram illustrating a communication network between an external terminal 100, an AR device 20, and an electronic device 10 that constitutes an infrastructure of a number of sensor devices installed on roads according to an embodiment of the present invention. [Figure 1c] 1 is a block diagram illustrating an external terminal 100 according to an embodiment of the present invention. [Figure 2] 1 is a diagram illustrating an embodiment of the present invention in which an AR device 20 receives data from an electronic device 10 that constitutes the infrastructure of a large number of sensor devices installed on roads. [Figure 3] 1 is a diagram illustrating an embodiment in which, when a pedestrian is located in a blind spot 31 of a vehicle 30 of a driver wearing an AR device according to an embodiment of the present invention, an AR device 20 receives data indicating that a pedestrian is located in the blind spot 31 from an electronic device 10 that constitutes an infrastructure of multiple sensor devices. [Figure 4a] This is a diagram for explaining an embodiment in which, when a collision occurs between vehicles 30-5 and 30-6 of drivers wearing AR equipment according to one embodiment of the present invention, an external terminal 100 recognizes this and transmits information on whether or not a collision has occurred to AR devices 20 of vehicles 30-1, 30-2, 30-3, and 30-4 of drivers wearing other AR equipment. [Figure 4b] This is a diagram showing an augmented reality image projected in the form of a 2D map into the field of view of a vehicle driver when a collision occurs between vehicles 30-5 and 30-6 of drivers wearing AR devices according to one embodiment of the present invention, the AR devices determine that it is a collision and communicate this to an external terminal 100, and the external terminal 100 recognizes this and transmits whether or not a collision has occurred to AR devices 20 of vehicles 30-1, 30-2, 30-3, and 30-4 of drivers wearing other AR devices. [Figure 4c] When a collision occurs between vehicles 30-5 and 30-6 of drivers wearing AR devices according to one embodiment of the present invention, the AR devices determine that it is a collision and communicate this to the external terminal 100, which then recognizes this and transmits information about whether or not a collision has occurred to the AR devices 20 of vehicles 30-1, 30-2, 30-3, and 30-4 of drivers wearing other AR devices. This is a diagram showing an augmented reality image 21-1 indicating the location of the collision that is projected into the field of view of the vehicle drivers. [Figure 5a] 1 is a diagram illustrating an embodiment in which information on how to use a vehicle 30 of a driver wearing an AR device is transmitted from an external terminal 100 to a vehicle driver according to an embodiment of the present invention. [Figure 5b] 10 is a diagram showing an augmented reality image 21-1 projected into the field of view of a vehicle driver when transmitting information on how to use a vehicle 30 of a driver wearing an AR device from an external terminal 100 according to an embodiment of the present invention to the vehicle driver. [Figure 6] 2 is a diagram illustrating an augmented reality image 21-1 that provides distance information to a preceding vehicle, among an embodiment of an augmented reality image 21-1 displayed on a display unit 21 of an AR device 20 according to the present invention. [Figure 7] 2 is a diagram illustrating an augmented reality image 21-1 that highlights lanes on a rainy day, among an embodiment of an augmented reality image 21-1 displayed on a display unit 21 of an AR device 20 according to the present invention. [Figure 8] This is a diagram for explaining an embodiment in which, when a vehicle 30 of a driver wearing an AR device according to the present invention enters a parking lot, the optimal parking space is guided taking into account the destination of the vehicle driver through communication with an external terminal 100 and electronic devices 10-1 and 10-2 that constitute the infrastructure of multiple sensor devices. [Figure 9] This is a diagram showing an augmented reality image 21-1 that guides the vehicle driver to the optimal parking space taking into account the vehicle driver's destination through communication with an external terminal 100 and electronic devices 10-1 and 10-2 that constitute the infrastructure of multiple sensor devices when the vehicle 30 of the driver wearing the AR device according to the present invention enters a parking lot. [Figure 10] This is a diagram showing an augmented reality image 21-1 that provides a route from a parking space to a destination in a building when a vehicle 30 of a driver wearing an AR device according to the present invention enters a parking lot and is guided to the optimal parking space taking into account the destination of the vehicle driver through communication with an external terminal 100 and electronic devices 10-1 and 10-2 that constitute the infrastructure of multiple sensor devices. [Figure 11] 1 is a diagram illustrating a traffic accident analysis system according to an embodiment of the present disclosure. [Figure 12] 12 is a block diagram illustrating a case where the video monitoring device or the management server shown in FIG. 11 operates as a general electronic device. [Figure 13] 1 is a diagram illustrating a block diagram of a traffic accident analysis system according to another embodiment of the present disclosure. [Figure 14] 1 is a diagram illustrating an example of two-dimensional images of standard traffic accidents classified by type and / or situation according to an embodiment of the present disclosure. [Figure 15] 10A-10C are diagrams illustrating examples of matching camera footage to a map as shown according to an embodiment of the present disclosure. [Figure 16]10A-10C are diagrams illustrating examples of matching camera footage to a map as shown according to an embodiment of the present disclosure. [Figure 17] 1 is a diagram illustrating an example of estimating a standard traffic accident type of an actual traffic accident situation according to an embodiment of the present disclosure. [Figure 18] 1 is a diagram illustrating an example of displaying a standard traffic accident estimation result of an actual traffic accident situation according to an embodiment of the present disclosure. [Figure 19] 10 is a flowchart illustrating a traffic accident analysis method according to another embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0047] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In describing the embodiments, technical content that is well known in the technical field to which the present invention pertains and is not directly related to the present invention will be omitted. This is to avoid obscuring the gist of the present invention by omitting unnecessary explanations and to more clearly communicate the gist of the present invention.

[0048] For the same reason, in the accompanying drawings, some components are exaggerated, omitted, or illustrated schematically, and the size of each component does not entirely reflect the actual size. The same or corresponding components in each drawing are given the same reference numerals.

[0049] The advantages and features of the present invention, as well as methods for achieving them, will become clearer with reference to the following detailed description of the embodiments in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, and may be embodied in various different forms. The present embodiments are provided solely to ensure that the disclosure of the present invention is complete and to fully convey the scope of the invention to those skilled in the art to which the present invention pertains. The present invention is defined only by the scope of the claims. Like reference numerals refer to like elements throughout the specification.

[0050] According to an embodiment of the present invention, a road-related event notification system using a sensor device may be provided. The road-related event notification system using a sensor device may include an electronic device 10 and a user terminal that constitute a sensor device infrastructure and communicate with each other via a communication protocol. In this case, the user terminal may include an AR device 20 or a user terminal 1140.

[0051] According to an embodiment of the present invention, an electronic device 10 constituting a sensor device infrastructure may include a sensor device 13 for capturing road images, a processor 11 for directly processing an image processing-based artificial intelligence model and determining road events using the road images captured by the sensor device as input to the image processing-based artificial intelligence model, and a memory 12. The sensor device 13 may include a camera 1112. The user terminal may include a user terminal processor unit for receiving data related to road events determined by the processor 11 via a communication protocol and an output unit for outputting the received data. The user terminal processor unit may include an AR device processor 23. The output unit may include a display unit 21. The output unit may output an augmented reality image associated with the road events. The processor 11 may classify the road events based on a classification model trained using actual road events as label data.

[0052] According to an embodiment of the present invention, the user terminal processor directly processes an image processing-based artificial intelligence model, receives the road image captured by the sensor device 13, and determines some or all of the road events as input to the image processing-based artificial intelligence model. In this case, the road events may be at least one of a traffic accident, a road congestion, road construction, a breakdown of road facilities, road restrictions, illegal parking or stopping on the road, and the occurrence of a vehicle violating traffic rules on the road.

[0053] According to yet another embodiment of the present invention, a road event notification system using a sensor device may include an electronic device 10, a user terminal, and an external terminal 100 that constitute an infrastructure of sensor devices that communicate with each other via a communication protocol. The external terminal 100 may include a management server 1120.

[0054] According to one embodiment of the present invention, an electronic device 10 constituting the infrastructure of a sensor device may include a sensor device 13 for capturing images of a road, a processor 11 that directly processes an image processing-based artificial intelligence model and determines some or all of the events occurring on the road by using the road images captured by the sensor device 13 as input to the image processing-based artificial intelligence model, and a memory 12.

[0055] According to one embodiment of the present invention, the external terminal 100 may include an external terminal processor 110 that processes a general-purpose artificial intelligence model, receives at least one of road images captured by the sensor device 13 and data related to road occurrence events determined by the processor 11, and determines some or all of the road occurrence events using the received road images captured by the sensor device 13 and the received data related to road occurrence events determined by the processor 11 as inputs to the general-purpose artificial intelligence model.

[0056] According to one embodiment of the present invention, the user terminal may include a user terminal processor unit that receives at least one of data regarding road occurrence events determined by the processor 11 through a communication protocol and data regarding road occurrence events determined by the external terminal processor 110, and an output unit that outputs the received data.

[0057] FIG. 1a is a block diagram illustrating a driver assistance system using edge AI according to the present invention.

[0058] Overall structure The overall configuration of the electronic device 10 that constitutes the infrastructure for a large number of sensor devices As shown in FIG. 1a, a driver assistance system using edge AI according to the present invention may include an electronic device 10 that constitutes an infrastructure of multiple sensor devices installed on roads, an AR device 20 that provides AR worn by a vehicle driver, and an external terminal 100.

[0059] In the following description of the present specification, a driver assistance system using edge AI is described as an AR device 20 that provides AR worn by a vehicle driver, but the present specification is not limited to the AR device 20 that provides AR worn by a vehicle driver, and also includes all embodiments in which an AR device is worn by a driver of a means of transportation such as an airplane, helicopter, bicycle, or motorcycle.

[0060] Sensor device 13 According to one embodiment of the present invention, a driver assistance system using edge AI may include an electronic device 10 that constitutes an infrastructure of numerous sensor devices installed on roads. The electronic device 10 that constitutes the infrastructure of numerous sensor devices installed on roads may include a processor 11, a memory 12, and a sensor device 13. Specifically, the sensor device 13 may be a camera that can capture still and video images and may include one or more image sensors, a lens, an image signal processor (ISP), or a flash (e.g., an LED or a xenon lamp).

[0061] According to one embodiment of the present invention, the sensor device 13 is installed on a smart pole and can capture various traffic images occurring on the road. As a specific example, when a vehicle 30 of a driver wearing an AR device, which is one of the embodiments described below, enters a blind spot 31, which is an area where the driver's field of view is obstructed and the driver cannot see, the sensor device 13 can sense that a pedestrian is located in the blind spot 31, and one camera in one embodiment of the sensor device 13 can capture an image of the pedestrian located in the blind spot 31. As another specific example, when a traffic jam occurs on a road on which a vehicle 30 of a driver wearing an AR device, which is one of the embodiments described below, is located due to an accident or the like on the route to the driver's destination, the sensor device 13 can sense the location of the accident on the road on the route to the driver's destination, and one camera in one embodiment of the sensor device 13 can capture an image of the location of the accident.

[0062] Processor 11 According to one embodiment of the present invention, in a driver assistance system using edge AI, an electronic device 10 constituting an infrastructure of numerous sensor devices installed on roads may include a processor 11. Specifically, the processor 11 executes a program to control the electronic device 10 constituting the infrastructure of numerous sensor devices installed on roads. The code of the program executed by the processor 11 may be stored in a memory 12. The processor 11 may also be connected to an external device via an input / output device to exchange data. The external device may be another electronic device constituting the infrastructure of numerous sensor devices installed on other roads, an external terminal 100 such as a server, or an AR device 20 worn by a vehicle driver that provides AR.

[0063] According to an embodiment of the present invention, the processor 11 included in the electronic device 10, which constitutes the infrastructure of multiple sensor devices installed on roads, can directly process an image processing-based artificial intelligence model or can independently process the image processing-based artificial intelligence model. Various artificial intelligence network models, such as a recurrent neural network (RNN), a deep neural network (DNN), and a dynamic recurrent neural network (DRNN), can be used for such learning. In this way, the processor 11 can process and analyze data within the device itself, without sending data to a central data center or cloud. Therefore, the electronic device 10, which constitutes the infrastructure of multiple sensor devices installed on roads, has high data processing efficiency and is extremely useful in situations where real-time responses are required.

[0064] Memory 12 According to an embodiment of the present invention, in a driver assistance system using edge AI, electronic device 10, which constitutes an infrastructure of numerous sensor devices installed on roads, may include memory 12. Specifically, when sensor device 13 mounted on electronic device 10, which constitutes an infrastructure of numerous sensor devices installed on roads, captures road images, etc., memory 12 stores the image data and may also store intermediate results and temporary data generated while processor 11 processes the data.

[0065] According to one embodiment of the present invention, memory 12 may be used to record sensed events, accidents, information related to traffic flow, etc., and processor 11 may use memory 12 to store and learn patterns of accidents, etc. occurring on the road.

[0066] According to one embodiment of the present invention, memory 12 may be volatile or non-volatile memory and may be referred to as a "database," "storage," or the like.

[0067] AR equipment As shown in FIG. 1a, an AR device 20 capable of communicating with an electronic device 10 constituting the infrastructure of the multiple sensor devices mentioned above may include a display unit 21, an AR device camera unit 22, and an AR device processor unit 23.

[0068] According to one embodiment of the present invention, the AR device 20 may be in a form that can be worn by the driver on the head, forehead, ear, or the like, but is not limited to this.

[0069] In the following, the present specification describes a driver assistance system using edge AI as an AR device 20 that provides AR worn by a vehicle driver, but it is not limited to the AR device 20 that provides AR worn by a vehicle driver, and also includes all embodiments in which an AR device is worn by a driver of a means of transportation such as an airplane, helicopter, bicycle, or motorcycle.

[0070] According to one embodiment of the present invention, an electronic device 10 that constitutes an infrastructure of multiple sensor devices can communicate with an AR device 20 via a communication protocol.

[0071] According to one embodiment of the present invention, the AR device 20 may include, but is not limited to, smart glasses that can be worn by a driver, a head-up display (HUD), an AR safety warning system, a smart helmet, a smart cap, a smart wearable device, an in-vehicle console display with AR capabilities, and the like.

[0072] The accompanying drawings in this specification will be described assuming smart glasses as one embodiment of the AR device 20, but this is not limited to this and may include any device that can communicate with an external device to provide information to a vehicle driver using AR.

[0073] According to an embodiment of the present invention, the AR device 20 worn by a vehicle driver may have a function of providing augmented reality. This functions to project information required while driving and visually convey the information to the driver. While the accompanying drawings of the present invention have been described as providing augmented reality, the AR device 20 may also provide not only augmented reality but also virtual reality images to project information required while driving and visually convey the information to the driver.

[0074] According to one embodiment of the present invention, the processor unit 23 of the AR device 20 worn by the vehicle driver can recognize the voice of the vehicle driver, so that the AR device 20 and the vehicle driver can communicate with each other by voice.

[0075] Display section 21 According to the present invention, the display unit 21 of the AR device 20 can project information into the driver's field of view, allowing the driver to check necessary information while keeping their eyes on the road. Specifically, the display unit 21 can project real-time road maps and navigation information into the driver's field of view. Information on road conditions, vehicle position, and surrounding environment can be provided to the driver's field of view by communicating with the electronic device 10 constituting the infrastructure of multiple sensor devices and receiving the information based on data collected and processed by the electronic device 10 constituting the infrastructure of multiple sensor devices.

[0076] According to an embodiment of the present invention, the display unit 21 can output data related to road conditions as an augmented reality image, and can output not only an augmented reality image but also a virtual reality image.

[0077] According to one embodiment of the present invention, the data related to road conditions output by the display unit 21 may include road conditions such as traffic accidents, road congestion, road construction, road facility failures, road restrictions, illegal parking and stopping on the road, and violations of road traffic rules.

[0078] According to one embodiment of the present invention, the display unit 21 communicates with and receives data collected and processed by the electronic device 10 constituting the infrastructure of multiple sensor devices, and projects real-time road traffic information, distance warnings to a preceding vehicle, whether or not a traffic light has been violated, the status of the traffic light, information on nearby shops or tourist attractions, major buildings, and monitoring of the driver's condition using the face recognition technology of the AR device 20 into the driver's field of view. As a result, the driver can sense the current road conditions or the driver's fatigue or drowsiness state from the display unit 21, and the AR device 20 can provide information not only visually but also audibly by outputting audio.

[0079] According to an embodiment of the present invention, a processor 11 included in an electronic device 10 constituting an infrastructure of a number of sensor devices installed on roads directly processes an image processing-based artificial intelligence model, and inputs road images captured by a sensor device 13 into the image processing-based artificial intelligence model to perform a situation assessment operation related to the road. At this time, an AR device processor unit 23 of an AR device 20 can receive data related to the situation assessment performed by the processor 11 through a communication protocol, and a display unit 21 of the AR device 20 can output the received data to provide information about road conditions to a driver.

[0080] As shown, the display unit 21 can output data related to maintaining a safe distance from other vehicles located within a critical distance while traveling as an augmented reality image.

[0081] As shown, the display unit 21 can output data related to the parking location with the shortest distance between the destination of the vehicle driver and the parking location in an augmented reality image.

[0082] AR equipment camera section 22 According to one embodiment of the present invention, the AR device camera unit 22 of the AR device 20 has a wide viewing angle, allowing the vehicle driver to effectively capture the surrounding environment of the vehicle while driving. Since the AR device processor unit 23 of the AR device 20, which will be described later, is equipped with image processing-based artificial intelligence, the road conditions captured through the AR device camera unit 22 can also be processed by the AR device processor unit 23.

[0083] According to one embodiment of the present invention, the AR device 20 can receive manuals, such as instructions on how to start the vehicle engine, from the external terminal 100. Accordingly, the AR device processor 23 can identify the face of the vehicle driver via the AR device camera 22 and check whether it matches the face of a pre-registered vehicle driver. The AR device processor 23 can also detect gestures of the vehicle driver via the AR device camera 22 and interpret specific actions as starting controls. The AR device processor 23 can also detect the driver's gaze using driver eye tracking technology via the AR device camera 22, and can recognize whether the driver is experiencing drowsiness or accumulated fatigue by detecting a specific gaze or pattern.

[0084] AR Device Processor 23 According to an embodiment of the present invention, the AR device 20 may include an AR device processor 23, which can communicate with an external terminal 100, an electronic device 10 constituting an infrastructure of multiple sensor devices, and a driver's vehicle 30. The AR device processor 23 can directly process an image processing-based artificial intelligence model or can independently process an image processing-based artificial intelligence model. Various artificial intelligence network models, such as a recurrent neural network (RNN), a deep neural network (DNN), and a dynamic recurrent neural network (DRNN), can be used for such learning. In this way, the AR device processor 23 can process and analyze data within the device itself, without sending data to a central data center or cloud. Therefore, the electronic device 10 constituting an infrastructure of multiple sensor devices installed on roads has high data processing efficiency and can be very useful in situations where real-time response is required.

[0085] According to an embodiment of the present invention, a processor 11 included in an electronic device 10 constituting an infrastructure of a number of sensor devices installed on roads directly processes an image processing-based artificial intelligence model, and inputs road images captured by a sensor device 13 into the image processing-based artificial intelligence model to perform a situation assessment operation related to the road. At this time, an AR device processor unit 23 of an AR device 20 can receive data related to the situation assessment performed by the processor 11 through a communication protocol, and a display unit 21 of the AR device 20 can output the received data to provide information about road conditions to a driver.

[0086] According to another embodiment of the present invention, the AR device processor 23 of the AR device 20 may directly process an image processing-based artificial intelligence model. In this case, the AR device processor 23 may receive a road image captured by the sensor device 13 via a communication protocol, and the AR device processor 23 may input the received road image into the image processing-based artificial intelligence model to perform a road situation determination operation. In this case, the display 21 of the AR device 20 may output data related to the situation determination performed by the AR device processor 23 to provide the driver with information about the road situation.

[0087] A specific embodiment of the AR device processor unit 23 will be described below.

[0088] External terminal 100 FIG. 1c is a block diagram illustrating an external terminal 100 according to an embodiment of the present invention.

[0089] As shown in FIG. 1c, the external terminal 100 may include an external terminal processor 110, an external terminal memory 120, and a communication module 170. The external terminal 100 may be an external server or a cloud server. The external server may be a digital device equipped with a processor and memory, such as a laptop computer, notebook computer, desktop computer, web pad, or mobile phone, and have computing capabilities. The external server may be a web server that processes services. The types of servers described above are merely examples, and the present invention is not limited thereto.

[0090] According to one embodiment of the present invention, the electronic device 10 constituting the infrastructure of a large number of sensor devices can communicate with the AR device 20 using a communication protocol. Furthermore, the electronic device 10 constituting the infrastructure of a large number of sensor devices can communicate with the external terminal 100 using the communication protocol. Furthermore, the AR device 20 and the external terminal 100 can communicate with each other using the communication protocol. That is, the electronic device 10 constituting the infrastructure of a large number of sensor devices, the AR device 20, and the external terminal 100 can communicate with each other using the communication protocol.

[0091] According to an embodiment of the present invention, the external terminal 100 may be a server providing a cloud computing service. More specifically, the external terminal 100 may be a server providing a cloud computing service, a type of Internet-based computing, that processes information on a computer connected to the Internet other than the user's computer. The cloud computing service may store data on the Internet and allow users to access the data or programs they need anytime and anywhere via an Internet connection without installing them on their own computers. The cloud computing service may also allow users to easily share and transfer data stored on the Internet with simple operations and clicks. The cloud computing service may not only simply store data on an Internet server, but also allow users to perform desired tasks using the functions of web-based applications without installing additional programs, and allow multiple users to share documents and work simultaneously. The cloud computing service may be implemented in at least one form of Infrastructure as a Service (IaaS), Platform as a Service (PaaS), Software as a Service (SaaS), a virtual machine-based cloud server, or a container-based cloud server. That is, the external terminal 100 of the present invention may be implemented in at least one form of the above-mentioned cloud computing services. The specific description of the cloud computing service mentioned above is merely an example and may include any platform that builds the cloud computing environment of the present invention.

[0092] According to an embodiment of the present invention, the external terminal processor 110 may control the external terminal 100 overall and may include an AI processor 115. Specifically, the external terminal processor 110 may process general artificial intelligence. Specifically, the AI ​​processor 115 may train a neural network using a program stored in the external terminal memory 120. In particular, the AI ​​processor 115 may train a neural network to recognize data related to the operation of the electronic device 10 and the AR device 20 that constitute the infrastructure of numerous sensor devices installed on roads. Here, the neural network may be designed to simulate the structure of a human brain (e.g., the neuron structure of a human neural network) on a computer. The neural network may include an input layer, an output layer, and at least one hidden layer. Each layer may include at least one neuron having a weight value, and the neural network may include synapses connecting neurons. In a neural network, each neuron can output an activation function value for a weight and / or bias of an input signal received via a synapse.

[0093] According to an embodiment of the present invention, multiple network modes may exchange data according to respective connection relationships to simulate synaptic activity of neurons, in which neurons exchange signals through synapses. Here, the neural network may include a deep learning model developed from a neural network model. In a deep learning model, multiple network nodes may be located in different layers and exchange data according to convolutional connections. Examples of neural network models include various deep learning techniques such as deep neural networks (DNNs), convolutional neural networks (CNNs), recurrent neural networks, restricted Boltzmann machines, deep belief networks, and deep Q-networks, and can be applied in fields such as vision recognition, speech recognition, natural language processing, and speech / signal processing.

[0094] Meanwhile, the external terminal processor 110 performing the above-mentioned functions may be a general-purpose processor (e.g., CPU), or may be an AI-dedicated processor (e.g., GPU, TPU) for artificial intelligence learning.

[0095] According to an embodiment of the present invention, the external device memory 120 may store various programs and data necessary for the operation of the electronic device 10 and the AR device 20, which constitute the infrastructure of numerous sensor devices installed on roads. The external device memory 120 is accessed by the AI ​​processor 115, and the AI ​​processor 115 may read, record, modify, delete, and update data. The external device memory 120 may also store a neural network model (e.g., a deep learning model) generated through a learning algorithm for data classification / recognition. The external device memory 120 may also store input data, learning data, learning history, and the like, in addition to the learning model 121.

[0096] According to one embodiment of the present invention, the communication module 170 can transmit the AI ​​processing results by the AI ​​processor 115 to the electronic device 10 or the AR device 20 that constitutes an infrastructure of numerous sensor devices installed on roads.

[0097] According to another embodiment of the present invention, the external device processor 110 of the external device 100 may directly process an image processing-based artificial intelligence model. In this case, the external device processor 110 may receive a road image captured by the sensor device 13 via a communication protocol, and the external device processor 110 may input the received road image into the image processing-based artificial intelligence model to perform a road situation determination operation. In this case, the AR device processor 23 may receive data related to the situation determination performed by the external device processor 110 via the communication protocol. The display unit 21 may output the received data related to the situation determination to provide the driver with information about the road situation.

[0098] Communication between electronic devices that make up the infrastructure of AR devices, external terminals, or numerous sensor devices installed on roads Embodiments of communication between multiple devices FIG. 1b is a diagram illustrating a communication network between an external terminal 100 according to an embodiment of the present invention, an AR device 20, and an electronic device 10 that constitutes an infrastructure of a number of sensor devices installed on roads.

[0099] According to one embodiment of the present invention, the electronic device 10 constituting the infrastructure of a large number of sensor devices can communicate with the AR device 20 using a communication protocol. Furthermore, the electronic device 10 constituting the infrastructure of a large number of sensor devices can communicate with the external terminal 100 using the communication protocol. Furthermore, the AR device 20 and the external terminal 100 can communicate with each other using the communication protocol. That is, the electronic device 10 constituting the infrastructure of a large number of sensor devices, the AR device 20, and the external terminal 100 can communicate with each other using the communication protocol.

[0100] 1b, an external terminal 100 can communicate with a plurality of AR devices 20, and the plurality of AR devices 20 can communicate with electronic devices 10 constituting an infrastructure of a plurality of sensor devices installed on a plurality of roads, and the electronic devices 10 constituting an infrastructure of a plurality of sensor devices installed on a plurality of roads can communicate with the external terminal 100. Specifically, the electronic devices 10 constituting an infrastructure of a plurality of sensor devices installed on a plurality of roads can capture and analyze video data generated on the roads in real time, and can process the data using image processing-based artificial intelligence implemented in the processors 11 of the electronic devices 10 constituting the infrastructure of a plurality of sensor devices installed on the roads to extract important information. The extracted information can indicate local events, traffic conditions, etc., and can be transmitted to the AR devices 20 or the external terminal 100 in real time.

[0101] According to one embodiment of the present invention, the AR device 20 can receive information about road conditions from the electronic device 10, which constitutes an infrastructure of multiple sensor devices installed on roads, and display the information to a vehicle driver. The AR device 20 can locally process part of the information received from the electronic device 10, which constitutes an infrastructure of multiple sensor devices installed on roads, and can also transmit the information to an external terminal 100 as needed.

[0102] According to an embodiment of the present invention, the external terminal 100 centrally processes data collected by the AR device 20 or the electronic device 10 that constitutes the infrastructure of multiple sensor devices installed on roads, and can perform high-level analysis and judgment. The external terminal 100 can understand a wider range of road conditions by integrating information collected by the AR device 20 or the electronic device 10 that constitutes the infrastructure of multiple sensor devices installed on roads. The external terminal 100 can transmit additional information required by the AR device 20 or the electronic device 10 that constitutes the infrastructure of multiple sensor devices installed on roads, and can analyze the data collected by the AR device 20 or the electronic device 10 that constitutes the infrastructure of multiple sensor devices installed on roads to generate comprehensive insights into road conditions. This information may be transmitted back to the AR device 20, and the vehicle driver can visually or audibly recognize the data based on which they can make judgments about road conditions, etc.

[0103] communication protocol According to one embodiment of the present invention, the external terminal 100 can communicate with a large number of AR devices 20, and the large number of AR devices 20 can communicate with the electronic device 10 that constitutes an infrastructure of a large number of sensor devices installed on a large number of roads. According to one embodiment of the present invention, the electronic device 10 constituting the infrastructure of a large number of sensor devices can communicate with the AR device 20 using a communication protocol. Furthermore, the electronic device 10 constituting the infrastructure of a large number of sensor devices can communicate with the external terminal 100 using the communication protocol. Furthermore, the AR device 20 and the external terminal 100 can communicate with each other using the communication protocol. That is, the electronic device 10 constituting the infrastructure of a large number of sensor devices, the AR device 20, and the external terminal 100 can communicate with each other using the communication protocol.

[0104] In this case, various communication protocols may be used. The protocol selected may vary depending on specific requirements, bandwidth, security level, etc., and the following protocols may also be used in combination. Specific examples include lightweight protocols such as MQTT (Message Queuing Telemetry transport) and CoAP (Constrained Application Protocol), which require fewer resources and are optimized for the Internet of Things. Other protocols that may be used include HTTP (HyperText Protocol) for quickly exchanging hypertext, HTTPS (HyperText Protocol Secure) for providing a secure connection and encrypting data, AMQP (Advanced Message Queuing Protocol) for exchanging and communicating messages, and DDS (Data Distribution Service) for exchanging data between real-time systems. Other protocols that may be used include, but are not limited to, FTP (File Transfer Protocol), RTSP (Real-Time Streaming Protocol), and RTP (Real-time Transport Protocol).

[0105] According to another embodiment of the present invention, the external terminal 100 may communicate with a plurality of AR devices 20, and the plurality of AR devices 20 may communicate with the electronic device 10 constituting an infrastructure of a plurality of sensor devices installed on a plurality of roads. In this case, various communication protocols may be used. Here, a wireless communication technology having an extremely wide bandwidth, such as UWB (Ultra-Wide Band), may be used, or a wireless communication protocol based on a method of transmitting data using an extremely wide frequency bandwidth may be used. This enables accurate search in a large space based on very precise spatial recognition and directionality, but is not limited thereto.

[0106] According to another embodiment of the present invention, the external terminal processor 110 of the external terminal 100 may process the general-purpose artificial intelligence model. At this time, the external terminal processor 110 may receive a road image captured by the sensor device 13 via a communication protocol, and the external terminal processor 110 may input the received road image to the general-purpose artificial intelligence model to perform a situation determination operation related to the road. At this time, the AR device processor 23 may receive data related to the situation determination performed by the external terminal processor 110 via the communication protocol. The display 21 may output the received data related to the situation determination to provide the driver with information about the road situation.

[0107] According to another embodiment of the present invention, the external terminal processor 110 of the external terminal 100 may process a general-purpose artificial intelligence model. Alternatively, the AR device processor 23 may directly process an image processing-based artificial intelligence model. Alternatively, the processor 11 may directly process an image processing-based artificial intelligence model. In this case, the processor 11 may perform some or all of the situation determination operations related to the road by inputting road images captured by the sensor device 13 into the image processing-based artificial intelligence model. In this case, the external terminal processor 110 may receive at least one of the road images captured by the sensor device 13 and some or all of the road situation determination operations performed by the processor 11 via a communication protocol, and may perform some or all of the road situation determination operations by inputting at least one of the road images captured by the sensor device 13 and some or all of the road situation determination operations performed by the processor 11. At this time, the AR device 20 can directly process an image processing-based artificial intelligence model to receive at least one of the partial or complete situation assessments regarding the road performed by the processor 11, the road image captured by the sensor device 13, and the partial or complete situation assessments performed by the external terminal processor 110, and perform partial or complete situation assessment operations regarding the road based on the above.

[0108] According to one embodiment of the present invention, the display unit 21 can output data related to road conditions as an augmented reality image, but can also output not only augmented reality but also virtual reality images.

[0109] According to one embodiment of the present invention, the data related to road conditions output by the display unit 21 may include road conditions such as traffic accidents, road congestion, road construction, road facility failures, road restrictions, illegal parking and stopping on the road, and violations of road traffic rules.

[0110] As shown, the display unit 21 can output data relating to the presence of a pedestrian or object in a blind spot 31 of a moving vehicle as an augmented reality image.

[0111] As shown, the display unit 21 can output data related to road lanes or road guide markings as an augmented reality image.

[0112] The embodiment of FIG. FIG. 2 is a diagram for explaining an embodiment in which an AR device 20 receives data transmission from an electronic device 10 that constitutes the infrastructure of a large number of sensor devices installed on roads according to the present invention.

[0113] In the following description of the present specification, a driver assistance system using edge AI is described as an AR device 20 that provides AR worn by a vehicle driver, but the present specification is not limited to the AR device 20 that provides AR worn by a vehicle driver, and also includes all embodiments in which an AR device is worn by a driver of a means of transportation such as an airplane, helicopter, bicycle, or motorcycle.

[0114] As shown in the drawings, according to one embodiment of the present invention, a vehicle driver can wear an AR device 20 while driving in the vehicle 30 of the driver wearing the AR device. As described above, the accompanying drawings illustrate an embodiment in which the AR device 20 is worn in the form of "smart glasses," but the AR device 20 is not limited to the form of "smart glasses."

[0115] According to one embodiment of the present invention, the electronic device 10 constituting the infrastructure of a large number of sensor devices can communicate with the AR device 20 using a communication protocol. Furthermore, the electronic device 10 constituting the infrastructure of a large number of sensor devices can communicate with the external terminal 100 using the communication protocol. Furthermore, the AR device 20 and the external terminal 100 can communicate with each other using the communication protocol. That is, the electronic device 10 constituting the infrastructure of a large number of sensor devices, the AR device 20, and the external terminal 100 can communicate with each other using the communication protocol.

[0116] According to another embodiment of the present invention, the external device processor 110 of the external device 100 may directly process an image processing-based artificial intelligence model. Furthermore, the processor 11 of the electronic device 10, which constitutes the infrastructure of multiple sensor devices, may also directly process an image processing-based artificial intelligence model. The sensor device 13 captures road images, and the processor 11 directly processes the image processing-based artificial intelligence model to perform a part or all of a road-related situation assessment operation. The external device processor 110 may receive the road images captured by the sensor device 13 and data related to the road-related situation assessment performed by the processor 11 via a communication protocol. The external device processor 110 may input the received road images and data related to the road-related situation assessment performed by the processor 11 into the image processing-based artificial intelligence model to perform a part or all of a road-related situation assessment operation. The AR device processor 23 may receive the data related to the situation assessment performed by the external device processor 110 and the data related to the situation assessment performed by the processor 11 via the communication protocol. The display unit 21 can output the received data relating to the situation assessment to provide the driver with information about road conditions.

[0117] According to another embodiment of the present invention, the external device processor 110 of the external device 100 may process a general-purpose artificial intelligence model. Furthermore, the AR device processor 23 may directly process an image processing-based artificial intelligence model. The external device processor 110 and the AR device processor 23 may receive a road image captured by the sensor device 13 via a communication protocol, and the external device processor 110 may input the received road image into the image processing-based artificial intelligence model to perform a part or all of a road-related situation determination operation. The AR device processor 23 may receive data related to the situation determination performed by the external device processor 110 and the road image captured by the sensor device 13 via the communication protocol. The AR device processor 23 may input the received data related to the situation determination performed by the external device processor 110 and the road image captured by the sensor device 13 into the image processing-based artificial intelligence model to perform a part or all of a road-related situation determination operation. Therefore, the display 21 may output data related to the performed situation determination to provide the driver with information about road conditions.

[0118] According to another embodiment of the present invention, the external terminal processor 110 of the external terminal 100 may process the general-purpose artificial intelligence model. At this time, the external terminal processor 110 may receive a road image captured by the sensor device 13 via a communication protocol, and the external terminal processor 110 may input the received road image to the general-purpose artificial intelligence model to perform a situation determination operation related to the road. At this time, the AR device processor 23 may receive data related to the situation determination performed by the external terminal processor 110 via the communication protocol. The display 21 may output the received data related to the situation determination to provide the driver with information about the road situation.

[0119] According to another embodiment of the present invention, the external terminal processor 110 of the external terminal 100 may process a general-purpose artificial intelligence model. Alternatively, the AR device processor 23 may directly process an image processing-based artificial intelligence model. Alternatively, the processor 11 may directly process an image processing-based artificial intelligence model. In this case, the processor 11 may perform some or all of the situation determination operations related to the road by inputting road images captured by the sensor device 13 into the image processing-based artificial intelligence model. In this case, the external terminal processor 110 may receive at least one of the road images captured by the sensor device 13 and some or all of the road situation determination operations performed by the processor 11 via a communication protocol, and may perform some or all of the road situation determination operations by inputting at least one of the road images captured by the sensor device 13 and some or all of the road situation determination operations performed by the processor 11. At this time, the AR device 20 can directly process an image processing-based artificial intelligence model to receive at least one of the partial or complete situation assessments regarding the road performed by the processor 11, the road image captured by the sensor device 13, and the partial or complete situation assessments performed by the external terminal processor 110, and perform partial or complete situation assessment operations regarding the road based on the above.

[0120] According to another embodiment of the present invention, the external device processor 110 of the external device 100 may process a general-purpose artificial intelligence model. Alternatively, the AR device processor 23 may directly process an image processing-based artificial intelligence model. In this case, the external device processor 110 may communicate a road image captured by the sensor device 13 via a communication protocol and perform a partial or complete road-related situation determination operation using the image as an input to the general-purpose artificial intelligence model. The AR device 20 may communicate a road image captured by the sensor device 13 and data related to partial or complete road-related situation determination performed by the external device processor 110 via the communication protocol and perform a partial or complete road-related situation determination operation using the data as an input to the image processing-based artificial intelligence model.

[0121] As shown in the figure, the electronic device 10 constituting the infrastructure of multiple sensor devices installed on roads may be installed in the form of being attached to a smart pole. The electronic device 10 constituting the infrastructure of multiple sensor devices installed on roads may capture images of road conditions, process the corresponding data, and perform communication 101 between the electronic device and the AR device 20 worn by a vehicle driver in real time. Specifically, communication 101 between the electronic device and the AR device is performed in the form of data processed by the electronic device 10 constituting the infrastructure of multiple sensor devices installed on major roads being transmitted to the AR device 20 using a communication protocol. However, an embodiment is also possible in which the AR device 20 analyzes the status of the vehicle 30 of the driver currently wearing the AR device or the status of the vehicle driver, and the electronic device 10 constituting the infrastructure of multiple sensor devices installed on roads receives the analyzed data. In addition, the AR device processor 23 of the AR device 20 worn by the vehicle driver may analyze the status of the vehicle 30 of the driver currently wearing the AR device or the status of the vehicle driver, and directly transmit information to the vehicle driver to assist the vehicle driver in safe driving.

[0122] For example, when the AR device processor 23 detects drowsy driving by a vehicle driver, it can transmit data indicating that drowsy driving has been detected to the electronic device 10, which constitutes an infrastructure of numerous sensor devices installed on roads, via the communication 101 between the electronic device and the AR device. The electronic device 10, which constitutes an infrastructure of numerous sensor devices installed on roads, receives the data and transmits the data to another electronic device 10-1 or an external terminal 100 shown in Fig. 1b, thereby transmitting information to vehicles 30-2, 30-3, etc., of drivers wearing other AR devices, to warn them about the vehicle 30 of the driver wearing the AR device. In addition, the AR device processor 23 of the AR device 20 worn by the vehicle driver can detect drowsy driving of the vehicle 30 of the driver currently wearing the AR device, analyze the detection, and directly transmit information regarding drowsy driving to the vehicle driver, thereby assisting the vehicle driver in safe driving. As a result, the electronic device 10, which constitutes the infrastructure of numerous sensor devices installed on roads, can receive the data directly from the AR device 20 and transmit information to vehicles 30-2, 30-3, etc., of drivers wearing other AR devices, instructing them to be careful of the vehicle 30 of a driver wearing an AR device.

[0123] The embodiment of FIG. FIG. 3 is a diagram illustrating an embodiment in which, when a pedestrian is located in a blind spot 31 of a vehicle 30 of a driver wearing an AR device according to an embodiment of the present invention, an AR device 20 receives data indicating that a pedestrian is located in the blind spot 31 from an electronic device 10 that constitutes an infrastructure of multiple sensor devices.

[0124] In the following description of the present specification, a driver assistance system using edge AI is described as an AR device 20 that provides AR worn by a vehicle driver, but the present specification is not limited to the AR device 20 that provides AR worn by a vehicle driver, and also includes all embodiments in which an AR device is worn by a driver of a means of transportation such as an airplane, helicopter, bicycle, or motorcycle.

[0125] According to one embodiment of the present invention, the electronic device 10 constituting the infrastructure of a large number of sensor devices can communicate with the AR device 20 using a communication protocol. Furthermore, the electronic device 10 constituting the infrastructure of a large number of sensor devices can communicate with the external terminal 100 using the communication protocol. Furthermore, the AR device 20 and the external terminal 100 can communicate with each other using the communication protocol. That is, the electronic device 10 constituting the infrastructure of a large number of sensor devices, the AR device 20, and the external terminal 100 can communicate with each other using the communication protocol.

[0126] As shown, the display unit 21 can output data relating to the presence of a pedestrian or object in a blind spot 31 of a moving vehicle as an augmented reality image.

[0127] As shown in the figure, a vehicle 30 in which a driver wearing an AR device is driving may have a blind spot 31. A blind spot 31 in a vehicle may refer to an area that is difficult for the driver to see directly from inside the vehicle, and may correspond to the sides and rear of the main vehicle. If an object or pedestrian suddenly appears in the blind spot 31 while the driver is driving, this may affect traffic safety, so a system that detects the blind spot 31 and warns the vehicle driver may be required.

[0128] As shown in the figure, the electronic device 10 constituting the infrastructure of multiple sensor devices installed on roads may be installed in the form of being attached to a smart pole. When a pedestrian or object is located in a blind spot 31 while the driver is driving, the electronic device 10 constituting the infrastructure of multiple sensor devices installed on roads may photograph the pedestrian or object and process the corresponding data. Thus, communication 101 between the electronic device and the AR device 20 worn by the vehicle driver may be performed in real time. For example, communication 101 between the electronic device and the AR device may be performed by transmitting data processed by the electronic device 10 constituting the infrastructure of multiple sensor devices installed on major roads to the AR device 20 using a communication protocol. The AR device 20 may display a warning message on the display unit 21, warning the driver that a pedestrian is currently located in the blind spot 31, and may transmit a guidance voice.

[0129] According to yet another embodiment of the present invention, the AR device processor 23 of the AR device 20 worn by a vehicle driver can detect and recognize in real time that a pedestrian is located in the blind spot 31. In this case, the AR device 20 can display a warning message on the display 21 urging the driver to be careful because a pedestrian is currently located in the blind spot 31 and can transmit a guidance voice. Furthermore, when the AR device 20 in a moving vehicle detects that a pedestrian is located in the blind spot 31, it can communicate with the AR devices 20 of other vehicles located within a critical distance from the blind spot 31 to display a warning message urging the driver to be careful because a pedestrian is currently located in the blind spot 31 and transmit a guidance voice. In addition, when the AR device 20 in a moving vehicle detects that a pedestrian is located in the blind spot 31, it can communicate with the electronic device 10 or an external terminal 100 constituting an infrastructure of multiple sensor devices installed on roads, so that the AR devices 20 of other vehicles located within a critical distance from the blind spot 31 can receive the corresponding data.

[0130] The embodiment of FIG. 4A is a diagram illustrating an embodiment in which, when a collision occurs between vehicles 30-5 and 30-6 of drivers wearing AR devices according to an embodiment of the present invention, an external device 100 recognizes the collision and transmits information about the occurrence of the collision to AR devices 20 of vehicles 30-1, 30-2, 30-3, and 30-4 of drivers wearing other AR devices. FIG. 4B is a diagram illustrating an augmented reality image projected in the form of a 2D map into the field of view of a vehicle driver when, when a collision occurs between vehicles 30-5 and 30-6 of drivers wearing AR devices according to an embodiment of the present invention, the AR devices determine that the collision occurred and communicate this to the external device 100, and the external device 100 recognizes this and transmits information about the occurrence of the collision to AR devices 20 of vehicles 30-1, 30-2, 30-3, and 30-4 of drivers wearing other AR devices.

[0131] Figure 4c is a diagram showing an augmented reality image 21-1 indicating the location of the collision accident projected into the field of view of the vehicle driver when a collision accident occurs between vehicles 30-5 and 30-6 of drivers wearing AR devices according to one embodiment of the present invention, the AR devices determine that it is a collision accident and communicate this to the external terminal 100, and the external terminal 100 recognizes this and transmits whether or not a collision accident has occurred to the AR devices 20 of vehicles 30-1, 30-2, 30-3, and 30-4 of drivers wearing other AR devices.

[0132] In the following description of the present specification, a driver assistance system using edge AI is described as an AR device 20 that provides AR worn by a vehicle driver, but the present specification is not limited to the AR device 20 that provides AR worn by a vehicle driver, and also includes all embodiments in which an AR device is worn by a driver of a means of transportation such as an airplane, helicopter, bicycle, or motorcycle.

[0133] According to the present invention, a driver wearing an AR device can verbally command the AR device 20, and the AR device processor unit 23 can recognize the voice of the driver wearing the AR device and perform the task according to the command. For example, if the driver commands, "Find a route to the destination," the AR device processor unit 23 can recognize the voice of the driver wearing the AR device and perform the task according to the command, and can project the route to the destination into the driver's field of view via the display unit 21. For another example, if the driver commands, "Find the nearest cafe on the route to the destination," the AR device processor unit 23 can recognize the voice of the driver wearing the AR device and perform the task according to the command.

[0134] As shown in the drawings, a case will be described in which a collision occurs between vehicles 30-5 and 30-6 of drivers wearing AR devices, or road conditions are stalled. While the attached drawings depict a collision between vehicles 30-5 and 30-6 of drivers wearing AR devices, the collision is not limited to a collision and may include all road events such as accidents and stalls. Also, although not shown in the attached drawings, all or some of the drivers of vehicles 30-5 and 30-6 of drivers wearing AR devices and vehicles 30-1, 30-2, 30-3, and 30-4 of drivers wearing other AR devices are wearing AR devices in this embodiment.

[0135] As shown in the figure, when a collision occurs between vehicles 30-5 and 30-6 whose drivers are wearing AR devices, the AR devices 20 worn by the drivers of the vehicles 30-5 and 30-6 whose drivers are wearing AR devices can take a picture of the collision, which the AR device processor 23 processes and transmits / receives to / from the external terminal 100. Upon receiving the data, the external terminal 100 can recognize through the AI ​​processor 115 of the external terminal processor 110 that a collision has occurred between the vehicles 30-5 and 30-6 whose drivers are wearing AR devices, and can transmit the collision occurrence data to the AR devices 20 worn by the drivers of the vehicles 30-1, 30-2, 30-3, and 30-4 whose drivers are wearing other AR devices through the communication module 170.

[0136] According to one embodiment of the present invention, the AR device 20 worn by the driver of each of the vehicles 30-1, 30-2, 30-3, and 30-4, whose driver is wearing another AR device, can visually display through the display unit 21 that a collision accident has occurred near the road on which the driver is currently traveling, and can provide information on whether there will be a certain amount of time delay in reaching the destination because a collision accident has occurred near the road on which the driver is currently traveling.

[0137] 4b, a driver wearing the AR device 20 can view the augmented reality image 21-1 through the display unit 21 and visually confirm that a collision accident has occurred ahead. According to an embodiment of the present invention, the display unit 21 can provide guidance, such as a detour to another road, and the AR device 20 can provide a function for providing route guidance in real time, so that the driver wearing the AR device 20 can prepare in advance for a collision accident ahead by using another road or driving safely through the augmented reality image 21-1.

[0138] As shown in FIG. 4c, a driver wearing the AR device 20 can view the augmented reality image 21-1 through the display unit 21 and visually check the location of the collision ahead in detail.

[0139] The embodiment of FIG. 5 FIG. 5a is a diagram illustrating an embodiment in which an external terminal 100 transmits information about how to use a vehicle 30 of a driver wearing an AR device 20 to a vehicle driver according to an embodiment of the present invention.

[0140] In the following description of the present specification, a driver assistance system using edge AI is described as an AR device 20 that provides AR worn by a vehicle driver, but the present specification is not limited to the AR device 20 that provides AR worn by a vehicle driver, and also includes all embodiments in which an AR device is worn by a driver of a means of transportation such as an airplane, helicopter, bicycle, or motorcycle.

[0141] FIG. 5b is a diagram showing an augmented reality image 21-1 projected into the field of view of a vehicle driver when transmitting information on how to use the vehicle 30 of a driver wearing an AR device from an external terminal 100 according to one embodiment of the present invention to the vehicle driver.

[0142] According to one embodiment of the present invention, a vehicle driver can wear an AR device 20, and the AR device 20 can communicate data with an external terminal 100. Therefore, the AR device 20 can receive vehicle operation methods for a specific vehicle of the vehicle driver from the external terminal 100 and sequentially display them on the display unit 21 of the AR device 20, thereby allowing the vehicle driver to learn how to drive the vehicle.

[0143] According to yet another embodiment of the present invention, a vehicle driver can wear an AR device 20, and the AR device processor unit 23 can detect and recognize the vehicle and sequentially display vehicle operation methods for the vehicle driver's specific vehicle on the display unit 21 of the AR device 20, allowing the vehicle driver to learn how to drive the vehicle.

[0144] According to a specific embodiment, as shown in FIG. 5b, when a vehicle driver is in a vehicle 30 wearing an AR device, an augmented reality image 21-1 may be projected into the field of view of the vehicle driver when the driver wears the AR device 20. If the vehicle driver is inexperienced in driving the vehicle, the projected augmented reality image 21-1 may provide guidance on how to operate the vehicle. For example, the augmented reality image 21-1 may display the words "First, please fasten your safety belt" along with an arrow pointing toward the safety belt, and a voice guidance may be simultaneously output from the AR device 20. This allows the vehicle driver to recognize that they must first fasten their safety belt. Next, when the vehicle driver fastens their safety belt, the AR device processor 23 may recognize that the vehicle driver has fastened their driver's belt, and the display 21 may provide guidance on the next step of the operation. In this case, the augmented reality image 21-1 may provide a phrase such as "Next, please adjust your side mirrors." If the vehicle driver adjusts the side mirrors appropriately, the AR device processor 23 may recognize that the vehicle driver has adjusted the side mirrors appropriately. Next, the augmented reality image 21-1 may provide a phrase such as "Next, release the parking brake," and in this case, the augmented reality image 21-1 may display the parking brake using an arrow or highlighting effect, etc., to enable the vehicle driver to recognize where the parking brake is located. In this case, when the vehicle driver releases the parking brake, the AR device processor 23 may recognize that the vehicle driver has released the parking brake. Next, the augmented reality image 21-1 may provide a phrase such as "Next, start the engine," and in this case, as shown in FIG. 5b, the augmented reality image 21-1 may display the start portion using an arrow or highlighting effect, etc., to enable the vehicle driver to recognize where the engine start portion is located.

[0145] However, the present invention is not limited to this embodiment, and since the operation method of a vehicle may differ depending on the type of vehicle and the model of the vehicle, the AR device 20 may receive information reflecting the type of vehicle, the model of the vehicle, the driver's level of driving inexperience, etc. from the external terminal 100 and provide the augmented reality image 21-1 accordingly. Also, instead of the AR device 20 receiving information reflecting the type of vehicle, the model of the vehicle, the driver's level of driving inexperience, etc. from the external terminal 100, the AR device processor unit 23 may itself detect and recognize the vehicle model, etc., and provide the augmented reality image 21-1 accordingly.

[0146] The embodiment of FIG. FIG. 6 is a diagram illustrating an augmented reality image 21-1 that provides distance information to a preceding vehicle, among an embodiment of the augmented reality image 21-1 displayed on the display unit 21 of the AR device 20 according to the present invention.

[0147] In the following description of the present specification, a driver assistance system using edge AI is described as an AR device 20 that provides AR worn by a vehicle driver, but the present specification is not limited to the AR device 20 that provides AR worn by a vehicle driver, and also includes all embodiments in which an AR device is worn by a driver of a means of transportation such as an airplane, helicopter, bicycle, or motorcycle.

[0148] As shown, the display unit 21 can output data related to maintaining a safe distance from other vehicles located within a critical distance while traveling as an augmented reality image.

[0149] According to one embodiment of the present invention, when a vehicle 30 of a driver wearing an AR device approaches another vehicle or a vehicle 30-1 of a driver wearing another AR device within a predetermined distance while traveling, an image displaying the distance to the preceding vehicle with an arrow may be provided along with a phrase such as "The distance to the preceding vehicle is getting closer. Please drive slowly" and a guidance voice via the augmented reality image 21-1 displayed on the display unit 21 of the AR device 20. According to yet another embodiment, when the distance to the preceding vehicle approaches within a predetermined distance, a phrase such as "The distance to the preceding vehicle is getting closer. Please slow down" may be simultaneously output via the augmented reality image 21-1 displayed on the display unit 21 of the AR device 20 along with a guidance voice from the AR device 20, thereby guiding the vehicle driver to recognize the distance to the preceding vehicle and pay attention to it. In this case, the AR device 20 can recognize when it approaches within a certain distance while driving through communication with the electronic device 10 or the external terminal 100 that constitutes the infrastructure of multiple sensor devices, but the AR device processor unit 23 can also determine by itself when it approaches within a certain distance while driving and simultaneously output a guidance voice from the AR device 20 via the augmented reality image 21-1 displayed on the display unit 21 of the AR device 20, along with a phrase such as "The distance to the vehicle in front is too close. Please slow down," to guide the vehicle driver to recognize the distance to the vehicle in front and pay attention.

[0150] The embodiment of FIG. FIG. 7 illustrates an augmented reality image 21-1 that highlights lanes on a rainy day, among an embodiment of the augmented reality image 21-1 displayed on the display unit 21 of the AR device 20 according to the present invention.

[0151] In the following description of the present specification, a driver assistance system using edge AI is described as an AR device 20 that provides AR worn by a vehicle driver, but the present specification is not limited to the AR device 20 that provides AR worn by a vehicle driver, and also includes all embodiments in which an AR device is worn by a driver of a means of transportation such as an airplane, helicopter, bicycle, or motorcycle.

[0152] As shown, the display unit 21 can output data related to road lanes or road guide markings as an augmented reality image.

[0153] According to one embodiment of the present invention, if the weather is bad, such as rain or snow, while the vehicle 30 of a driver wearing an AR device is traveling, the driver's view may be obstructed and the lanes may appear cloudy. In this case, the lanes may be highlighted or a clear image of the lanes may be provided on the lanes via the augmented reality image 21-1 displayed on the display unit 21 of the AR device 20. According to yet another embodiment, if the weather is bad, such as rain or snow, while the vehicle 30 of a driver wearing an AR device is traveling, the road signs may be highlighted or a clear image of the signs may be provided on the AR device via the augmented reality image 21-1 displayed on the display unit 21 of the AR device 20. In this case, the AR device 20 can recognize that the weather is bad, such as when it is raining or snowing, through communication with the electronic device 10 or the external terminal 100 that constitutes the infrastructure of a large number of sensor devices. However, it is also possible for the AR device processor unit 23 to determine on its own that the weather is bad, such as when it is raining or snowing, and highlight or clearly display the road signs through the augmented reality image 21-1 displayed on the display unit 21 of the AR device 20.

[0154] The embodiment of Figs. 8, 9 and 10 FIG. 8 is a diagram illustrating an embodiment in which, when a vehicle 30 of a driver wearing an AR device according to the present invention enters a parking lot, the optimal parking space is guided taking into account the destination of the vehicle driver through communication with an external terminal 100 and electronic devices 10-1 and 10-2 that constitute an infrastructure of multiple sensor devices.

[0155] In the following description of the present specification, a driver assistance system using edge AI is described as an AR device 20 that provides AR worn by a vehicle driver, but the present specification is not limited to the AR device 20 that provides AR worn by a vehicle driver, and also includes all embodiments in which an AR device is worn by a driver of a means of transportation such as an airplane, helicopter, bicycle, or motorcycle.

[0156] FIG. 9 is a diagram showing an augmented reality image 21-1 that guides the driver of a vehicle 30 wearing an AR device according to the present invention to the optimal parking space taking into account the driver's destination through communication with an external terminal 100 and electronic devices 10-1 and 10-2 that constitute the infrastructure of multiple sensor devices when the driver enters a parking lot.

[0157] FIG. 10 is a diagram showing an augmented reality image 21-1 that provides a route from the parking spot to the destination in the building when the vehicle 30 of a driver wearing the AR device according to the present invention enters a parking lot and the optimal parking space is guided taking into account the destination of the vehicle driver through communication with an external terminal 100 and electronic devices 10-1 and 10-2 that constitute the infrastructure of multiple sensor devices.

[0158] As shown, the display unit 21 can output data related to the parking location with the shortest distance between the destination of the vehicle driver and the parking location in an augmented reality image.

[0159] As shown in the figure, according to one embodiment of the present invention, when a vehicle driver is in a building such as a shopping complex such as a department store or outlet mall and has a destination 40 within the building, the distance to the destination 40 within the building for the vehicle driver may be relatively close depending on the parking location. Therefore, as shown in Fig. 8, an external terminal 100 communicates with an AR device 20 or an electronic device 10 constituting an infrastructure of multiple sensor devices for the building such as the shopping complex, and the AR device 20 may provide guidance on an optimal parking location taking into account the current parking situation and an optimal route to the destination 40 within the building for the vehicle driver.

[0160] As shown in FIG. 9, the vehicle driver may receive guidance to the optimal parking location determined by the AR device 20 through the augmented reality image 21-1, taking into consideration the current parking situation and the optimal route to the specific vehicle driver's destination 40 within the building. Specifically, the vehicle driver may receive visual guidance to the optimal parking location in real time through an arrow on the augmented reality image 21-1, which indicates the vehicle 30 of the driver currently wearing the AR device. As shown in FIG. 10, for example, if the specific vehicle driver's destination 40 within the building is a movie theater within the building, the vehicle driver may recognize that if the AR device 20 parks at the current parking location provided through the augmented reality image 21-1, a route can be taken to reach the specific vehicle driver's destination 40 within the building, and the vehicle driver may recognize that the parking location is the shortest distance to reach the specific vehicle driver's destination 40 within the building. Therefore, the augmented reality image 21-1 may provide the specific vehicle driver with a route to the specific vehicle driver's destination 40 within the building, and may simultaneously provide a voice guidance message such as, "If you're going to the movie theater, parking here is the closest place to go!"

[0161] According to yet another embodiment of the present invention, if an optimal parking location is not available, the AR device processor unit 23 can recommend to the vehicle driver a parking location from among the remaining parking locations that is the shortest distance to the destination 40 within the building of the specific vehicle driver, and the vehicle driver can recognize that the parking location is the shortest distance to reach the destination 40 within the building of the specific vehicle driver. Therefore, the augmented reality image 21-1 can provide a route to the destination 40 within the building of the specific vehicle driver.

[0162] According to another embodiment of the present invention, as shown in FIG. 8 , a plurality of electronic devices 10-1 and 10-2 constituting the infrastructure of a plurality of sensor devices may be located in a parking lot. Communication may be established between the electronic devices 10-1 and 10-2 constituting the infrastructure of a plurality of sensor devices, communication may be established between an external terminal 100 and the electronic devices 10-1 and 10-2 constituting the infrastructure of a plurality of sensor devices, communication may be established between an AR device 20 and the electronic devices 10-1 and 10-2 constituting the infrastructure of a plurality of sensor devices, and communication may be established between the AR device 20 and the external terminal 100. In this case, the external terminal 100 may communicate with the AR device 20 in real time, taking into consideration the availability of vacant spaces in the parking lot, the vehicle driver's preferred parking location, and the vehicle driver's final destination, to determine an optimal parking location. Alternatively, an augmented reality image 21-1 may be provided by highlighting the optimal parking location along with the current location of the driver's vehicle 30 wearing the AR device in the parking lot. In addition, the AR device processor unit 23 can also determine the optimal parking location through communication with the electronic devices 10-1 and 10-2 that constitute the infrastructure of multiple sensor devices, without going through communication between the external terminal 100 and the electronic devices 10-1 and 10-2 that constitute the infrastructure of multiple sensor devices.

[0163] FIG. 11 is a diagram illustrating a traffic accident analysis system according to one embodiment of the present disclosure.

[0164] As shown in FIG. 11, the traffic accident analysis system 1100 includes a camera 1112, a video monitoring device 1114, a management server 1120, a database (DB) 1130, and a user terminal 1140.

[0165] The camera 1112 is installed on the smart pole 1110 and captures various traffic images occurring on the road. The camera 1112 is, for example, a device capable of capturing still and video images, and according to an embodiment, may include one or more image sensors, lenses, an image signal processor (ISP), or a flash (e.g., LED or xenon lamp). The camera 1112 can generate a road traffic image signal corresponding to the input external image and output it to the image monitoring device 1114. The image of the camera 1112 used in the present disclosure is road traffic image captured from above by the camera 1112 installed on the smart pole 1110, and can obtain information on a wider space than vehicle black box images.

[0166] The video monitoring device 1114 may also be installed on the smart pole 1110 and may monitor traffic accidents occurring on the road, including vehicle accidents, pedestrian appearances, or a combination thereof, from road traffic footage captured by the camera 1112. The video monitoring device 1114 may calculate the distance and angle to the vehicle from the footage of the camera 1112 through planar processing of the footage at the time of the traffic accident, generate planar traffic accident footage, and provide the planar traffic accident footage to the management server 1120. The video monitoring device 1114 may also compare the planar traffic accident footage with standard traffic accident footage to determine how similar the planar traffic accident footage is to at least one of the standard traffic accident footage.

[0167] The management server 1120 can manage the traffic accident-related video together with the road traffic video received from each of the video monitoring devices 1114 installed on the multiple smart poles 1110. The management server 1120 can store the traffic accident-related video together with the received road traffic video in the database 1130.

[0168] A large number of standard traffic accident videos may be stored in the database 1130. Accordingly, a large number of standard traffic accident information and a large number of actual traffic accident videos may be stored in the database 1130. Here, the standard traffic accident information is traffic accident data provided by the Road Traffic Authority, etc., and may include the percentage of fault established by court precedents, etc.

[0169] The user terminal 1140 is an electronic device for displaying the similarity of actual traffic accident videos, for example, thought patterns and accident causes, and can receive and display actual traffic accident videos and standard traffic accident videos similar to the actual traffic accident videos from the video monitoring device 1114 or the management server 1120, and can also display the similarity and the percentage of fault. Here, the user terminal 1140 may be an electronic device of the General Insurance Association of Japan or an electronic device of the person involved in the accident.

[0170] 11 shows that the video monitoring device 1114 estimates the similarity of traffic accidents, but the estimation of the similarity of traffic accidents can also be processed by the management server 1120. Therefore, the traffic accident analysis system may be the video monitoring device 1114 or the management server 1120. Meanwhile, in FIG. 11, the management server 1120 and the database 1130 are shown separately, but the database function may be embodied in the management server 1120.

[0171] FIG. 12 is a block diagram for explaining the case where the video monitoring device or the management server shown in FIG. 11 operates as a general electronic device.

[0172] As shown in FIG. 12 , an electronic device 1200 coupled to a network is described. The electronic device 1200 may include a bus 1200, a processor 1220, a memory 1230, an input / output interface 1250, a display 1260, and a communication interface 1270. In some embodiments, the electronic device 1200 may omit at least one of the components or include additional components. The bus 1200 may include circuitry that couples the components 1220-1270 to one another and transmits communications (e.g., control messages or data) between the components. The processor 1220 may include one or more of a central processing unit, an application processor, or a communication processor (CP). The processor 1220 may perform computations and data processing related to, for example, control and / or communication with at least one other component of the electronic device 1200.

[0173] Memory 1230 may include volatile and / or non-volatile memory. Memory 1230 may store, for example, instructions or data related to at least one other component of electronic device 1200. According to one embodiment, memory 1230 may store software and / or programs 1240. Programs 1240 may include, for example, kernel 1241, middleware 1243, application programming interface (API) 1245, and / or application program (or “application”) 1247. At least a portion of kernel 1241, middleware 1243, or API 1245 may be referred to as an operating system. Kernel 1241 may, for example, control or manage system resources (e.g., bus 1200, processor 1220, or memory 1230) used to execute operations or functions embodied in other programs (e.g., middleware 1243, API 1245, or application program 1247). The kernel 1241 may also provide interfaces that allow control or management of system resources by accessing individual components of the electronic device 1200 in middleware 1243, API 1245, or application programs 1247.

[0174] The middleware 1243 may act as an intermediary to enable, for example, the API 1245 or the application program 1247 to communicate with the kernel 1241 and exchange data. The middleware 1243 may also process one or more work requests received from the application program 1247 according to a priority order. For example, the middleware 1243 may assign a priority for using system resources (e.g., the bus 1200, the processor 1220, or the memory 1230) of the electronic device 1200 to at least one of the application programs 1247, and process the one or more work requests. The API 1245 is an interface through which the application 247 controls functions provided by the kernel 1241 or the middleware 1243, and may include at least one interface or function (e.g., command) for file control, window control, video processing, character control, etc. The input / output interface 1250 may, for example, transmit commands or data input from a user or other external device to other components of the electronic device 1200, or output commands or data received from other components of the electronic device 1200 to a user or other external device.

[0175] Display 1260 may include, for example, a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a microelectromechanical system (MEMS) display, or an electronic paper display. Display 1260 may, for example, display various content (e.g., text, images, videos, icons, and / or symbols, etc.) to a user. Display 1260 may include a touchscreen and may receive touch, gesture, proximity, or hover input, for example, using an electronic pen or a part of the user's body.

[0176] The communication interface 1270 may establish communication with, for example, another electronic device (not shown). For example, the communication interface 1270 may be connected to a network via wireless or wired communication to communicate with other electronic devices. Here, the wireless communication may include cellular communication using at least one of LTE, LTE Advance (LTE-A), code division multiple access (CDMA), wideband CDMA (WCDMA), universal mobile telecommunications system (UMTS), wireless broadband (WiBro), or Global System for Mobile Communications (GSM). According to an embodiment, the wireless communication may include at least one of wireless fidelity (WiFi), Bluetooth, Bluetooth Low Energy (BLE), Zigbee, near field communication (NFC), magnetic secure transmission, radio frequency (RF), or a body area network (BAN). According to an embodiment, the wireless communication may include Global Navigation Satellite System (GNSS). The GNSS may be, for example, the Global Positioning System (GPS), the Global Navigation Satellite System (Glonass), the Beidou Navigation Satellite System (hereinafter "Beidou"), or Galileo, the European global satellite-based navigation system. Hereinafter, in this document, "GPS" may be used interchangeably with "GNSS." The wired communication may include, for example, at least one of universal serial bus (USB), high definition multimedia interface (HDMI), recommended standard 232 (RS-232), power line communication, or plain old telephone service (POTS), etc.Network 1262 may include a telecommunications network, such as at least one of a computer network (eg, a LAN or WAN), the Internet, or a telephone network.

[0177] FIG. 13 is a diagram specifically showing a block diagram of a traffic accident analysis system according to an embodiment of the present disclosure, FIG. 14 is a diagram showing examples of two-dimensional images of standard traffic accident types and / or situations according to an embodiment of the present disclosure, FIGS. 15 and 16 are diagrams showing examples of matching camera images to maps as shown according to an embodiment of the present disclosure, FIG. 17 is a diagram showing an example of estimating a standard traffic accident type of an actual traffic accident situation according to an embodiment of the present disclosure, and FIG. 18 is a diagram showing an example of displaying a standard traffic accident estimation result of an actual traffic accident situation according to an embodiment of the present disclosure.

[0178] As shown in FIG. 13, the traffic accident analysis system 1100 may include a camera 1112, a video monitoring device 1114, and a management server 1120.

[0179] The management server 1120 may include a standard traffic accident information storage unit 1310 and a data set construction unit 1312. The standard traffic accident information storage unit 1310 stores a large number of commonly used standard traffic accident information. Here, the standard traffic accident information is traffic accident data provided by the Road Traffic Authority or the like, and may include the percentage of fault established by court precedent or the like. FIG. 14 shows an example of a 2D image of a standard traffic accident categorized by type and / or situation according to an embodiment of the present disclosure.

[0180] The dataset construction unit 1312 constructs a dataset to be used in the traffic accident estimation unit 1330. To do so, the dataset construction unit 1312 may first classify possible traffic accident types, such as vehicle-to-vehicle (car and car), vehicle-to-pedestrian (car and pedestrian), vehicle-to-motorcycle (car and motorcycle), and vehicle-to-bicycle (car and bicycle), using standard traffic accident information video. The dataset construction unit 1312 may perform labeling on the traffic accident video using the video in which the traffic accident type has been classified. Here, the labeling may be classified into categories such as passenger cars, trucks, buses, and ambulances when vehicles are used as the criterion, and adult, elderly person, and child when people are used as the criterion. The dataset construction unit 1312 may further use text data, which is explanatory material explaining the traffic accident, in addition to the traffic accident video. In this case, the dataset may include both video and text. Meanwhile, the dataset construction unit 1312 may construct a dataset by comparing and evaluating a large number of actual traffic accident data using artificial intelligence modeling.

[0181] The video monitoring device 1114 may include a road traffic information receiving unit 1320 , a road traffic information storage unit 1322 , a traffic accident identification unit 1324 , a traffic accident information storage unit 1326 , an omniscient representation unit 1328 , and a traffic accident estimation unit 1330 .

[0182] The road traffic information receiving unit 1320 receives real-time road traffic images provided by the camera 1112 .

[0183] The road traffic information storage unit 1322 stores road traffic videos received by the road traffic information receiving unit 1320. The videos stored in the road traffic information storage unit 1322 may be stored in a new file at predetermined time intervals, for example, every two minutes. That is, a two-minute video may be stored in a first file, the next two-minute video may be stored in a second file, and another two-minute video may be stored in a third file. In addition, only the previous two video files may be maintained, while the remaining files may be deleted. Therefore, the files stored in the road traffic information storage unit 1322 may include the previous two files and one file currently being recorded.

[0184] The traffic accident identification unit 1324 identifies whether there is a traffic accident in the road traffic video being recorded. If the traffic accident identification unit 1324 identifies a traffic accident, the traffic accident identification unit 1324 can copy and store the traffic accident information stored in the road traffic information storage unit 1322 at the time of the traffic accident occurrence in the traffic accident information storage unit 1326. The traffic accident information stored in the traffic accident information storage unit 1326 may be provided to the management server 1120.

[0185] The omniscient representation unit 1328 converts the traffic accident video from the actual traffic accident information stored in the traffic accident information storage unit 1326 into an omniscient representation for a predetermined time, for example, 1 to 5 seconds, from the identification of the accident vehicle to the occurrence of the accident.

[0186] For the omni-intelligent representation of the video, the omni-intelligent representation unit 1328 may first display circular distance lines at predetermined distance intervals, for example, 1 to 10 meters, on the traffic accident video. Here, the circular distance lines may be values ​​previously obtained through video captured by a camera 1112 installed on the smart pole 1110. The omni-intelligent representation unit 1328 may then display a bounding box using the frame of the identified object. Here, the bounding box for the vehicle may be used to calculate the vehicle's traveling direction and angle. The omni-intelligent representation unit 1328 may then match the 2D map with the actual location of the accident site. Here, the 2D planar map may be a map as shown in the figure. As a result, the vehicle position in the traffic accident video may be mapped onto the 2D planar map taking into account the distance and angle on the smart pole 1110. The omni-intelligent representation unit 1328 may display the accident video in an omni-intelligent representation on the 2D planar map within a predetermined time, for example, 1 to 5 seconds, from the identification of the accident vehicle to the occurrence of the accident. An example of camera footage is shown in FIG. 15 and an example of matching to the map as shown is shown in FIG. 16, according to an embodiment of the present disclosure.

[0187] The traffic accident estimation unit 1330 can estimate similar standard traffic accident information by learning actual traffic accident information using the standard traffic accident dataset constructed in this manner. On the other hand, the traffic accident estimation unit 1330 can estimate standard traffic accidents similar to the traffic accident in question and their similarity by learning from an actual traffic accident dataset constructed from actual traffic accident information. In this case, the traffic accident estimation unit 1330 can analyze the video by applying a convolutional neural network (CNN). The CNN consists of a part that extracts image features by effectively recognizing and emphasizing features with neighboring images while maintaining the spatial information of the image, and a part that classifies the image. The feature extraction region may consist of a convolutional layer that searches for image features while minimizing the number of shared parameters using a filter, and a pooling layer that enhances and collects features. FIG. 17 illustrates an example of estimating a standard traffic accident type from an actual traffic accident situation according to an embodiment of the present disclosure.

[0188] Therefore, the traffic accident estimation unit 1330 can compare the two-dimensional planar traffic accident video represented by the omniscient representation unit 1328 with standard traffic accident videos to estimate how similar the planar traffic accident video is to at least one of the standard traffic accident videos. A diagram showing an example of displaying a standard traffic accident estimation result of an actual traffic accident situation according to an embodiment of the present disclosure is shown in FIG.

[0189] FIG. 19 is a flowchart illustrating a traffic accident analysis method according to another embodiment of the present disclosure.

[0190] The dataset construction unit 1312 of the management server 1120 constructs a dataset to be used by the traffic accident estimation unit 1330 from a plurality of standard traffic accident information stored in the standard traffic accident information storage unit 1310 (S810). To do this, the dataset construction unit 1312 first classifies traffic accident types that may occur on roads, such as vehicle-to-vehicle (car and car), car-to-person (car and pedestrian), car-to-motorcycle (car and motorcycle), and car-to-bicycle (car and bicycle), using video of the standard traffic accident information. The dataset construction unit 1312 can perform labeling on the traffic accident video using the video in which the traffic accident type has been classified. Here, the labeling may be classified into categories such as passenger cars, trucks, buses, and ambulances when the vehicle is used as a criterion, and categories such as adults, elderly people, and children when the person is used as a criterion. The dataset construction unit 1312 may further use text data, which is explanatory material explaining traffic accidents, in addition to the traffic accident video. In this case, the dataset may include both video and text.

[0191] The road traffic information receiving unit 1320 of the video monitoring device 1114 receives real-time road traffic video provided from the camera 1112 (S820). The road traffic information storage unit 1322 stores the road traffic video received by the road traffic information receiving unit 1320 (S830). The video stored in the road traffic information storage unit 1322 may be stored in a new file every two minutes, for example.

[0192] The traffic accident identification unit 1324 identifies whether there is a traffic accident in the road traffic video being recorded (S840). If the traffic accident identification unit 1324 identifies a traffic accident, the traffic accident identification unit 1324 copies and stores the traffic accident information stored in the road traffic information storage unit 1322 at the time of the traffic accident in the traffic accident information storage unit 1326 (S850). The traffic accident information stored in the traffic accident information storage unit 1326 may be provided to the management server 1120.

[0193] The omni-intelligent representation unit 1328 converts the traffic accident video from the actual traffic accident information stored in the traffic accident information storage unit 1326 into an omni-intelligent representation in a predetermined time interval, for example, 1 to 5 seconds, from the identification of the accident vehicle to the occurrence of the accident (S860). To generate the omni-intelligent representation of the video, the omni-intelligent representation unit 1328 may first display circular distance lines at predetermined distance intervals, for example, 1 to 10 meters, on the traffic accident video. Here, the circular distance lines may be values ​​previously obtained through video captured by a camera 1112 installed on the smart pole 1110. The omni-intelligent representation unit 1328 may then display a bounding box using the frame of the identified object. Here, the bounding box for the vehicle may be used to calculate the vehicle's traveling direction and angle. The omni-intelligent representation unit 1328 may then match the 2D map with the actual location of the accident site. Here, the 2D planar map may be a map as shown in the figure. As a result, the vehicle positions in the traffic accident video may be mapped onto a two-dimensional planar map taking into account the distance and angle at the smart pole 1110. The omniscient representation unit 1328 can display the accident video on a two-dimensional planar map in an omniscient representation for a predetermined time, for example, one to five seconds, from the identification of the accident vehicle to the occurrence of the accident.

[0194] The traffic accident estimation unit 1330 estimates similar standard traffic accident information by learning actual traffic accident information using the standard traffic accident dataset constructed in this way (S870). On the other hand, the traffic accident estimation unit 1330 can estimate standard traffic accidents similar to the traffic accident in question and their similarity by learning from the actual traffic accident dataset constructed from actual traffic accident information. In this case, the traffic accident estimation unit 1330 can analyze the video by applying a convolutional neural network (CNN). The CNN consists of a part that extracts image features by effectively recognizing and emphasizing features with neighboring images while maintaining the spatial information of the image, and a part that classifies the image. The feature extraction region may consist of a convolutional layer that searches for image features while minimizing the number of shared parameters using a filter, and a pooling layer that enhances and collects features.

[0195] The above-described embodiments may be implemented using hardware components, software components, and / or a combination of hardware and software components. For example, the devices, methods, and components described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as a processor, controller, arithmetic logic unit (ALU), digital signal processor, microcomputer, field programmable gate array (FPGA), programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. A processing device may execute an operating system (OS) and one or more software applications running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to the execution of software. For ease of understanding, a processing device may be described as being a single device; however, those skilled in the art will recognize that a processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing device may include multiple processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

[0196] Methods according to the embodiments may be embodied in the form of program instructions that can be executed by various computer means and stored on a computer-readable medium. The computer-readable medium may include, alone or in combination, program instructions, data files, data structures, and the like. The program instructions stored on the medium may be specially designed and constructed for the embodiments, or may be publicly available to those skilled in the art of computer software. Examples of computer-readable storage media include magnetic media such as hard disks, floppy disks, and magnetic tape; optical media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include not only machine code, such as produced by a compiler, but also high-level language code that can be executed by a computer using an interpreter, for example. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, or vice versa.

[0197] Software may include a computer program, code, instructions, or a combination of one or more of these, which can configure a processing device to operate as desired or instruct the processing device, either individually or collectively. The software and / or data may be permanently or temporarily embodied in some type of machine, component, physical device, virtual device, computer storage medium or device, or transmitted signal wave, to be interpreted by the processing device or to provide instructions or data to the processing device. The software may be distributed across computer systems coupled to a network, stored and executed in a distributed manner. The software and data may be stored on one or more computer-readable recording media.

[0198] Although the embodiments have been described above with reference to limited drawings, those skilled in the art may apply various technical modifications and variations based on the foregoing. For example, the described techniques may be performed in a different order than described, and / or the components of the described systems, structures, devices, circuits, etc. may be combined or combined in a different manner than described, or may be replaced or substituted by other components or equivalents, and still achieve suitable results.

[0199] Therefore, other implementations, other embodiments, and equivalents of the claims are within the scope of the following claims.

Claims

1. In a road event notification system using a sensor device, an electronic device forming an infrastructure of sensor devices communicating with each other according to a communication protocol; a user terminal; The electronic device constituting the infrastructure of the sensor device includes: a sensor device for capturing images of a road; a processor that directly processes an image processing-based artificial intelligence model and determines an event occurring on the road using the image of the road captured by the sensor device as an input of the image processing-based artificial intelligence model; a memory, The user terminal: a user terminal processor unit for receiving data relating to an event occurring on the determined road via the communication protocol; an output unit that outputs the received data, A road event notification system using sensor devices.

2. the user terminal processor unit directly processes an image processing-based artificial intelligence model, receives the image of the road captured from the sensor device, and determines some or all of the events occurring on the road as an input of the image processing-based artificial intelligence model; A road event notification system using the sensor device according to claim 1.

3. The road occurrence event is at least one of a traffic accident on the road, a congestion on the road, construction work on the road, a breakdown of facilities on the road, a restriction on the road, illegal parking and stopping on the road, and an occurrence of a vehicle violating traffic rules on the road. A road event notification system using the sensor device according to claim 1 or 2.

4. the output unit outputs an augmented reality image associated with an event occurring on the road. A road event notification system using the sensor device according to claim 1 or 2.

5. The processor classifies the road occurrence events based on a classification model trained using actual road occurrence events as label data. A road event notification system using the sensor device according to claim 1 or 2.

6. In a road event notification system using a sensor device, The system includes an electronic device constituting an infrastructure of sensor devices communicating with each other through a communication protocol, a user terminal, and an external terminal, The electronic device constituting the infrastructure of the sensor device includes: a sensor device for capturing images of a road; a processor that directly processes an image processing-based artificial intelligence model and determines some or all of the events occurring on the road using the image of the road captured by the sensor device as an input of the image processing-based artificial intelligence model; a memory, the external terminal includes an external terminal processor that processes a general-purpose artificial intelligence model, receives at least one of an image of the road captured by the sensor device and data relating to events occurring on the road determined by the processor, and determines some or all of the events occurring on the road using at least one of the received image of the road captured by the sensor device and the received data relating to events occurring on the road determined by the processor as inputs to the general-purpose artificial intelligence model; The user terminal: a user terminal processor unit that receives at least one of data relating to road events determined by the processor through the communication protocol and data relating to road events determined by the external terminal processor; an output unit that outputs the received data, A road event notification system using sensor devices.

7. the user terminal processor unit directly processes an image processing-based artificial intelligence model, receives the image of the road captured from the sensor device, and determines some or all of the events occurring on the road as an input of the image processing-based artificial intelligence model; A road event notification system using the sensor device according to claim 6.

8. The road occurrence event is at least one of a traffic accident on the road, a congestion on the road, construction work on the road, a breakdown of facilities on the road, a restriction on the road, illegal parking and stopping on the road, and an occurrence of a vehicle violating traffic rules on the road. A road event notification system using the sensor device according to claim 6 or 7.

9. the output unit outputs an augmented reality image associated with an event occurring on the road. A road event notification system using the sensor device according to claim 6 or 7.

10. The processor classifies the road occurrence events based on a classification model trained using actual road occurrence events as label data. A road event notification system using the sensor device according to claim 6 or 7.

Images