MEMORY-LINKABLE HIERARCHICALLY STRUCTURE-BASED EVENT DETECTION METHOD AND APPARATUS - Patent application

Abstract

The system can detect key events in videos by analyzing and learning features that appear in event footage using a deep learning model, and provides an efficient processing structure for long-term events using a hierarchical structure. [Solution] The present invention relates to a memory-linkable hierarchical structure-based event detection method and device, which includes a first event processing step on a terminal device, which includes a step of collecting real-time driving footage from a moving vehicle and applying the real-time driving footage to a first event detection model to generate first event information related to the vehicle's driving, and a second processing step on the terminal device, which includes a step of collecting the first event information and applying the first event information to a second event detection model to generate second event information related to the vehicle's driving.

Description

[Technical Field]

[0001] The present invention relates to a first-person event type determination technology, and to a system that can effectively detect various events with a hierarchical structure that can be linked to memory from real-time driving images collected from multiple cameras inside a vehicle. [Background technology]

[0002] With the development of digital technology, various sensors and devices are being applied to vehicles. For example, cameras installed in vehicles record images of the vehicle's surroundings, which is very useful for identifying the causes of events such as traffic accidents. However, vehicle images are mostly used for recording driving conditions rather than for real-time analysis, and are not widely used for real-time situation analysis and risk prediction.

[0003] In other words, if various abnormal phenomena that occur while a vehicle is running can be accurately detected in advance, it will be possible to warn the driver in advance or prevent accidents by using the vehicle's own automatic control function.To this end, research into predicting abnormal situations in advance using video information collected from various cameras installed in vehicles is becoming more active using artificial intelligence technology. [Prior art documents] [Patent documents]

[0004] Korean Patent No. 10-2105954 Summary of the Invention [Problem to be solved by the invention]

[0005] One embodiment of the present invention aims to provide a system that can detect key events in a video by analyzing and learning features that appear in event footage using a deep learning model, and that includes an efficient processing structure for long-term events using a hierarchical structure. [Means for solving the problem]

[0006] Among the embodiments, the memory-interlockable hierarchical structure-based event detection method includes a first event processing step including a step of collecting real-time driving video from a moving vehicle on a terminal device and applying the real-time driving video to a first event detection model to generate first event information related to the driving of the vehicle; and a second event processing step including a step of collecting the first event information on the terminal device and applying the first event information to a second event detection model to generate second event information related to the driving of the vehicle.

[0007] Here, the second event information may correspond to upper layer information derived using the first event information as lower layer information of the hierarchical structure while forming a hierarchical structure together with the first event information.

[0008] The first event information may be an abnormal event occurring during the running of the vehicle, and may include an accident event, a bump event, an anomaly motion event, and a dangerous event.

[0009] Each of the first and second event processing steps may include a step of transmitting the real-time driving video or the first event information to a cloud server and storing the same on the terminal device; a step of generating, on the cloud server, learning data for additional learning of an event detection model for the corresponding step from the real-time driving video or the first event information; a step of additionally learning the event detection model for the corresponding step by learning the learning data on the cloud server; and a step of receiving, on the terminal device, the additionally learned event detection model for the corresponding step from the cloud server and updating the event detection model.

[0010] The step of generating the learning data may include a step of receiving label information regarding the real-time driving video or the first event information from a user terminal and generating the learning data; and a step of generating the learning data by auto-labeling using an event detection model of the corresponding step.

[0011] The second event processing step may include generating the second event information for each second event section that is extended in at least one of the spatial domain and the time domain from the first event section in which the first event information is collected, according to a hierarchical structure formed based on the spatial domain and the time domain.

[0012] The event detection method may further include a hierarchical structure expansion step of iteratively expanding the hierarchical structure by using the second event information as lower hierarchical information of the hierarchical structure to generate third event information corresponding to upper hierarchical information of the second event information.

[0013] The hierarchical structure expanding step may include a step of selectively generating event information of each event section from the real-time driving video through an event detection model of each event processing step formed by iteratively expanding the hierarchical structure.

[0014] In one embodiment, the memory-interlockable hierarchical structure-based event detection device includes a first event processing unit that operates on a terminal device and performs the steps of collecting real-time driving video from a moving vehicle and applying the real-time driving video to a first event detection model to generate first event information related to the driving of the vehicle; and a second processing unit that operates on the terminal device and performs the steps of collecting the first event information and applying the first event information to a second event detection model to generate second event information related to the driving of the vehicle. [Effects of the Invention]

[0015] The disclosed technology has the following effects. However, this does not mean that a particular embodiment should include all of the following effects or only the following effects, and therefore the scope of the disclosed technology should not be understood to be limited thereby.

[0016] A memory-linkable hierarchical structure-based event detection method and apparatus according to an embodiment of the present invention is a memory-linkable hierarchical structure-based event detection method and apparatus that can receive images acquired from multiple cameras in a vehicle and determine the type of a first-person event.

[0017] The memory-linkable hierarchical structure-based event detection method and apparatus according to an embodiment of the present invention is a memory-linkable hierarchical structure-based event detection method and apparatus that can provide an efficient hierarchical processing structure for short-term, long-term, and ultra-long-term memories depending on the type and utility of the event.

[0018] Therefore, the present invention can distinguish between vehicle events such as speed bumps or potholes and accidents caused by collisions with vehicles or facilities, detect events that require long-term memory such as tailgating, solve problems that require very long-term memory such as a driver's driving score or habits, and provide information for efficient management of vehicle control, driver information, and event status. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a diagram illustrating an event detection system according to the present invention; [Figure 2] FIG. 2 is a diagram illustrating the system configuration of the terminal device of FIG. [Figure 3] FIG. 1 is a diagram illustrating a functional configuration of an event detection device according to the present invention. [Figure 4]1 is a flowchart illustrating a memory-interlockable hierarchical structure-based event detection method according to the present invention. [Figure 5] FIG. 1 illustrates a hierarchical structure for efficient handling of events according to the present invention. [Figure 6] FIG. 2 illustrates an embodiment of an event detection process according to the present invention. [Figure 7] FIG. 2 is a diagram illustrating the operation of an event detection model according to the present invention. [Figure 8] FIG. 2 illustrates one embodiment of an event hierarchy in accordance with the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0020] The description of the present invention is merely an embodiment for the purpose of structural or functional description, and therefore the scope of the present invention should not be construed as being limited by the embodiments described herein. In other words, since the embodiments are subject to various modifications and can have various forms, the scope of the present invention should be understood to include equivalents that can realize the technical idea. Furthermore, the objectives or effects presented in the present invention do not mean that a particular embodiment should include all of these or that it should not include only such effects, and therefore the scope of the present invention should not be understood as being limited thereby.

[0021] Meanwhile, the meanings of the terms used in this application should be understood as follows.

[0022] Terms such as "first" and "second" are used to distinguish one component from another and should not be used to limit the scope of rights. For example, a first component may be called a second component, and similarly, a second component may be called a first component.

[0023] When a component is said to be "connected" to another component, it should be understood that it may be directly connected to the other component, but there may also be other components in between. On the other hand, when a component is said to be "directly connected" to another component, it should be understood that there are no other components in between. Meanwhile, other expressions describing the relationship between components, such as "between" and "immediately between," or "adjacent to" and "directly adjacent to," should be interpreted similarly.

[0024] Singular expressions should be understood to include plural expressions unless the context clearly dictates otherwise, and terms such as "comprise" or "have" are intended to specify the presence of embodied features, numerals, steps, operations, components, parts, or combinations thereof, but should be understood not to preclude the presence or possible addition of one or more other features, numerals, steps, operations, components, parts, or combinations thereof.

[0025] In each step, identifiers (e.g., a, b, c, etc.) are used for convenience of explanation, and do not describe the order of each step. Unless the context clearly dictates a specific order, each step may occur in a different order than specified. That is, each step may occur in the same order as specified, may occur substantially simultaneously, or may occur in the reverse order.

[0026] The present invention can be embodied as computer-readable code on a computer-readable recording medium, which includes all types of recording devices that store data that can be read by a computer system. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc. Furthermore, the computer-readable recording media can be distributed among computer systems connected via a network, so that the computer-readable code can be stored and executed in a distributed manner.

[0027] Unless otherwise defined, all terms used herein have the same meaning as commonly understood by a person of ordinary skill in the art to which the present invention pertains. Terms commonly used and defined in advance are interpreted to be consistent with the meaning they have in the context of the relevant art, and are not interpreted as having an ideal or overly formal meaning unless expressly defined herein.

[0028] FIG. 1 is a diagram illustrating an event detection system according to the present invention.

[0029] 1, the event detection system 100 is configured to perform a memory-interlockable hierarchical structure-based event detection method according to the present invention, and may include a terminal 110, a cloud server 130, and a database 150. For convenience of explanation, the terminal 110, the cloud server 130, and the database 150 are described as independent devices, but this is not necessarily limited thereto, and at least two different devices may be integrated into one device according to various embodiments for event detection.

[0030] The terminal 110 may correspond to a computing device that can generate and store images and transmit the images to the cloud server 130. In one embodiment, the terminal 110 may be implemented with a camera module that can capture images and videos. For example, the terminal 110 may be implemented as a camera sensor or black box installed in a vehicle. The terminal 110 may also be implemented as a smartphone, tablet PC, laptop, etc., but is not limited thereto, and may be implemented as various devices including a camera.

[0031] The terminal 110 may also be implemented as one device constituting the event detection system 100 according to the present invention. The terminal 110 may also be implemented so that it is connected to the cloud server 130 via a network, and multiple terminals 110 are simultaneously connected to one cloud server 130 as needed.

[0032] The cloud server 130 may be implemented as a server corresponding to a computer or program that receives and stores data transmitted from the terminal 110 and generates and stores information necessary for event detection through data analysis. For example, the cloud server 130 generates learning data for an event detection model based on vehicle driving videos captured by the terminal 110 and detected event information. The cloud server 130 may perform additional learning on the event detection model using the learning data and distribute the updated event detection model to each terminal 110. To this end, the cloud server 130 is connected to the terminal 110 via a wired network or a wireless network such as Bluetooth®, Wi-Fi®, or LTE®, and can transmit and receive data to and from the terminal 110 via the network.

[0033] In addition, the cloud server 130 may be implemented to operate in a cloud environment, or may be implemented to operate in connection with an independent external system (not shown in FIG. 1) according to various embodiments for performing the event detection method according to the present invention. For example, the cloud server 130 may operate as a server in a cloud environment, and may preferably be implemented to have computing performance superior to that of the terminal 110.

[0034] On the other hand, each of the terminal 110 and the cloud server 130 may include multiple modules implemented independently to perform related operations, or may be implemented with a control module that manages the control and data flow for the multiple modules.

[0035] The database 150 corresponds to a storage device that stores various information required in the operation process of the cloud server 130. For example, the database 150 may store short-term, long-term, and ultra-long-term event information received by at least one terminal 110, or may store information related to a learning algorithm and learning data for additional learning of an event detection model, but is not necessarily limited thereto. The database 150 may store information collected or processed in various forms in the course of performing the memory-interoperable hierarchical structure-based event detection method according to the present invention in conjunction with the terminal 110.

[0036] FIG. 2 is a diagram illustrating the system configuration of the terminal device of FIG.

[0037] As shown in FIG. 2, the terminal 110 may include a processor 210, a memory 230, a user input / output unit 250, and a network input / output unit 270.

[0038] The processor 210 may execute procedures for performing a memory-interlockable hierarchical structure-based event detection method according to an embodiment of the present invention, may manage the memory 230 that is read or created during this process, and may schedule a synchronization time between the volatile memory and the non-volatile memory in the memory 230. The processor 210 may control the overall operation of the terminal 110, and may be electrically connected to the memory 230, the user input / output unit 250, and the network input / output unit 270 to control the flow of data therebetween. The processor 210 may be implemented as a central processing unit (CPU) or a graphics processing unit (GPU) of the terminal 110 or the cloud server 130.

[0039] The memory 230 may include a secondary storage device implemented as a non-volatile memory such as a solid state disk (SSD) or a hard disk drive (HDD) and used to store all data required by the terminal 110, or a primary storage device implemented as a volatile memory such as a random access memory (RAM). The memory 230 may also store a set of instructions that are executed by the processor 210 electrically connected thereto to perform the memory-interoperable hierarchical structure-based event detection method according to the present invention.

[0040] The user input / output unit 250 includes an environment for receiving user input and an environment for outputting specific information to a user, and may include, for example, an input device including an adapter such as a touchpad, touchscreen, visual keyboard, or pointing device, and an output device including an adapter such as a monitor or touchscreen. In one embodiment, the user input / output unit 250 corresponds to a computing device connected via a remote connection, and in such a case, the terminal 110 may correspond to an independent node of a network to which the computing device is connected.

[0041] The network input / output unit 270 provides a communication environment for connection with other devices via a network, and may include adapters for communication with, for example, a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a value-added network (VAN), etc. The network input / output unit 270 may also be implemented to provide a short-range communication function such as Wi-Fi or Bluetooth® or a wireless communication function of 4G or higher for wireless data transmission.

[0042] FIG. 3 is a diagram illustrating the functional configuration of the event detection device according to the present invention.

[0043] As shown in FIG. 3, the event detection device 300 may include a first event processing unit 310, a second event processing unit 330, a hierarchical structure extending unit 350, and a control unit 370.

[0044] Here, embodiments of the present invention do not necessarily include all of the above functional components at the same time, but may omit some of the components or selectively include some or all of the components depending on the embodiment. Furthermore, an embodiment of the present invention may be implemented as independent modules selectively including some of the components, and the memory-interlockable hierarchical structure-based event detection method according to the present invention may be performed by interlocking between the modules. The operation of each component will now be described in detail.

[0045] The first event processing unit 310 operates on the terminal 110 and may collect real-time driving images from a moving vehicle and generate first event information related to the vehicle's driving by applying the real-time driving images to a first event detection model. The first event processing unit 310 may be composed of a plurality of modules that independently perform each step.

[0046] More specifically, the first event processing unit 310 may be implemented within the terminal 110 installed and operated in a vehicle, and may collect vehicle surroundings images collected through the terminal 110 during vehicle driving as real-time driving images. For example, the vehicle surroundings images may include a front image in the direction in which the vehicle is driving, a rear image in the opposite direction to the driving direction, and a side image in the direction of the side of the vehicle. The first event processing unit 310 may directly interface with a camera module in the terminal 110 to directly collect images captured by the camera module.

[0047] In addition, the first event processing unit 310 may store the collected real-time driving video in a memory and input the real-time driving video to a pre-constructed first event detection model to acquire first event information. Here, the first event detection model may correspond to a deep learning model that receives the real-time driving video as input and generates first event information detected in the real-time driving video as output. Here, each frame image of the driving video may be used as input data depending on the definition of the first event detection model, but is not necessarily limited to this.

[0048] In one embodiment, the first event information may be an abnormal event occurring during the vehicle's driving process, such as an accident event, a bump event, an abnormal motion event, or a dangerous event. The first event information corresponds to event information derived as a result of analyzing real-time driving video, and may be detected based on the smallest unit interval of time and space.

[0049] For example, the first event information may include a short-term event that has a low probability of occurring during normal vehicle driving. An accident event may include an event related to a collision between vehicles or a collision with an external obstacle. A bump event may include an event caused by a road step, a speed bump, or a small obstacle. An abnormal motion event may include an event caused by an abnormal driving operation such as sudden acceleration, sudden stopping, or sharp turning. A risk event may include an event corresponding to a dangerous situation during driving such as lane departure, rainy road, snowy road, or fog.

[0050] That is, the first event may correspond to a short-term event with the smallest time unit and a local area event with the smallest spatial unit. Therefore, the first event detection model may correspond to a short-term event detection model that is detected based on the smallest unit in the time and spatial domains.

[0051] In one embodiment, the first event processing unit 310 may perform the steps of transmitting and storing real-time driving video or first event information to the cloud server 130 on the terminal 110, generating training data for additional training of an event detection model for the corresponding step from the real-time driving video or the first event information on the cloud server 130, training the training data on the cloud server 130 to additionally train the event detection model for the corresponding step, and receiving and updating the additionally trained event detection model for the corresponding step from the cloud server 130 on the terminal 110.

[0052] More specifically, the first event processing unit 310 can transmit real-time driving video collected from the terminal 110 to the cloud server 130 connected to the terminal 110. The cloud server 130 can receive and store the data and perform additional learning of the first event detection model. That is, the cloud server 130 can perform a labeling operation to generate learning driving data based on the real-time driving video in conjunction with the first event processing unit 310. The labeling operation corresponds to an operation of assigning labels to learning data extracted from the real-time driving video. In addition, the cloud server 130 can generate learning driving data from label information of the first event information detected by the first event detection model.

[0053] Thereafter, the cloud server 130 can additionally learn the learning driving data through the first event detection model, and the first event detection model updated through the additional learning can be transmitted from the cloud server 130 to the corresponding terminal 110. The terminal 110 can receive the additionally learned first event detection model and update the previously stored first event detection model to the additionally learned model.

[0054] In one embodiment, the first event processing unit 310 may generate the learning data by receiving label information about real-time driving video from the user terminal and generating the learning data through auto-labeling using the first event detection model. Meanwhile, the second event processing unit 330 may generate the learning data in a similar manner. That is, the second event processing unit 330 may receive label information about the first event information from the user terminal or generate the learning data through auto-labeling using the second event detection model.

[0055] The second event processing unit 330 operates on the terminal 110 and may collect first event information and generate second event information related to vehicle driving by applying the first event information to a second event detection model. Here, the second event information may form a hierarchical structure with the first event information and correspond to upper-level information derived using the first event information as lower-level information in the hierarchical structure. That is, the second event processing unit 330 may form an efficient hierarchical processing structure for event detection in conjunction with the first event processing unit 310.

[0056] For example, if the first event processing unit 310 detects first event information, which is collision event information, from a vehicle driving video in a video section of 1 second in length, the second event processing unit 330 can generate second event information such as a tailgating event or a chain reaction collision event based on the collision event information detected in a video section extended to 10 seconds.

[0057] In one embodiment, the second event processor 330 may perform an update operation on the event detection model performed in the first event processor 310, and the specific operation related to this may be the same as the operation for the first event processor 310.

[0058] In one embodiment, the second event processor 330 may generate second event information for each second event section, which is extended in at least one of the spatial and time domains from the first event section in which the first event information is collected, according to a hierarchical structure formed based on the spatial domain and the time domain. For example, the first event processor 310 may detect a first event from a driving video every 10 seconds in the time domain, and the second event processor 330 may detect a second event based on the first event detected every 5 minutes. In other words, the second event information may correspond to event information detected as a result of integrating the first event information. Event detection in the spatial domain may be performed in the same manner as event detection in the time domain, and a description thereof will be omitted.

[0059] The hierarchical structure expanding unit 350 can iteratively expand the hierarchical structure by using the second event information as lower hierarchical information of the hierarchical structure to generate third event information corresponding to upper hierarchical information of the second event information. The hierarchical structure expanding unit 350 can operate on the terminal 110, but is not limited thereto, and can also operate on the cloud server 130 as necessary. That is, the hierarchical structure of event detection can be selectively expanded by iteration.

[0060] In addition, the hierarchical structure of event detection can be implemented in conjunction with a memory structure, so that event information collected in each event detection step can be independently stored and managed in a corresponding hierarchical memory area.

[0061] For example, a third event processing unit may be added based on the event detection structure between the first event processing unit 310 and the second event processing unit 330. The third event processing unit may convert the second event information detected by the second event processing unit 330 into lower layer information and generate third event information corresponding to upper layer information.

[0062] That is, if the first event information is a short-term event and the second event information is a long-term event, the third event information may correspond to an ultra-long-term event. Also, the first event information may be stored in a short-term memory area, the second event information may be stored in a long-term memory area, and the third event information may be stored in an ultra-long-term memory area. In this way, the hierarchical structure extension unit 350 can iteratively add event processing steps of higher layers to an existing hierarchical structure to extend the hierarchical structure.

[0063] In one embodiment, the hierarchical structure expansion unit 350 can selectively generate event information for each event section from real-time driving video through an event detection model for each event processing step formed by iteratively expanding the hierarchical structure. For example, a hierarchical structure can be formed by sequentially connecting first through third event processing units to detect short-term events, long-term events, and ultra-long-term events related to vehicle driving video. The hierarchical structure expansion unit 350 can provide selective event detection results by operating specific event processing steps according to the hierarchical structure.

[0064] The control unit 370 controls the overall operation of the event detection device 300 and can manage the flow of control or data between the first event processing unit 310, the second event processing unit 330, and the hierarchical structure extension unit 350.

[0065] FIG. 4 is a flowchart illustrating a memory-interlockable hierarchical structure-based event detection method according to the present invention.

[0066] 4, the event detection device 300 collects real-time driving video from a moving vehicle on the terminal 110 via the first event processing unit 310 (step S410). The event detection device 300 applies the real-time driving video to a first event detection model on the terminal 110 via the first event processing unit 310 to generate first event information related to the vehicle's driving (step S420). The event detection device 300 transmits the real-time driving video and the first event information on the terminal 110 to the cloud server 130 via the first event processing unit 310 (step S430).

[0067] Furthermore, the event detection device 300 generates learning driving data from the real-time driving video or the first event information on the cloud server 130 via the first event processing unit 310 (step S440). The event detection device 300 updates the first event detection model by additionally learning the learning driving data on the cloud server 130 via the first event processing unit 310 (step S450). The event detection device 300 distributes the updated first event detection model to the terminals 110 on the cloud server 130 via the first event processing unit 310, so that the event detection model is updated in each terminal 110 (step S460).

[0068] FIG. 5 is a diagram illustrating a hierarchical structure for efficient handling of events according to the present invention.

[0069] 5, the event detection device 300 can build an efficient hierarchical processing structure corresponding to short-term, long-term, and ultra-long-term memories according to the type and utility of an event. That is, the event detection device 300 can perform hierarchical event detection from real-time driving images collected during the driving process of a vehicle.

[0070] In FIG. 5, in the short-term event processing step (S510), based on the connection between the terminal 110 and the cloud server 130, operations related to collecting real-time driving images and detecting short-term events can be performed through the terminal 110, and operations related to collecting driving data for learning and additional learning of the short-term event detection model can be performed through the cloud server 130.

[0071] In addition, in the long-term event processing step (S530), operations related to collecting compressed short-term event information and detecting long-term events can be performed via the terminal 110 based on the short-term event information generated in the short-term event processing step (S510), and operations related to collecting short-term data for learning and additional learning of the long-term event detection model can be performed via the cloud server 130.

[0072] In addition, in the ultra-long-term event processing step (S550), operations related to collecting compressed long-term event information and detecting ultra-long-term events can be performed via the terminal 110 based on the long-term event information generated in the long-term event processing step (S530), and operations related to collecting long-term data for learning and additional learning of the ultra-long-term event detection model can be performed via the cloud server 130.

[0073] In one embodiment, the event detection device 130 may logically divide the memory 230 on the terminal 110 into a short-term memory area, a long-term memory area, an ultra-long-term memory area, etc. according to a hierarchical structure. In this case, the first event processing unit 310 may operate in conjunction with the short-term memory area, the second event processing unit 330 may operate in conjunction with the long-term memory area, and the third event processing unit may operate in conjunction with the ultra-long-term memory area.

[0074] Furthermore, the event detection device 130 may be implemented to operate in conjunction with an independent instance created by the cloud server 130. In this case, the first event processing unit 310 may operate in conjunction with a short-term event detection instance, the second event processing unit 330 may operate in conjunction with a long-term event detection instance, and the third event processing unit may operate in conjunction with an ultra-long-term event detection instance.

[0075] FIG. 6 is a diagram illustrating one embodiment of an event detection process according to the present invention.

[0076] As shown in FIG. 6, the event detection device 300 can effectively detect events occurring during vehicle travel through a hierarchical event detection model.

[0077] 6, the event detection device 300 may divide a real-time vehicle driving video into predetermined time intervals. The divided video 610 may be input to a pre-constructed event detection model to generate an output indicating whether an event has occurred. That is, the event detection model may provide an output indicating whether an event has occurred for each divided video received as an input.

[0078] The event image 630 in which an event is determined to have occurred by the event detection model is then transmitted to a higher-level event detection model and can be used for event detection over a longer time period than the current time period. In Fig. 6, a dangerous event may occur in a divided image 610 based on the distance to the preceding vehicle, and the event detection device 300 may transmit information about four event images 630 in which a dangerous event is detected among the six divided images 610 to the next step, so that event detection can be performed in a higher level.

[0079] FIG. 7 is a diagram illustrating the operation of the event detection model according to the present invention.

[0080] 7, the event detection device 300 can perform end-to-end learning and event detection using a video recorded for a certain period of time from a moving vehicle as an input. The event detection device 300 can learn spatial and temporal domain features that appear in the event video.

[0081] For example, the event detection device 300 can learn hierarchically extended features in the spatial domain in a step-by-step manner based on front, rear, and side images captured through multiple cameras in a vehicle, thereby enabling the event detection device 300 to effectively distinguish between accident-like events such as speed bumps and potholes and actual accident-related events in a video.

[0082] In addition, the event detection device 300 can acquire compressed event information as upper layer information through an event detection model based on lower layer information extracted from a video. At this time, the compressed event information can include the type, availability, and status of the event.

[0083] FIG. 8 is a diagram illustrating one embodiment of an event hierarchy according to the present invention.

[0084] 8, the event detection device 300 can effectively detect major events in real-time driving videos by analyzing and learning features that appear in the event video using a deep learning model. In particular, the event detection device 300 can build an efficient processing structure for long-term events using a hierarchical structure.

[0085] 8, the event detection device 300 can detect events that are not typical driving situations as short-term event information from real-time driving video through a short-term detection model. For example, the short-term event information may include Accident (presence or absence of a collision, collision direction, etc.), Bump (road step, speed bump, small obstacle), Anomaly motion (sudden acceleration, sudden stop, sharp turn), Dangerous (dangerous situation, lane departure), etc.

[0086] In addition, the event detection device 300 can detect events that require information for a certain period of time as long-term event information from short-term event information through the long-term detection model. For example, the long-term event information may include aggressive driving, chain-rear-end collisions, secondary accidents, short-term driving behavior, collected short-term events, etc.

[0087] In addition, the event detection device 300 can detect events that require long-term information as ultra-long-term event information from the long-term event information through the ultra-long-term detection model. For example, the ultra-long-term event information may include a driver's driving score, driving characteristics (such as aggressive driving), main driving roads and environments, etc.

[0088] While the present invention has been described above with reference to preferred embodiments, those skilled in the art will appreciate that the present invention can be modified and changed in various ways without departing from the spirit and scope of the present invention as set forth in the claims. [Explanation of symbols]

[0089] 100 Event Detection System 110 Terminal 130 Cloud Server 150 databases 210 processor 230 memory 250 User input / output unit 270 Network Input / Output Unit 300 Event Detection Device 310 First event processing section 330 Second Event Processing Unit 350 Hierarchical Structure Extension 370 Control Unit

Claims

1. a first event processing step including the steps of: collecting real-time driving images from a moving vehicle on a terminal; and applying the real-time driving images to a first event detection model to generate first event information related to the driving of the vehicle; a second event processing step including the steps of: collecting the first event information on the terminal; and applying the first event information to a second event detection model to generate second event information related to the running of the vehicle; The second event information corresponds to upper layer information derived using the first event information as lower layer information of the hierarchical structure while forming a hierarchical structure with the first event information.

2. The first event information is 2. The memory-interlockable hierarchical structure-based event detection method of claim 1, wherein the abnormal events occurring during the vehicle driving process include an accident event, a bump event, an abnormal motion event, and a dangerous event.

3. Each of the first and second event processing steps includes: transmitting the real-time driving video or the first event information to a cloud server on the terminal and storing the video; generating, on the cloud server, learning data for additional learning of an event detection model of a corresponding step from the real-time driving video or the first event information; a step of additionally learning the event detection model of the corresponding step by learning the learning data on the cloud server; and 2. The memory-interlockable hierarchical structure-based event detection method of claim 1, further comprising: receiving, on the terminal, an event detection model for the corresponding step that has been additionally learned from the cloud server, and updating the event detection model.

4. The step of generating learning data includes: receiving label information relating to the real-time driving video or the first event information from a user terminal and generating the learning data; and 4. The memory-interlockable hierarchical structure-based event detection method of claim 3, further comprising: generating the training data by auto-labeling using the event detection model of the corresponding step.

5. The second event processing step includes:

2. The method of claim 1, further comprising: generating the second event information for each second event interval extended from a first event interval in which the first event information is collected to at least one of the spatial domain and the time domain in accordance with a hierarchical structure formed based on the spatial domain and the time domain.

6. 2. The memory-interlockable hierarchical structure-based event detection method of claim 1, further comprising: a hierarchical structure expansion step of iteratively expanding the hierarchical structure by using the second event information as lower-level information of the hierarchical structure to generate third event information corresponding to upper-level information of the second event information.

7. The hierarchical structure expansion step 7. The memory-interlockable hierarchical structure-based event detection method of claim 6, further comprising: selectively generating event information for each event section from the real-time driving video through an event detection model for each event processing step formed by iteratively expanding the hierarchical structure.

8. a first event processing unit that operates on a terminal device and performs the steps of collecting real-time driving images from a moving vehicle and applying the real-time driving images to a first event detection model to generate first event information related to the driving of the vehicle; a second event processing unit that operates on the terminal and performs the steps of collecting the first event information and applying the first event information to a second event detection model to generate second event information related to the running of the vehicle; The second event information corresponds to upper hierarchical information derived using the first event information as lower hierarchical information of the hierarchical structure while forming a hierarchical structure with the first event information.

Images