Memory-linkable hierarchical structure-based event detection method and apparatus

Abstract

The system can detect key events in videos by analyzing and learning features that appear in event footage using a deep learning model, and provides an efficient processing structure for long-term events using a hierarchical structure. [Solution] The present invention relates to a memory-linkable hierarchical structure-based event detection method and device, which includes a first event processing step on a terminal device, which includes a step of collecting real-time driving footage from a moving vehicle and applying the real-time driving footage to a first event detection model to generate first event information related to the vehicle's driving, and a second processing step on the terminal device, which includes a step of collecting the first event information and applying the first event information to a second event detection model to generate second event information related to the vehicle's driving.

Description

Technical Field

[0001] The present invention relates to a first-person event type determination technology, and relates to a system that can effectively detect various events with a hierarchical structure capable of memory linkage from real-time driving videos collected from a plurality of cameras in a vehicle.

Background Art

[0002] Due to the development of digital technology, various sensors and devices are applied to vehicles. For example, cameras installed in vehicles leave videos around the vehicle as records, which are very useful for understanding the causes of event situations such as traffic accidents. However, videos related to vehicles are mostly utilized for recording driving situations rather than real-time analysis, and in fact, their utilization rate in real-time situation analysis and danger prediction is low.

[0003] That is, if various abnormal phenomena occurring during vehicle driving can be accurately detected in advance, it will be possible to warn the driver in advance or prevent accidents through the automatic control function of the vehicle itself. For this purpose, research for predicting abnormal situations in advance using video information collected from various cameras mounted on the vehicle is tending to be activated using artificial intelligence technology. [[ID=…]]

Prior Art Documents

Patent Documents

[0004] Korean Registered Patent No. 10-2105954

Summary of the Invention

Problems to be Solved by the Invention

[0005] One embodiment of the present invention aims to provide a system that can detect major events in a video by analyzing and learning features appearing in event videos with a deep learning model, and includes an efficient processing structure for long-term events using a hierarchical structure.

Means for Solving the Problems

[0006] Among the embodiments, a memory-linkable hierarchical structure-based event detection method includes a first event processing step on a terminal device, which includes the steps of: collecting real-time driving video from a moving vehicle; and applying the real-time driving video to a first event detection model to generate first event information relating to the vehicle's movement; and a second event processing step on the terminal device, which includes the steps of: collecting the first event information; and applying the first event information to a second event detection model to generate second event information relating to the vehicle's movement.

[0007] Here, the second event information may correspond to higher-level information derived by using the first event information as lower-level information in the hierarchical structure, while forming a hierarchical structure with the first event information.

[0008] The first event information is an abnormal event that occurs during the vehicle's driving process, and may include accident events, bump events, anomaly motion events, and dangerous events.

[0009] Each of the first and second event processing steps may include, on the terminal device, a step of transmitting the real-time driving video or the first event information to a cloud server for storage; on the cloud server, a step of generating training data for further training of the event detection model for the relevant step from the real-time driving video or the first event information; on the cloud server, a step of learning the training data to further train the event detection model for the relevant step; and on the terminal device, a step of receiving and updating the further trained event detection model for the relevant step from the cloud server.

[0010] The step of generating the training data may include a step of generating the training data by receiving label information relating to the real-time driving video or the first event information from the user terminal; and a step of generating the training data by auto-labeling using the event detection model of the relevant step.

[0011] The second event processing step may include generating the second event information for each second event interval that is extended from the first event interval in which the first event information is collected to at least one of the spatial domain and the time domain, in accordance with a hierarchical structure formed based on the spatial domain and the time domain.

[0012] The event detection method may further include a hierarchical structure extension step that iteratively extends the hierarchical structure by using the second event information as lower-level information of the hierarchical structure to generate a third event information that corresponds to higher-level information of the second event information.

[0013] The hierarchical structure extension step may include a step of selectively generating event information for each event section from the real-time driving video via an event detection model of each event processing step formed by the iterative extension of the hierarchical structure.

[0014] In one embodiment, a memory-linkable hierarchical structure-based event detection device includes a first event processing unit that operates on a terminal and performs the steps of: collecting real-time driving video from a moving vehicle; and applying the real-time driving video to a first event detection model to generate first event information relating to the vehicle's movement; and a second processing unit that operates on the terminal and performs the steps of: collecting the first event information; and applying the first event information to a second event detection model to generate second event information relating to the vehicle's movement. [Effects of the Invention]

[0015] The disclosed technology has the following effects. However, this does not mean that any particular embodiment should include all of the following effects or only the following effects, and the scope of rights of the disclosed technology should not be understood to be limited by this.

[0016] The memory-linkable, hierarchical structure-based event detection method and apparatus according to an embodiment of the present invention is a memory-linkable, hierarchical structure-based event detection method and apparatus that can receive video footage acquired from multiple cameras inside a vehicle and determine the type of first-person event.

[0017] The memory-linked, hierarchical structure-based event detection method and apparatus according to embodiments of the present invention is a memory-linked, hierarchical structure-based event detection method and apparatus that can provide an efficient hierarchical processing structure for short-term, long-term, and ultra-long-term memory depending on the type and usability of the event.

[0018] Therefore, the present invention can distinguish between vehicle events such as speed bumps or potholes and accidents resulting from collisions with vehicles or facilities, can detect events requiring long-term memory such as aggressive driving, can solve problems requiring very long-term memory such as a driver's driving record or habits, and can provide information for efficient management of vehicle control, driver information, and event status. [Brief explanation of the drawing]

[0019] [Figure 1] This is a diagram illustrating the event detection system according to the present invention. [Figure 2] This diagram illustrates the system configuration of the terminal device shown in Figure 1. [Figure 3] This diagram illustrates the functional configuration of the event detection device according to the present invention. [Figure 4]It is a flowchart for explaining a memory-linked hierarchical structure-based event detection method according to the present invention. [Figure 5] It is a diagram for explaining a hierarchical structure for efficient processing of events according to the present invention. [Figure 6] It is a diagram for explaining an embodiment of an event detection process according to the present invention. [Figure 7] It is a diagram for explaining the operation of an event detection model according to the present invention. [Figure 8] It is a diagram for explaining an embodiment of an event hierarchy according to the present invention.

Embodiments for Carrying Out the Invention

[0020] The description regarding the present invention is merely an embodiment for structural or functional description, and the scope of rights of the present invention should not be construed as being limited by the embodiments described in the text. That is, since the embodiments can be variously changed and can have various forms, the scope of rights of the present invention should be understood to include equivalents that can realize the technical idea. Also, the objects or effects presented in the present invention do not mean that a specific embodiment should include all of them or should not include only such effects, so the scope of rights of the present invention should not be construed as being limited thereby.

[0021] On the other hand, the meanings of the terms described in the present application should be understood as follows.

[0022] Terms such as "first", "second", etc. are for distinguishing one component from other components, and the scope of rights should not be limited by these terms. For example, the first component may be named the second component, and similarly, the second component may also be named the first component.

[0023] When it is mentioned that one component is "connected" to another, it should be understood that it may be directly connected to that other component, or that there may be another component in between. On the other hand, when it is mentioned that one component is "directly connected" to another, it should be understood that there is no other component in between. Similarly, other expressions describing the relationship between components, such as "between" and "immediately between," or "adjacent to" and "directly adjacent to," should be interpreted in the same way.

[0024] Unless the context clearly indicates otherwise, singular expressions should be understood to include plural expressions, and terms such as “includes” or “possesses” should be understood to indicate the presence of the implemented features, figures, steps, actions, components, parts, or combinations thereof, without prejudice against the existence or possibility of adding one or more other features, figures, steps, actions, components, parts, or combinations thereof.

[0025] In each step, the identification codes (e.g., a, b, c, etc.) are used for explanatory purposes only and do not indicate the order of the steps. Unless the context explicitly states a specific order, the steps may occur in a different order than that specified. That is, the steps may occur in the same order as specified, substantially simultaneously, or in the reverse order.

[0026] The present invention can be implemented as computer-readable code on a computer-readable recording medium, and a computer-readable recording medium includes all types of recording devices that store data readable by a computer system. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage devices. Furthermore, computer-readable recording media can be distributed across computer systems connected via a network, allowing computer-readable code to be stored and executed in a distributed manner.

[0027] All terms used herein have the same meaning as those generally understood by a person of ordinary skill in the art to which the present invention pertains, unless otherwise specifically defined. Terms that are generally used and predefined are to be interpreted as having the same meaning as they do in the context of the relevant art, and are not to be interpreted as having an ideal or overly formal meaning unless expressly defined herein.

[0028] Figure 1 illustrates an event detection system according to the present invention.

[0029] As shown in Figure 1, the event detection system 100 is configured for performing the memory-linkable hierarchical structure-based event detection method according to the present invention, and may include a terminal 110, a cloud server 130, and a database 150. For the sake of explanation, the terminal 110, cloud server 130, and database 150 are described here as independent devices, but are not necessarily limited to this, and it is of course possible to integrate at least two different devices into one device depending on various embodiments for event detection.

[0030] The terminal device 110 may be a computing device capable of generating and storing video and transmitting the video to a cloud server 130. In one embodiment, the terminal device 110 may be implemented including a camera module capable of capturing images and videos. For example, the terminal device 110 may be implemented as a camera sensor installed in a vehicle, a black box, etc. Furthermore, the terminal device 110 may be implemented as a smartphone, tablet PC, notebook computer, etc., and is not necessarily limited to these, and can of course be implemented as a variety of devices including cameras.

[0031] Furthermore, the terminal 110 may be implemented as one of the devices constituting the event detection system 100 according to the present invention. Also, the terminal 110 can be connected to the cloud server 130 via a network, and multiple terminals 110 can be connected to a single cloud server 130 simultaneously as needed.

[0032] The cloud server 130 may be implemented as a server corresponding to a computer or program that receives and stores data transmitted from the terminal 110, and generates and stores information necessary for event detection through data analysis. For example, the cloud server 130 generates training data for an event detection model based on vehicle driving video and detected event information captured by the terminal 110. The cloud server 130 can use the training data to perform additional training on the event detection model and can also distribute the updated event detection model to each terminal 110. For this purpose, the cloud server 130 is connected to the terminal 110 by a wired network or a wireless network such as Bluetooth®, Wi-Fi®, or LTE®, and can send and receive data with the terminal 110 via the network.

[0033] Furthermore, the cloud server 130 may be implemented to operate in a cloud environment, and may be implemented to operate in conjunction with an independent external system (not shown in Figure 1) depending on various embodiments for performing the event detection method according to the present invention. For example, the cloud server 130 can operate as a server in a cloud environment, and may preferably be implemented to have performance superior to the computing performance of the terminal device 110.

[0034] On the other hand, each of the terminal device 110 and the cloud server 130 may include multiple modules implemented independently to perform related operations, and may also include a control module that manages the control and data flow to the multiple modules.

[0035] The database 150 is a storage device that stores various types of information necessary for the operation of the cloud server 130. For example, the database 150 can store short-term, long-term, and very long-term event information received by at least one terminal 110, or it can store information related to learning algorithms and training data for additional training of the event detection model. However, it is not necessarily limited to this, and it can store information collected or processed in various forms during the process in which the cloud server 130 works in conjunction with the terminal 110 to perform the memory-linkable hierarchical structure-based event detection method according to the present invention.

[0036] Figure 2 is a diagram illustrating the system configuration of the terminal shown in Figure 1.

[0037] As shown in Figure 2, the terminal 110 may include a processor 210, memory 230, user input / output unit 250, and network input / output unit 270.

[0038] The processor 210 can execute procedures for performing a memory-linkable, hierarchical structure-based event detection method according to embodiments of the present invention, manage the memory 230 that is read or created in such a process, and schedule synchronization times between volatile and non-volatile memory in the memory 230. The processor 210 can control the overall operation of the terminal 110 and is electrically connected to the memory 230, user input / output unit 250, and network input / output unit 270 to control the flow of data between them. The processor 210 may be implemented as the CPU (Central Processing Unit) or GPU (Graphics Processing Unit) of the terminal 110 or the cloud server 130.

[0039] The memory 230 may include an auxiliary storage device implemented with non-volatile memory such as an SSD (Solid State Disk) or HDD (Hard Disk Drive) and used to store all the data necessary for the terminal 110, or it may include a main memory implemented with volatile memory such as RAM (Random Access Memory). Furthermore, the memory 230 can store a set of instructions that execute the memory-linkable hierarchical structure-based event detection method according to the present invention, by being executed by an electrically connected processor 210.

[0040] The user input / output unit 250 includes an environment for receiving user input and an environment for outputting specific information to the user, and may include an input device including an adapter such as a touchpad, touchscreen, image keyboard, or pointing device, and an output device including an adapter such as a monitor or touchscreen. In one embodiment, the user input / output unit 250 corresponds to a computing device connected via a remote connection, in which case the terminal 110 may correspond to an independent node in the network to which the computing device is connected.

[0041] The network input / output unit 270 provides a communication environment for connecting with other devices via a network, and may include adapters for communication such as LAN (Local Area Network), MAN (Metropolitan Area Network), WAN (Wide Area Network), and VAN (Value Added Network). Furthermore, the network input / output unit 270 can be configured to provide short-range communication functions such as Wi-Fi and Bluetooth®, or wireless communication functions of 4G or higher, for wireless data transmission.

[0042] Figure 3 illustrates the functional configuration of the event detection device according to the present invention.

[0043] As shown in Figure 3, the event detection device 300 may include a first event processing unit 310, a second event processing unit 330, a hierarchical structure extension unit 350, and a control unit 370.

[0044] Herein, embodiments of the present invention do not necessarily have to include all of the above functional configurations simultaneously. Depending on the embodiment, some of the configurations may be omitted, or some or all of the configurations may be selectively included. Furthermore, one embodiment of the present invention may be implemented as an independent module that selectively includes some of the above configurations, and the memory-linkable hierarchical structure-based event detection method according to the present invention can be performed by linking each module. The operation of each configuration will be described in detail below.

[0045] The first event processing unit 310 operates on the terminal device 110 and can perform the steps of collecting real-time driving video from a moving vehicle and applying the real-time driving video to a first event detection model to generate first event information related to the vehicle's movement. The first event processing unit 310 may consist of multiple modules that perform each step independently.

[0046] More specifically, the first event processing unit 310 may be implemented inside a terminal unit 110 that is installed and operates in the vehicle, and can collect vehicle surrounding video footage collected via the terminal unit 110 during the vehicle's driving process as real-time driving video. For example, the vehicle surrounding video footage may include forward video footage in the direction of the vehicle's driving, rear video footage in the opposite direction of the driving, and side video footage in the direction of the vehicle's sides. The first event processing unit 310 can also directly collect video footage captured by a camera module inside the terminal unit 110 in direct conjunction with the camera module.

[0047] Furthermore, the first event processing unit 310 can store the collected real-time driving video in memory and input the real-time driving video into a pre-built first event detection model to acquire first event information. Here, the first event detection model may correspond to a deep learning model that receives real-time driving video as input and generates first event information detected within the real-time driving video as output. Here, depending on the definition of the first event detection model, each frame image of the driving video can be used as input data, but it is not necessarily limited to this.

[0048] In one embodiment, the first event information is an abnormal event that occurs during the vehicle's driving process, and may include accident events, bump events, anomaly motion events, and dangerous events. The first event information corresponds to event information derived as a result of analyzing real-time driving video, and can be detected based on the smallest unit interval of time and space in event detection.

[0049] For example, the first event information may include events with a low probability of occurring during normal vehicle operation as short-term events. In the case of accident events, it may include events related to collisions between vehicles or collisions with external obstacles, and in the case of bump events, it may include events caused by road surface irregularities, speed bumps, and small obstacles. In the case of abnormal motion events, it may include events caused by abnormal driving operations such as sudden acceleration, sudden braking, and sudden turns. Dangerous events may include events corresponding to dangerous driving conditions such as lane departure, driving on rainy or snowy roads, or driving in fog.

[0050] In other words, the first event may correspond to the short-term event with the smallest time unit and the localized event with the narrowest spatial unit. Therefore, the first event detection model may correspond to a short-term event detection model that detects events based on the smallest units in the time and spatial domains.

[0051] In one embodiment, the first event processing unit 310 can perform the following steps: transmit real-time driving video or first event information on the terminal device 110 to a cloud server 130 for storage; generate training data for additional training of the event detection model for the relevant step from the real-time driving video or first event information on the cloud server 130; train the training data on the cloud server 130 to further train the event detection model for the relevant step; and receive and update the event detection model for the relevant step that has been further trained from the cloud server 130 on the terminal device 110.

[0052] More specifically, the first event processing unit 310 can transmit real-time driving video collected from the terminal device 110 to a cloud server 130 connected to the terminal device 110. The cloud server 130 can receive and store the data and perform additional training on the first event detection model. In other words, the cloud server 130 can work in conjunction with the first event processing unit 310 to perform labeling operations for generating training driving data based on the real-time driving video. The labeling operation corresponds to the operation of assigning labels to the training data extracted from the real-time driving video. Furthermore, the cloud server 130 can generate training driving data from the label information of the first event information detected by the first event detection model.

[0053] Subsequently, the cloud server 130 can further train the training driving data via the first event detection model, and the first event detection model updated by the additional training can be transmitted from the cloud server 130 to the terminal device 110. The terminal device 110 can receive the additionally trained first event detection model and update the existing stored first event detection model with the additionally trained model.

[0054] In one embodiment, the first event processing unit 310 can perform the steps of generating training data, including receiving label information related to real-time driving video from a user terminal and generating training driving data, and generating training driving data by auto-labeling using the first event detection model. On the other hand, the second event processing unit 330 can generate short-term training data in a similar manner. That is, the second event processing unit 330 can generate short-term training data by receiving label information related to the first event from a user terminal or by auto-labeling using the second event detection model.

[0055] The second event processing unit 330 operates on the terminal device 110 and can perform the steps of collecting first event information and applying the first event information to the second event detection model to generate second event information related to vehicle movement. Here, the second event information may correspond to higher-level information derived by using the first event information as lower-level information in the hierarchical structure, while forming a hierarchical structure with the first event information. In other words, the second event processing unit 330 can work in conjunction with the first event processing unit 310 to form an efficient hierarchical processing structure for event detection.

[0056] For example, if the first event processing unit 310 detects collision event information, which is first event information, from the vehicle driving video in units of 1-second video segments, the second event processing unit 330 can generate second event information, such as road rage events or chain-rear collision events, based on the collision event information detected in units of video segments extended to 10 seconds.

[0057] In one embodiment, the second event processing unit 330 can perform update operations on the event detection model performed by the first event processing unit 310, and the specific operations related thereto may be the same as those performed on the first event processing unit 310.

[0058] In one embodiment, the second event processing unit 330 can generate second event information for each second event interval that is extended from the first event interval in which the first event information is collected to at least one of the spatial domain and the time domain, according to a hierarchical structure formed based on the spatial domain and the time domain. For example, the first event processing unit 310 can detect first events from driving video in 10-second increments in the time domain, and the second event processing unit 330 can detect second events based on the first events detected in 5-minute increments. That is, the second event information may correspond to event information detected as a result of integrating the first event information. Event detection in the spatial domain can be performed in a similar manner to event detection in the time domain, and a detailed explanation of this will be omitted.

[0059] The hierarchical structure extension unit 350 can iteratively extend the hierarchical structure by using the second event information as lower-level information in the hierarchical structure to generate a third event information that corresponds to higher-level information than the second event information. The hierarchical structure extension unit 350 can operate on the terminal device 110, but is not necessarily limited to this, and can also operate on the cloud server 130 if necessary. In other words, the hierarchical structure of event detection can be selectively extended through iteration.

[0060] Furthermore, the hierarchical structure of event detection can be implemented in conjunction with the memory structure. This allows event information collected at each event detection step to be independently stored and managed in the corresponding hierarchical memory area.

[0061] For example, a third event processing unit can be added based on the event detection structure between the first event processing unit 310 and the second event processing unit 330. The third event processing unit can take the second event information detected by the second event processing unit 330 and convert it into lower-level information to generate third event information corresponding to the higher-level information.

[0062] In other words, if the first event information is a short-term event and the second event information is a long-term event, the third event information may be a very long-term event. Furthermore, the first event information can be stored in the short-term memory area, the second event information can be stored in the long-term memory area, and the third event information can be stored in the very long-term memory area. In this way, the hierarchical structure extension unit 350 can extend the hierarchical structure by iteratively adding higher-level event processing steps to the existing hierarchical structure.

[0063] In one embodiment, the hierarchical structure extension unit 350 can selectively generate event information for each event segment from real-time driving video via an event detection model of each event processing step formed by the iterative extension of the hierarchical structure. For example, by sequentially linking the first to third event processing units, a hierarchical structure can be formed to detect short-term events, long-term events, and very long-term events related to vehicle driving video, respectively. The hierarchical structure extension unit 350 can provide selective event detection results by operating a specific event processing step according to the hierarchical structure.

[0064] The control unit 370 controls the overall operation of the event detection device 300 and can manage the flow of control or data between the first event processing unit 310, the second event processing unit 330, and the hierarchical structure extension unit 350.

[0065] Figure 4 is a flowchart illustrating the memory-linkable, hierarchical structure-based event detection method according to the present invention.

[0066] As shown in Figure 4, the event detection device 300 collects real-time driving video from a moving vehicle on the terminal device 110 via the first event processing unit 310 (step S410). The event detection device 300 applies the real-time driving video to a first event detection model on the terminal device 110 via the first event processing unit 310 to generate first event information related to the vehicle's movement (step S420). The event detection device 300 transmits the real-time driving video and the first event information to the cloud server 130 on the terminal device 110 via the first event processing unit 310 (step S430).

[0067] Furthermore, the event detection device 300 generates training driving data from real-time driving video or first event information on the cloud server 130 using the first event processing unit 310 (step S440). The event detection device 300 updates the first event detection model by additionally learning the training driving data on the cloud server 130 using the first event processing unit 310 (step S450). The event detection device 300 distributes the updated first event detection model to the terminals 110 on the cloud server 130 using the first event processing unit 310 so that the event detection model is updated on each terminal 110 (step S460).

[0068] Figure 5 illustrates a hierarchical structure for efficient event processing according to the present invention.

[0069] As shown in Figure 5, the event detection device 300 can construct an efficient hierarchical processing structure that supports short-term, long-term, and ultra-long-term memory depending on the type and usability of the event. In other words, the event detection device 300 can perform hierarchical event detection from real-time driving video collected during the vehicle's journey.

[0070] In Figure 5, during the short-term event processing step (S510), based on the coordination between the terminal 110 and the cloud server 130, operations related to real-time driving video collection and short-term event detection can be performed via the terminal 110, and operations related to the collection of training driving data and additional training of the short-term event detection model can be performed via the cloud server 130.

[0071] Furthermore, in the long-term event processing step (S530), based on the short-term event information generated in the short-term event processing step (S510), operations related to the collection of compressed short-term event information and long-term event detection can be performed via the terminal 110, and operations related to the collection of short-term training data and additional training of the long-term event detection model can be performed via the cloud server 130.

[0072] Furthermore, in the ultra-long-term event processing step (S550), based on the long-term event information generated in the long-term event processing step (S530), operations related to the collection of compressed long-term event information and the detection of ultra-long-term events can be performed via the terminal device 110, and operations related to the collection of long-term data for training and the additional training of the ultra-long-term event detection model can be performed via the cloud server 130.

[0073] In one embodiment, an event detection device 300 The memory 230 on the terminal 110 can be logically divided into short-term memory area, long-term memory area, and ultra-long-term memory area according to its hierarchical structure. In this case, the first event processing unit 310 can operate in conjunction with the short-term memory area, the second event processing unit 330 can operate in conjunction with the long-term memory area, and the third event processing unit can operate in conjunction with the ultra-long-term memory area.

[0074] Also, event detection device 300This can be implemented to operate in conjunction with independent instances generated by the cloud server 130. In this case, the first event processing unit 310 can operate in conjunction with a short-term event detection instance, the second event processing unit 330 can operate in conjunction with a long-term event detection instance, and the third event processing unit can operate in conjunction with an ultra-long-term event detection instance.

[0075] Figure 6 illustrates one embodiment of the event detection process according to the present invention.

[0076] As shown in Figure 6, the event detection device 300 can effectively detect events occurring during the vehicle's driving process through a hierarchical event detection model.

[0077] For example, in Figure 6, the event detection device 300 can divide real-time vehicle driving video into pre-set time intervals. The divided video 610 is input to a pre-built event detection model, which can generate an output indicating whether or not an event has occurred. In other words, the event detection model can provide an output indicating whether or not an event has occurred for each divided video received as input.

[0078] The event video 630, which has been determined by the event detection model to have occurred, is subsequently transmitted to a higher-level event detection model and can be used for event detection in longer time intervals than currently possible. In Figure 6, based on the distance to the car in front among the segmented video 610, there is a possibility that a dangerous event may occur, and the event detection device 300 7 Information regarding the four event videos 630 in which dangerous events were detected from the two segmented video 610 can be transmitted to the next step, enabling higher-level event detection.

[0079] Figure 7 illustrates the operation of the event detection model according to the present invention.

[0080] As shown in Figure 7, the event detection device 300 can perform end-to-end learning and event detection using video recorded for a certain period of time via a moving vehicle as input. The event detection device 300 can learn spatial and temporal domain features that appear in the event video.

[0081] For example, the event detection device 300 can progressively learn hierarchically extended features in the spatial domain based on forward, rear, and side images acquired through multiple cameras inside the vehicle. This allows the event detection device 300 to effectively distinguish between events similar to accidents, such as speed bumps and potholes, and events related to actual accidents in the video.

[0082] Furthermore, the event detection device 300 can acquire compressed event information as higher-level information via an event detection model based on lower-level information extracted from the video. In this case, the compressed event information may include information such as the type of event, whether it occurred or not, and the status.

[0083] Figure 8 illustrates one embodiment of the event hierarchy according to the present invention.

[0084] As shown in Figure 8, the event detection device 300 can effectively detect major events in real-time driving footage by analyzing and learning features appearing in event video using a deep learning model. In particular, the event detection device 300 can construct an efficient processing structure for long-term events by using a hierarchical structure.

[0085] In Figure 8, the event detection device 300 can detect events that are not typical driving conditions as short-term event information from real-time driving video via a short-term detection model. For example, short-term event information may include Accident (presence or absence of collision, direction of collision, etc.), Bump (road surface unevenness, speed bump, small obstacle), Anomaly motion (sudden acceleration, sudden stop, sharp turn), Dangerous (dangerous situation, lane departure), etc.

[0086] Furthermore, the event detection device 300 can detect events requiring information over a certain period from short-term event information via a long-term detection model. For example, long-term event information may include road rage, chain-reaction collisions, secondary accidents, short-term driving characteristics, and collected short-term events.

[0087] Furthermore, the event detection device 300 can detect events requiring information over a long period as ultra-long-term event information from long-term event information via an ultra-long-term detection model. For example, ultra-long-term event information may include the driver's driving score, driving characteristics (such as reckless driving), major driving roads, and the environment.

[0088] While preferred embodiments of the present invention have been described above with reference to those skilled in the art, a person skilled in the art will understand that the present invention can be modified and altered in various ways, without departing from the spirit and scope of the invention as described in the claims. [Explanation of Symbols]

[0089] 100 Event Detection Systems 110 terminals 130 Cloud Servers 150 databases 210 processors 230 memory 250 User Input / Output Section 270 Network Input / Output Section 300 Event Detection Devices 310 First Event Processing Unit 330 Second Event Processing Unit 350 Hierarchical structural extension section 370 Control Unit

Claims

1. A first event processing step includes the steps of: collecting real-time driving video from a moving vehicle on a terminal device; applying the real-time driving video to a first event detection model to generate first event information related to the vehicle's movement; and The terminal device includes a second event processing step which includes the steps of: collecting the first event information; and applying the first event information to a second event detection model to generate second event information relating to the vehicle's movement. The second event information forms a hierarchical structure with the first event information, and corresponds to higher-level information derived by using the first event information as lower-level information in the hierarchical structure. The aforementioned first event information is, A memory-linkable, hierarchical structure-based event detection method characterized by including accident events, bump events, abnormal motion events, and dangerous events as abnormal events occurring during the driving process of the vehicle.

2. Each of the first and second event processing steps is: The terminal device transmits the real-time driving video or the first event information to a cloud server for storage. On the cloud server, a step of generating training data for additional training of the event detection model for the relevant step from the real-time driving video or the first event information, On the aforementioned cloud server, the steps include: training the training data to further train the event detection model for the relevant step, and The memory-linkable hierarchical structure-based event detection method according to claim 1, characterized in that it includes the step of receiving and updating the event detection model for the relevant step, which has been further learned from the cloud server, on the terminal device.

3. The step of generating the aforementioned training data is: A step of receiving label information relating to the real-time driving video or the first event information from the user terminal and generating the learning data, and The memory-linkable hierarchical structure-based event detection method according to claim 2, characterized by including a step of generating the training data by auto-labeling using the event detection model of the aforementioned step.

4. The second event processing step is: The memory-linkable hierarchical structure-based event detection method according to claim 1, characterized in that it includes the step of generating second event information for each second event interval which is extended from the first event interval in which the first event information is collected to at least one of the spatial domain and the time domain, according to a hierarchical structure formed based on a spatial domain and a time domain.

5. A first event processing unit that operates on a terminal device and performs the steps of collecting real-time driving video from a moving vehicle and applying the real-time driving video to a first event detection model to generate first event information related to the vehicle's movement, and The terminal includes a second event processing unit that operates on the terminal and performs the steps of collecting the first event information and applying the first event information to a second event detection model to generate second event information relating to the vehicle's movement, The second event information forms a hierarchical structure with the first event information, and corresponds to higher-level information derived by using the first event information as lower-level information in the hierarchical structure. The aforementioned first event information is, A memory-linkable, hierarchical structure-based event detection device characterized by including accident events, bump events, abnormal motion events, and dangerous events as abnormal events occurring during the driving process of the vehicle.

Images