Electronic device and method for detecting driving events of a vehicle

Abstract

A method and an electronic device are provided for determining whether a driving event of a vehicle occurs based on positions of objects in a plurality of frames using a plurality of training models.

Description

Technical Field

[0001] The present disclosure relates to an artificial intelligence (AI) system configured to use a machine learning algorithm such as deep learning to simulate functions of human intelligence such as recognition and determination, and applications of the AI ​​system, and the present disclosure relates, for example, to a method and apparatus for using the AI ​​system to determine an event that occurs based on the position of an object while driving a vehicle. Background Art

[0002] With the development of multimedia technology and network technology, users have been able to use electronic devices to receive various services. With the development of technology applied to vehicles, various methods of recognizing whether a driving event of a vehicle occurs have been developed.

[0003] Meanwhile, a large amount of driving data is required in order to recognize whether a driving event of a vehicle occurs, and thus, demands for technology configured to efficiently obtain driving data at little cost have increased.

[0004] An artificial intelligence (AI) system is a computer system that can simulate human intelligence. Unlike previous rule-based intelligent systems, AI systems perform self-learning and determination, thereby becoming more intelligent. The more an AI system is used, the higher the recognition rate of the AI ​​system becomes, and the more accurately the user's taste becomes understood. Therefore, previous rule-based intelligent systems have gradually been replaced by AI systems based on deep learning.

[0005] AI technologies include machine learning (deep learning) and elemental technologies that leverage machine learning.

[0006] Machine learning can be an algorithmic technology that self-classifies and learns the features of input data. Element technology can be a technology configured to simulate the functions of the human brain (e.g., recognition, determination, etc.) using machine learning algorithms such as deep learning, and includes language understanding, visual understanding, inference / prediction, knowledge expression, operational control, etc.

[0007] The various fields to which AI technology is applied may include, for example, the following. Language understanding may refer to the technology of identifying and applying / processing human language / characters, and includes natural language processing, machine translation, dialogue systems, questions and responses, speech recognition / synthesis, etc. Visual understanding may refer to the technology of identifying and processing objects as if they were human visually, and includes object recognition, object tracking, image search, character recognition, scene understanding, spatial understanding, image improvement, etc. Inference / prediction may refer to the technology of determining information and performing logical inference and prediction on it, and includes knowledge / probability-based inference, optimized prediction, preference-based planning, recommendation, etc. Knowledge expression may refer to the technology of performing automation to automatically convert human experience information into knowledge data, and includes knowledge construction (data generation / classification), knowledge management (data utilization), etc. Operational control may refer to the technology of controlling the autonomous driving and robot motion of a vehicle, and includes motion control (navigation methods, collision, driving), manipulation control (behavior control), etc. Summary of the invention

[0008] Solution to the problem

[0009] A method and an electronic device are provided for determining whether a driving event of a vehicle occurs based on positions of objects in a plurality of frames using a plurality of training models. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings, in which:

[0011] Figure 1 is a diagram illustrating an example in which an electronic device according to an embodiment detects an object in front of a vehicle and determines whether an event occurs;

[0012] Figure 2 is a flow chart illustrating a method for determining whether an event occurs via an electronic device according to an embodiment;

[0013] Figure 3 is a diagram illustrating an operation of generating a training model configured to determine whether a driving event of a vehicle occurs according to an embodiment;

[0014] Figure 4 is a diagram illustrating an example of detecting an object using a first training model according to an embodiment;

[0015] Figure 5 is a diagram illustrating an operation of determining whether a detected object is noise or an object based on a distance between the detected object and a vehicle according to an embodiment;

[0016] Figure 6is a diagram illustrating an example of using a second training model to determine whether an event occurs based on sequential movement of an object according to an embodiment;

[0017] Figure 7 is a diagram illustrating an example of obtaining a frame of an object detected in front of a vehicle from among a plurality of data frames in order to detect an occurrence of an event according to time according to an embodiment;

[0018] Figure 8 is a diagram illustrating an example of using a training model to detect whether an event occurs through sequence frame analysis according to an embodiment;

[0019] Fig. 9 is a diagram illustrating an example of obtaining driving data of a vehicle using virtual reality according to an embodiment;

[0020] Fig.10 is a diagram illustrating an example of determining a notification message and a command for controlling a vehicle based on a type of an event according to an embodiment;

[0021] Fig.11 is a diagram illustrating an example of outputting a notification message on a head-up display (HUD) of a vehicle according to an embodiment;

[0022] Fig.12 is a diagram illustrating an example of outputting a notification message in a vehicle according to an embodiment;

[0023] Fig.13 is a block diagram illustrating components of an electronic device according to an embodiment;

[0024] Fig.14 is a block diagram illustrating components of an electronic device according to an embodiment;

[0025] Fig.15 is a block diagram illustrating a processor according to an embodiment;

[0026] Fig.16 is a block diagram illustrating a data learner according to an embodiment;

[0027] Fig.17 is a block diagram illustrating a data recognizer according to an embodiment; and

[0028] Fig.18 is a diagram illustrating an example in which an electronic device and a server synchronize with each other to learn and recognize data according to an embodiment. DETAILED DESCRIPTION

[0029] A method and an electronic device are provided for determining whether a driving event of a vehicle occurs based on positions of objects in a plurality of frames using a plurality of training models.

[0030] Additional aspects will be set forth in part in the description which follows and, in part, will be apparent from the description.

[0031] According to one aspect of the present disclosure, an electronic device includes: a processor; and a memory for storing instructions that can be executed by the processor, wherein the processor is configured to execute the instructions to enable the electronic device to: obtain a video sequence from a vehicle, the video sequence including multiple frames captured while driving the vehicle, identify positions of objects included in the multiple frames, analyze sequential changes in the positions of the objects in the multiple frames, and determine whether a driving event of the vehicle occurs.

[0032] According to another aspect of the present disclosure, a method includes: obtaining a video sequence from a vehicle, the video sequence including a plurality of frames captured while driving the vehicle, identifying positions of objects included in the plurality of frames, analyzing sequential changes in the positions of the objects in the plurality of frames, and determining whether a driving event of the vehicle occurs.

[0033] According to another aspect of the present disclosure, a computer program product includes a non-transitory computer-readable recording medium, which includes instructions that, when executed by a processor, cause an electronic device to: obtain a video sequence from a vehicle, the video sequence including multiple frames captured from the vehicle while driving the vehicle; identify a position of an object included in at least one of the multiple frames; analyze sequential changes in the position of the object in the multiple frames; and determine whether a driving event of the vehicle has occurred.

[0034] Invention Mode

[0035] Hereinafter, various example embodiments of the present disclosure will be described in more detail with reference to the accompanying drawings to convey the concepts of the present disclosure to those of ordinary skill in the art. However, the present disclosure can be implemented in many different forms and should not be construed as being limited to the embodiments set forth herein. Moreover, in order to clearly describe the present disclosure, functions or configurations that may obscure the main points of the present disclosure may be omitted in the accompanying drawings, and the same reference numerals refer to the same elements throughout the disclosure. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items. For example, expressions such as "at least one of...", when following a list of elements, modify the entire list of elements without modifying the individual elements in the list.

[0036] Throughout the present disclosure, it should be understood that when an element is referred to as being "connected" to another element, it may be "directly connected" to the other element or "electrically connected" to the other element with an intermediate element interposed therebetween. It should also be understood that when a component "includes" or "comprises" an element, unless otherwise defined, the component may also include other elements without excluding other elements.

[0037] Hereinafter, embodiments will be described in more detail with reference to the accompanying drawings.

[0038] Figure 1 is a diagram illustrating an example in which the electronic device 1000 detects an object in front of a vehicle and determines whether an event occurs according to an embodiment;

[0039] refer to Figure 1 , the electronic device 1000 may be a device installed in the vehicle 100. The electronic device 1000 may include various circuits and components configured to receive a video sequence from a camera installed in the vehicle 100 and determine the occurrence of various events. In addition, the electronic device 1000 may provide a notification message to the user based on the occurrence of the event.

[0040] According to an embodiment, the electronic device 1000 may be a device installed in a driving device using a virtual environment, such as a simulated driving device, a driving game console, etc., but is not limited thereto. The electronic device 1000 may receive virtual environment data with good graphics performance and determine the occurrence of various events of the vehicle 100. Although it is described that the electronic device 1000 receives a video sequence from a camera installed in the vehicle 100, it is not limited thereto. The electronic device 1000 may receive a video sequence from a camera configured to capture the surroundings of the vehicle 100. The surroundings of the vehicle 100 may include, for example, the front side, the side, and the rear side.

[0041] According to an embodiment, the electronic device 1000 may detect an event using coordinate information, and thus may differently recognize the importance of the type of the object based on the location of the object, thereby determining whether an event occurs based on the location of the object.

[0042] According to an embodiment, the electronic device 1000 may detect at least one video sequence 103 including an object from a plurality of video sequences. The electronic device 1000 may detect, for example and without limitation, a road 102 including a fixed pattern and another vehicle 101 moving according to time by analyzing at least one video sequence 103. According to an embodiment, the electronic device 1000 may analyze the position of another vehicle 101 by analyzing the coordinates of the other vehicle 101 in the at least one video sequence 103 to determine the occurrence of an event based on the other vehicle 101.

[0043] The electronic device 1000 may include various circuits, including but not limited to a head unit or embedded board in a vehicle, a smart phone, a tablet personal computer (PC), a PC, a smart television (TV), a cellular phone, a personal digital assistant (PDA), a laptop, a vehicle, a media player, a micro server, a global positioning system (GPS) device, an e-book terminal, a digital broadcast terminal, a navigation device, an information kiosk, an MP3 player, a digital camera, a home appliance, and other mobile or non-mobile computing devices, etc., but not limited thereto. Moreover, the electronic device 1000 may include a wearable device having a communication function and a data processing function, such as a watch, glasses, a hairband, a ring, etc. However, the electronic device 1000 is not limited thereto, and may include all types of devices configured to obtain images (e.g., videos and still images) from a camera and provide a notification message to a user based on the obtained images.

[0044] According to an embodiment, the electronic device 1000 may be a module installed in a vehicle including various circuits and components. The electronic device 1000 may be configured to control the operation of the vehicle and communicate with other modules installed in the vehicle via a certain network.

[0045] According to an embodiment, the electronic device 1000 may be a device different from the vehicle, such as a smart phone, etc., but is not limited thereto. In this case, the electronic device 1000 may obtain a video sequence using a camera of the electronic device 1000, and may receive the video sequence from a camera configured to capture the surroundings of the vehicle via a certain network. The electronic device 1000 may communicate with a module included in the vehicle. For example, the electronic device 1000 may obtain a video sequence using a camera of the electronic device 1000, and may receive the video sequence from a camera configured to capture the surroundings of the vehicle. Moreover, when the electronic device 1000 is a device different from the vehicle, the electronic device 1000 may communicate with a module installed in the vehicle to control the operation of the vehicle.

[0046] According to an embodiment, a vehicle may include any means of transportation that provides communication functions, data processing functions and / or transportation functions, such as but not limited to a car, bus, truck, train, bicycle, motorcycle, etc.

[0047] According to an embodiment, the vehicle may include, for example but not limited to, a driving simulator system configured to detect an event of the vehicle using virtual reality and a real-time vehicle analysis program. According to an embodiment, the simulator may be configured to test and evaluate a real or virtual mobile system model via a computer program after making the real or virtual mobile system model, and thus may be used to effectively obtain driving data based on the movement of the vehicle in virtual reality.

[0048] Furthermore, the electronic device 1000 can communicate with the server 2000 (see, for example, Fig.18 ) and another electronic device (not shown) to receive a video sequence, send a notification message, and send a command for controlling the operation of another electronic device. In this case, the network may include, but is not limited to, for example, a local area network (LAN), a wide area network (WAN), a value-added network (VAN), a mobile radio communication network, a satellite communication network, etc. and any combination thereof. The network may be a comprehensive data communication network configured to enable components included in the network to communicate smoothly with each other, and may include wired Internet, wireless Internet, and mobile wireless communication networks. Wireless communication may include, but is not limited to, for example, wireless LAN (Wi-fi), Bluetooth, Bluetooth low energy, Zigbee, Wi-fi direct (WFD), ultra-wideband (UWB), infrared data association (IrDA), near field communication (NFC), etc., but is not limited thereto.

[0049] Figure 2 is a flowchart illustrating a method of determining whether an event occurs via the electronic device 1000 according to an embodiment.

[0050] In operation 210, the electronic device 1000 may obtain a video sequence including a plurality of frames from, for example, a camera installed in a vehicle. According to an embodiment, the electronic device 1000 may receive the video sequence by communicating with the camera installed in the vehicle via a certain network. For example, the video sequence may be a black box image of the vehicle or an image received from a stereo camera of the vehicle. According to an embodiment, the electronic device 1000 may include a camera, and the video sequence may be obtained from the camera included in the electronic device 1000.

[0051] A video sequence may include a series of still images. Each still image may be called a picture or a frame.

[0052] In operation 220, the electronic device 1000 may recognize the position of an object included in a plurality of frames. According to an embodiment, the electronic device 1000 may detect one or more objects from a frame included in a video sequence. One or more objects detected from a frame may be detected from another frame included in the same video sequence. One or more objects detected from a frame may not be detected from another frame included in the same video sequence. For example, a road, a sidewalk, a first vehicle, a second vehicle, a third vehicle, and a traffic sign may be detected from a first frame, while only a road, a sidewalk, a first vehicle, and a third vehicle may be detected from a second frame included in the same video sequence as the first frame, and the second vehicle and the traffic sign may not be detected. Moreover, a motorcycle not detected from the first frame may be detected from the second frame.

[0053] According to an embodiment, the electronic device 1000 may use, for example, an artificial intelligence (AI) learning model to obtain the position information of the object. For example, the electronic device 1000 may identify the location of the first vehicle in the first frame based on the coordinate information of the first vehicle in the first frame. Moreover, the electronic device 1000 may use the position of the first vehicle in the first frame and the position of the third vehicle to identify the distance between the first vehicle and the third vehicle. Moreover, the electronic device 1000 may use the position of the first vehicle in the third frame and the position of the third vehicle to identify the change in the distance between the first vehicle and the third vehicle in the third frame.

[0054] According to an embodiment, the electronic device 1000 may determine the type of the object. Based on the information about the available size of the type of the object in the position where the object is recognized, the electronic device 1000 may determine whether the object is noise. According to an embodiment, the electronic device 1000 may determine whether the object is noise by comparing the information about the reference size of the type of the object in the recognized position of the object with the information about the size of the object in the recognized position of the object.

[0055] According to an embodiment, the information about the available size of the type of the object may include size information set according to a predetermined shape based on the type of the object. According to an embodiment, the size information may include information predefining a range of an area of ​​the frame that the object will occupy based on a distance between the vehicle and the object.

[0056] According to an embodiment, the size information may include information about the extent of the area of ​​the frame that the object will occupy. According to an embodiment, as the distance between the vehicle and the object decreases, the area in the frame that will be occupied by the object may increase, and as the distance between the vehicle and the object increases, the area in the frame that will be occupied by the object may decrease. According to an embodiment, the size information may be predetermined based on the type of the object. According to an embodiment, the AI ​​may learn the size information based on the type of the object.

[0057] According to an embodiment, the electronic device 1000 may determine the type of the object.

[0058] Types of objects may include, for example, but are not limited to, roads, sidewalks, buildings, walls, fences, utility poles, traffic signals, traffic signs, vegetation, terrain, sky, people, riders, cars, trucks, buses, trains, motorcycles, bicycles, etc.

[0059] For example, the electronic device 1000 may detect a plurality of objects from one frame, and may determine the type of each of the plurality of objects. Moreover, even if some of the plurality of objects are of the same type, the electronic device 1000 may distinguish objects of the same type from each other. For example, when three vehicles are detected in one frame, the electronic device 1000 may distinguish the three vehicles into a first vehicle, a second vehicle, and a third vehicle.

[0060] According to an embodiment, the electronic device 1000 may use a first training model to detect an object included in a frame and the position of the object. According to an embodiment, the first training model may be obtained based on a result of learning the position of an object corresponding to the coordinates of the object in a plurality of frames by detecting the object in a video sequence including a plurality of frames captured during driving of a vehicle, and simultaneously mapping the coordinate information to the object.

[0061] Therefore, when a frame obtained from a video sequence is input in the first training model, position information of an object detected in the frame can be output from the first training model.

[0062] According to an embodiment, with respect to one frame, the electronic device 1000 may determine where an object is located in the frame. For example, the electronic device 1000 may determine where an object is located in another frame. Moreover, for example, the electronic device 1000 may determine where multiple objects are located in one frame. For example, the electronic device 1000 may determine where multiple objects are located in another frame. That is, the electronic device 1000 may determine at which coordinates in each frame multiple objects are located. According to an embodiment, the coordinates may represent XY coordinates, and the position value of the object may be output as a matrix value.

[0063] According to an embodiment, the electronic device 1000 may determine the position of the object via a pixel unit. For example, the electronic device 1000 may determine the pixel indicating the object from the pixels included in the frame. For example, when a frame includes multiple objects, the electronic device 1000 may determine the pixels indicating the multiple objects respectively. For example, the electronic device 1000 may determine which detected object a certain pixel included in the frame indicates.

[0064] The method performed by the electronic device 1000 to accurately obtain the position information of the object in the pixel unit without using the bounding box can be applied to the technical field that requires accurate recognition function. For example, since the electronic device 1000 can obtain the position information of the object via the pixel unit, the electronic device 1000 can analyze the sequential changes of the position of the object and use the analysis results for autonomous driving technology that requires fast and accurate recognition.

[0065] According to an embodiment, the electronic device 1000 may obtain information about the position of an object using a first training model. When a plurality of frames are input into the first training model, pixel information may be output. The pixel information may be information about which object each pixel included in the plurality of frames indicates.

[0066] Operations 210 and 220 are described above as separate operations. However, the present disclosure is not limited thereto. For example, when multiple frames and coordinate information corresponding to the multiple frames are input in the first training model, information about objects detected from the multiple frames and the coordinate information of the objects may be output together. As another example, only pixel information about the object may be output.

[0067] According to an embodiment, in operation 230 , the electronic device 1000 may determine whether a driving event of a vehicle occurs by analyzing sequential changes in positions of objects in a plurality of frames.

[0068] According to an embodiment, the electronic device 1000 may analyze the change of the position of the object between the previous frame and the next frame based on the display order of the video sequence. For example, by comparing the position information of the object included in the first frame (displayed first) with the position information of the same object included in the second frame (displayed next), the electronic device 1000 may analyze the change of the position of the object. For example, the electronic device 1000 may determine whether an event has occurred by analyzing the change of the position of each of the multiple objects according to time. That is, the electronic device 1000 may identify the change of the position of multiple objects in a complex manner via the change of the coordinates of the object, rather than tracking the bounding box relative to the region of interest (ROI). Therefore, the electronic device 1000 may more accurately determine whether an event has occurred. For example, when the first vehicle and the second vehicle stop in front and the third vehicle, the fourth vehicle and the fifth vehicle continuously change lanes to the right, the electronic device 1000 may determine that an event has occurred in which the vehicle in front encounters an accident. Moreover, since there is a vehicle in front that has an accident, the electronic device 1000 may determine that it is best to change the lane to the right action guide, which corresponds to the event.

[0069] According to an embodiment, the electronic device 1000 may determine the type of an event by analyzing sequential changes in the position of an object in a plurality of frames.

[0070] The type of driving event of the vehicle may include, but is not limited to, a change in traffic signal, a prediction of an accident, a change in road conditions, a change in terrain, etc. Examples of changes in traffic signals may include a change in traffic from green to red or from red to green. Examples of accident predictions may include an inability to ensure a safe distance relative to the vehicle ahead and / or the vehicle behind, the appearance of an unpredicted shape, etc. Examples of changes in road conditions may include traffic congestion due to an accident vehicle ahead. Examples of changes in terrain may include a winding road ahead, a hill ahead, etc.

[0071] According to an embodiment, the electronic device 1000 may determine the risk level of driving by analyzing sequential changes in the positions of objects in a plurality of frames.

[0072] For example, the risk level of driving may be indicated via a numerical value, wherein as the numerical value increases, the risk level may also increase. For example, the risk level may be indicated as an integer between a value equal to or greater than 1 and a value equal to or less than 100. When the risk level is equal to or greater than a critical value, it may be set in the electronic device 1000 to include the risk level of driving in the notification message. Moreover, for example, the risk level of driving may be indicated as high, medium, and low.

[0073] According to an embodiment, the electronic device 1000 may use the second training model to determine whether an event occurs. When an output value related to an object output from the first training model is input in the second training model, whether an event occurs may be output.

[0074] According to an embodiment, a plurality of training models may be used to perform an operation of detecting an object, an operation of obtaining information about a location of an object, and an operation of determining whether an event occurs.

[0075] Figure 3 is a diagram illustrating an operation of generating a training model configured to determine whether a driving event of a vehicle occurs according to an embodiment.

[0076] According to an embodiment, by training a first training model (e.g., including a processing circuit and / or a program element) 301 and a second training model (e.g., including a processing circuit and / or a program element) 302 using coordinate information 303 and a video sequence 304 including the sequence information 303, an event detection model (e.g., including a processing circuit and / or a program element) 305 configured to detect a driving event of a vehicle based on the position of an object can be generated.

[0077] According to an embodiment, the first training model 301 may include various processing circuits and / or program elements, and is generated by learning a reference for determining the type of an object and a reference for determining the position of the object in each of multiple frames using, for example but not limited to, a fully convolutional network (FCN), etc.

[0078] According to an embodiment, the electronic device 1000 may input a video sequence 304 including frames (which include RGB channels) together with the coordinate information 303 into the first training model 301. According to an embodiment, the first training model 301 may include various processing circuits and / or program elements, and is trained to recognize the position of an object in multiple frames in the video sequence 304 using the coordinate information 303 explicitly input in a previous model configured to learn using an encoder and a decoder.

[0079] According to an embodiment, the electronic device 1000 may convert a frame including RGB channels into a luminance-chrominance (Lab) channel. In addition to color information, the L value of the converted Lab channel is a luminance value of an image indicating the brightness of the image. The electronic device 1000 may perform preprocessing, whereby a median filter for flattening the L values ​​of the multiple frames is applied to the multiple frames included in the video sequence 304 before the multiple frames are input into the first training model. Via preprocessing, objects can be easily detected and multiple objects can be distinguished from each other, whether it is dark or rainy.

[0080] According to an embodiment, the electronic device 1000 may determine the types of objects included in a plurality of frames using the first training model 301. For example, the electronic device 1000 may detect a plurality of objects from a frame and determine the types of the plurality of objects using the first training model 301. For example, when a frame is input in the first training model 301, different values ​​may be output based on the types of objects included in the frame. For example, the sky may be set to 12, the plant to 10, the road to 4, the sidewalk to 3, the vehicle to 6, the person to 8, and so on in the first training model 301. For example, when plants, roads, vehicles, and people are included in the frame input in the first training model 301, the output of the first training model 301 may include 4, 6, 8, and 10. Therefore, the first training model 301 may be used to detect which objects are included in the frame. Moreover, for example, when a frame is input in the first training model 301, pixel information indicating which object is indicated for each pixel included in the frame may be output instead of a value corresponding to the type of the object. The pixel information may be a matrix in which the value corresponding to the type of the object matches the position of the object in the frame.

[0081] According to an embodiment, the electronic device 1000 may reduce the size of an output of the first training model 301 in order to input the output to the second training model 302 .

[0082] According to an embodiment, the second training model 302 may be generated by analyzing sequential changes in positions of an object in a plurality of frames using, for example but not limited to, a recurrent neural network (RNN) or the like to learn a reference for determining whether a driving event of a vehicle occurs.

[0083] According to an embodiment, the output of the first training model 301 may be used as an input of the second training model 302. According to another embodiment, the electronic device 1000 may use a matrix generated by reducing the size of the matrix output from the first training model as an input of the second training model in order to reduce the amount of operation of the second training model 302 configured to determine whether an event occurs. For example, the size of the matrix may be reduced using dilated convolution or the like.

[0084] Also, according to an embodiment, in order to reduce the amount of operation of the first training model 301 , the electronic device 1000 may perform 1×1 convolution filtering on the output of the layer included in the first training model 301 to match the size of each layer included in the first training model 301 .

[0085] According to an embodiment, when the first and second vehicles stop in front and the third to fifth vehicles continuously change lanes to the right, the electronic device 1000 may determine that an event that the front vehicle is involved in an accident occurs. Also, since there is a front vehicle involved in an accident, the electronic device 1000 may determine that it is best to change lanes to the right, and the action guideline corresponds to the event.

[0086] As described above, the electronic device 1000 can obtain information about the type of the object and information about the position of the object of the entire screen using a small amount of operations via the first training model 301 of the input coordinate information 303. Therefore, the electronic device 1000 can quickly and accurately analyze the sequential changes in the position of the object without setting ROI (as in the method of tracking the object).

[0087] Therefore, the electronic device 1000 may determine the occurrence of an event that can be detected by analyzing sequential changes in the positions of the objects, and the driving event of the vehicle using the second training model 302 input to the event detection model 305 .

[0088] Figure 4 is a diagram illustrating an example of detecting an object using the first training model 1010 according to an embodiment.

[0089] According to an embodiment, the electronic device 1000 may detect objects in frames and recognize the locations of the objects using a first training model (e.g., including a processing circuit and / or a program element) 1010 learned using a video sequence 402 including a plurality of frames obtained during driving of a vehicle as an input value.

[0090] According to an embodiment, the coordinate information may represent a two-dimensional coordinate of (x, y). For example, the position of the object may be identified using the coordinate information of the object in the frame output from the first training model 1010. For example, when plants, roads, vehicles, and people are included in the frame input into the first training model 1010, the output of the first training model 1010 may include 4 (1, 4), 6 (3, 4), 8 (10, 23), and 10 (15, 11). Therefore, the first training model 1010 may be used to detect where a certain object is located in the frame. According to an embodiment, information about a plurality of objects learned using the first training model 1010 may be output as a matrix value.

[0091] According to an embodiment, for example, the electronic device 1000 may determine the position of an object in units of pixels using the first training model 1010. For example, the electronic device 1000 may determine which objects are indicated by the pixels included in the frame, and therefore, when a frame including an object is input in the first training model 1010, a matrix in which a value corresponding to the type of the object matches the position of the object in the frame may be output. The position of the object is determined in units of pixels, and therefore, when, for example, the size of the frame is 512×256, the size of the matrix may also be 512×256. That is, as an output of the first training model regarding the input of the frame, a matrix including information about the type of the object and information about the position of the object may be obtained.

[0092] According to an embodiment, since the first training model 1010 uses FCN, when the electronic device 1000 inputs the video sequence 402 in the first training model 1010, the type of the object and the position of the object can be output. According to an embodiment, based on the coordinate information explicitly input in the first training model 1010, the first training model 1010 can detect the position of the object with a small number of operations by learning to detect the object and recognize the position of the object at the same time.

[0093] According to an embodiment, when a series of matrices output from the first training model 1010 are generated as an image, a video sequence 405 may be obtained in which objects included in the video sequence 405 are indicated in different colors based on the object type. For example, a road forming a constant pattern and a vehicle as a moving object may be indicated in different colors. According to an embodiment, the electronic device 1000 may detect the type of the object and the accuracy of the object recognition. For example, the electronic device 1000 may determine the type and position of the first object 403 and the second object 404 in the video sequence 405 output from the first training model 1010.

[0094] According to an embodiment, the electronic device 1000 may use information about the shape and position of the first object 403 to recognize that the first object 403 is a bus with 75% accuracy, and use information about the shape and position of the second object 404 to recognize that the second object 404 is a car with 97% accuracy.

[0095] Figure 5 is a diagram illustrating an operation of determining whether a detected object is an object or noise based on a distance between the detected object and a vehicle according to an embodiment.

[0096] According to an embodiment, the electronic device 1000 may detect the first object 501, the second object 502, and the third object 503, and recognize their positions in the video sequence 505 output using, for example, the first training model 1010. The electronic device 1000 may determine that the distance between the driving vehicle and the third object 503 is greater than the distance between the driving vehicle and the first object 501 and the distance between the driving vehicle and the second object 502. The electronic device 1000 may determine a low level of importance of the occurrence of an event with respect to the third object 503 located relatively far from the driving vehicle.

[0097] According to an embodiment, the electronic device 1000 may determine the type of the first object 501 and the second object 502. According to an embodiment, the electronic device 1000 may determine the type of the first object 501 and the second object 502 as a human being, and thus the first object 501 and the second object 502 may be displayed in the same color in the video sequence 505. According to an embodiment, the electronic device 1000 may detect noise based on the position coordinates of the first object 501 and the second object 502. For example, the position of the second object 502 is similar to the position of the first object 501, and therefore, the electronic device 1000 may determine that the distance between the first object 501 and the driving vehicle and the distance between the second object 502 and the driving vehicle are similar to each other. However, the electronic device 1000 may determine that the size of the second object 502 is significantly smaller than the size of the first object 501. According to an embodiment, the electronic device 1000 may determine that the second object 502 is noise information rather than a human being based on information about the size of a person recognized in the position of the second object 502. Therefore, the electronic device 1000 may display the second object 502 in a different color from the first object 501.

[0098] Figure 6 is a diagram illustrating an example of determining whether an event occurs based on sequential movement of an object using the second training model 1020 according to an embodiment.

[0099] According to an embodiment, when a frame including an object including position information output from the first training model 1010 is input in the second training model 1020, it may be determined whether an event related to the object occurs.

[0100] According to an embodiment, the second training model 1020 may use, for example but not limited to, an RNN, which may, for example, refer to a neural network in which nodes are cyclically connected to each other in different time partitions. The RNN according to an embodiment may recognize sequential data. Sequential data is data with a time characteristic or order, such as voice data, image data, biological data, handwriting data, etc. For example, the recognition model of the RNN may recognize a pattern based on which image data of the input changes.

[0101] The RNN can be trained via supervised learning, wherein learning data and output data corresponding thereto are input into the neural network, and the connection weights of the connection lines are modified and improved so that the output corresponds to the output data of the learning data. For example, the RNN can modify and improve the connection weights between neurons based on incremental rules and back-propagation learning.

[0102] According to an embodiment, the second training model 1020 may include various processing circuits and / or program elements, and use an RNN including a long short-term memory (LSTM) network. Therefore, the second training model 1020 can use a structure related to all of the previous step, the current step, and the next step of learning, and can send information of the current step to the next step to affect the output value. Therefore, the electronic device 1000 can determine an event that may occur while driving a vehicle based on a learning result for understanding the context of a driving image of a vehicle.

[0103] For example, the second training model 1020 may recognize that the object 601 is located closer to the driving vehicle in the next frame than in the previous frame, and may determine that a collision has occurred between the object 601 and the driving vehicle.

[0104] According to an embodiment, the second training model 1020 may predict the probability of an event occurring based on an object by analyzing the sequential changes in the position of the object. For example, the second training model 1020 may determine the probability of an accident occurring based on the distance between the object 601 and the vehicle, the distance being determined based on the position of the object 601. According to an embodiment, when the second training model 1020 determines that the distance between the object 601 and the vehicle is large, the second training model 1020 may determine that the probability of an accident occurring is 10%, as described in operation 602. When the second training model 1020 determines that the distance between the vehicle and the object 601 decreases as the vehicle and the object 601 move according to time, the second training model 1020 may determine that the probability of an accident occurring is 64%, as described in operation 603. According to an embodiment, the second training model 1020 may learn the probability of an accident occurring based on the movement of the vehicle and the object 601 according to time.

[0105] For convenience of explanation, a description is given by separating the first training model 1010 and the second training model 1020. However, the first training model 1010 and the second training model 1020 may include a plurality of training models based on their functions, or one integrated training model.

[0106] Figure 7 is a diagram illustrating an example of obtaining a frame of an object detected in front of a vehicle from among a plurality of frames in order to detect whether an event occurs according to time according to an embodiment.

[0107] According to an embodiment, the electronic device 1000 may segment a frame 702 including an object using a video sequence including a plurality of frames 701. According to an embodiment, the electronic device 1000 may input the plurality of frames 701 in a first training model including a combination of an FCN and a convolutional neural network (CNN), and may output a segmented frame 702 including the object while recognizing the position of the object. Next, the electronic device 1000 may input the segmented frame 702 including the object in a second training model including an LSTM network to determine whether an event occurs in the frame 702 including the object.

[0108] According to an embodiment, the electronic device 1000 may recognize sequential motions of an object using the first training model and the second training model, and may determine occurrences of various events based on the object.

[0109] Figure 8 is a diagram illustrating an example of using a training model to detect whether an event occurs through sequence frame analysis according to an embodiment.

[0110] CNN includes a fully connected layer in the latter layer to classify the category of the image data. Here, when the position information of the object in the input image passes through the fully connected layer, the position information is lost. In order to improve on this problem, FCN can regard the fully connected layer of CNN as a 1×1 convolution so that the position information can be maintained.

[0111] According to an embodiment, the first training model uses FCN, and therefore, when the video sequence 810 is input in the first training model, the electronic device 1000 may output a series of matrices including information about the type and position of the object. According to an embodiment, the video sequence 810 preprocessed to be input in the first training model may be input in the first training model based on the display order. The output order of the matrices output from the first training model may be the same as the input order of the video sequence 810 input in the first training model.

[0112] When a series of matrices output from the first training model are made into images, a video sequence 820 can be obtained in which objects included in the video sequence 810 are displayed in different colors based on their types. Here, the position of the object can be recognized via the coordinate information of the object included in the video sequence 810 output from the first training model. According to an embodiment, when a dilated convolution is performed on a video sequence 830 generated by dividing the video sequence 820 in units of pixels, a matrix 840 can be obtained in which the size of the matrix output from the first training model is reduced. Dilated convolution is a method of performing convolution using only some pixels included in the video sequence 830. For example, convolution is performed by skipping one or more pixel units, thereby dilating the size of the receptive field (RF) and reducing the size of the matrix and the amount of operation.

[0113] According to an embodiment, when the matrix 840 is input in the second training model, an object-based event 860 can be detected. The second training model uses an RNN, which is a neural network in which nodes are cyclically connected to each other in different time partitions. The RNN according to an embodiment can recognize sequential data. Sequential data is data with a time characteristic or order, such as voice data, image data, biological data, handwriting data, etc. For example, the recognition model of the RNN can recognize a pattern based on which image data of the input changes.

[0114] The RNN can be trained via supervised learning, wherein learning data and output data corresponding thereto are input into the neural network, and the connection weights of the connection lines are modified and improved so that the output corresponds to the output data of the learning data. For example, the RNN can modify and improve the connection weights between neurons based on incremental rules and back-propagation learning.

[0115] The RNN may be a structure including an LSTM network 850. The LSTM network 850 may be a type of RNN capable of performing long-term dependency learning. An RNN that does not include the LSTM network 850 may connect previous information to a current operation, but may have difficulty connecting information from past operations to a current operation. The LSTM network 850 may be a structure designed to avoid this long-term dependency problem. The LSTM network 850 may extract a relative change amount that changes over time in the input data as a characteristic value, and therefore, may analyze the sequential changes in the position of an object in order to determine whether an event has occurred.

[0116] The second training model may use an RNN including the LSTM network 850, and thus may use a structure related to all of the previous step, the current step, and the next step of learning, and may send information of the current step to the next step and affect the output value.

[0117] According to an embodiment, the size-reduced matrix 840 of the output of the first training model may be input into the second training model based on the order of the output from the first training model. The second training model may generate a notification message or command to control the driving vehicle by considering whether an event occurs, the type of event, the risk level of driving, etc.

[0118] For ease of explanation, a description is given by separating the first training model from the second training model. However, the first training model and the second training model may include a plurality of training models based on their functions, or one integrated training model.

[0119] Fig. 9 is a diagram illustrating an example of obtaining vehicle driving data using virtual reality according to an embodiment.

[0120] According to an embodiment, the electronic device 1000 may be a driving simulator system configured to detect an event of a vehicle using virtual reality and a real-time vehicle analysis program. According to an embodiment, the electronic device 1000 may use a virtual environment to effectively obtain data that is difficult to predict in real life, such as an accident.

[0121] According to an embodiment, using the driving simulation device 900, the electronic device 1000 can effectively obtain driving data by detecting objects with consistent patterns (such as roads, buildings, etc.) and objects with inconsistent patterns (such as the speed of the driving vehicle, the distance between the driving vehicle and another vehicle, etc.).

[0122] According to an embodiment, the electronic device 1000 may obtain driving data of a vehicle in the driver's field of view using the driving simulation device 900. According to an embodiment, the driving data of a vehicle in the driver's field of view obtained from the driving simulation device 900 may be sent to the CREST UDP sender 901 and the HDMI splitter 902.

[0123] According to an embodiment, the CREST UDP sender 901 may also use the User Datagram Protocol (UDP) to send the driving data of the vehicle in the driver's field of view to the driving data crawler 905 .

[0124] According to an embodiment, the HDMI splitter 902 may distribute the driving data of the vehicle in the field of view of the driver to the image capture card 903, and the image capture card 903 may capture (904) a frame image included in the driving data of the vehicle.

[0125] According to an embodiment, the electronic device 1000 may synchronize the driving data from the driving data crawler 905 with the captured image received from the image capture card 903 in operation 906, and then may store the synchronized driving data and the captured image in a database in operation 907. According to an embodiment, the electronic device 1000 may store the driving data image in a database and manage the database.

[0126] According to an embodiment, the electronic device 1000 may detect an event from the driving data stored in the database in operation 908. According to an embodiment, the electronic device 1000 may detect an event occurring during driving of the vehicle by analyzing the driving data using the vehicle data analyzer 918.

[0127] According to an embodiment, the electronic device 1000 may generate a tag by classifying the type of the event detected in operation 908. According to an embodiment, the electronic device 1000 may assign the generated tag as a tag of the event.

[0128] As a result, in operation 909 , the electronic device 1000 may obtain a plurality of events classified based on the type, accuracy, etc. of the event.

[0129] Therefore, the electronic device 1000 can effectively obtain driving data including a plurality of events that are difficult to obtain in reality, taking into account the state of the driving vehicle and the states of other vehicles in a virtual driving environment having a certain pattern.

[0130] Fig.10 is a diagram illustrating an example of determining a notification message and a vehicle control command based on a type of event according to an embodiment.

[0131] According to an embodiment, when the electronic device 1000 processes the output of the first training model so that the operation amount is reduced and the processed output is input into the second training model, a notification message may be output. Alternatively, a control operation corresponding to the event may be output, or a control operation corresponding to the event and a notification message may be output.

[0132] The electronic device 1000 may generate a notification message notifying the event based on the determination of whether the event occurs.

[0133] According to an embodiment, different notification messages may be generated based on the type of object, sequential changes in the location of the object, whether an event occurs, and the like.

[0134] When it is determined that an event has occurred, the electronic device 1000 may generate a notification message notifying the event. Moreover, when it is determined that no event has occurred, the electronic device 1000 may not generate a notification message notifying the event. According to an embodiment, when the electronic device 1000 determines not to generate a notification message notifying the event, the electronic device 1000 may not generate a notification message at all, or may generate a preset notification message that does not include an event notification. For example, the electronic device 1000 may generate a notification message including at least one of the current temperature, the rpm value of the vehicle, the vehicle's forward direction, traffic conditions, and a driving risk level. According to an embodiment, a notification message not including an event notification may be set as a default value in the electronic device 1000.

[0135] According to an embodiment, the electronic device 1000 may generate a notification message based on the type of event and the driving risk level. The notification message according to the embodiment may be generated as a text message or a voice message, but is not limited thereto. Moreover, for example, the message generated as text may be converted into voice via text to speech (TSS) to obtain a notification message in voice format.

[0136] The electronic device 1000 may output the generated notification message. According to an embodiment, the notification message may be output as sound, text, image and / or vibration. According to an embodiment, the electronic device 1000 may display the notification message on a head-up display (HUD) or a dashboard. According to an embodiment, when the notification message has a voice format, the electronic device 1000 may present the notification message through a speaker of the vehicle. For example, the electronic device 1000 may send a command to a speaker of the vehicle to control the speaker of the vehicle to output a notification message in a voice format.

[0137] According to an embodiment, a command for controlling the operation of a module installed in a vehicle may be sent to the module based on the type of event and the risk level of driving. When the electronic device 1000 determines that it better controls the operation of the module installed in the vehicle based on the type of event and the risk level of driving, the electronic device 1000 may send a command for controlling the operation of the module installed in the vehicle to the module installed in the vehicle. For example, the electronic device 1000 may output a notification message based on the type of event and the risk level of driving, and simultaneously control the operation of the module installed in the vehicle. Alternatively, when a user input for controlling the operation of the module installed in the vehicle is not received within a preset time period after the output of the notification message, the electronic device 1000 may send a command for controlling the operation of the module installed in the vehicle to the module installed in the vehicle. Alternatively, the electronic device 1000 may not output a notification message, and may send a command for controlling the operation of the module installed in the vehicle to the module installed in the vehicle based on the type of event and the risk level of driving.

[0138] The user input according to the embodiment may include at least one of a step on input, a steering input, a voice input, a key input, a touch input, a bending input, and a multimodal input, but is not limited thereto. The step on input may represent a user input of stepping on a brake of the vehicle to control the brake. The steering input may represent a user input of rotating the steering wheel of the vehicle to control the steering wheel.

[0139] According to an embodiment, the notification message generated using the second training model may be changed based on the type of event and the risk level of driving. For example, the content included in the notification message may be changed based on the type of event and the risk level of driving. For example, the content included in the notification message may include a notification of the event, an action guide corresponding to the event, an alarm sound, etc.

[0140] Fig.10 is a table illustrating an example of how to determine the content of a notification message and the content of a vehicle control command based on the type of event and the risk level of driving. The table according to the embodiment is merely an example, and multiple training models can be continuously modified and improved. Therefore, the output value based on the input value in the multiple training models can be continuously modified and improved. The electronic device 1000 can use the second training model to output different notification messages and different vehicle control commands based on the type of event and the risk level of driving.

[0141] According to the embodiment, Fig.10 As shown in, for example, when the type of event is an expected accident due to failure to ensure a safe distance from the vehicle ahead and the risk level of driving is high, the electronic device 1000 can generate a notification message including content "immediately step on the brakes" and an alarm sound, the content being an action guide corresponding to the event. Furthermore, the electronic device 1000 can operate the brakes of the vehicle by applying a certain force to the brakes to generate a command to control the speed of the vehicle.

[0142] Moreover, according to an embodiment, the electronic device 1000 may determine the time at which the user input for executing the action guide will be received based on the type of event and the risk level of driving. For example, the time at which the user input will be received may be determined based on the risk level of driving. Moreover, for example, based on learning according to a preset reference, the determination of which data will be used in order to determine the time at which the user input will be received may be set and changed. For example, when it is determined that the driving risk level is high and the user input for controlling the operation of the module installed in the vehicle is not received within the preset time, the electronic device 1000 may send a command to the module installed in the vehicle for controlling the operation of the module installed in the vehicle.

[0143] For example, when the type of event is that the terrain changes to include a curved road ahead and the driving risk level is medium, the electronic device 1000 may generate a notification message including the content "Warning, the road ahead is curved" so that an action guide corresponding to the event is included in the notification message together with the notification of the event. Furthermore, the electronic device 1000 may control the steering wheel at a certain angle, thereby generating a control command to change the moving direction of the vehicle.

[0144] For example, when the type of event is a change in road conditions in which the vehicle is involved in an accident ahead and the driving risk level is low, the electronic device 1000 may generate a notification message including the content "A vehicle ahead is involved in an accident, change lanes to the right", so that an action guide corresponding to the event is included in the notification message together with the notification of the event. Furthermore, the electronic device 1000 may generate a control command to change the direction of the steering wheel in order to avoid the vehicle being involved in an accident.

[0145] For example, when the type of event is that the traffic signal changes from green to red and the risk level of driving is high, the electronic device 1000 may generate a notification message including the content "Traffic signal changes, please stop", so that the action guide corresponding to the event is included in the notification message together with the notification of the event. Moreover, the electronic device 1000 may gradually operate the brake, thereby generating a control command to stop the vehicle at the stop line.

[0146] For example, when the type of event is that the traffic signal changes from red to green and the risk level of driving is low, the electronic device 1000 may generate a notification message including the content "Traffic signal changes, please start", so that the action guide corresponding to the event is included in the notification message together with the notification of the event. Moreover, the electronic device 1000 may gradually operate the accelerator, thereby generating a control command to start the vehicle.

[0147] Fig.11 is a diagram illustrating an example of displaying a notification message on a HUD of a vehicle according to an embodiment.

[0148] According to an embodiment, the electronic device 1000 may display a notification message on the HUD of the vehicle.

[0149] For example, when the electronic device 1000 determines that an event of a predicted accident occurs due to failure to ensure a safe distance relative to the vehicle 1101 in front and the driving risk level 1103 is 35, the electronic device 1000 can control the HUD of the vehicle to display a notification message, including the driving risk level 1103 and a virtual image 1104 for ensuring a safe distance. Moreover, the electronic device 1000 can output a notification message including the content "Please step on the brake immediately" (which is an action guide corresponding to the event) as a voice together with an alarm sound. Moreover, for example, when no user's stepping input of stepping on the brake is received within a preset time period after the notification message is output, the electronic device 1000 can send a command to the brake to control the operation of the brake. For example, the preset time period can be set based on learning and can be changed based on the driving risk level 1103. For example, as the level 1103 of driving risk increases, the preset time period can be reduced. For example, when an accident is expected to occur unless the brakes are immediately depressed because the distance between the user's vehicle and the preceding vehicle 1101 is too small, a notification message may be output and a command to control the operation of the brakes may be sent to the brakes at the same time.

[0150] Fig.12 is a diagram illustrating an example of displaying a notification message in a vehicle according to an embodiment.

[0151] According to an embodiment, the electronic device 1000 may transmit a command for controlling an operation of the module installed in the vehicle to the module installed in the vehicle.

[0152] For example, when the electronic device 1000 determines that an event related to a change in traffic conditions occurs so that it is impossible to drive in the corresponding lane due to the front vehicle being involved in an accident, the electronic device 1000 may control the HUD of the vehicle to display a preset notification message that does not include an event notification. For example, the electronic device 1000 may display a notification message including at least one of the current temperature, the rpm value of the vehicle, the forward direction of the vehicle, the traffic conditions, and the driving risk level on the HUD. Moreover, the electronic device 1000 may output a notification message "The front vehicle is involved in an accident, change the lane to the right" as a voice format. Moreover, for example, when a user's steering input to rotate the steering wheel 1210 is not received from the user within a preset time period after the notification message in the voice format is output, a command to rotate the steering wheel 1210 may be sent to the steering wheel 1210. Therefore, the electronic device 1000 may prompt the user to control the steering wheel 1210, or may send a command for controlling the steering wheel 1210 to the steering wheel 1210 so as to autonomously control the driving path.

[0153] Fig.13 is a block diagram illustrating components of an electronic device 1000 according to an embodiment

[0154] Fig.14 is a block diagram illustrating components of an electronic device 1000 according to an embodiment.

[0155] like Fig.13 As described in the embodiment, the electronic device 1000 according to the embodiment may include a processor (eg, including a processing circuit) 1300, a communication interface (eg, including a communication circuit) 1500, and a memory 1700. However, Fig.13 Not all components described in the electronic device 1000 are essential components. Fig.13 The electronic device 1000 is implemented by the components described in FIG.

[0156] For example, Fig.14 It is described in the figure that according to an embodiment, the electronic device 1000 may further include an input interface (e.g., including an input circuit) 1100, an output interface (e.g., including an output circuit) 1200, a sensor 1400 and an audio / video (A / V) input interface (e.g., including an A / V input circuit) 1600, as well as a processor 1300, a communication interface 1500 and a memory 1700.

[0157] For example, the electronic device 1000 according to an embodiment may be a vehicle board including a processor 1300, a communication interface 1500 and a memory 1700, and the electronic device 1000 according to an embodiment may be a vehicle further including at least one of the following items: an input interface 1100, an output interface 1200, a sensor 1400 and an A / V input interface 1600, as well as a processor 1300, a communication interface 1500 and a memory 1700.

[0158] The input interface 1100 refers to a device including various input circuits, through which a user can input data for controlling the electronic device 1000. For example, the input interface 1100 may include various input circuits, such as but not limited to a keypad, a dome switch, a touch pad (using a touch capacitance method, a withstand voltage method, an infrared sensing method, a surface ultrasonic conduction method, an integral tension measurement method, a piezoelectric effect method, etc.), a roller, a roller switch, etc., but not limited thereto.

[0159] The input interface 1100 may receive an input for controlling operations of modules installed in the vehicle.

[0160] The output interface 1200 may include various circuits for outputting audio signals, video signals, and / or vibration signals, and may include a display 1210, a sound output interface (e.g., including a sound output circuit) 1220, and a vibration motor 1230. According to an embodiment, the output interface 1200 may output the notification message as audio, video, and / or vibration.

[0161] The display 1210 may display and output information processed in the electronic device 1000. For example, the electronic device 1210 may display a notification message on a head-up display (HUD) of a vehicle.

[0162] The sound output interface 1220 may include various circuits to output audio data received from the communication interface 1500 or stored in the memory 1700. Also, the sound output interface 1220 may output a sound signal (e.g., a call signal reception sound, a message reception sound, a notification sound) related to a function performed in the electronic device 1000. For example, the sound output interface 1220 may output an alarm sound for notifying the occurrence of an event.

[0163] The processor 1300 may include various processing circuits and generally control the operation of the electronic device 1000. For example, the processor 1300 may control the user input interface 1100, the output interface 1200, the sensor 1400, the communication interface 1500, the A / V input interface 1600, etc., generally by executing a program stored in the memory 1700. Moreover, the processor 1300 may control the functions of the electronic device 1000 by using different programs stored in the memory 1700. The processor 1300 may include at least one processor. The processor 1300 may include a plurality of processors or an integrated processor based on its functions and operations. According to an embodiment, the processor 1300 may include at least one processor configured to execute at least one program stored in the memory 1700 to provide a notification message.

[0164] According to an embodiment, the processor 1300 may obtain a video sequence including a plurality of frames from a camera installed in a vehicle via the communication interface 1500. According to an embodiment, the processor 1300 may send a command configured to control the operation of the module installed in the vehicle to the module installed in the vehicle via the communication interface 1500 based on the type of event and the risk level of driving the vehicle.

[0165] According to an embodiment, the processor 1300 may detect an object included in a plurality of frames from a plurality of frames. According to an embodiment, the processor 1300 may use coordinate information to identify information about the position of the object with respect to each of the plurality of frames. The processor 1300 may determine the position of the object in units of pixels. According to an embodiment, the processor 1300 may determine whether a driving event of a vehicle occurs by analyzing the sequential changes in the position of the object in the plurality of frames. According to an embodiment, the processor 1300 may determine the type of event and the risk level of driving the vehicle by analyzing the sequential changes in the position of the object in the plurality of frames. According to an embodiment, the processor 1300 may generate a notification message for notifying the event based on the determination of whether the event occurs. According to an embodiment, the processor 1300 may generate a notification message for notifying the event based on the type of event and the risk level of driving the vehicle. According to an embodiment, the processor 1300 may control the output interface 1200 to output the generated notification message. According to an embodiment, the processor 1300 may control the display 1210 to display the generated notification message. According to an embodiment, using different training models, the processor 1300 may detect an object, obtain information about the position of the object, determine whether an event occurs, and generate a notification message.

[0166] According to an embodiment, by adding coordinate information to the FCN, with respect to each of the plurality of frames, a first training model may be generated by learning a reference for determining the type of the object and a reference for determining the position of the object in the plurality of frames. The processor 1300 may use the first training model to determine the type of the object and determine the position of the object in the plurality of frames.

[0167] According to an embodiment, using RNN, a second training model can be generated by learning a reference for determining whether a driving event of a vehicle occurs by analyzing sequential changes in the position of an object in a plurality of frames. The processor 1300 can use the second training model to determine whether an event related to the driving of the vehicle occurs and determine the content of the notification message.

[0168] According to an embodiment, the processor 1300 may apply a filter for flattening a measure of a plurality of frames to the plurality of frames so as to input the plurality of frames into a first training model, and may reduce a size of an output of the first training model so as to input the plurality of frames into a second training model.

[0169] The sensor 1400 may include various sensors and sensing circuits to sense a state of the electronic device 1000 , a state of a user, or a state around the electronic device 1000 , and may transmit the sensed information to the processor 1300 .

[0170] The sensor 1400 may include various sensing circuits, such as but not limited to at least one of the following: a magnetic sensor 1410, an acceleration sensor 1420, a temperature / humidity sensor 1430, an infrared sensor 1440, a gyroscope sensor 1450, a position sensor (e.g., a global positioning system (GPS)) 1460, an atmospheric pressure sensor 1470, a proximity sensor 1480, and an RGB sensor 1490, but is not limited thereto. The function of each sensor can be intuitively inferred from its name by a person of ordinary skill in the art, and therefore a detailed description thereof will be omitted.

[0171] The communication interface 1500 may include various communication circuits, including at least one component configured to enable the electronic device 1000 to communicate with another electronic device (not shown) and the server 2000. The other electronic device may be a computing device or a sensor, but is not limited thereto. Moreover, for example, the other electronic device may be a module included in a vehicle together with the electronic device 1000. For example, the communication interface 1500 may include a short-range wireless communication interface 1510, a mobile communication interface 1520, and a broadcast reception interface 1530.

[0172] The short-range wireless communication interface 1510 may include a Bluetooth communication interface, a Bluetooth low energy communication interface, a near field communication interface (NFC / RFID), a WLAN (Wi-Fi) communication interface, a Zigbee communication interface, an Infrared Data Association (IrDA) communication interface (not shown), a Wi-Fi Direct (WFD) communication interface, an ultra-wideband (UWB) communication interface, an Ant+ communication interface, etc., but is not limited to these.

[0173] The mobile communication interface 1520 may include various mobile communication circuits to transmit wireless signals to at least one of the following or receive wireless signals from at least one of the following in a mobile communication network: a base station, an external terminal, and a server. Here, the wireless signal may include a voice call signal, a video phone call signal, or various types of data based on the exchange of text / multimedia messages.

[0174] The broadcast receiving interface 1530 may include various broadcast receiving circuits to receive broadcast signals and / or information related to broadcasting from the outside via a broadcast channel. The broadcast channel may include a satellite channel and a ground wave channel. According to an embodiment, the electronic device 1000 may not include the broadcast receiving interface 1530.

[0175] According to an embodiment, the communication interface 1500 may receive a video sequence including a plurality of frames from a camera installed in a vehicle. According to an embodiment, the communication interface 1500 may transmit a command for controlling an operation of the module installed in the vehicle to a module installed in the vehicle.

[0176] The A / V input interface 1600 may include various A / V interface circuits and be configured to input an audio signal or a video signal, and may include a camera 1610, a microphone 1620, etc. The camera 1610 may obtain an image frame such as a still image or a video via an image sensor in a video phone mode or a shooting mode. The image captured by the image sensor may be processed by the processor 1300 or an additional image processor (not shown). For example, the image captured by the camera 1610 may be used as information for determining whether an event has occurred.

[0177] The microphone 1620 may receive an external sound signal and process the external sound signal into electrical sound data. For example, the microphone 1620 may receive a sound signal from an external device or a user. The microphone 1620 may use various noise elimination algorithms to eliminate noise generated during the process of receiving the external sound signal.

[0178] The memory 1700 may store a program for processing and controlling the processor 1300 , and may store data input to and / or output from the electronic device 1000 .

[0179] Memory 1700 may include a storage medium from at least one of the following: a flash memory type memory, a hard disk type memory, a multimedia card micro memory, a card type memory (e.g., SD or XD memory), a random access memory (RAM), a static RAM (SRAM), a read-only memory (ROM), an electrically erasable programmable ROM (EEPROM), a programmable ROM (PROM), a magnetic memory, a magnetic disk, and an optical disk.

[0180] The program stored in the memory 1700 may be divided into a plurality of modules based on its function. For example, the program may be divided into a user interface (UI) module 1710, a touch screen module 1720, and a notification module 1730.

[0181] The UI module 1710 may provide a dedicated UI, a graphical user interface (GUI), etc., for each application that is synchronized with the electronic device 1000. The touch screen module 1720 may sense a touch gesture on the touch screen via a user, and send information about the touch gesture to the processor 1300. The touch screen module 1720 according to an embodiment may recognize and analyze a touch code. The touch screen module 1720 may be implemented as additional hardware including a processor.

[0182] The notification module 1730 may generate a signal to notify the occurrence of an event. The notification module 1730 may output the notification signal as a video signal via the display 1210 , as an audio signal via the sound output interface 1220 , or as a vibration signal via the vibration motor 1230 .

[0183] Fig.15 is a block diagram illustrating components of a processor 1300 according to an embodiment.

[0184] refer to Fig.15 , the processor 1300 may include a data learner (eg, including a processing circuit and / or a program element) 1310 and a data recognizer (eg, including a processing circuit and / or a program element) 1320 .

[0185] The data learner 1310 may include various processing circuits and / or program elements, and use the coordinate information to learn a reference for identifying the location of an object. The data learner 1310 may use the coordinate information to learn a reference about which data to use to identify the location of an object, how to use the data to obtain location information, and how to determine whether an event has occurred. By obtaining data to be used for learning and applying the obtained data to a data recognition model described below, the data learner 1310 can learn a reference for identifying the location of an object using coordinate information.

[0186] According to an embodiment, operations of detecting an object, identifying the location information of an object, determining the type of an object, determining the location of an object, determining whether an event occurs, determining the type of an event, determining the risk level of driving a vehicle, generating a notification message, generating a vehicle control command, etc. may be performed as separate operations. However, the embodiment is not limited thereto. At least two of the operations of detecting an object, identifying the location information of an object, determining the type of an object, determining the location of an object, determining whether an event occurs, determining the type of an event, determining the risk level of driving a vehicle, generating a notification message, generating a vehicle control command, etc. may be performed based on learning according to a preset reference.

[0187] The data identifier 1320 may include various processing circuits and / or program elements, and recognize the location information of the object based on the data, and determine whether an event occurs based on the recognition of the location information. The data identifier 1320 may use a learned data recognition model to recognize the location information of the object based on certain data. The data identifier 1320 may obtain specific data based on a preset reference according to learning, and use the data recognition model with the obtained data as an input value to determine how to obtain the location information of the object based on the specific data, and determine how to generate a notification message and a vehicle control command based on whether an event occurs. Moreover, the result value output by the data recognition model using the obtained data as an input value can be used to modify and improve the data recognition model.

[0188] At least one of the data learner 1310 and the data recognizer 1320 may be manufactured as at least one hardware chip and may be installed in an electronic device. For example, at least one of the data learner 1310 and the data recognizer 1320 may be manufactured as a dedicated hardware chip for AI, or may be manufactured as a part of an existing general-purpose processor (e.g., a central processing unit (CPU) or an application processor) or a graphics-specific processor (e.g., a graphics processing unit (GPU)), and may be installed in the various electronic devices described above.

[0189] In this case, the data learner 1310 and the data identifier 1320 may be installed in the same electronic device, or each may be installed in a different electronic device. For example, one of the data learner 1310 and the data identifier 1320 may be included in one electronic device 1000, and the other may be included in the server 2000 (see, for example, Fig.18 ). Moreover, the data learner 1310 and the data identifier 1320 may be connected to each other in a wired or wireless manner so that the model information established by the data learner 1310 may be provided to the data identifier 1320, or the data input to the data identifier 1320 may be provided to the data learner 1310 as additional learning data.

[0190] Meanwhile, at least one of the data learner 1310 and the data recognizer 1320 may be implemented as a software module including various program elements. When at least one of the data learner 1310 and the data recognizer 1320 is implemented as a software module (or a program module including instructions), the software module may be stored in a non-transitory computer-readable recording medium. Moreover, in this case, one or more software modules may be provided by an operating system (OS) or a predetermined application. Some of the one or more software modules may be provided by the OS, while others may be provided by a predetermined application.

[0191] Fig.16 is a block diagram illustrating a data learner 1310 according to an embodiment.

[0192] refer to Fig.16 According to an embodiment, the data learner 1310 may include a data obtainer (e.g., including a processing circuit and / or a program element) 1310-1, a preprocessor (e.g., including a processing circuit and / or a program element) 1310-2, a learning data selector (e.g., including a processing circuit and / or a program element) 1310-3, a model learner (e.g., including a processing circuit and / or a program element) 1310-4 and a model evaluator (e.g., including a processing circuit and / or a program element) 1310-5.

[0193] The data obtainer 1310-1 may include various processing circuits and / or program elements, and obtains data necessary for determining how to use coordinate information to obtain the location information of an object and how to determine whether an event has occurred. The data obtainer 1310-1 may obtain data necessary for learning configured to determine how to use coordinate information to obtain the location information of an object and how to determine whether an event has occurred.

[0194] For example, the data obtainer 1310-1 may obtain voice data, image data, text data, or bio-signal data. For example, the data obtainer 1310-1 may receive data via an input device (e.g., a microphone, a camera, a sensor, etc.) of the electronic device 1000. Alternatively, the data obtainer 1310-1 may obtain data via another electronic device that communicates with the electronic device 1000. Alternatively, the data obtainer 1310-1 may obtain data via a server 2000 that communicates with the electronic device 1000.

[0195] For example, the data obtainer 1310-1 may receive a video sequence from a camera installed in a vehicle. Also, for example, the data obtainer 1310-1 may receive a video sequence from a camera configured to capture the surrounding environment of the vehicle. Also, for example, the data obtainer 1310-1 may obtain a video sequence from a camera included in the electronic device 1000.

[0196] The preprocessor 1310-2 may include various processing circuits and / or program elements, and preprocess the obtained data so that the obtained data can be used in a learning operation to determine how to obtain pixel information and how to generate a notification message. The preprocessor 1310-2 may process the obtained data in a preset format so that the obtained data can be used in a learning operation to determine how to obtain pixel information and how to generate a notification message. For example, the preprocessor 1310-2 may perform preprocessing, whereby a filter configured to flatten the brightness of multiple frames included in a video sequence is applied to the multiple frames.

[0197] The learning data selector 1310-3 may include various processing circuits and / or program elements, and selects data necessary for learning from the preprocessed data. The selected data may be provided to the model learner 1310-4. The learning data selector 1310-3 may select data necessary for learning from the preprocessed data based on a predetermined reference for determining how to obtain pixel information and how to generate a notification message. Moreover, the learning data selector 1310-3 may select data based on a reference (which is predetermined based on the learning of the model learner 1310-4).

[0198] The model learner 1310-4 may include various processing circuits and / or program elements, and learn references on how to obtain the position information of the object based on the learning data and how to determine whether an event has occurred. In addition, the model learner 1310-4 may learn references on how to use coordinate information to obtain the position information of the object and which learning data will be used to determine whether an event has occurred.

[0199] Moreover, the model learner 1310-4 can train a data recognition model, which is used to use the coordinate information based on the learning data to determine how to obtain the location information of the object and how to determine whether an event occurs. In this case, the data recognition model can be a pre-established model. For example, the data recognition model can be a model pre-established by receiving basic learning data (e.g., a black box image of a vehicle, etc.).

[0200] The data recognition model can be established by considering the field in which the data recognition model is implemented, the purpose of learning, the computing performance of the device, etc. The data recognition model may include, for example, a model based on a neural network. For example, a model such as a deep neural network (DNN), RNN, or a bidirectional recurrent deep neural network (BRDNN) can be used as a data recognition model, but the present disclosure is not limited thereto.

[0201] According to various embodiments, when there are multiple pre-established data recognition models, the model learner 1310-4 can determine a data recognition model whose basic learning data has a high correlation with the input learning data as a data recognition model for learning. In this case, the basic learning data can be pre-classified based on the type of data, and the data recognition model can be pre-established based on the type of data. For example, the basic learning data can be pre-classified based on various references, such as the area where the learning data is generated, the time when the learning data is generated, the amount of the learning data, the type of the learning data, the generator of the learning data, etc.

[0202] Moreover, the model learner 1310-4 can train the data recognition model using, for example, a learning algorithm such as error back propagation or gradient descent.

[0203] Moreover, the model learner 1310-4 can, for example, identify the model via supervised learning training data with the learning data as input. Moreover, in the absence of instructions, the model learner 1310-4 can, for example, identify the model via unsupervised learning training data, in which, by learning its own data types, it discovers references for determining how to obtain the position information of an object based on the learning data using coordinate information and how to determine whether an event has occurred, and these data types are necessary for determining how to obtain the position information of an object based on the learning data using coordinate information and how to determine whether an event has occurred. Moreover, using feedback on whether the results of determining how to obtain the position information of an object based on the learning data using coordinate information and the results of determining how to determine whether an event has occurred based on learning are correct, the model learner 1310-4 can, for example, identify the model via reinforcement learning training data.

[0204] Moreover, when learning the data recognition model, the model learner 1310-4 can store the learned data recognition model. In this case, the model learner 1310-4 can store the learned data recognition model in the memory of the electronic device 1000 including the data identifier 1320. Alternatively, the model learner 1310-4 can store the learned data recognition model in the memory of the server 2000 connected to the electronic device 1000 via a wired or wireless network.

[0205] In this case, for example, the memory storing the learned data recognition model may also store commands or data associated with at least one other component of the electronic device 1000. Moreover, the memory may store software and / or programs. The programs may include, for example, a kernel, middleware, an application programming interface (API), and / or an application program (or "application").

[0206] The model evaluator 1310-5 may include various processing circuits and / or program elements and input evaluation data to the data recognition model, and when the result of the recognition output from the evaluation data does not meet the predetermined reference, the model learner 1310-4 may be allowed to learn again. In this case, the evaluation data may be data predetermined for evaluating the data recognition model.

[0207] For example, when the number or ratio of segments of evaluation data for which the recognition results are incorrect among the recognition results of the learned data recognition model about the evaluation data is greater than a predetermined threshold, the model evaluator 1310-5 may evaluate that the predetermined reference standard is not met. For example, when the predetermined threshold is defined as a ratio of 2%, and when the learned data recognition model outputs erroneous recognition results for more than 20 evaluation data segments among a total of 1000 evaluation data segments, the model evaluator 1310-5 may evaluate that the learned data recognition model is inappropriate.

[0208] Meanwhile, when there are multiple learned data recognition models, the model evaluator 1310-5 may evaluate whether each learned data recognition model satisfies a predetermined reference standard, and may determine the learned data recognition model that satisfies the predetermined reference standard as the final data recognition model. In this case, when there are multiple learned data recognition models that satisfy the predetermined reference standard, the model evaluator 1310-5 may determine any one or a predetermined number of learned data recognition models as the final data recognition model in descending order of evaluation scores.

[0209] Meanwhile, at least one of the data acquirer 1310-1, the preprocessor 1310-2, the learning data selector 1310-3, the model learner 1310-4, and the model evaluator 1310-5 in the data learner 1310 may be manufactured as at least one hardware chip and may be installed on the electronic device 1000. For example, at least one of the data acquirer 1310-1, the preprocessor 1310-2, the learning data selector 1310-3, the model learner 1310-4, and the model evaluator 1310-5 may be manufactured as a dedicated hardware chip for AI, or may be manufactured as part of an existing general-purpose processor (e.g., a CPU or an application processor) or a graphics-dedicated processor (e.g., a GPU), and may be installed in the various electronic devices described above.

[0210] Moreover, the data acquirer 1310-1, the preprocessor 1310-2, the learning data selector 1310-3, the model learner 1310-4, and the model evaluator 1310-5 may be installed in one electronic device, or each may be installed in a different electronic device. For example, some of the data acquirer 1310-1, the preprocessor 1310-2, the learning data selector 1310-3, the model learner 1310-4, and the model evaluator 1310-5 may be included in the electronic device 1000, and the others may be included in the server 2000.

[0211] Moreover, at least one of the data acquirer 1310-1, the preprocessor 1310-2, the learning data selector 1310-3, the model learner 1310-4, and the model evaluator 1310-5 may be implemented as a software module. When at least one of the data acquirer 1310-1, the preprocessor 1310-2, the learning data selector 1310-3, the model learner 1310-4, and the model evaluator 1310-5 is implemented as a software module (or a program module including instructions), the software module may be stored in a non-transitory computer-readable recording medium. Moreover, in this case, one or more software modules may be provided by the OS or a predetermined application. Alternatively, some of the one or more software modules may be provided by the OS, while others may be provided by a predetermined application.

[0212] Fig.17 is a block diagram illustrating the data identifier 1320 according to an embodiment.

[0213] refer to Fig.17 According to an embodiment, the data identifier 1320 may include a data obtainer (e.g., including a processing circuit and / or a program element) 1320-1, a preprocessor (e.g., including a processing circuit and / or a program element) 1320-2, an identification data selector (e.g., including a processing circuit and / or a program element) 1320-3, an identification result provider (e.g., including a processing circuit and / or a program element) 1320-4, and a model modifier (e.g., including a processing circuit and / or a program element) 1320-5.

[0214] The data obtainer 1320 - 1 may include various processing circuits and / or program elements, and obtain data necessary for determining how to obtain position information of an object based on learning data and how to determine whether an event occurs.

[0215] The preprocessor 1320-2 may include various processing circuits and / or program elements, and preprocess the obtained data so that the obtained data can be used to determine how to obtain the position information of the object based on the learning data and how to determine whether an event occurs. The preprocessor 1320-2 may process the obtained data in a predetermined format so that the recognition result provider 1320-4, which will be described below, can use the obtained data to determine how to obtain the position information of the object based on the learning data and how to determine whether an event occurs.

[0216] The recognition data selector 1320-3 may include various processing circuits and / or program elements, and selects data necessary for determining how to obtain the position information of the object based on the learning data and how to determine whether an event has occurred from the preprocessed data. The selected data may be provided to the recognition result provider 1320-4. Based on a predetermined reference for determining how to obtain the position information of the object based on the learning data and how to determine whether an event has occurred, the recognition data selector 1320-3 may select part or all of the preprocessed data. Moreover, the recognition data selector 1320-3 may select data based on a predetermined reference according to the learning of the model learner 1310-4 described above.

[0217] The recognition result provider 1320-4 may include various processing circuits and / or program elements, and apply the selected data to the data recognition model to determine how to obtain pixel information and how to generate notification messages. The recognition result provider 1320-4 may provide recognition results based on data recognition purposes. The recognition result provider 1320-4 may use the data selected by the recognition data selector 1320-3 as input to apply the selected data to the data recognition model. Moreover, the recognition result may be determined by the data recognition model.

[0218] The model modifier 1320-5 may include various processing circuits and / or program elements and allow modification and improvement of the data recognition model based on the evaluation of the recognition results provided by the recognition result provider 1320-4. For example, the model modifier 1320-5 may provide the recognition results provided by the recognition result provider 1320-4 to the model learner 1310-4 so that the model learner 1310-4 can modify and improve the data recognition model.

[0219] Meanwhile, at least one of the data obtainer 1320-1, the preprocessor 1320-2, the recognition data selector 1320-3, the recognition result provider 1320-4, and the model modifier 1320-5 in the data recognizer 1320 may be manufactured as at least one hardware chip and may be installed on the electronic device 1000. For example, at least one of the data obtainer 1320-1, the preprocessor 1320-2, the recognition data selector 1320-3, the recognition result provider 1320-4, and the model modifier 1320-5 may be manufactured as a dedicated hardware chip for AI, or may be manufactured as a part of an existing general-purpose processor (e.g., a CPU or an application processor) or a graphics-dedicated processor (e.g., a GPU), and may be installed in the various electronic devices described above.

[0220] Moreover, the data obtainer 1320-1, the preprocessor 1320-2, the recognition data selector 1320-3, the recognition result provider 1320-4, and the model modifier 1320-5 may be installed in one electronic device, or each may be installed in a different electronic device. For example, some of the data obtainer 1320-1, the preprocessor 1320-2, the recognition data selector 1320-3, the recognition result provider 1320-4, and the model modifier 1320-5 may be included in the electronic device 1000, and the others may be included in the server 2000.

[0221] Moreover, at least one of the data obtainer 1320-1, the preprocessor 1320-2, the identification data selector 1320-3, the identification result provider 1320-4, and the model modifier 1320-5 may be implemented as a software module. When at least one of the data obtainer 1320-1, the preprocessor 1320-2, the identification data selector 1320-3, the identification result provider 1320-4, and the model modifier 1320-5 is implemented as a software module (or a program module including instructions), the software module may be stored in a non-transitory computer-readable recording medium. Moreover, in this case, one or more software modules may be provided by the OS or a predetermined application. Alternatively, some of the one or more software modules may be provided by the OS, while others may be provided by a predetermined application.

[0222] Fig.18 is a diagram illustrating an example in which the electronic device 1000 and the server 2000 synchronize with each other to learn and recognize data according to an embodiment.

[0223] refer to Fig.18 , the server 2000 may learn a reference for determining how to obtain the location information of the object and how to determine whether an event occurs. The electronic device 1000 may determine how to obtain the location information of the object and how to determine whether an event occurs based on the result of the learning of the server 2000.

[0224] In this case, the model learner 2340 of the server 2000 may include various processing circuits and / or program elements, and execute Fig.16 The model learner 2340 of the server 2000 can learn a reference about which data will be used to determine how to obtain the location information of the object and how to determine whether an event has occurred, and a reference to using data to determine how to obtain the location information of the object and how to determine whether an event has occurred. By obtaining data to be used for learning and applying the obtained data to a data recognition model, the model learner 2340 can learn a reference for determining how to obtain the location information of the object and how to determine whether an event has occurred.

[0225] Moreover, the recognition result provider 1320-4 of the electronic device 1000 may include various processing circuits and / or program elements, and apply the data selected by the recognition data selector 1320-3 to the data recognition model generated by the server 2000 to determine how to obtain the location information of the object and how to determine whether an event occurs. For example, the recognition result provider 1320-4 may include various processing circuits and / or program elements and send the data selected by the recognition data selector 1320-3 to the server 2000, and may request the server 2000 to apply the data selected by the recognition data selector 1320-3 to the data recognition model to determine how to obtain pixel information and how to generate a notification message. Moreover, the recognition result provider 1320-4 may include various processing circuits and / or program elements, and receive information from the server 2000 about the determination via the server 2000, the determination being about how to obtain the location information of the object and how to determine whether an event occurs.

[0226] The recognition result provider 1320-4 of the electronic device 1000 may receive the data recognition model generated by the server 2000 from the server 2000, and may use the received data recognition model to determine how to obtain the location information of the object and how to determine whether an event has occurred. In this case, by applying the data selected by the recognition data selector 1320-3 to the data recognition model received from the server 2000, the recognition result provider 1320-4 of the electronic device 1000 may determine how to obtain the location information of the object and how to determine whether an event has occurred.

[0227] Some embodiments may be implemented via a recording medium including computer executable instructions, such as a program module executed by a computer. A computer readable recording medium may be any available medium that can be accessed by a computer, and examples thereof include all volatile and non-volatile media and separable and non-separable media. In addition, examples of computer readable media may include computer storage media and communication media. Examples of computer storage media include all volatile and non-volatile media and separable and non-separable media that have been implemented by any method or technology to store information, such as computer readable instructions, data structures, program modules, or other data.

[0228] Furthermore, in the present disclosure, an “interface” may include a hardware component such as a processor or a circuit, and / or a software component executed by a hardware component such as a processor.

[0229] Although the present disclosure has been illustrated and described with reference to various exemplary embodiments of the present disclosure, it will be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the present disclosure as defined by the following claims. Therefore, it should be understood that the above-described embodiments do not limit the scope of the present disclosure. For example, each component described in a single type may be executed in a distributed manner, and components described in a distributed form may also be executed in an integrated form.

[0230] The scope of the present disclosure is indicated by the claims rather than the detailed description of the present disclosure, and it should be understood that the claims and all modifications or modified forms derived from the concepts of the claims are included in the scope of the present disclosure.

Claims

1. An electronic device, comprising: processor; and a memory for storing instructions executable by the processor, The processor is configured to execute the instructions to cause the electronic device to: obtaining a video sequence from a vehicle, the video sequence comprising a plurality of frames captured while the vehicle is traveling, recognizing a location of an object in a first frame and a second frame of the plurality of frames by using a first training model, wherein the first training model outputs a matrix including information about a type of the object and a location of the object based on each of the first frame and the second frame included in the video sequence, When a matrix generated by reducing the size of a matrix output from a first training model is input to a second training model, sequential changes in the position of the object in a first frame and a second frame are analyzed by using the second training model, wherein the position of the object in the plurality of frames is recognized in units of pixels, and, wherein the first training model uses a fully convolutional network (FCN), and the size of the matrix output from the first training model is reduced by performing dilated convolution, wherein the dilated convolution is performed by dilating the size of a receptive field (RF), and determining whether a driving event of the vehicle occurs based on an output of the second training model, determining a moving speed and a moving direction of the object based on sequential changes in the position of the object in the plurality of frames, determining a type of driving event of the vehicle and a risk level of driving the vehicle based on a moving speed and a moving direction of the object, generating commands to control driving of the vehicle based on the determined type of driving event of the vehicle and the determined risk level of driving the vehicle, and operating at least one of a steering wheel, a brake, and an accelerator based on a command to control driving of the vehicle, The processor is further configured to execute instructions to enable the electronic device to: A filter is applied to the plurality of frames, the filter being configured to flatten brightness of the plurality of frames for inputting the plurality of frames into the first training model.

2. The electronic device as claimed in claim 1, wherein: The processor is further configured to execute the instructions to cause the electronic device to: determining the type of the object, and Whether the object is noise is determined by comparing information about a reference size of the type of the object in the recognized position of the object with information about a size of the type of the object in the recognized position of the object.

3. The electronic device as claimed in claim 2, wherein: The information about the size includes information predefining a range of an area that the object will occupy in the plurality of frames based on a distance between the vehicle and the object.

4. The electronic device of claim 1, wherein the processor is further configured to execute the instructions to cause the electronic device to: detecting a learning object from a video sequence including a plurality of learning frames captured while driving a learning vehicle, and simultaneously mapping coordinate information to the learning object to obtain a first training model based on a result of learning a position of the learning object, wherein the position of the learning object corresponds to the coordinate information of the learning object in the plurality of learning frames, and The obtained first training model is used to recognize a location of the object in at least one of the plurality of frames.

5. The electronic device as claimed in claim 1, wherein: The electronic device is configured to generate the first training model by learning a reference for determining an importance based on a position of a learning object in at least one of a plurality of learning frames using coordinate information added to a fully convolutional network (FCN).

6. The electronic device as claimed in claim 1, wherein: The processor is further configured to execute instructions to cause the electronic device to: When the objects detected in the plurality of frames include a plurality of objects, a likelihood of occurrence of a driving event of the vehicle is determined as a numerical value, the driving event being based on each of the plurality of objects.

7. The electronic device as claimed in claim 1, wherein: The processor is further configured to execute instructions to cause the electronic device to: using a recurrent neural network (RNN) to learn whether a driving event of a learning vehicle occurs based on sequential changes in positions of a learning object in a plurality of learning frames to obtain a second training model based on a result of the learning, and The obtained second training model is used to determine whether a driving event of the vehicle with respect to the object occurs.

8. The electronic device as claimed in claim 1, wherein: The processor is further configured to execute instructions to cause the electronic device to: Driving data of a learning vehicle is obtained from a virtual simulation device including a virtual simulation circuit, wherein the driving data of the learning vehicle includes at least one of the following: a driving image captured when driving the learning vehicle, information about the virtual simulation device, information about the state of the learning vehicle, information about the driving of the learning vehicle, information about the driving of other vehicles, and information about the surrounding environment.

9. The electronic device as claimed in claim 8, wherein: The driving data of the learning vehicle includes driving data obtained by driving the learning vehicle in a virtual driving environment in which a driving environment including an inconsistent pattern is added to a driving environment including a consistent pattern.

10. A method comprising: obtaining a video sequence from a vehicle, the video sequence comprising a plurality of frames captured while the vehicle is traveling, recognizing the position of an object in a first frame and a second frame included in the plurality of frames by using a first training model, wherein the first training model outputs a matrix including information about the type of the object and the position of the object based on each of the first frame and the second frame included in the video sequence, The method comprises analyzing sequential changes in the position of the object in the first frame and the second frame by using a second training model, wherein the second training model is used based on a matrix generated by reducing the size of the matrix output from the first training model being used as an input to the second training model, wherein the position of the object in the plurality of frames is recognized in units of pixels, and wherein the first training model uses a fully convolutional network (FCN), and the size of the matrix output from the first training model is reduced by performing dilated convolution, wherein the dilated convolution is performed by dilating the size of a receptive field (RF), and determining whether a driving event of the vehicle occurs based on an output of the second training model, determining a moving speed and a moving direction of the object based on sequential changes in the position of the object in the plurality of frames, determining a type of driving event of the vehicle and a risk level of driving the vehicle based on a moving speed and a moving direction of the object, generating commands to control driving of the vehicle based on the determined type of driving event of the vehicle and the determined risk level of driving the vehicle, and operating at least one of a steering wheel, a brake, and an accelerator based on a command to control driving of the vehicle, Wherein, the method further comprises: A filter is applied to the plurality of frames, the filter being configured to flatten brightness of the plurality of frames for inputting the plurality of frames into the first training model. 11 . A non-transitory computer-readable recording medium having stored thereon instructions which, when executed by a processor, cause an electronic device to execute the method of claim 10 .

Images