Hazard detection method and electronic device for performing this method

Abstract

A method and an electronic device for detecting a danger to a vehicle are provided. The method for detecting a danger to a vehicle includes an operation of acquiring an original image captured using a camera mounted on the exterior of the vehicle, an operation of generating a first processed image for a deep learning image recognition method and a second processed image for a computer vision image recognition method based on the original image, an operation of generating deep learning recognition data for the first processed image using a deep learning image recognition model, an operation of generating computer vision recognition data for the second processed image using the computer vision image recognition model, and an operation of detecting a danger to the vehicle based on the deep learning recognition data and the computer vision recognition data.

Description

[Technical Field]

[0001] The present invention relates to a method and an electronic device for detecting danger in a vehicle using a camera. [Background technology]

[0002] For safe driving, sufficient visibility around the vehicle must be ensured. In particular, if the vehicle is large or the driver's seat is high, it is necessary to ensure the driver's visibility in blind spots such as the lower front or side of the passenger seat to prevent dangerous situations and accidents.

[0003] To ensure the driver's field of vision, a function is provided to output images captured by a camera installed outside the vehicle to a monitor inside the vehicle. However, there is a need to install front and side cameras separately, or for the driver to directly view the captured images and determine dangerous situations. Summary of the Invention [Problem to be solved by the invention]

[0004] An electronic device according to one embodiment provides a method for detecting a danger using a camera installed in a vehicle. [Means for solving the problem]

[0005] In one embodiment, a method for detecting danger to a vehicle, which is performed by an electronic device installed in the vehicle, includes an operation of acquiring original video captured using a camera mounted on the outside of the vehicle, an operation of generating a first processed video for a deep learning video recognition method and a second processed video for a computer vision video recognition method based on the original video, an operation of generating deep learning recognition data for the first processed video using a deep learning video recognition model, an operation of generating computer vision recognition data for the second processed video using a computer vision video recognition model, and an operation of detecting danger to the vehicle based on the deep learning recognition data and the computer vision recognition data.

[0006] The operation of generating the first processed image for the deep learning image recognition method and the second processed image for the computer vision image recognition method based on the original image may include the operation of generating the first processed image by processing RGB or YUV conversion based on the original image, and generating the second processed image by processing YUV conversion based on the original image.

[0007] The operation of generating the deep learning recognition data for the first processed image using the deep learning image recognition model may include an operation of performing preprocessing on the first processed image, including image quantization, and an operation of generating the deep learning recognition data by determining a target object for the preprocessed first processed image using the deep learning image recognition model.

[0008] The operation of generating the deep learning recognition data for the first processed image using the deep learning image recognition model may further include an operation of performing post-processing on the deep learning recognition data using an anchor box algorithm and non-maximum suppression (NMS).

[0009] The operation of generating the computer vision recognition data for the second processed image using the computer vision image recognition model may include an operation of performing preprocessing on the second processed image, including noise reduction and edge enhancement, and an operation of generating the computer vision recognition data by determining a target object for the preprocessed second image using the computer vision image recognition model.

[0010] The operation of generating the computer vision recognition data for the second processed image using the computer vision image recognition model may further include an operation of post-processing the computer vision recognition data using Intersection over Union (IoU) tracking.

[0011] The operation of generating the computer vision recognition data for the second processed image using the computer vision image recognition model may include the operation of generating the computer vision recognition data for the second processed image using the computer vision image recognition model, the computer vision recognition data including correction data based on camera parameters.

[0012] The camera parameters may include characteristics of the sensor and lens that constitute the camera, or the coordinates and installation angle at which the camera is mounted on the vehicle.

[0013] The deep learning recognition data and the computer vision recognition data may include the type and coordinates of the target object.

[0014] The operation of detecting a danger to the vehicle based on the deep learning recognition data and the computer vision recognition data may include an operation of redetermining a target object by applying a weighting value to the deep learning recognition data and the computer vision recognition data, and an operation of detecting a danger to the vehicle based on the redetermined target object and vehicle data of the vehicle.

[0015] The vehicle data may include at least one of a measured distance between the vehicle and a target object, the vehicle speed / acceleration, the vehicle position, or driving information.

[0016] The method may further include an operation of outputting a monitoring screen for detecting a danger to the vehicle to a user in real time.

[0017] The method may further include sounding a danger alarm if a danger to the vehicle is detected.

[0018] In one embodiment, an electronic device for detecting danger to a vehicle may include a communication unit that acquires original video captured using a camera mounted on the outside of the vehicle, an image processing unit that generates a first processed image for a deep learning image recognition method and a second processed image for a computer vision image recognition method based on the original video, an image recognition unit that generates deep learning recognition data for the first processed image using a deep learning image recognition model and generates computer vision recognition data for the second processed image using a computer vision image recognition model, and a danger detection software unit that detects danger to the vehicle based on the deep learning recognition data and the computer vision recognition data. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a schematic diagram of a hazard detection system according to one embodiment; [Figure 2] FIG. 10 is a diagram for explaining a danger detection situation according to an embodiment. [Figure 3] 1 is a flowchart of a hazard detection method according to an embodiment. [Figure 4] 1 is a flowchart of a processed image generating method according to an embodiment. [Figure 5] 1 is a flowchart of a deep learning recognition data generation method according to an example. [Figure 6] 1 is a flowchart of a computer vision recognition data generation method according to an example. [Figure 7] 1 is a flowchart of a hazard detection method according to an example. [Figure 8] FIG. 1 is a block diagram of an electronic device according to an embodiment. [Figure 9] 1 is a schematic diagram of a hazard detection system according to one embodiment; DETAILED DESCRIPTION OF THE INVENTION

[0020] Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only and may be modified in various forms. Therefore, the embodiments are not limited to the specific disclosed forms, and the scope of this specification includes modifications, equivalents, or alternatives that fall within the technical ideas described in the embodiments.

[0021] Although terms such as "first" or "second" may be used to describe multiple components, such terms should be construed only to distinguish one component from the other components. For example, a first component may be designated as a second component, and similarly, a second component may be designated as a first component.

[0022] When any component is referred to as being "connected" to another component, it is directly linked or connected to the other component, but it should be understood that there may be other components in between.

[0023] The singular expression includes the plural expression unless the context clearly dictates otherwise. In this specification, the words "comprise" or "have" and the like indicate the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0024] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art. Commonly used, predefined terms should be interpreted as having a meaning consistent with the meaning they have in the context of the relevant art, and should not be interpreted as having an ideal or overly formal meaning unless expressly defined herein.

[0025] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. When describing with reference to the drawings, the same components will be given the same reference numerals regardless of the reference numerals, and redundant description thereof will be omitted.

[0026] FIG. 1 is a schematic diagram of a hazard detection system according to one embodiment.

[0027] The danger detection system includes an electronic device 100 and a camera 110. The electronic device 100 includes an image processing unit 120, an image recognition unit 130, and a danger detection software unit 140. For example, the danger detection system may be provided in a vehicle.

[0028] According to one embodiment, camera 110 is equipped with a wide-angle lens (e.g., a fisheye lens) and can capture an angle of 180° or more. For example, camera 110 may capture an angle of 200° or more. Therefore, even if only one camera is installed on the exterior of the passenger side of the vehicle, it is possible to simultaneously ensure visibility of blind spots such as the lower front and lower sides of the passenger seat.

[0029] According to one embodiment, the image processing unit 120 can generate a first processed image for a deep learning image recognition method and a second processed image for a computer vision image recognition method based on the original image captured by the camera 110.

[0030] According to one embodiment, the video recognition unit 130 includes a deep learning recognition unit 132 and a computer vision recognition unit 134. The deep learning recognition unit 132 can generate recognition data by determining a target object using a pre-stored and / or trained deep learning video recognition model. The computer vision recognition unit 134 can generate recognition data by determining a target object using a pre-stored and / or trained computer vision video recognition model. For example, each recognition model may be trained to determine the target object using reinforcement learning, supervised learning, or the like.

[0031] The danger detection software unit 140 collects and stores various data including vehicle data and camera parameters, and can perform comprehensive danger detection based on all the data, as will be described in detail with reference to FIG.

[0032] According to an embodiment, the electronic device 100 includes a video recognition engine including a video processing unit 120 and a video recognition unit 130. The video recognition engine may further include a video input unit (not shown) that receives an original video from the camera 110 and a video output unit (not shown) that outputs recognition data.

[0033] Depending on the embodiment, there may be integrated software for both the video recognition engine and the danger detection software unit 140, and one processor (or two or more processors) may execute the integrated software. The software embodiments are not limited to those described above, and there may be software corresponding to each of the video recognition engine and the danger detection software unit 140, and one processor (or two or more processors) may execute each piece of software.

[0034] The electronic device 100 may output a monitoring screen for detecting danger to the vehicle to the driver in real time via a display. The output image is provided based on the first processed image and / or the second processed image. For example, the output image may be the first processed image and / or the second processed image after correcting the lens distortion rate of the camera 110. For example, the output image may be the first processed image and / or the second processed image after correcting deviations related to the coordinates and installation angle at which the camera 110 is mounted on the vehicle.

[0035] According to an embodiment, the electronic device 100 may be expressed as including the camera 110 because the electronic device 100 and the camera 110 can transmit and receive images even when they are not integrated but are separate.

[0036] FIG. 2 is a diagram for explaining a danger detection situation according to one embodiment.

[0037] According to one embodiment, camera 210 (e.g., camera 110 in FIG. 1) may be provided on the exterior of vehicle 200 in the passenger seat direction. Camera 210 is provided to capture blind spots in front of and on the side of the vehicle in the passenger seat direction. For example, camera 210 may be provided to capture blind spots on the front right side and lower side of the vehicle. In particular, if vehicle 200 is large, camera 210 may capture images in the lower right corner of the vehicle, which is difficult for the driver to see. Camera 210 may be provided close to a side mirror in the passenger seat direction of vehicle 200. For example, camera 210 may be provided below the side mirror in the passenger seat direction of vehicle 200, but is not limited thereto.

[0038] According to one embodiment, the shaded area in FIG. 2 indicates an area corresponding to the captureable angle (e.g., 200°) of camera 210. If necessary, an auxiliary camera may be provided to the left, right, above, or below camera 210. When an auxiliary camera is provided, the area captured by the camera covers a wider area in addition to the sector shown in FIG. 2. Alternatively, the auxiliary camera may be the same as camera 210 or may focus on capturing an area smaller than that.

[0039] An electronic device (e.g., electronic device 100 of FIG. 1) installed inside vehicle 200 may detect objects 220, 221, 222, 230, and 231 based on images captured by camera 210. The electronic device may generate recognition data, such as deep learning recognition data and computer vision recognition data, including the types and coordinates of objects 220, 221, 222, 230, and 231. The object types may include, for example, dynamic / static objects, pedestrians, pedestrians in vehicles, utility poles, roadside trees, traffic lights, animals, etc. For example, the electronic device may determine dynamic objects 220, 221, and 222 and static objects 230 and 231, respectively. For example, the electronic device may determine the probability that each object corresponds to a particular type.

[0040] The electronic device detects a target object from among the objects 220, 221, 222, 230, and 231. The target object refers to an object among the objects 220, 221, 222, 230, and 231 that may cause a dangerous situation for the vehicle 200. For example, referring to FIG. 2, the electronic device may determine, as target objects, the object 220 moving straight in the direction of the vehicle 200 making a right turn and the object 222 located in the path of the vehicle 200. Depending on the embodiment, a series of processes of detecting an object and determining the target object may be treated as a single process for determining the target object or may be processed at one time.

[0041] The electronic device can apply weights to the recognition data to redetermine the target object, which may be the same as or different from the previously determined target object.

[0042] The electronic device detects a danger to the vehicle 200 based on the target object and vehicle data of the vehicle 200. The vehicle data may include at least one of a measured distance between the vehicle and the target object, vehicle speed / acceleration, vehicle position, or driving information (e.g., left / right turn, braking / acceleration operation, gear change, turn signal on / off, etc.). For example, referring to FIG. 2, the electronic device may detect a danger based on the coordinates of the object 220 determined as the target object and the speed and right turn information of the vehicle 200.

[0043] FIG. 3 is a flowchart of a hazard detection method according to one embodiment.

[0044] According to one embodiment, the following operations 310-350 may be performed by an electronic device (eg, electronic device 100 of FIG. 1).

[0045] In operation 310, the electronic device acquires original video captured using a camera mounted on the exterior of the vehicle.

[0046] According to one embodiment, the electronic device can acquire original video captured using one camera (eg, camera 110 in FIG. 1 or camera 210 in FIG. 2).

[0047] According to one embodiment, if the vehicle is further equipped with an auxiliary camera, the electronic device can acquire original images captured by the two cameras.

[0048] In operation 320, the electronic device generates a first processed image for the deep learning image recognition method and a second processed image for the computer vision image recognition method based on the original image.

[0049] The electronic device may crop or resize the original image to generate the first processed image and the second processed image, respectively.

[0050] For example, the electronic device may generate a first processed image by performing RGB or YUV conversion based on the original image. After performing RGB or YUV conversion based on the original image, the electronic device may additionally perform appropriate cropping, resizing, and / or cropping to generate the first processed image. The first processed image is transmitted to a deep learning recognition unit (e.g., deep learning recognition unit 132 in FIG. 1).

[0051] For example, the electronic device may generate the second processed image by performing YUV conversion based on the original image. After performing YUV conversion based on the original image, the electronic device may additionally perform appropriate cropping, resizing, and / or cropping to generate the second processed image. The second processed image is transmitted to a computer vision recognition unit (e.g., computer vision recognition unit 134 in FIG. 1).

[0052] According to one embodiment, when there are multiple original images captured using an additional auxiliary camera, the electronic device can perform multiple image processing and transmission for two or more channels.

[0053] At operation 330, the electronic device generates deep learning recognition data for the first processed image using the deep learning image recognition model.

[0054] The electronic device may perform pre-processing, including image quantization, on the first processed image. The electronic device may generate deep learning recognition data by determining a target object from the pre-processed first processed image using a deep learning image recognition model.

[0055] For example, the electronic device may perform post-processing on the deep learning recognition data using an anchor box algorithm and non-maximum suppression (NMS).

[0056] At operation 340, the electronic device generates computer vision recognition data for the second processed image using the computer image recognition model.

[0057] The electronic device can perform preprocessing, including noise reduction and edge enhancement, on the second processed image. The electronic device can generate computer vision recognition data by determining a target object in the preprocessed second image using a computer vision image recognition model.

[0058] For example, the electronic device may post-process the computer vision recognition data using Intersection over Union (IoU) tracking.

[0059] According to one embodiment, the electronic device can use the computer vision image recognition model to generate computer vision recognition data for the second processed image, including calibration data based on camera parameters, which may include characteristics of a sensor and a lens constituting a camera (or a camera module), or coordinates and an installation angle at which the camera is mounted on the vehicle.

[0060] According to one embodiment, operations 330 and 340 may be performed simultaneously.

[0061] In operation 350, the electronic device detects a danger to the vehicle based on the deep learning recognition data and the computer vision recognition data, as will be described in more detail with reference to FIG.

[0062] FIG. 4 is a flowchart of a processed image generating method according to an embodiment.

[0063] According to one embodiment, the next operation 410 is performed by a video processor (e.g., video processor 120 of FIG. 1) of an electronic device (e.g., electronic device 100 of FIG. 1) after operation 310 described with reference to FIG. 3.

[0064] In operation 410, the electronic device generates a first processed image by processing RGB conversion (or YUV conversion) based on the original image, and generates a second processed image by processing YUV conversion based on the original image.

[0065] For example, the image processor may generate the first processed image by performing RGB or YUV conversion based on the original image, and then the image processor may generate the first processed image by additionally appropriately cropping, resizing, and / or cropping the original image after performing RGB or YUV conversion based on the original image.

[0066] For example, the image processor may generate the second processed image by performing YUV conversion based on the original image, and the image processor may generate the second processed image by additionally appropriately cropping, resizing, and / or cropping the original image after performing YUV conversion based on the original image.

[0067] FIG. 5 is a flowchart of a deep learning recognition data generation method according to an example.

[0068] According to one embodiment, operation 330 illustrated in Figure 3 includes the following operations 510-530. The following operations 510-530 are performed by a deep learning recognizer (e.g., deep learning recognizer 132 in Figure 1) of an electronic device (e.g., electronic device 100 in Figure 1).

[0069] In operation 510, the deep learning recognizer performs pre-processing, including image quantization, on the first processed image.

[0070] In operation 520, the deep learning recognition unit generates deep learning recognition data by determining a target object for the pre-processed first processing image using the deep learning image recognition model. The deep learning recognition data may include the type and coordinates of the target object.

[0071] In operation 530, the deep learning recognizer performs post-processing on the deep learning recognition data using the anchor box algorithm and NMS.

[0072] FIG. 6 is a flowchart of a computer vision recognition data generation method according to an example.

[0073] According to one embodiment, the operations 340 illustrated in Figure 3 may include the following operations 610-630. The following operations 610-630 are performed by a computer vision recognizer (e.g., computer vision recognizer 134 of Figure 1) of an electronic device (e.g., electronic device 100 of Figure 1).

[0074] In operation 610, the computer vision recognizer performs pre-processing on the second processed image, including noise reduction and edge enhancement.

[0075] At operation 620, the computer vision recognizer uses the computer vision image recognition model to determine a target object for the pre-processed second processed image, thereby generating computer vision recognition data. The computer vision recognition data can include the type and coordinates of the target object.

[0076] According to one embodiment, the computer vision recognition unit may reference vehicle data obtained from the hazard detection software unit to generate the computer vision recognition data, as will be described in more detail with reference to FIG.

[0077] In operation 630, the computer vision recognizer performs post-processing on the computer vision recognition data using IoU tracking.

[0078] According to one embodiment, the computer vision recognition unit generates computer vision recognition data for the second processed image using a computer vision image recognition model, the computer vision recognition data including correction data based on camera parameters. The camera parameters may include characteristics of a sensor and a lens constituting a camera (or a camera module) or coordinates and an installation angle at which the camera is mounted on the vehicle. The camera parameters may be pre-stored in the electronic device, for example, by user input via a UI. The computer vision recognition unit transmits the correction data to a hazard detection software unit (e.g., hazard detection software unit 140 in FIG. 1 ).

[0079] According to an embodiment, the computer vision recognition unit may be implemented using a computer vision algorithm (e.g., feature extraction, background subtraction, etc.). For example, the computer vision recognition unit may be implemented to take into account environmental conditions such as rain, snow, etc. The computer vision recognition unit may also correct for distortion of a lens used for capturing images.

[0080] FIG. 7 is a flowchart of a hazard detection method according to an example.

[0081] According to one embodiment, the following operations 710 and 720 are performed by a threat detection software portion (e.g., threat detection software portion 140 of FIG. 1) of an electronic device (e.g., electronic device 100 of FIG. 1) after operation 340 described in FIG. 3.

[0082] According to one embodiment, the hazard detection software unit can collect vehicle data in real time to detect hazards to the vehicle. For example, the vehicle data may include at least one of a measured distance between the vehicle and a target object, vehicle speed / acceleration, vehicle position, or driving information (e.g., left / right turn, braking / acceleration operation, gear change, turn signal on / off, etc.). For example, the hazard detection software unit may acquire vehicle data via various sensors, such as a GPS sensor, acceleration / speed sensor, or a Controller Area Network (CAN) receiver, included in an electronic device or vehicle. The hazard detection software unit can directly use the vehicle data for hazard detection and transmit it to a computer vision recognition unit (e.g., computer vision recognition unit 134 in FIG. 1 ).

[0083] According to one embodiment, the hazard detection software unit may collect and store camera parameters. The camera parameters may include characteristics of the sensor and lens constituting the camera (or camera module) or the coordinates and installation angle at which the camera is mounted on the vehicle. The camera parameters may be pre-stored in the hazard detection software unit, for example, by user input via a UI. The hazard detection software unit may set the area of ​​the vehicle in the image using the camera parameters. For example, the camera parameters may include distortion rate data for the camera lens. The hazard detection software unit may use the distortion rate data to generate a distance table between the target object and the vehicle and / or between the target objects.

[0084] At operation 710, the hazard detection software unit re-determines the target object by applying weightings to the deep learning recognition data and the computer vision recognition data, where the weighting ratios may be predetermined or adjusted as needed.

[0085] Specifically, the danger detection software unit may redetermine the target object by applying weights to detailed data items included in the deep learning recognition data and the computer vision recognition data. The redetermined target object may be the same as or different from the previously determined target object. The redetermining of the target object may be selectively performed.

[0086] In operation 720, the hazard detection software unit detects a hazard to the vehicle based on the re-determined target object and vehicle data of the vehicle. The hazard detection software unit can detect a hazard to the vehicle by referring to the target object, the vehicle data, and all other data stored in or collected by the hazard detection software unit.

[0087] For example, the hazard detection software unit may detect a hazard to the vehicle based on the distance between the target object and the vehicle and the vehicle speed. For example, the hazard detection software unit may detect a hazard based on the real-time coordinates and type of the target object and the driving behavior of the vehicle. The coordinates of the target object, which change in real time, indicate the speed of the target object.

[0088] According to one embodiment, the electronic device may sound a danger alarm if it detects a danger to the vehicle. In addition to the danger alarm, the electronic device may also notify the driver of the danger using other audiovisual means, such as displaying the danger on a display or playing a recorded verbal voice.

[0089] FIG. 8 is a block diagram of an electronic device according to one embodiment.

[0090] Electronic device 800 represents electronic device 100 of FIG.

[0091] The electronic device 800 includes a communication unit 810 , a memory 820 , a processor 830 , and a display 840 .

[0092] The communication unit 810 is connected to the video recognition engine, danger detection software (e.g., danger detection software 140 in FIG. 1), and memory 820 to transmit and receive data. The communication unit 810 can connect to other external devices to transmit and receive data. Hereinafter, the expression "transmitting and receiving "A" means transmitting and receiving "information or data indicating A."

[0093] The communication unit 810 can be implemented as a circuitry within the electronic device 800. For example, the communication unit 810 may include an internal bus and an external bus. As another example, the communication unit 810 may be an element that connects the electronic device 800 to an external device. The communication unit 810 may be an interface. The communication unit 810 may receive data from an external device (e.g., the camera 110 in FIG. 1 ) and transmit the data to the memory 820, the processor 830, the video recognition engine, and the hazard detection software.

[0094] The memory 820 can store data received by the communication unit 810 and data processed by the video recognition engine and the danger detection software. The memory 820 may also store a program (or application, software). The program may be coded to generate recognition data or detect danger, and may be a set of syntax executable by the processor 830, the video recognition engine, or the danger detection software.

[0095] Memory 820 may include one or more of volatile memory, non-volatile memory, and random access memory (RAM), flash memory, a hard disk drive, and an optical disk drive.

[0096] The memory 820 stores a set of instructions (e.g., software) that operate the electronic device 800. The set of instructions that operate the electronic device 800 is executed by the processor 830, the video recognition engine, and the hazard detection software.

[0097] The processor 830 processes data received by the communication unit 810 and data stored in the memory 820. The processor 220 may be an ISP (Image Signal Processor).

[0098] A "processor" may be a data processing device implemented in hardware having circuits with physical structures for performing desired operations. For example, the desired operations may include code or instructions contained in a program. For example, a data processing device implemented in hardware may include a microprocessor, a central processing unit, a processor core, a multi-core processor, a multiprocessor, an application-specific integrated circuit (ASIC), or a field programmable gate array (FPGA).

[0099] Processor 830 can execute computer-readable code (eg, software) stored in a memory (eg, memory 820) and instructions triggered by processor 830.

[0100] The display 840 can output a monitoring screen to the user in real time to detect danger to the vehicle. By providing the blind spot image of the vehicle in real time on the display 840, the driver's field of vision is secured, thereby reducing the probability of an accident. Even if the driver does not stare at the display 840, the electronic device 800 can notify the driver of the danger by other audible means, such as sounding a danger alarm when a danger is detected.

[0101] FIG. 9 is a schematic diagram of a hazard detection system according to one embodiment.

[0102] Referring to both Figures 1 and 9, the danger detection system of Figure 9 shows a system configured so that the components included in the electronic device 100 of the danger detection system of Figure 1, i.e., the image processing unit 120, the image recognition unit 130, and the danger detection software unit 140, are included in the camera 910.

[0103] According to the embodiment shown in Figure 9, the electronic device 900 represents the electronic device 800 of Figure 8. That is, the electronic device 900 includes a communication unit, a memory, a processor, and a display (e.g., the communication unit 810, the memory 820, the processor 830, and the display 840 of Figure 8).

[0104] The operations / functions performed by the electronic devices described with reference to Figures 1 to 8 (electronic device 100 of Figure 1 and electronic device 800 of Figure 8) may be performed / processed by camera 910 of Figure 9. That is, camera 910 may generate a first processed image for a deep learning image recognition method and a second processed image for a computer vision image recognition method based on an original image captured using camera 910, generate deep learning recognition data for the first processed image using a deep learning image recognition model, generate computer vision recognition data for the second processed image using the computer vision image recognition model, and detect a danger to the vehicle based on the deep learning recognition data and the computer vision recognition data. Electronic device 900 may receive each of these series of operations and / or the danger detection result via a communication unit.

[0105] The above-described embodiments may be implemented using hardware components, software components, and / or a combination of hardware and software components. For example, the devices, methods, and components described in the embodiments may be implemented using a general-purpose computer or a special-purpose computer, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or other adaptive supersampling device capable of executing and responding to commands. The processing device may execute an operating system (OS) and software applications that run on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of software. For ease of understanding, a single processing device may be described as being used, but those skilled in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, the processing device may include multiple processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

[0106] Software may include computer programs, code, instructions, or any combination thereof, capable of configuring a processing device or instructing the processing device, either individually or collectively, as desired. The software and / or data may be permanently embodied in any type of machine, component, physical device, virtual device, computer storage medium, or device, or transmitted signal wave, to be interpreted by the processing device or to provide instructions or data to the processing device. The software may be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on a computer-readable recording medium.

[0107] The method according to the present invention may be embodied in the form of program instructions that can be executed by various computer means and recorded on a computer-readable recording medium. The recording medium may include program instructions, data files, data structures, and the like, alone or in combination. The recording medium and program instructions may be specially designed and constructed for the purposes of the present invention, or may be well-known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tape, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language code, such as that generated by a compiler, but also high-level language code that is executed by a computer using an interpreter, for example.

[0108] The hardware adaptive supersampling devices described above may be configured to operate as one or more software models to perform the operations shown in this invention, and vice versa.

[0109] Although the embodiments have been described above with reference to limited drawings, those skilled in the art may apply various technical modifications and variations based on the above description. For example, the described techniques may be performed in a different order than described, and / or the components of the described systems, structures, devices, circuits, etc. may be combined or combined in a different manner than described, and may be replaced or substituted with other components or equivalents, while still achieving suitable results.

[0110] Accordingly, other implementations, other embodiments, and equivalents of the claims are intended to fall within the scope of the following claims.

Claims

1. 1. A method for detecting a danger to a vehicle, the method being performed by an electronic device provided in the vehicle, the method comprising: acquiring an original image captured using a camera mounted on the exterior of the vehicle; generating a first processed image for a deep learning image recognition method and a second processed image for a computer vision image recognition method based on the original image; generating deep learning recognition data for the first processed video using a deep learning video recognition model; generating computer vision recognition data for the second processed image using a computer vision image recognition model; detecting a danger to the vehicle based on the deep learning recognition data and the computer vision recognition data; A method comprising:

2. 2. The method of claim 1, wherein the operation of generating the first processed image for a deep learning image recognition method and the second processed image for a computer vision image recognition method based on the original image includes the operation of generating the first processed image by processing RGB or YUV conversion based on the original image, and generating the second processed image by processing YUV conversion based on the original image.

3. The operation of generating the deep learning recognition data for the first processed video using the deep learning video recognition model includes: performing pre-processing including image quantization on the first processed image; generating the deep learning recognition data by determining a target object for the pre-processed first processed video using the deep learning video recognition model; The method of claim 1 , comprising:

4. 4. The method of claim 3, wherein the operation of generating the deep learning recognition data for the first processed video using the deep learning video recognition model further comprises performing post-processing on the deep learning recognition data using an anchor box algorithm and non-maximum suppression (NMS).

5. generating the computer vision recognition data for the second processed image using the computer vision image recognition model, performing pre-processing on the second processed image, including noise reduction and edge enhancement; generating the computer vision recognition data by determining a target object for the pre-processed second image using the computer vision image recognition model; The method of claim 1 , comprising:

6. 6. The method of claim 5, wherein generating the computer vision recognition data for the second processed video using the computer vision video recognition model further comprises performing post-processing on the computer vision recognition data using Intersection over Union (IoU) tracking.

7. 6. The method of claim 5, wherein the operation of generating the computer vision recognition data for the second processed image using the computer vision image recognition model includes an operation of generating the computer vision recognition data for the second processed image using the computer vision image recognition model, the computer vision recognition data including correction data based on camera parameters.

8. The camera parameters are: The characteristics of the sensor and lens that make up the camera, or The method of claim 7 , including coordinates and installation angles at which the cameras are mounted on the vehicle.

9. The method of claim 3 , wherein the deep learning recognition data and the computer vision recognition data include the type and coordinates of the target object.

10. The operation of detecting a danger to the vehicle based on the deep learning recognition data and the computer vision recognition data includes: redetermining a target object by applying weights to the deep learning recognition data and the computer vision recognition data; detecting a danger to the vehicle based on the redefined target object and vehicle data of the vehicle; The method of claim 1 , comprising:

11. The method of claim 10 , wherein the vehicle data includes at least one of a measured distance between the vehicle and a target object, the vehicle speed / acceleration, the vehicle position, or driving information.

12. The method of claim 1 , further comprising the operation of outputting a monitoring screen to a user in real time to detect a danger to the vehicle.

13. The method of claim 1 further comprising sounding a hazard alarm upon detecting a hazard to the vehicle.

14. 1. An electronic device for detecting a danger to a vehicle, comprising: a communication unit that acquires original video captured using a camera mounted on the exterior of the vehicle; an image processing unit that generates a first processed image for a deep learning image recognition method and a second processed image for a computer vision image recognition method based on the original image; an image recognition unit that generates deep learning recognition data for the first processed image using a deep learning image recognition model and generates computer vision recognition data for the second processed image using a computer vision image recognition model; a danger detection software unit that detects a danger to the vehicle based on the deep learning recognition data and the computer vision recognition data; 2. An electronic device comprising:

Images