Electronic device for controlling a driving vehicle and operating method of electronic device

Abstract

An electronic device configured to control a host vehicle includes an image sensor configured to photograph a surrounding environment of the host vehicle, and a processor configured to perform an image processing operation based on a first image captured by the image sensor and to control the host vehicle based on a processing result, wherein the processor determines whether to use a high-speed performance of the image processing operation based on a speed of the host vehicle, and the electronic device is configured such that, when the high-speed performance is not used, the processor performs the image processing operation by using a first image processing module, and when the high-speed performance is used, the processor performs the image processing operation by using a second image processing module having a data throughput less than that of the first image processing module.

Description

[0001] Cross-references to related applications

[0002] This application claims the benefit of Korean Patent Application No. 10-2020-0048305, filed on April 21, 2020, with the Korean Intellectual Property Office, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] The disclosed embodiments relate to electronic equipment for controlling a main vehicle and a method of operating the electronic equipment. More specifically, they relate to electronic equipment for determining whether high-speed performance of image processing operations is necessary based on information about the main vehicle, and, when high-speed performance is required, performing the image processing operations using an image processing module with a small data throughput. A method of operating the electronic equipment is also disclosed. Background Technology

[0004] Autonomous driving systems (or Advanced Driver Assistance Systems (ADAS)) perform safe driving by acquiring information about the primary vehicle and its surrounding environment from various types of sensors and controlling the primary vehicle based on that information. On the other hand, autonomous driving systems need to process large amounts of data to control the primary vehicle, and given that the primary vehicle can travel at high speeds, a fast data processing speed is typically required. Specifically, when the primary vehicle is moving at low speeds, the reaction speed of the autonomous driving system is relatively less important, but in dangerous situations such as when the primary vehicle is moving at high speeds or when a nearby vehicle suddenly overtakes, a fast response speed is essential. Summary of the Invention

[0005] The disclosed embodiments provide an electronic device and a method of operating the electronic device, the electronic device determining whether high-speed performance of an image processing operation is required, and performing the image processing operation by using an image processing module corresponding to the determination result.

[0006] According to one aspect of the present invention, an electronic device configured to control a main vehicle includes: an image sensor configured to capture images of the surrounding environment of the main vehicle; and a processor configured to perform image processing operations based on a first image captured by the image sensor and to control the main vehicle based on the processing result, wherein the processor determines whether to use high-speed performance of the image processing operations based on the speed of the main vehicle, and the electronic device is configured such that: when high-speed performance is not used, the processor performs image processing operations by using a first image processing module, and when high-speed performance is used, the processor performs image processing operations by using a second image processing module with a data throughput less than that of the first image processing module.

[0007] According to another aspect of the present inventive concept, an electronic device configured to control a host vehicle includes an image sensor configured to photograph a surrounding environment of the host vehicle, and a processor configured to perform an image processing operation based on a first image captured by the image sensor and to control the host vehicle based on a processing result, wherein the processor is configured to select an operation parameter of the image processing operation based on travel information about the host vehicle or information about the surrounding environment of the host vehicle, and to perform the image processing operation according to the selected operation parameter.

[0008] According to another aspect of the present inventive concept, an operation method of an electronic device configured to control a host vehicle includes obtaining a first image of a surrounding environment of the host vehicle, obtaining speed information about the host vehicle, identifying whether a speed of the host vehicle is greater than or equal to a threshold speed, processing the first image by using a first image processing method when the speed of the host vehicle is less than the threshold speed, processing the first image by using a second image processing method having a data throughput less than that of the first image processing method when the speed of the host vehicle is greater than or equal to the threshold speed, and controlling the host vehicle based on a processing result of the first image. BRIEF DESCRIPTION OF DRAWINGS

[0009] Embodiments of the present inventive concept will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0010] Figure 1 is a block diagram of an electronic device according to an example embodiment of the present inventive concept;

[0011] Figure 2 is a diagram of a host vehicle including an electronic device according to an example embodiment of the present inventive concept;

[0012] Figure 3 is a flowchart of an operation method of an electronic device according to an example embodiment of the present inventive concept;

[0013] Figure 4 is a flowchart of an image processing operation according to an example embodiment of the present inventive concept;

[0014] Figure 5 is a flowchart of an image processing operation according to an example embodiment of the present inventive concept;

[0015] Figures 6A-6C is a diagram illustrating a preset event according to an example embodiment of the present inventive concept;

[0016] Figure 7 is a block diagram of an image processing module according to an example embodiment of the present inventive concept;

[0017] Figure 8A and Figure 8B is a diagram of a region of interest (ROI) extraction operation according to an example embodiment of the inventive concept;

[0018] Figure 9A and Figure 9B is a diagram illustrating a generation operation of a plurality of pyramid images according to an example embodiment of the inventive concept;

[0019] Figure 10 is a diagram of an image processing operation of a second image processing module according to an embodiment of the inventive concept;

[0020] Figure 11 is a block diagram of an image processing module according to an example embodiment of the inventive concept;

[0021] Figure 12 is a diagram for explaining an image processing operation of a second image processing module according to an embodiment of the inventive concept;

[0022] Figure 13 is a timing diagram illustrating an image processing operation using a plurality of image processing modules according to an example embodiment of the inventive concept;

[0023] Figure 14 is a block diagram illustrating an electronic device according to an example embodiment of the inventive concept;

[0024] Figure 15 is a flowchart of an operation method of an electronic device according to an example embodiment of the inventive concept;

[0025] Figure 16 is a flowchart of a selection operation of an operation parameter according to an example embodiment of the inventive concept; and

[0026] Figure 17 is a block diagram of an autonomous driving device according to an example embodiment of the inventive concept. DETAILED DESCRIPTION

[0027] Figure 1 is a block diagram illustrating an electronic device 10 according to an example embodiment of the inventive concept.

[0028] Referring to Figure 1 , the electronic device 10 can include an image sensor 100, a memory 200, and a processor 300.

[0029] The electronic device 10 can include a device that performs an image processing operation on an image. In this case, the image processing operation can be referred to as an operation of performing object detection or object segmentation on at least one object in the image by analyzing the image. The type of the image processing operation is not limited to the above-described example and can include various operations.

[0030] The electronic device 10 can be implemented as or can include, for example, a personal computer (PC), an Internet of Things (IoT) device, or a portable electronic device. The portable electronic device can be provided in various devices, for example, a laptop computer, a mobile phone, a smart phone, a tablet PC, a personal digital assistant (PDA), an enterprise digital assistant (EDA), a digital camera, a digital video recorder, an audio device, a portable multimedia player (PMP), a personal navigation device (PND), an MP3 player, a handheld game console, an electronic book, and a wearable device. Various devices among the devices can be combined and communicate to implement the electronic device 10.

[0031] In one embodiment, the electronic device 10 can include hardware, software, and / or firmware and can be implemented as a device that controls a host carrier (for example, a carrier device such as an advanced driver assistance system (ADAS)). The electronic device 10 can perform an image processing operation based on a captured image of a surrounding environment of the host carrier and control the host carrier based on a processing result (for example, an object recognition or segmentation result). Hereinafter, for convenience of description, it is assumed that the electronic device 10 is a device that controls a host carrier.

[0032] The image sensor 100 can be embedded in the electronic device 10, receive an image signal related to a surrounding environment of the electronic device 10, and output the received image signal as an image. For example, the image sensor 100 can generate an image by converting light of an external environment in front or each direction into electric energy and output the generated image to the processor 300. In one embodiment, when the electronic device 10 is a device that controls a host carrier, the image sensor 100 can receive a video signal related to a surrounding environment of the host carrier and output the received video signal as an image.

[0033] The memory 200 can include a storage location for storing data and can store data generated by using the image sensor 100 and various data generated in a process of performing an operation of the processor 300. For example, the memory 200 can store an image obtained by using the image sensor 100. In addition, as described later in connection with an operation of the processor 300, the memory 200 can store a processing result according to an image processing operation of the processor 300.

[0034] The memory 200 according to an example embodiment of the present inventive concept can include information Info_V about a host vehicle. The information Info_V about the host vehicle can include driving information about the host vehicle or surrounding environment information about the host vehicle. The driving information and the surrounding environment information can be real-time active information about the vehicle or its surrounding environment. In this case, the driving information Info_V about the host vehicle can include at least one of a speed, an acceleration, a moving distance, and a position of the host vehicle. For example, the driving information about the host vehicle can be obtained by using a speedometer (not shown) for detecting a speed of the host vehicle or an accelerometer (not shown) for detecting an acceleration of the host vehicle. In another example, the driving information about the host vehicle can be obtained by calculating the driving information about the host vehicle based on a processing result of an image processing operation that has been performed (e.g., a moving distance between frames of a fixed object included in a previous image). In another example, the driving information about the host vehicle can be obtained based on global positioning system (GPS) information about the host vehicle received through communication with the outside. However, the type and the obtaining method of the driving information about the host vehicle of the present inventive concept are not limited to the above-described examples.

[0035] Here, the surrounding environment information about the host vehicle can include at least one of a speed, an acceleration, a moving distance, and a position of an object located around the host vehicle, and a normal speed or a speed limit of a road on which the host vehicle travels. For example, the surrounding environment information about the host vehicle can be calculated based on a processing result of an image processing operation that has been performed. In another example, the surrounding environment information about the host vehicle can be received through communication (e.g., electronic communication) with the outside. However, the type and the obtaining method of the surrounding environment of the host vehicle of the present inventive concept are not limited to the above-described examples.

[0036] In addition, the memory 200 can store a first image processing module 220 and a second image processing module 230. The first image processing module 220 and the second image processing module 230 can perform an image processing operation based on an image captured by the image sensor 100. In one embodiment, the second image processing module 230 can be designed to have a smaller data throughput than a data throughput of the first image processing module 220. The second image processing module 230 can process an image according to the same image processing operation as the first image processing module 220, but can be designed such that an amount of input data in each image processing operation is reduced. Accordingly, the second image processing module 230 can perform an image processing operation at a faster speed than the first image processing module 220. Accordingly, the first image processing module 220 can perform a first image processing method, and the second image processing module 230 can perform a second image processing method.

[0037] The first image processing module 220 and the second image processing module 230 can be implemented in firmware or software, and can be loaded into the memory 200 and executed by the processor 300. However, embodiments are not limited thereto, and the first image processing module 220 and the second image processing module 230 can be implemented in hardware or a combination of software and hardware. The first image processing module 220 and the second image processing module 230 can be implemented in computer program codes stored on a computer readable medium, and can be implemented as two separate instruction sets or a single combined instruction set. Details of the image processing operations of the first image processing module 220 and the second image processing module 230 will be given later with reference to FIGS. 4 and 5. Figure 8A and Figure 8B Detailed descriptions of the image processing operations of the first image processing module 220 and the second image processing module 230 are given.

[0038] The processor 300 can control all operations of the electronic device 10. The processor 300 can include various arithmetic processing devices, such as a central processing unit (CPU), a graphics processing unit (GPU), an application processor (AP), a digital signal processor (DSP), a field-programmable gate array (FPGA), a neural network processing unit (NPU), an electronic control unit (ECU), and an image signal processor (ISP).

[0039] The processor 300 can perform image processing operations based on images captured by the image sensor 100, and can control the host vehicle based on the processing results. In one embodiment, the processor 300 can receive images from the image sensor 100, and can perform object detection or object segmentation by analyzing the received images using the first image processing module 220 or the second image processing module 230 stored in the memory 200. In addition, the processor 300 can control the host vehicle based on the object detection or segmentation results.

[0040] The processor 300 according to an example embodiment of the inventive concept can determine whether high-speed performance of image processing operations is required based on the information Info_V about the host vehicle, and can select one of the first image processing module 220 and the second image processing module 230 according to the determination result, and perform image processing operations by using the selected image processing module. In this case, the case where high-speed performance is required can mean a case where the host vehicle is moving at high speed, a vehicle in the vicinity is very close to the host vehicle, or the speed of a vehicle in the vicinity is rapidly decreasing, and thus it is required or desired that the host vehicle has a fast response speed for safety.

[0041] When it is determined that high-speed performance is not required, the processor 300 can select the first image processing module 220. On the other hand, when it is determined that high-speed performance is required, the processor 300 can select the second image processing module 230 having a smaller data throughput than the first image processing module 220. When the processor 300 performs an image processing operation by using the second image processing module 230, the processor 300 can obtain a processing result faster than in the case where the image processing operation is performed by using the first image processing module 220.

[0042] The processor 300 according to an embodiment of the inventive concept can determine whether high-speed performance of an image processing operation is required or should be used according to various methods based on the information Info_V about the host vehicle. For example, the processor 300 can determine whether high-speed performance of an image processing operation is required or should be used based on a speed of the host vehicle in the information Info_V about the host vehicle. As another example, the processor 300 can determine whether a preset event has occurred based on the surrounding environment information and the information Info_V about the host vehicle, and can determine the need for performing a high-speed image processing operation according to whether the preset event has occurred. Later, reference will be made to Figures 4-7 A detailed description is provided of operations for determining the need for high-speed performance of an image processing operation of the processor 300.

[0043] In the above-described example, it is described that, when the processor 300 receives an image IMG from the image sensor 100, the processor 300 determines the need for high-speed performance of an image processing operation, selects the first image processing module 220 or the second image processing module 230 according to a determination result, and performs an image processing operation by using the selected image processing module. However, embodiments are not limited thereto. The processor 300 can periodically determine the need for high-speed performance of an image processing operation. For example, the processor 300 can be implemented to determine the need for high-speed performance whenever an image processing operation is completed for a preset number of images.

[0044] The electronic device 10 according to the technical idea of the inventive concept can determine whether high-speed performance of an image processing operation is required or desired, and when high-speed performance is required or desired, can quickly control a host vehicle by performing an image processing operation using an image processing module having a smaller amount of data throughput.

[0045] In Figure 1In the meantime, although the electronic device 10 is illustrated and described as including the image sensor 100 according to an embodiment, the electronic device 10 and the image sensor 100 can be separate components. The electronic device 10 can also be implemented by receiving an image from an external image sensor 100 and performing an image processing operation on the received image. In addition, according to certain embodiments, the electronic device 10 can not include the image sensor 100 and can receive an image via a communication device (not shown) and can be implemented to perform an image processing operation on the received image.

[0046] In addition, in Figure 1 In the meantime, although the electronic device 10 is illustrated and described as including one image sensor 100, the electronic device 10 can include a plurality of image sensors 100. The electronic device 10 can perform an image processing operation on a plurality of images received from the plurality of image sensors 100 and control the host carrier based on the processing result.

[0047] In addition, in Figure 1 In the meantime, although the electronic device 10 is illustrated and described as including the memory 200 according to an embodiment, the electronic device 10 and the memory 200 can be separate components, and the electronic device 10 can also be implemented to load the first image processing module 220 or the second image processing module 230 stored in an external memory 200.

[0048] In addition, in Figure 1 In the meantime, although the electronic device 10 is illustrated and described as including one memory 200, the electronic device 10 can include a plurality of memories 200. In addition, according to one embodiment, the electronic device 10 can be implemented to store images captured by the image sensor 100 in one memory and store the first image processing module 220 or the second image processing module 230 in the remaining memories.

[0049] Figure 2 is a diagram of a host carrier including the electronic device 10 according to an example embodiment of the inventive concept. Figure 2 is a diagram illustrating an example of a host carrier 400 of the electronic device 10 including Figure 1

[0050] Referring to Figure 1 and Figure 2 ​The main carrier 400 can include the electronic device 10 and a carrier controller 410. At least a portion of the electronic device 10 can be disposed at the top of the main carrier 400, and the image sensor 100 can photograph the front of the main carrier 400. However, the disposition position of the electronic device 10 is not limited to this embodiment, and according to an embodiment, the electronic device 10 can be disposed at one or more different positions in the main carrier 400. For example, different portions of the electronic device 10 can be at different positions within the carrier 400. In one embodiment, the electronic device 10 can be implemented to perform only an image processing operation on an image captured by the image sensor 100 and provide a result of the image processing operation to the carrier controller 410.

[0051] The carrier controller 410 can control the overall travel of the main carrier 400. The carrier controller 410 can determine the surrounding situation of the main carrier 400 and control the travel direction or travel speed of the main carrier 400 based on the determination result. In one embodiment, the carrier controller 410 can receive a result of the image processing operation of the electronic device 10 (e.g., an object recognition result and an analysis result), determine the surrounding situation of the main carrier 400 based on the received processing result, and control the travel direction, travel speed, etc. of the main carrier 400 by transmitting a control signal to a driving unit (not shown) of the main carrier 400 based on the determination result. The carrier controller 410 can be implemented in hardware, software, and firmware, and can include, for example, a computer including a processor that performs control of the main carrier 400.

[0052] In Figure 2 , the carrier controller 410 and the electronic device 10 are shown and described as separate components. However, according to an embodiment, the electronic device 10 can be implemented to include the carrier controller 410 or include the processor 300 of the electronic device 10 and the carrier controller 410 in a single configuration.

[0053] Figure 3 is a flowchart of an operation method of the electronic device 10 according to an example embodiment of the inventive concept. Figure 3 is Figure 1 a flowchart of an operation method of the electronic device 10. Figure 3 At least some of the operations in

[0054] Referring to Figure 1 and Figure 3The electronic device 10 can obtain a first image of the surrounding environment of the host vehicle (S110). For example, the electronic device 10 can obtain the first image of the surrounding environment of the host vehicle via the image sensor 100. As another example, when the electronic device 10 is implemented to not include the image sensor 100, the electronic device 10 can also be implemented to obtain the first image from the outside.

[0055] Then, the electronic device 10 can determine whether high-speed performance of the image processing operation is required (S120). The electronic device 10 can determine whether high-speed performance of the image processing operation is required based on driving information about the host vehicle or surrounding environment information about the host vehicle. A situation requiring high-speed performance can mean a situation in which the host vehicle is moving at high speed, a vehicle in the vicinity is very close to the host vehicle, or a speed of a vehicle in the vicinity is rapidly decreasing, and thus a fast response speed of the host vehicle is required for safety. For example, the electronic device 10 can refer to a stored parameter set corresponding to a situation requiring high-speed performance (e.g., a situation in which it would be particularly useful or desirable to control the vehicle to avoid damage or an accident). The driving information about the host vehicle or the surrounding environment information about the host vehicle can be substantially the same as the descriptions given above with respect to the driving information about the host vehicle or the surrounding environment information about the host vehicle, and thus redundant descriptions thereof are omitted. Figure 1

[0056] When high-speed performance of the image processing operation is not required (S120-NO), the electronic device 10 can perform the image processing operation by using the first image processing module 220 (S130). For example, when a speed of the host vehicle is less than a first threshold speed, the processor 300 can select the first image processing module 220 among the first and second image processing modules 220 and 230 and perform the image processing operation by using the selected first image processing module 220.

[0057] In this case, the first image processing module 220 can include an image processing module designed to have more data throughput than data throughput of the second image processing module 230. In this case, the image processing operation can be referred to as an operation of performing object detection or object segmentation on at least one object in an image by analyzing the image. The type of the image processing operation is not limited to the above-described example and can include various operations.

[0058] ​On the other hand, when high-speed performance of the image processing operation is required (S120-Yes), or the high-speed performance of the image processing operation would be particularly useful or desirable, the electronic device 10 can perform the image processing operation by using the second image processing module 230 having a data throughput less than that of the first image processing module 220 (S140). When the image processing operation is performed by using the second image processing module 230, the electronic device 10 can obtain a processing result faster than when the image processing operation is performed by using the first image processing module 220. Then, the electronic device 10 can control the host vehicle based on the processing result (S150). The electronic device 10 can determine a surrounding situation of the host vehicle based on an object recognition result or an object segmentation result as a result of performing the image processing operation, and can control a travel direction or a travel speed based on the determination result.

[0059] Figure 4 is a flowchart of an image processing operation according to an exemplary embodiment of the inventive concept. Figure 4 is a diagram illustrating necessity or desirability of high-speed performance of an image processing operation in Figure 3 (S120) and operations of the image processing operation (operation S130 and operation S140) in detail.

[0060] Referring to Figure 1 , Figure 3 and Figure 4 , after obtaining the first image of the surrounding environment of the host vehicle (from S110), the electronic device 10 can determine whether the speed of the host vehicle is greater than or equal to a first threshold speed (S121). In this case, the first threshold speed can be a speed that becomes a reference for determining that the host vehicle moves at high speed, and can be set by a manufacturer or a user.

[0061] In one embodiment, the electronic device 10 can obtain the travel speed of the host vehicle from a speedometer that detects the travel speed of the host vehicle. In another embodiment, the electronic device 10 can obtain the travel speed of the host vehicle according to a processing result of an image processing operation that has been performed. For example, the electronic device 10 can calculate the travel speed of the host vehicle based on a moving distance between frames of a fixed object included in a previous image according to an object recognition result of the previous image.

[0062] When the speed of the host vehicle is less than the first threshold speed (S121-NO), the electronic device 10 can perform the image processing operation by using the first image processing module 220 (S130). If the speed of the host vehicle is less than the first threshold speed, since the host vehicle moves at a low speed, the necessity of a fast response speed can be relatively low from the viewpoint of controlling the host vehicle by using the image processing operation and processing result of the electronic device 10. Accordingly, the electronic device 10 can perform the image processing operation by using the first image processing module 220.

[0063] On the other hand, when the speed of the host vehicle is greater than or equal to the first threshold speed (S121-YES), the electronic device 10 can perform the image processing operation by using the second image processing module 230 having a data processing throughput less than that of the first image processing module 220. When the speed of the host vehicle is greater than or equal to the first threshold speed, since the host vehicle is currently moving at a high speed and thus moves a large distance during a unit time period, the necessity of a fast response speed can be relatively high from the viewpoint of controlling the host vehicle by using the image processing operation and processing result of the electronic device 10. Accordingly, the electronic device 10 can perform the image processing operation by using the second image processing module 220 having a less data processing throughput.

[0064] Then, when the image processing operation is completed, the electronic device 10 can proceed to the next operation S150.

[0065] As such, the electronic device 10 according to the inventive concept can select an image processing module for an image processing operation according to whether the host vehicle moves at a low speed or a high speed. In other words, the electronic device 10 according to the inventive concept can adaptively apply an image processing method according to the driving speed of the host vehicle.

[0066] Figure 5 is a flowchart of an image processing operation according to an example embodiment of the inventive concept. Figure 5 is Figure 4 is a flowchart of a modifiable embodiment of Figure 3 is a flowchart of the necessity or desirability of a high-speed performance of the image processing operation (S120) in and the operation of the image processing operation (S130 and S140).

[0067] Referring to Figure 1 , Figure 3 and Figure 5, the electronic device 10 can identify whether the speed of the host vehicle is equal to or greater than a first threshold speed (S123). When the speed of the host vehicle is greater than or equal to the first threshold speed (S123-Yes), the electronic device 10 can perform an image processing operation by using the second image processing module 230 having less data processing throughput than the first image processing module 220.

[0068] However, when the speed of the host vehicle is less than the first threshold speed (S121-No), the electronic device 10 can determine whether a preset event has occurred (S125). The electronic device 10 can determine whether the preset event has occurred based on at least one of travel information about the host vehicle and surrounding environment information about the host vehicle. In this case, the preset event refers to various events in which a dangerous situation is detected, for example, a nearby vehicle is very close to the host vehicle, a speed of a nearby vehicle is rapidly reduced, and a normal speed or a speed limit of a road on which the host vehicle moves exceeds a threshold speed. The type of the preset event is not limited to the above-described examples, and can be set by a manufacturer or a user. Detailed examples of the determination operation of the preset event are described later with reference to FIGS. 6 to 8. Figures 6A-6C Detailed examples of the determination operation of the preset event are described later with reference to FIGS. 6 to 8.

[0069] When the preset event has not occurred (S125-No), the electronic device 10 can perform an image processing operation by using the first image processing module 220 (S130). When the preset event has not occurred, the current host vehicle can not be in a dangerous situation, and thus the demand for a fast reaction speed can be relatively low from the viewpoint of controlling the host vehicle by using the image processing operation and processing results of the electronic device 10. Accordingly, the electronic device 10 can perform the image processing operation by using the first image processing module 220.

[0070] However, when the preset event has occurred (S125-Yes), the electronic device 10 can perform an image processing operation by using the second image processing module 230 (S140). When the preset event has occurred, the current host vehicle can be in a dangerous situation, and thus the demand for a fast reaction speed can be relatively high from the viewpoint of controlling the host vehicle by using the image processing operation and processing results of the electronic device 10. For example, even when the host vehicle 400 moves at a low speed less than the first threshold speed and a vehicle in front of the host vehicle 400 suddenly applies a brake, the necessity of a fast reaction by increasing an image processing speed can be required. Accordingly, the electronic device 10 can perform the image processing operation by using the second image processing module 220 having less data processing throughput.

[0071] Then, when the image processing operation is completed, the electronic device 10 can proceed to the next operation S150.

[0072] The electronic device 10 according to the present embodiment can select an image processing module for an image processing operation according to whether a preset event occurs. For example, the electronic device 10 according to the present embodiment can adaptively apply an image processing method according to whether the host vehicle is in a dangerous situation.

[0073] In Figure 5 , operation S123 is shown and described as preceding operation S125, however the present disclosure is not limited thereto. For example, according to one embodiment, the present disclosure can be implemented such that operation S125 precedes operation S123, or can be implemented such that only operation S125 is performed.

[0074] Figures 6A-6C is a diagram illustrating a preset event according to an example embodiment of the present inventive concept. Figure 6A is a diagram illustrating a situation in which a speed of a nearby vehicle adjacent to a host vehicle rapidly decreases, Figure 6B is a diagram illustrating a situation in which a nearby vehicle is very close to a host vehicle, and Figure 6C is a diagram illustrating a situation in which a speed limit of a road on which a host vehicle moves exceeds a threshold speed.

[0075] The electronic device 10 can determine whether a preset event has occurred by using a processing result of an image processing operation that has been performed. Hereinafter, a method of determining the occurrence of various types of preset events is described with reference to Figures 6A-6C .

[0076] The electronic device 10 can calculate a speed change of a nearby vehicle based on information about the nearby vehicle that is commonly included in previous images. For example, with reference to Figure 6A , the electronic device 10 can calculate a speed change of a nearby vehicle OB1 based on a moving distance between frames of the nearby vehicle OB1 that is commonly included in previous images IMG in an object recognition result that has been performed. As another example, the electronic device 10 can calculate a speed change of a nearby vehicle OB1 based on a size change of the nearby vehicle OB1 between frames that are commonly included in previous images IMG. However, a method of calculating a speed change of a nearby vehicle OB1 is not limited to the above-described examples, and various methods can be applied.

[0077] In addition, when the electronic device 10 recognizes that a speed of a nearby vehicle OB1 rapidly decreases in Figure 6A , there is a risk of collision with a host vehicle, and thus it can be determined that a preset event has occurred.

[0078] Additionally, the electronic device 10 can determine whether the main vehicle is within a threshold distance from the nearby vehicle OB1, based on the position of the nearby vehicle OB1 included in the previous image IMG. In this case, the threshold distance may refer to a stopping distance to avoid collisions between vehicles, can be set proportionally to the current speed of the main vehicle, and can be set by the manufacturer or the user. For example, refer to... Figure 6B The electronic device 10 can determine whether the main vehicle is within a threshold distance from the nearby vehicle OB1 based on the location of the nearby vehicle OB1 included in the previous image IMG in the already performed object recognition results.

[0079] Additionally, when the electronic device 10 detects that the distance between the nearby vehicle OB1 and the main vehicle is within a threshold distance, there may be a risk of collision with the main vehicle, thus confirming that a preset event has occurred.

[0080] Additionally, the electronic device 10 can determine whether the speed limit of the road on which the main vehicle is moving is greater than or equal to a second threshold speed based on traffic signs in the previous image IMG. In this case, the second threshold speed can typically refer to the minimum speed limit of the road on which the vehicle is moving at high speed, and can be set by the manufacturer or the user. For example, refer to... Figure 6C The electronic device 10 can identify the speed limit (e.g., about 120 km / h) of the traffic sign TS1 included in the previous image IMG from the already performed object recognition results, and determine whether the speed limit (e.g., about 120 km / h) is greater than or equal to a second threshold speed (e.g., about 100 km / h).

[0081] Additionally, when identified Figure 6C When the speed limit of traffic sign TS1 is greater than or equal to the second threshold speed, electronic device 10 can determine that a preset event has occurred, either because the main vehicle is likely moving at high speed or because other vehicles may be traveling at high speed.

[0082] However, the operation of electronic device 10 in determining the occurrence of a preset event by recognizing traffic sign TS1 is not limited to the examples described above. For example, electronic device 10 may determine that a preset event has occurred when the normal speed of traffic sign TS1 is greater than or equal to a first threshold speed (i.e., a speed used as a reference for determining whether the main vehicle is moving at high speed). In another example, electronic device 10 may use Global Positioning System (GPS) information about the main vehicle to identify the road on which the main vehicle is moving based on map information, identify information about the normal speed or speed limit of the road based on the map information, and determine whether a preset event has occurred.

[0083] Figure 7is a block diagram of an image processing module 250 according to an example embodiment of the present inventive concept.

[0084] Figure 7 The image processing module 250 can correspond to the first image processing module 220 or the second image processing module 230 in Figure 1 Referring to Figure 7 , the image processing module 250 can include a region of interest (ROI) extractor 251, a pyramid image generator 253, a feature extractor 255, and an object detector 257.

[0085] The ROI extractor 251 can extract an ROI from an image. In this case, the ROI can be referred to as a region selected for object detection. In one embodiment, the ROI extractor 251 can receive an image from the image sensor 100 in Figure 1 and extract an ROI from the received image.

[0086] The ROI extractor 251 can extract an ROI in various ways. In one embodiment, the ROI extractor 251 can extract an ROI based on vanishing points of an image. For example, the ROI extractor 251 can extract a region of a certain size including the vanishing points of the image as an ROI. However, the method of extracting an ROI performed by the ROI extractor 251 is not limited to the above-described example, and can extract an ROI in various ways.

[0087] The pyramid image generator 253 can generate a plurality of pyramid images having different scales from each other. In one embodiment, the pyramid image generator 253 can generate a preset number of first to nth pyramid images P_IMG_1 to P_IMG_n having different scales from each other by using an ROI.

[0088] The feature extractor 255 can generate feature information Info_F by extracting features of an image using a plurality of images. In one embodiment, the feature extractor 255 can extract features of an image by moving a search window of a fixed size on the first to nth pyramid images P_IMG_1 to P_IMG_n, and generate the feature information Info_F.

[0089] The object detector 257 can perform an object recognition operation or a segmentation operation based on the feature information Info_F, and can generate object recognition information or object segmentation information. The generated object recognition information or the generated object segmentation information can be used to control a host carrier.

[0090] The ROI extractor 251, the pyramid image generator 253, the feature extractor 255, and the object detector 257 can be respectively implemented in firmware or software, loaded into the memory 200, and then executed by the processor 300. However, the present disclosure is not limited thereto, and the ROI extractor 251, the pyramid image generator 253, the feature extractor 255, and the object detector 257 can be respectively implemented in hardware or a combined type of software and hardware.

[0091] Hereinafter, reference will be made to Figures 8A-10 The operation method of the image processing module 250 will be described in detail.

[0092] Figure 8A and Figure 8B is a diagram of an ROI extraction operation according to an example embodiment of the present inventive concept. Specifically, Figure 8A is a diagram of an ROI extraction operation of the first image processing module 220 in Figure 1 , and Figure 8B is a diagram of an ROI extraction operation of the second image processing module 230 in Figure 1 .

[0093] In one embodiment, the first image processing module 220 can extract an ROI by using a vanishing point of an image. For example, referring to Figure 8A , the first image processing module 220 can identify a vanishing point VP of the image IMG, identify parallel lines including the vanishing point VP, and extract a region under the identified parallel lines of the image IMG as a first ROI ROI_1.

[0094] The second image processing module 230 can extract an ROI having a size smaller than that of the ROI extracted by the first image processing module 230. For example, referring to Figure 8B , the second image processing module 230 can identify a road region of the image IMG based on a vanishing point VP of the image IMG, and extract the identified road region as a second ROI ROI_2. Since the vanishing point VP is obtained by extending edges of a road in the image IMG and identifying an overlapping point, the second ROI ROI_2 in Figure 8B may have less features than the first ROI ROI_1 in Figure 8A .

[0095] However, the method in which the first image processing module 220 and the second image processing module 230 extract the first ROI ROI_1 and the second ROI ROI_2 is not limited to the above-described example, and can be implemented to extract an ROI according to various methods. As long as the ROI of the second image processing module 230 has less features than the ROI of the first image processing module 220, various methods can be sufficient.

[0096] Figure 9A and Figure 9B is a diagram showing a generation operation of a plurality of pyramid images according to an example embodiment of the inventive concept. Figure 9A is Figure 1 is a diagram of a generation operation of a plurality of pyramid images of the first image processing module 220 in Figure 9B is Figure 1 is a diagram of a generation operation of a plurality of pyramid images of the second image processing module 230 in

[0097] Referring to Figure 1 , Figure 7 , Figure 9A and Figure 9B , the first image processing module 220 and the second image processing module 230 can receive an image IMG from the image sensor 100 and generate a plurality of pyramid images based on the received image IMG. For example, the first image processing module 220 and the second image processing module 230 can generate pyramid images having a scale identical to or different from a scale of the image IMG received from the image sensor 100.

[0098] However, the present disclosure is not limited thereto, and the first image processing module 220 and the second image processing module 230 can receive an ROI from the ROI extractor 251 and generate a plurality of pyramid images based on the received ROI. For example, the first image processing module 220 and the second image processing module 230 can generate pyramid images having a scale identical to or different from a scale of the received ROI.

[0099] Hereinafter, for convenience of description, a method of generating a plurality of pyramid images based on an image IMG received from the image sensor 100 is described. In one embodiment, the first image processing module 220 can generate M (M is a positive integer) pyramid images. For example, referring to Figure 9A , the first image processing module 220 can generate a first pyramid image P_IMG_1, a second pyramid image P_IMG_2, a third pyramid image P_IMG_3, and a fourth pyramid image P_IMG_4 having different scales from each other.

[0100] In addition, the second image processing module 230 can generate N (N is a positive integer smaller than M) pyramid images. For example, referring to Figure 9B , the second image processing module 230 can generate a first pyramid image P_IMG_1 and a fourth pyramid image P_IMG_4 having different scales from each other. The first pyramid image P_IMG_1 and the fourth pyramid image P_IMG_4 can have a scale identical to or different from a scale of the first pyramid image P_IMG_1 and the fourth pyramid image P_IMG_4 generated by the first image processing module 220. Figure 9AThe first image processing module 220 in the first image processing module 220 and the second image processing module 230 can generate the first pyramid image P_IMG_1 and the fourth pyramid image P_IMG_4, respectively, in the same scale. In this way, the second image processing module 230 can generate at least some of the plurality of pyramid images generated by the first image processing module 220.

[0101] On the other hand, in a modifiable embodiment, the second image processing module 230 can generate a smaller number (e.g., 2) of pyramid images than the number (e.g., 4) of pyramid images generated by the first image processing module 220, while generating pyramid images having different scales from the scales of the pyramid images generated by the first image processing module 220. For example, when the first image processing module 220 generates four pyramid images having scales of about 100%, about 50%, about 25%, and about 12.5%, respectively, the second image processing module 230 can generate two pyramid images having scales of about 80% and about 40%, respectively. However, the scales of the pyramid images generated by the first image processing module 220 and the second image processing module 230 are not limited to the above-described example, and pyramid images having various scales can be generated.

[0102] In Figure 9A and Figure 9B , the first image processing module 220 and the second image processing module 230 are illustrated and described as generating four pyramid images and two pyramid images, respectively, but the present disclosure is not limited thereto. As long as the number of pyramid images generated by the second image processing module 230 is smaller than the number of pyramid images generated by the first image processing module 220, the embodiment can be sufficient.

[0103] Figure 10 is a diagram of an image processing operation of the second image processing module 250 according to an embodiment of the inventive concept. Figure 10 is a diagram of an image processing operation of the image processing module 250 of Figure 7 . Figure 10 The image processing module 250 of Figure 1 corresponds to the second image processing module 230 in Figure 10 Hereinafter, the image processing module 250 of Figure 1Numerals such as "first," "second," "third," etc. can simply be used as labels for certain elements, steps, etc. to distinguish these elements, steps, etc. from one another. Terms described in the specification without "first," "second," etc. can still be referred to as "first" or "second" in the claims. In addition, terms referred to with a particular numeral (e.g., "first" in a particular claim) can be described elsewhere with a different numeral (e.g., "second" in the specification or another claim).

[0104] Referring to Figure 1 , Figure 7 , Figure 8B , Figure 9B and Figure 10 , the ROI extractor 251 of the second image processing module 250 can receive the image IMG from the image sensor 100 and extract a second ROI ROI_2 from the received image IMG. For example, the ROI extractor 251 can identify a road region of the image IMG based on the vanishing point VP of the image IMG and extract the identified road region as the second ROI ROI_2. Then, the ROI extractor 251 can transmit the extracted second ROI ROI_2 to the pyramid image generator 253.

[0105] The pyramid image generator 253 can generate a plurality of pyramid images having different scales from each other based on the received second ROI ROI_2. For example, the pyramid image generator 253 can generate a first pyramid image P_IMG_1 and a fourth pyramid image P_IMG_4 having the same or different scales from a scale of the received second ROI ROI_2. In addition, the pyramid image generator 253 can transmit the generated first pyramid image P_IMG_1 and fourth pyramid image P_IMG_4 to the feature extractor 255.

[0106] The feature extractor 255 can obtain feature information Info_F based on the received first pyramid image P_IMG_1 and fourth pyramid image P_IMG_4. In addition, the feature extractor 255 can transmit the obtained feature information Info_F to the object detector 257. The object detector 257 can perform an object recognition operation or a segmentation operation based on the received feature information Info_F. A result of the object recognition operation or the segmentation operation of the object detector 257 can be used to control the host vehicle.

[0107] The second image processing module 250 according to an embodiment of the inventive concept can extract a ROI having a smaller size than a ROI of the first image processing module 220 and generate a pyramid image having a smaller scale than a pyramid image of the first image processing module 220. Figure 1 Figure 1 ​The number of pyramid images (220 in the original text) is less than the number of pyramid images, thus reducing the overall data throughput of image processing operations. It should be noted that, compared to the first image processing module (e.g., ...), Figure 1 Compared to 220 in the first image processing module, the second image processing module (e.g., Figure 1 230 or Figure 10 250) does not need to include all the simplified image processing steps mentioned above, but may include only some of these image processing steps, so that the second image processing module produces less data throughput than the first image processing module.

[0108] Figure 11 This is a block diagram of an image processing module 260 according to an exemplary embodiment of the present invention. Figure 11 It shows Figure 7 A diagram illustrating a modifiable embodiment.

[0109] Figure 11 The image processing module 260 can correspond to Figure 1 The first image processing module 220 or the second image processing module 230 in the image. (See reference) Figure 11 The image processing module 260 may include an ROI extractor 263, a pyramid image generator 261, a feature extractor 265, and an object detector 267.

[0110] Figure 11 The image processing module 260 can be used with Figure 7 The image processing module 250 performs image processing operations in different ways. These will be described in detail below. Figure 11 Image processing operation of the image processing module 260.

[0111] The pyramid image generator 261 can generate multiple pyramid images with different scales. In one embodiment, the pyramid image generator 261 can receive images from the image sensor 100 and generate a predetermined number of first pyramid images P_IMG_1 to nth pyramid images P_IMG_n with different scales using the received images. The pyramid image generator 261 can send the generated first pyramid images P_IMG_1 to nth pyramid images P_IMG_n to the ROI extractor 263 and the feature extractor 265.

[0112] The ROI extractor 263 can extract ROIs using one of the first pyramid images P_IMG_1 to the nth pyramid image P_IMG_n received from the pyramid image generator 261. The method for extracting ROIs performed by the ROI extractor 263 can be compared with the reference... Figure 7 The methods of description are basically the same, so redundant descriptions are omitted.

[0113] The feature extractor 265 can generate feature information Info_F by extracting features of an image based on the first to nth pyramid images P_IMG_1 to P_IMG_n. The feature extractor 265 can mask a remaining area except for a target area corresponding to an ROI of each of the first to nth pyramid images P_IMG_1 to P_IMG_n, extract features of an image from the target area of each of the first to nth pyramid images P_IMG_1 to P_IMG_n, and generate the feature information Info_F.

[0114] The object detector 267 can perform an object recognition operation or a segmentation operation based on the feature information Info_F, and can generate object recognition information or object segmentation information. The generated object recognition information or the generated object segmentation information can be used to control the host carrier.

[0115] Figure 12 is a diagram of an image processing operation of the second image processing module 260 according to an embodiment of the inventive concept. Figure 12 is Figure 11 a diagram of an image processing operation of the image processing module 260. Figure 12 The image processing module 260 of Figure 1 is the second image processing module 230 in Hereinafter, Figure 12 The image processing module 260 of Figure 1 is referred to as the second image processing module 260 to distinguish the image processing module 260 from the first image processing module 220 in

[0116] Figure 1 Referring to Figure 11 and Figure 12 , the pyramid image generator 261 of the second image processing module 260 can generate a plurality of pyramid images having different scales from each other based on an image IMG received from the image sensor 100. For example, the pyramid image generator 261 can generate a first pyramid image P_IMG_1 and a fourth pyramid image P_IMG_4 having the same or different scales from a scale of the received image IMG. In addition, the pyramid image generator 261 can transmit the generated first pyramid image P_IMG_1 and fourth pyramid image P_IMG_4 to the ROI extractor 263 and the feature extractor 265.

[0117] The ROI extractor 263 can receive a first pyramid image P_IMG_1 and a fourth pyramid image P_IMG_4 received from the pyramid image generator 261, and extract a second ROI ROI_2 using one of the first pyramid image P_IMG_1 and the fourth pyramid image P_IMG_4. For example, the ROI extractor 263 can identify a road region based on the vanishing point VP of the first pyramid image P_IMG_1, and extract the identified road region as the second ROI ROI_2. Then, the ROI extractor 263 can send the extracted second ROI ROI_2 to the feature extractor 265.

[0118] Feature extractor 265 can generate feature information Info_F by extracting features of image IMG based on the first pyramid image P_IMG_1, the fourth pyramid image P_IMG_4, and the second ROI ROI_2. Feature extractor 265 can mask the regions other than the target regions (i.e., road regions) corresponding to the second ROI ROI_2 of each of the first and fourth pyramid images P_IMG_1 and P_IMG_4, extract features of image IMG from the target regions (i.e., road regions) of each of the first and fourth pyramid images P_IMG_1 and P_IMG_4, and generate feature information Info_F. Furthermore, feature extractor 265 can send the generated feature information Info_F to object detector 267. Object detector 267 can perform object recognition or segmentation operations based on the received feature information Info_F. The results of the object recognition or segmentation operations of object detector 267 can be used to control the main vehicle.

[0119] Figure 13 A timing diagram illustrating image processing operations using multiple image processing modules according to an example embodiment of the present invention is shown. Figure 13 It shows Figure 1 The timing diagram shows the image processing operations performed by the processor 300 using the first image processing module 220 and the second image processing module 230. (See reference...) Figure 13 The first image processing (image processing 1) can be represented as a timing diagram in which image processing operations are performed using only the first image processing module 220, the second image processing (image processing 2) can be represented as a timing diagram in which image processing operations are performed using only the second image processing module 230, and the third image processing (image processing 3) can be represented as a timing diagram in which image processing operations are performed by both the first image processing module 220 and the second image processing module 230.

[0120] refer to Figure 1 and Figure 13, the processor 300 can perform the image processing operation by alternately using the first image processing module 220 and the second image processing module 230. When the first image IMG1 is received, the processor 300 can determine a need for high-speed performance of the image processing operation by using the information Info_V about the host vehicle. When it is determined or indicated that the high-speed performance is needed or desired, the processor 300 can perform the image processing operation on a preset number of continuous images including the first image IMG1 by using the second image processing module 230. For example, referring to Figure 13 , the processor 300 can perform the image processing operation on two continuous images (i.e., the first image IMG1 and the second image IMG2) including the first image IMG1 by using the second image processing module 230, where the two continuous images are merely an example.

[0121] When the image processing operation using the second image processing module 230 is completed, the processor 300 can perform the image processing operation on the next image in order by using the first image processing module 220. For example, referring to Figure 13 When the image processing operation on the first image IMG1 and the second image IMG2 is completed by using the second image processing module 230, the processor 300 can perform the image processing operation on the next third image IMG3 in order by using the first image processing module 220. The processor 300 can perform the image processing operation by using the first image processing module 220 to obtain additional information (e.g., information about a remaining area other than a road area in the image IMG) that is not obtained due to the characteristics of the second image processing module 230.

[0122] When the image processing operation using the first image processing module 220 is completed, the processor 300 can determine the need for high-speed performance of the image processing operation again for the next image IMG in order. Alternatively, as described above, the processor 300 can alternately use the second image processing module 230 and the first image processing module 220 to repeat the image processing operation on the next image IMG in order.

[0123] As described above, when the first image processing module 220 and the second image processing module 230 are alternately used, the processor 300 can obtain a fast response speed by using the second image processing module 230, and can obtain additional information by using the first image processing module 220.

[0124] In Figure 13In the above, it is described that when the processor 300 determines that high-speed performance of the image processing operation is required, the image processing operation is performed by alternately using the first image processing module 220 and the second image processing module 230, but the present disclosure is not limited thereto. For example, when it is determined that high-speed performance is not required, the processor 300 can perform the image processing operation by alternately using the first image processing module 220 and the second image processing module 230, and when it is determined that high-speed performance is required, the processor 300 can be implemented to perform the image processing operation by using only the second image processing module 230.

[0125] Figure 14 is a block diagram illustrating an electronic device 10a according to an example embodiment of the present inventive concept. Figure 14 is a block diagram illustrating an electronic device 10a according to an example embodiment of the present inventive concept. Figure 1 is a block diagram illustrating an electronic device 10a according to an example embodiment of the present inventive concept. Figure 14 Referring to Figure 14 , the electronic device 10a can include an image sensor 100, a memory 200a, and a processor 300a. Because the image sensor 100 in Figure 1 is substantially the same as the image sensor 100 in , a redundant description thereof is omitted.

[0126] The memory 200a can include one image processing module 220a. The image processing module 220a can be a module designed to flexibly change an operation parameter for determining an image processing method. In this case, the operation parameter can include a size of an ROI, a number of pyramid images, etc. The image processing module 220a can perform an image processing operation based on an operation parameter determined under the control of the processor 300a.

[0127] The memory 200a can include a mapping table 240a. The mapping table 240a can include mapping information about a target latency of an image processing operation and mapping information about a mapping relationship between operation parameters. The processor 300a can determine a target latency of an image processing operation based on information Info_V 210 about a host vehicle, select an operation parameter corresponding to the determined target latency by using the mapping table 240a, and perform an image processing operation by using the selected operation parameter and the image processing module 200a.

[0128] However, according to a modifiable embodiment, the memory 200a can include a plurality of image processing modules 220a instead of one image processing module 220a. In this case, unlike one image processing module 220a, the plurality of image processing modules 220a can be designed such that the operation parameters are fixed. The processor 300a can determine a target latency of an image processing operation based on the information 210 about the host vehicle, identify an image processing module 220a of the plurality of image processing modules 220a corresponding to the determined target latency, and perform the image processing operation by using the identified image processing module 220a.

[0129] Figure 15 is a flowchart of an operation method of the electronic device 10a according to an example embodiment of the inventive concept. Figure 15 is Figure 14 a flowchart of an operation method of the electronic device 10a. Figure 15 At least some of the operations in Figure 14 may be performed by

[0130] First, the electronic device 10a can obtain a first image IMG1 of a surrounding environment of a host vehicle (S210). Then, the electronic device 10a can obtain information Info_V about the host vehicle (S220). The electronic device 10a can obtain driving information about the host vehicle or surrounding environment information about the host vehicle. The method of obtaining the driving information about the host vehicle or the surrounding environment information about the host vehicle performed by the electronic device 10a can be substantially the same as the method described above with reference to Figure 1 and a redundant description thereof is omitted.

[0131] Next, the electronic device 10a can select an operation parameter of an image processing operation (S230). The electronic device 10a can determine a current situation of the host vehicle based on the information 210 about the host vehicle and determine an operation parameter corresponding to the determined current situation. For example, the electronic device 10a can determine whether high-speed performance of a current image processing operation is required (or desired) based on the driving information of the host vehicle or the surrounding environment information of the host vehicle and, when the high-speed performance is required, the electronic device 10a can determine an operation parameter corresponding to the high-speed performance. The operation parameter corresponding to a situation in which the high-speed performance is required can be designed to perform the image processing operation using less data throughput than an operation parameter corresponding to a situation in which the high-speed performance is not required. For example, when the high-speed performance is required, the electronic device 10a can perform the image processing operation faster based on less data throughput by reducing the size of an ROI and reducing the number of pyramid images.

[0132] Next, the electronic device 10a can perform an image processing operation according to the selected operation parameter (S240). The electronic device 10a can control the image processing module 220a to perform the image processing operation according to the selected operation parameter, and the image processing module 220a can perform the image processing operation according to the control of the electronic device 10a. Next, the electronic device 10a can control the main vehicle based on the processing result (S250).

[0133] Figure 16 is a flowchart of a selection operation of an operation parameter according to an example embodiment of the inventive concept. Figure 16 is a flowchart illustrating a detailed selection operation S230 of an operation parameter according to an example embodiment of the inventive concept.

[0134] Reference Figures 14-16 After obtaining the information Info_V about the main vehicle (from S220), the electronic device 10a can determine a target delay (S231). In this case, the target delay can refer to a delay corresponding to a reaction speed required to safely control the main vehicle in the current situation of the main vehicle. For example, when the main vehicle is in a dangerous situation requiring a fast response speed (i.e., the main vehicle is moving at a high speed), the target delay can have a relatively small value. However, when the main vehicle is in a dangerous situation not requiring a fast response speed (i.e., the main vehicle is moving at a low speed), the target delay can have a relatively large value.

[0135] The electronic device 10a can determine the target delay in various ways. In one embodiment, the electronic device 10a can identify a travel speed of the main vehicle and determine a target delay corresponding to a speed range to which the identified travel speed belongs among a plurality of speed ranges. For example, the electronic device 10a can identify that a speed range to which a travel speed of the main vehicle or about 100 km / h belongs corresponds to a high-speed range of about 80 km / h or more and a low-speed range less than about 80 km / h, and determine a delay corresponding to the high-speed range as the target delay. However, the technical idea of the present disclosure is not limited to the embodiment for determining the target delay among delays corresponding to the two ranges of the high-speed region or the low-speed region, and according to one embodiment, one of three or more delays corresponding to one of three or more speed regions can be determined as the target delay.

[0136] In another embodiment, when a preset event (e.g., a nearby vehicle is very close to the host vehicle or a speed of the nearby vehicle is suddenly reduced) is identified based on the information about the host vehicle Info_V, the electronic device 10a can determine a delay corresponding to the identified preset event as the target delay. In this case, the memory 200a of the electronic device 10a can store information about a plurality of preset events, and the electronic device 10a can identify an event corresponding to the current state of the host vehicle based on the information about the plurality of preset events and the information about the host vehicle Info_V. In addition, the electronic device 10a can determine a delay corresponding to the identified preset event among a plurality of delays as the target delay.

[0137] Next, the electronic device 10a can select a size of the ROI corresponding to the determined target delay (S233). The electronic device 10a can select a size of the ROI corresponding to the target delay that has been determined based on the mapping table 240a. The mapping information included in the mapping table 240a can be designed such that the shorter the target delay, the smaller the size of the ROI, and the greater the target delay, the greater the size of the ROI.

[0138] For example, when the determined target delay is large, the electronic device 10a can select a size of the ROI large enough to include a road and a peripheral area of the road in the image. On the other hand, when the determined target delay is small, the electronic device 10a can select a size of the ROI to include only the road or at least a portion of the road in the image. As another example, the electronic device 10a can identify a size corresponding to the target delay based on information about a preset size of the ROI for each delay and select the identified size as the size of the ROI. In this case, the information about the preset size of the ROI for each delay can be stored when the electronic device 10a is manufactured or can be received from the outside. However, the method of setting the size of the ROI is not limited to the above-described example, and the size of the ROI can be set in various ways.

[0139] Next, the electronic device 10a can select a number of pyramid images corresponding to the determined target delay (S235). The electronic device 10a can select a number of pyramid images corresponding to the target delay that has been determined based on the mapping table 240a. The mapping information included in the mapping table 240a can be designed such that the shorter the target delay, the fewer the number of pyramid images, and the greater the target delay, the greater the number of pyramid images.

[0140] For example, the electronic device 10a can identify the number corresponding to the target delay based on information about the preset number of pyramid images for each delay, and select the identified number as the number of pyramid images to be generated later. In this case, the information about the preset number of pyramid images for each delay can be stored when the electronic device 10a is manufactured or received from the outside. However, the method of setting the number of pyramid images is not limited to the above-described example, and the number of pyramid images can be set in various ways.

[0141] Next, the electronic device 10a can perform the next operation after selecting the number of pyramid images (to S240). The electronic device 10a can extract an ROI having the selected size, generate the selected number of pyramid images, and perform an object recognition operation on the generated pyramid images. In the above-described embodiment, the size of the ROI corresponding to the determined target delay is selected first, and the number of pyramid images is selected later, but the embodiment is not limited thereto. According to various embodiments, the electronic device 10a can select the number of pyramid images first and select the size of the ROI later, or can select both the number of pyramid images and the size of the ROI simultaneously in parallel.

[0142] Figure 17 is a block diagram of an autonomous driving device 500 according to an example embodiment of the inventive concept.

[0143] Referring to Figure 17 , the autonomous driving device 500 can include a sensor 510, a memory 520, a processor 530, a RAM 540, a main processor 550, a driver 560, and a communication interface 570. The components of the autonomous driving device 500 can be communicatively connected to each other via a bus. The image sensor 511 included in the sensor 510 can correspond to the image sensor 100 of the above-described embodiment, the memory 520 can correspond to the memory 200 of the above-described embodiment, and the processor 530 can correspond to the processor 300 of the above-described embodiment. In addition, the main processor 550 can correspond to the vehicle controller 410 in Figure 2 . In some embodiments, the image sensor 511, the memory 520, and the processor 530 can be implemented by using the above-described embodiments with reference to Figures 1-16 .

[0144] The autonomous driving device 500 can perform situation determination and vehicle operation control by using real-time analysis of data on the surrounding environment of an autonomous host vehicle based on a neural network.

[0145] The sensor 510 can include a plurality of sensors that generate information about the surrounding environment of the autonomous driving device 500. For example, the sensor 510 can include a plurality of sensors that receive an image signal about the surrounding environment of the autonomous driving device 500 and output the received image signal as an image. The sensor 510 can include an image sensor 511 (e.g., a charge-coupled device (CCD), a complementary metal-oxide semiconductor (CMOS)), a depth camera 513, a light detection and ranging (LiDAR) sensor 515, and a radio detection and ranging (RaDAR) sensor 517, etc. In one embodiment, the image sensor 511 can generate a front image of the autonomous driving device 500 and provide the front image to the processor 530.

[0146] In addition, the sensor 510 can include a plurality of sensors that generate information about the driving of the autonomous driving device 500. For example, the sensor 510 can measure a speed meter 518 that measures the driving speed of the autonomous driving device 500 and outputs the measured driving speed, an accelerometer 519 that measures the driving acceleration of the autonomous driving device 500 and outputs the measured driving acceleration, etc. However, the present disclosure is not limited thereto, and the sensor 510 can include an ultrasonic sensor (not shown), an infrared sensor (not shown), etc.

[0147] The memory 520 (as a storage location for storing data) can store, for example, various data generated in the operation execution of the main processor 550 and the processor 530.

[0148] When the processor 530 receives an image from the image sensor 511, the processor 530 can determine whether high-speed performance of an image processing operation is required based on the information Info_V about the host vehicle, and when the high-speed performance is required, the processor 530 can control the host vehicle at a fast reaction speed in a dangerous situation by performing an image processing operation using an image processing module having less data throughput. The method of the processor 530 for extracting the cutout point VP can be substantially the same as the method described above with reference to FIGS. 2 to 4, and thus a repeated description is omitted. Figures 1-16

[0149] The main processor 550 can control the overall operation of the autonomous driving device 500. For example, the main processor 550 can control the functions of the processor 530 by executing a program stored in the RAM 540. The RAM 540 can temporarily store a program, data, an application, or an instruction.

[0150] ​In addition, the main processor 550 can control the operation of the autonomous driving apparatus 500 based on the calculation result of the processor 530. In one embodiment, the main processor 550 can receive information about the vanishing point VP from the processor 530, and control the operation of the driver 560 based on the received information about the vanishing point VP.

[0151] The driver 560 configured to drive the autonomous driving apparatus 500 can include an engine and a motor 561, a steering unit 563, and a brake unit 565. In one embodiment, the driver 560 can adjust the thrust, braking, speed, direction, etc. of the autonomous driving apparatus 500 by using the engine and the motor 561, the steering unit 563, and the brake unit 565 under the control of the processor 530.

[0152] The communication interface 570 can communicate with an external device by using a wired or wireless communication method. For example, the communication interface 570 can communicate by using a wired communication method such as Ethernet, or by using a wireless communication method such as Wi-Fi and Bluetooth.

[0153] While the present inventive concept has been particularly shown and described with reference to embodiments thereof, it will be understood that various changes in form and details can be made therein without departing from the spirit and scope of the appended claims.

Claims

1. An electronic device configured to control a host vehicle, the electronic device comprising: an image sensor configured to photograph a surrounding environment of the host vehicle; and a processor configured to perform an image processing operation based on a first image captured by the image sensor and to control the host vehicle based on a processing result, wherein the processor determines whether to use a high-speed performance of the image processing operation based on a speed of the host vehicle, and the electronic device is configured such that: when the high-speed performance is not used, the processor performs the image processing operation based on a first region of interest (ROI) by using a first image processing module, and when the high-speed performance is used, the processor performs the image processing operation based on a second region of interest (ROI) by using a second image processing module having a data throughput less than that of the first image processing module, wherein the first image processing module is configured to extract the first ROI from the first image, wherein the second image processing module is configured to extract the second ROI from the first image, and wherein a size of the second ROI is less than a size of the first ROI. 2.The electronic device of claim 1, wherein, the processor determines to use the high-speed performance when the speed of the host vehicle is greater than or equal to a first threshold speed, and the processor determines not to use the high-speed performance when the speed of the host vehicle is less than the first threshold speed. 3.The electronic device of claim 1, wherein, the first image processing module and the second image processing module are respectively configured to generate a plurality of pyramid images having different scales from each other based on the extracted ROI and to perform an object detection operation on the generated plurality of pyramid images. 4.The electronic device of claim 1, wherein, the second image processing module is configured to determine a road region of the first image based on vanishing point information and to extract the determined road region as the ROI.

5. The electronic device of claim 3, wherein, the first image processing module is configured to generate M pyramid images, where M is a positive integer, and the second image processing module is configured to generate Y pyramid images, where Y is a positive integer less than M. 6.The electronic device of claim 1, wherein, the processor is configured to determine whether the high-speed performance is required by additionally considering information about the surrounding environment of the host vehicle.

7. The electronic device of claim 6, wherein, the processor is configured to determine whether to use the high-speed performance of the image processing operation based on information about the surrounding environment of the host vehicle when the speed of the host vehicle is less than a first threshold speed.

8. The electronic device of claim 6, wherein, the information about the surrounding environment of the host vehicle includes information about at least one of a speed, an acceleration, a moving distance, and a position of an object located in a vicinity of the host vehicle and a normal speed or a speed limit of a road on which the host vehicle travels.

9. The electronic device of claim 8, wherein, the processor is configured to estimate a collision risk between the host vehicle and the object based on at least one of the speed, the acceleration, the moving distance, and the position of the object located in the vicinity of the host vehicle, and to determine that the high-speed performance of the image processing operation is required when the collision risk is estimated. 10.The electronic device of claim 8, wherein, the processor is configured to determine to use a high-speed performance of the image processing operation when a normal speed or a speed limit of a road on which the host vehicle travels is greater than or equal to a second threshold speed. 11.The electronic device of claim 1, wherein, the processor is configured to perform the image processing operation on a preset number of continuous images captured by the image sensor by using the second image processing module when the high-speed performance is required, and the continuous images include the first image.

12. The electronic device of claim 1, wherein, the processor is configured to perform the image processing operation by alternately applying the second image processing module and the first image processing module to the preset number of continuous images captured by the image sensor when the high-speed performance is used. 13.An electronic device configured to control a host vehicle, the electronic device comprising: an image sensor configured to photograph a surrounding environment of the host vehicle; and a processor configured to perform an image processing operation based on a first image captured by the image sensor and to control the host vehicle based on a processing result, wherein the processor is configured to determine a target latency of the image processing operation based on travel information about the host vehicle or information about the surrounding environment of the host vehicle, to select a size of a region of interest (ROI) corresponding to the determined target latency, and to perform the image processing operation according to the selected size of the ROI.

14. The electronic device of claim 13, wherein, the processor is configured to select an operation parameter according to the determined target latency, the operation parameter including a number of pyramid images having different scales from each other, wherein the processor is configured to select the number of pyramid images corresponding to the determined target latency.

15. The electronic device of claim 14, wherein, the processor is configured to extract an ROI having the selected size from the first image, to generate the selected number of pyramid images based on the extracted ROI, and to perform an object detection operation on the generated pyramid images. 16.The electronic device of claim 13, further comprising a memory storing a plurality of image processing modules corresponding to a plurality of target latencies, wherein the processor is configured to select an image processing module corresponding to the determined target latency from among the plurality of image processing modules and to perform the image processing operation by using the selected image processing module. 17.An operation method of an electronic device configured to control a host vehicle, the operation method comprising: obtaining a first image of a surrounding environment of the host vehicle; obtaining speed information about the host vehicle; identifying whether a speed of the host vehicle is greater than or equal to a threshold speed; when the speed of the host vehicle is less than the threshold speed, processing the first image by using a first image processing method; when the speed of the host vehicle is greater than or equal to the threshold speed, processing the first image by using a second image processing method having a data throughput less than that of the first image processing method; and controlling the host vehicle based on a processing result of the first image, The operation method further includes determining a target delay of an image processing operation based on travel information about the host vehicle or information about a surrounding environment of the host vehicle, selecting a size of a region of interest (ROI) corresponding to the determined target delay, and performing the image processing operation according to the selected size of the ROI.

Images