Driving assistance system, electronic device and operating method thereof

Abstract

A system and method for automated driving assistance are described. The system may include: a first sensor unit configured to provide an input image including other vehicles and a background; an image processor configured to generate a plurality of pyramid images by downsampling the input image, obtain a depth map including depth values ​​from the first sensor unit to the other vehicles and the background, generate a plurality of mask data having different average depths of the depth values ​​based on the depth map, and output a plurality of masked images representing different average distances from the first sensor unit based on the plurality of mask data and the plurality of pyramid images; a feature extractor configured to output feature data for each of the plurality of masked images; and a detector configured to detect the other vehicles included in the input image based on the feature data.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of Korean Patent Application No. 10-2019-0108932, filed on September 3, 2019, in the Korean Intellectual Property Office, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] The present inventive concept relates to a driving assistance system, an electronic device for detecting an object included in an image, and an operating method thereof. Background Art

[0004] Modern vehicles include numerous electronic components. For example, sensors and computers are often used to detect objects, calculate distance or speed, and predict future events. Driver assistance technologies can use these components to assist the driver with maneuvers such as lane changes, adaptive cruise control, blind spot monitoring, and night-time object detection. In some cases, vehicles equipped with driver assistance technologies can react automatically to external objects and events without driver input.

[0005] Vehicles that include driver-assistance technologies can collect and process large amounts of data. This places significant computational demands on the vehicle's computers. In some cases, this can result in slower computations, which can reduce the performance of the driver-assistance technology and compromise vehicle safety. Therefore, there is a need in the art for vehicle-assistance technology systems that require less data and computation without sacrificing performance. Summary of the Invention

[0006] The present disclosure describes a driving assistance system and electronic device for efficiently detecting objects from high-resolution images using an artificial neural network, and a method of operating the same.

[0007] According to an aspect of the present inventive concept, a driving assistance system may include: a first sensor unit configured to provide an input image including other vehicles and a background; an image processor configured to generate a plurality of pyramid images by downsampling the input image, identify a depth map including depth values ​​to the other vehicles and the background, generate a plurality of mask data having different average depths of the depth values ​​based on the depth map, and output a plurality of masked images representing different average distances based on the plurality of mask data and the plurality of pyramid images; a feature extractor configured to output feature data of each of the plurality of masked images; and a detector configured to detect the other vehicles included in the input image based on the feature data.

[0008] According to another aspect of the inventive concept, an electronic device for detecting an object from an input image may include: an image processor configured to generate a first pyramid image by downsampling the input image, generate a second pyramid image by downsampling the first pyramid image, identify a partial region in the first pyramid image where an average distance indicates a first value as a first region, identify a partial region in the second pyramid image where an average distance indicates a second value as a second region, and output images of the first region and the second region; a first core configured to generate first feature data including a feature value of the first region and detect the object in the first region based on the first feature data; and a second core configured to generate second feature data including the feature value of the second region and detect the object in the second region based on the second feature data.

[0009] According to another aspect of the inventive concept, an operating method of an electronic device may include: generating a first pyramid image by downsampling an input image including an object and a background; generating a second pyramid image by downsampling the first pyramid image; masking an area remaining by excluding a first area having an average distance of a first value from the first pyramid image; masking an area remaining by excluding a second area having an average distance of a second value from the second pyramid image, wherein the second value is smaller than the first value; acquiring a plurality of feature data from a plurality of masked images generated based on a plurality of masking operations; and detecting an object based on the plurality of feature data.

[0010] According to another embodiment of the present inventive concept, an image processing method may include: receiving input data, the input data including image data and distance data corresponding to the image data; generating a plurality of downsampled images based on the image data, wherein each of the plurality of downsampled images corresponds to a different image resolution; generating a plurality of image masks based on the distance data, wherein each of the plurality of image masks corresponds to a different average distance value; generating a plurality of masked images, wherein each of the plurality of masked images is based on one of the plurality of downsampled images and one of the plurality of image masks; generating feature data based on the plurality of masked images; and detecting an object based on the feature data. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Embodiments of the present invention will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0012] Figure 1 is a block diagram of an electronic device according to an example embodiment of the inventive concept;

[0013] Figure 2 is a block diagram of a driving assistance system according to an example embodiment of the present inventive concept;

[0014] Figure 3 An artificial neural network and a computational process using the artificial neural network are shown;

[0015] Figure 4 is a block diagram of an electronic device according to an example embodiment of the inventive concept;

[0016] Figure 5 is a block diagram of an electronic device according to an example embodiment of the inventive concept;

[0017] Figure 6 is a block diagram of an electronic device according to an example embodiment of the inventive concept;

[0018] Figure 7 and Figure 8 shows region of interest (RoI) information and mask data according to an example embodiment of the inventive concept;

[0019] Figure 9 An operation of masking a pyramid image and generating a masked image according to an example embodiment of the inventive concept is shown;

[0020] Figure 10A 、 Figure 10B and Figure 10C shows an image processing operation of a pre-processor according to an example embodiment of the inventive concept;

[0021] Figure 11A and Figure 11B is a block diagram for describing a feature extractor and a kernel unit according to an example embodiment of the inventive concept;

[0022] Figure 12 is a block diagram of an electronic device according to an example embodiment of the inventive concept;

[0023] Figure 13 is a block diagram of an electronic device for describing a sensor fusion operation according to an example embodiment of the present inventive concept;

[0024] Figure 14 is a flowchart of an operating method of an electronic device according to an exemplary embodiment of the inventive concept;

[0025] Figure 15 is a flowchart of an operating method of an electronic device according to an exemplary embodiment of the inventive concept;

[0026] Figure 16 is a flowchart of an operating method of an electronic device according to an exemplary embodiment of the inventive concept; and

[0027] Figure 17 is a block diagram of an electronic system according to an example embodiment of the inventive concept. DETAILED DESCRIPTION

[0028] This disclosure describes systems and methods for object detection. For example, object detection systems can be used to improve vehicle driving stability and efficiency by identifying objects from the perspective of driver assistance systems. As technology improves, the amount of data collected by vehicle sensors tends to increase. High-resolution images are particularly useful for identifying distant objects. Due to the use of these high-resolution images, vehicle object detection models can leverage increased computational capabilities to process image data in real time.

[0029] In some cases, the increasing computational demands may exceed the capabilities of the onboard computer system. When this happens, the real-time operation of the vehicle or system may be affected. Addressing this challenge by reducing the complexity of the feature extractor in the system backbone can reduce computational demands, but this also reduces object recognition accuracy.

[0030] A driver assistance system can operate by identifying an image that includes a set of objects (both near and far). Next, the size and distance of each object are determined (i.e., whether it is a large object such as a vehicle or a person, or a small object such as a bird or a ball). In some cases, the size of the object can be used to determine the distance between the object and the vehicle. After determining the distance to each object, one or more regions of interest (RoIs) are determined based on the objects and the distance to each object.

[0031] According to embodiments of the present disclosure, an RoI can be divided into several sub-regions, which can be downsampled to a lower resolution based on distance information. The modified image, including some downsampled portions, can be used as input for feature extractors, trackers, and the like. By using images in which some portions have been downsampled, the total amount of data can be reduced. However, since high resolution is used for key portions of the image, the overall performance of the system can be maintained.

[0032] Hereinafter, embodiments of the inventive concept will be described in detail with reference to the accompanying drawings.

[0033] Figure 1 is a block diagram of an electronic device 1 according to an exemplary embodiment of the inventive concept.

[0034] The electronic device 1 according to the exemplary embodiment of the present invention can extract valid information by analyzing input data. In addition, the electronic device 1 can generate output data based on the extracted information. For example, the input data can be an image obtained by shooting a front view from the electronic device 1. The valid information can be an object (another vehicle, a pedestrian, etc.). The output data can be data of the object detected from the image. For example, the electronic device 1 can be an application processor. The application processor can perform various types of computing processing. The neural processing unit (NPU) 12 included in the application processor can perform computing processing using an artificial neural network.

[0035] Reference Figure 1 The electronic device 1 may include a processor 10, a random access memory (RAM) 20, a storage device 30, a sensor unit 40, a communication module 50, and a bus 90 for performing communication therewith. The processor 10 may include a central processing unit (CPU) 11 and an NPU 12. The electronic device 1 may further include an input / output module, a security module, a power control device, etc. In addition, the electronic device 1 may also include various types of computing devices.

[0036] The CPU 11 controls the overall operation of the electronic device 1. The CPU 11 may include a single-core processor or a multi-core processor. The CPU 11 may process or execute programs and / or data stored in the storage device 30. For example, the CPU 11 may control the functions of the NPU 12 by executing programs stored in the storage device 30.

[0037] The NPU 12 can receive input data, perform arithmetic operations using an artificial neural network, and provide output data based on the arithmetic operation results. The NPU 12 can perform computational processing based on various types of networks, such as convolutional neural networks (CNNs), regions with convolutional neural networks (R-CNNs), region proposal networks (RPNs), recurrent neural networks (RNNs), fully convolutional networks, long short-term memory (LSTM) networks, and classification networks. However, the NPU 12 is not limited thereto and can perform various types of computational processing that simulate human neural networks.

[0038] The RAM 20 can temporarily store programs, data, or instructions. For example, the program and / or data stored in the storage device 30 can be temporarily stored in the RAM 20 according to the control or boot code of the CPU 11. For example, the RAM 20 includes dynamic RAM (DRAM), static RAM (SRAM), synchronous DRAM (SDRAM), etc.

[0039] The storage device 30 is a storage space for storing data, and can store an operating system (OS), various programs, and various data. The storage device 30 can be a DRAM, but is not limited thereto. The storage device 30 can include at least one of a volatile memory or a non-volatile memory. The non-volatile memory can include a read-only memory (ROM), a flash memory, a phase change RAM (PRAM), a magnetic RAM (MRAM), a resistive RAM (RRAM), a ferroelectric RAM (FRAM), etc. According to an embodiment, the storage device 30 can be implemented by a hard disk drive (HDD), a solid-state drive (SSD), etc.

[0040] The sensor unit 40 can collect information about objects sensed by the electronic device 1. For example, the sensor unit 40 can be an image sensor unit. In this case, the sensor unit 40 can include at least one image sensor. The sensor unit 40 can sense or receive image signals from outside the electronic device 1 and convert the image signals into image data, i.e., image frames. As another example, the sensor unit 40 can be a distance sensor unit. In this case, the sensor unit 40 can include at least one distance sensor. The distance sensor can include at least one of various types of sensing devices, such as a light detection and ranging (LIDAR) sensor, a radio detection and ranging (RADAR) sensor, a time of flight (ToF) sensor, an ultrasonic sensor, and an infrared sensor. LIDAR sensors and RADAR sensors can be distinguished based on their effective measurement distance. For example, LIDAR sensors can be distinguished into long-range LIDAR sensors and short-range LIDAR sensors, and RADAR sensors can be distinguished into long-range RADAR sensors and short-range RADAR sensors. The sensor unit 40 is not limited thereto. The sensor unit 40 may further include at least one of a magnetic sensor, a position sensor (e.g., GPS), an acceleration sensor, an atmospheric pressure sensor, a temperature / humidity sensor, a proximity sensor, and a gyroscope sensor, but is not limited thereto. A person skilled in the art can intuitively infer the function of each sensor from its name. Therefore, a detailed description thereof is omitted here.

[0041] The communication module 50 can send and / or receive data of the electronic device 1. For example, the communication module 50 can communicate with an external target of the electronic device 1. In this case, the communication module 50 can perform communication according to the vehicle-to-everything (V2X) scheme. For example, the communication module 50 can perform communication according to the vehicle-to-vehicle (V2V), vehicle-to-infrastructure (V2I), vehicle-to-pedestrian (V2P) and vehicle-to-mobile device (V2N) schemes. However, the communication module 50 is not limited to this. The communication module 50 can send and receive data according to various known communication schemes. For example, the communication module 50 can perform communication by including third generation (3G), long term evolution (LTE), Bluetooth, Bluetooth low energy (BLE), ZigBee, near field communication (NFC), ultrasound or similar communication methods. In addition, the communication module 50 can perform both short-range communication and long-range communication.

[0042] According to an exemplary embodiment of the present inventive concept, the processor 10 may receive an input image and then generate multiple images by downsampling the input image. The multiple images may have different sizes. The size may indicate the resolution. The processor 10 may identify the farthest object from the image with a relatively large size and the closest object from the image with a relatively small size. During this process, the processor 10 may mask the area remaining after excluding the object. Thus, the object can be identified based on the masked-out area, reducing the computational complexity of the processor 10.

[0043] Figure 2 is a block diagram of a driving assistance system 2 according to an exemplary embodiment of the present invention. Figure 1 .

[0044] Reference Figure 2 The driving assistance system 2 may include a processor 10, a sensor unit 40, a communication module 50, a driving control unit 60, an autonomous driving unit 70, and a user interface (IF) 80. The processor 10, the sensor unit 40, and the communication module 50 are similar to those in the above reference. Figure 1 The same or similar ones as those described above are omitted here.

[0045] According to an exemplary embodiment of the present invention, the driving assistance system 2 can detect an object. Object detection is performed using information about the external environment acquired by the sensor unit 40. For example, the sensor unit 40 can capture an image and send the captured image to the processor 10. The processor 10 can detect an object (e.g., another vehicle) based on the captured image (hereinafter, the input image) and control the driving control unit 60 and the autonomous driving unit 70. As an example description, the processor 10 detects an object based on the input image, but the processor 10 is not limited thereto. For example, the processor 10 can detect an object based on depth information output from a distance sensor.

[0046] The driving control unit 60 may include a vehicle steering device and a throttle control device. The vehicle steering device is configured to control the direction of travel of the vehicle. The throttle control device is configured to control acceleration and / or deceleration by controlling the vehicle's motor or engine. The driving control unit 60 may also include a brake device configured to control the braking of the vehicle, external lighting devices, etc.

[0047] The autonomous driving unit 70 may include a computing device configured to implement autonomous control of the driving control unit 60. For example, the autonomous driving unit 70 may include at least one of the components of the electronic device 1. The autonomous driving unit 70 may include a memory storing a plurality of program instructions and one or more processors configured to execute the program instructions. The autonomous driving unit 70 may be configured to control the driving control unit 60 based on sensing signals output from the sensor unit 40. The user interface 80 may include various electronic and mechanical devices included in the driver's seat, passenger seat, etc., such as a display showing the vehicle instrument panel.

[0048] The processor 10 uses various sensory data such as input images and depth information to detect objects. In this case, the processor 10 can use artificial neural networks for efficient computational processing. For example, the NPU 12 can perform the following reference Figure 3 Describe the calculation method.

[0049] Figure 3 An artificial neural network NN and a calculation process using the artificial neural network NN are shown.

[0050] refer to Figure 3 , the artificial neural network NN may include multiple layers, for example, a first layer L1 to an nth layer Ln. Each of the multiple layers L1 to Ln may be a linear layer or a nonlinear layer. According to an embodiment of the present inventive concept, at least one linear layer and at least one nonlinear layer may be combined and referred to as one layer. For example, a linear layer may include a convolutional layer and a fully connected layer. A nonlinear layer may include a sampling layer, a pooling layer, and an activation layer.

[0051] For example, the first layer L1 may be a convolutional layer, and the second layer L2 may be a sampling layer. The artificial neural network NN may further include activation layers, and may further include layers configured to perform other types of arithmetic operations.

[0052] Each of the multiple layers L1 to Ln can receive input image data or a feature map generated in the previous layer as an input feature map, and perform arithmetic operations on the input feature map to generate an output feature map. In this case, the feature map indicates data representing various features of the input data. The first to third feature maps FM1, FM2, and FM3 may have, for example, a two-dimensional (2D) matrix or a 3D matrix format. The first to third feature maps FM1 to FM3 may have a width (or column) W, a height (or row) H, and a depth D corresponding to the x-axis, y-axis, and z-axis on the coordinate system, respectively. Here, the depth D may be referred to as the number of channels.

[0053] The first layer L1 can generate a second feature map FM2 by convolving the first feature map FM1 and the weight map WM. The weight map WM can filter the first feature map FM1 and can be called a filter or a kernel. For example, the depth of the weight map WM, that is, the number of channels, is the same as the depth of the first feature map FM1. In addition, the weight map WM and the same channel of the first feature map FM1 can be convolved. By using the first feature map FM1 as a sliding window, the weight map WM can be shifted in a traversal manner. The shift amount can be referred to as a "stride length" or "stride". During each shift, each weight value included in the weight map WM can be multiplied and added with the pixel value in the area overlapping with the first feature map FM1. According to the convolution of the first feature map FM1 and the weight map WM, one channel of the second feature map FM2 can be generated. Although Figure 3 One weight map WM is shown, but in practice multiple weight maps can be convolved with the first feature map FM1 to generate multiple channels of the second feature map FM2. In other words, the number of channels of the second feature map FM2 can correspond to the number of weight maps.

[0054] The second layer L2 can generate a third feature map FM3 by changing the spatial size of the second feature map FM2. For example, the second layer L2 can be a sampling layer. The second layer L2 can perform upsampling or downsampling. The second layer L2 can select a portion of the data included in the second feature map FM2. For example, the 2D window WD can be shifted on the second feature map FM2 in units of the size of the window WD (e.g., a 4*4 matrix), and the value of a specific position (e.g., the first row and first column) in the area overlapping with the window WD can be selected. The second layer L2 can output the selected data as data for the third feature map FM3. As another example, the second layer L2 can be a pooling layer. In this case, the second layer L2 can select the maximum value (or the mean value) of the eigenvalue in the area overlapping with the window WD on the second feature map FM2. The second layer L2 can output the selected data as data for the third feature map FM3.

[0055] As a result, a third feature map FM3 having a changed spatial size compared to the second feature map FM2 can be generated. The number of channels of the third feature map FM3 can be the same as the number of channels of the second feature map FM2. According to an example embodiment of the present inventive concept, the sampling layer can have an arithmetic operation speed faster than the pooling layer. The sampling layer can improve the quality of the output image (for example, in terms of peak signal-to-noise ratio (PSNR)). For example, the arithmetic operation of the pooling layer can calculate the maximum value or the mean. Therefore, the pooling layer can have a longer arithmetic operation time than the sampling layer.

[0056] According to an embodiment of the present inventive concept, the second layer L2 is not limited to a sampling layer or a pooling layer. For example, the second layer L2 may be a convolutional layer similar to the first layer L1. The second layer L2 may generate a third feature map FM3 by convolving the second feature map FM2 with a weight map. In this case, the weight map on which the convolution operation has been performed in the second layer L2 may be different from the weight map WM on which the convolution operation has been performed in the first layer L1.

[0057] By passing through multiple layers including a first layer L1 and a second layer L2, an Nth feature map can be generated in the Nth layer. The Nth feature map can be input to a reconstruction layer located at the back end of the artificial neural network NN, from which output data is output. The reconstruction layer can generate an output image based on the Nth feature map. Alternatively, the reconstruction layer can receive multiple feature maps. The multiple feature maps can include the Nth feature map, which can be a first feature map FM1, a second feature map FM2, and so on. In addition, the reconstruction layer can generate an output image based on the multiple feature maps.

[0058] The third layer L3 can classify the category CL of the input data by combining the features of the third feature map FM3. Furthermore, the third layer L3 can generate a recognition signal REC corresponding to the category. For example, the input data can be image frames or video frames. In this case, the third layer L3 can identify an object included in the image indicated by the frame data by extracting the category corresponding to the object based on the third feature map FM3 provided by the second layer L2. The third layer L3 can then generate a recognition signal REC corresponding to the identified object.

[0059] In artificial neural networks, lower-level layers (such as convolutional layers) can extract lower-level features (such as the outline or gradient of a vehicle) from the input data or input feature map. Higher-level layers (such as fully connected layers) can extract or detect higher-level features, i.e., categories (such as the taillights, rear window glass, etc. of a vehicle) from the input feature map.

[0060] Figure 4 is a block diagram of an electronic device 3 according to an example embodiment of the inventive concept.

[0061] Reference Figure 4 , the electronic device 3 may include a sensor unit 100, an image processor 200, a feature extractor 300, and a detector 510. The image processor 200 may further include a pre-processor 210, a region of interest (RoI) network 220, a mask generator 230, and a masking unit 240. The electronic device 3 may be included as the above reference Figure 1 and Figure 2 For example, the image processor 200, the feature extractor 300, and the detector 510 of the electronic device 3 may be included in the processor 10, and commands, instructions, or program codes for executing the operations and functions of the image processor 200, the feature extractor 300, and the detector 510 may be stored in the storage device 30. The sensor unit 100 is similar to the above-mentioned Figure 1 and Figure 2 The sensor unit 40 described above is the same. Therefore, its detailed description is omitted here. In addition, the electronic device 3 can be included as the above reference Figure 2 The electronic device 3 may be configured as at least partially as described in the driving assistance system 2. For example, the electronic device 3 may acquire images of other vehicles from the sensor unit 100. The image processor 200 may perform image processing to detect other vehicles. The feature extractor 300 may extract features of the image, and the detector 510 may detect other vehicles. For example, the electronic device 3 may be a driving assistance system for assisting a driver or for autonomous driving. Alternatively, the electronic device 3 may be included as part of the driving assistance system.

[0062] According to an exemplary embodiment of the present invention, the sensor unit 100 may capture a front view and output an input image IM. The input image IM may include an object and a background. For example, the input image IM may be data about a 2D image in RGB format, but is not limited thereto. The sensor unit 100 may be referred to as a first sensor unit 100 to be used in conjunction with the first sensor unit 100 described below. Figure 13 The sensor unit 520 is described separately. The sensor unit 520 may be referred to as a second sensor unit 520.

[0063] The image processor 200 may receive an input image IM, mask at least a partial region of the input image IM, and output a masked image IMK.

[0064] The preprocessor 210 may receive an input image IM, downsample the input image IM, and generate and output pyramid images PIM. For example, the preprocessor 210 may generate a first pyramid image by downsampling the horizontal and vertical lengths of the input image IM at a specific ratio. The preprocessor 210 may then generate a second pyramid image by downsampling the first pyramid image again at a specific ratio. As another example, the first pyramid image may be generated by downsampling the input image IM once. The second pyramid image may be generated by downsampling the input image IM twice. For example, the preprocessor 210 may generate a plurality of pyramid images PIM derived from the input image IM, each having a gradually smaller size than the input image IM.

[0065] The preprocessor 210 may mask the region remaining after excluding the object of interest (e.g., at least one of another vehicle, a road, or a pedestrian) from the input image IM, and generate a pyramid image PIM based on the masked image. For example, although not shown, the preprocessor 210 may receive RoI information RID and mask the data of the region remaining after excluding the object of interest based on the RoI information RID.

[0066] The preprocessor 210 may receive an input image IM and then acquire and output a depth map DP including depth data regarding objects and background included in the input image IM. The depth data may include depth values ​​indicating, for example, the distance from a user or their own vehicle to an object or other vehicles. The depth map DP indicates a map including, for example, depth values ​​from the sensor unit 100 to other vehicles and the background. For example, the sensor unit 100 may include a stereo camera. In this case, the input image IM may include a left-eye image and a right-eye image. The preprocessor 210 may calculate parity using the left-eye image and the right-eye image and acquire the depth map DP based on the calculated parity. However, the sensor unit 100 is not limited thereto and may be a combination of a single camera and a range sensor rather than a stereo camera. For example, the sensor unit 100 may output 2D information using a single camera and 3D information using a range sensor. In this case, the preprocessor 210 may acquire a depth map DP regarding both the objects and background included in the input image IM using both the 2D and 3D information. The depth map DP can be generated by the pre-processor 210 and the depth generator ( Figure 5 250) is generated, and the following will refer to Figure 5 For example, the depth generator 250 may generate the depth map DP by using the parity of the stereo image received from the sensor unit 100. Alternatively, the depth map DP may be generated by the sensor unit 100, which will be referred to below. Figure 6 For example, the sensor unit 100 may output a stereoscopic image, and may also output a depth map DP based on the stereoscopic image.

[0067] The RoI network 220 may identify a partial region included in the input image IM as a RoI based on the input image IM and output RoI information RID including data regarding the RoI. For example, the RoI information RID may include at least one of 2D information of the RoI (e.g., a partial region of the input image IM) and 3D information of the RoI (e.g., partial data of the depth map DP). For example, when the input image IM includes a left-eye image and a right-eye image, the RoI network 220 may calculate a depth value based on the parity of the left-eye image and the right-eye image. The RoI network 220 may then output the RoI information RID, which may include a depth value. As another example, the RoI network 220 may identify a RoI from the input image IM based on the depth map DP. For example, the RoI network 220 may identify elements useful for driving based on multiple depth values ​​included in the depth map DP. For example, when the electronic device 3 is included in a driving assistance system, the RoI network 220 may analyze the input image IM and / or the depth map DP to identify a region that may include information useful for vehicle driving as a RoI. For example, the RoI may be an area including a road where the vehicle is traveling, other vehicles, traffic lights, crosswalks, etc. The RoI may include multiple areas.

[0068] The mask generator 230 may generate and output a plurality of mask data MK having different average depths based on the RoI information RID and the depth map DP. The mask generator 230 may generate and output a plurality of mask data MK based on the RoI information RID, in which regions other than the RoI in the pyramid image PIM are masked.

[0069] The mask data MK may be data used to mask areas of the pyramid image PIM other than the meaningful area. For example, the smaller the number of downsampling times used to generate the pyramid image PIM, the more distant the meaningful area may be. Conversely, the larger the number of downsampling times used to generate the pyramid image PIM, the more distant the meaningful area may be. For example, when the pyramid image PIM to which the mask data MK is applied is relatively large, the mask data MK may include data for masking areas at relatively close distances. Conversely, when the pyramid image PIM to which the mask data MK is applied is relatively small, the mask data MK may include data for masking areas at relatively far distances.

[0070] The mask generator 230 may receive a depth map DP and RoI information RID and generate a plurality of mask data MK based on the depth values. The mask generator 230 may identify a portion of the region included in the depth map DP as a RoI based on the RoI information RID and generate the mask data MK by excluding the depth value of the RoI. Thus, the RoI of the mask data MK may include a meaningful depth value, and the data of the region other than the RoI of the mask data MK may include a null value or an invalid value.

[0071] The mask generator 230 may divide the RoI in the depth map DP into a plurality of regions and output mask data MK including each of the plurality of regions. For example, the mask data MK may include depth values ​​of a portion of the RoI in the depth map DP. Figure 7 The mask data MK is described in detail.

[0072] The masking unit 240 may generate and output masked images IMK representing different average distances based on the plurality of pyramid images PIM and the plurality of mask data MK. The average distance indicates the average value of the distances from the sensor unit 100 to the real object corresponding to the pixels represented on the image. In other words, the masking unit 240 may apply the plurality of mask data MK to the plurality of pyramid images PIM, respectively.

[0073] The masked image IMK may represent a partial region of the input image IM or the pyramid image PIM. Figure 9 The IMK1) may include a first region (eg, Figure 9 , wherein the first region is a first pyramid image (e.g., Figure 9 In addition, the first masked image may include a second masked image (eg, Figure 9 IMK2), the second masked image may include a second region (eg, Figure 9 C2), wherein the second region is a second pyramid image (e.g., Figure 9 In this case, the contexts formed by the first region C1 and the second region C2 can be different. For example, the first region C1 may include the vanishing point and the end of the road, while the second region C2 may include the middle of the road and the vehicle ahead. The image context may indicate information about the situation depicted by the image. For example, images captured simultaneously may have the same context.

[0074] The masking unit 240 may mask the region remaining by excluding the first region C1 in the first pyramid image PIM1 and mask the region remaining by excluding the second region C2 in the second pyramid image PIM2. The masking unit 240 may output the masked first pyramid image (i.e., the first masked image) and the masked second pyramid image (i.e., the second masked image).

[0075] The first area C1 and the second area C2 include areas at different distances, but they may not be mutually exclusive. For example, the first area C1 and the second area C2 may include overlapping areas. For example, a portion of an image included in the first area C1 (e.g., a specific portion of a road) may also be included in the second area C2. Furthermore, at least some of the plurality of mask data MK may mask overlapping areas in the pyramid image PIM. When the first area C1 and the second area C2 do not overlap, portions of objects included in the pyramid image PIM may be omitted. To prevent this, the masking unit 240 may mask the pyramid image PIM so that the boundaries of different areas overlap.

[0076] The mask generator 230 may write information indicating the first pyramid image into the header portion of the first mask data, so that the first mask data is applied to the first pyramid image, which is the largest pyramid image. Therefore, the masking unit 240 can mask the first pyramid image using the first mask data. As another example, the masking unit 240 may recognize that the first mask data has a size corresponding to the first pyramid image and mask the first pyramid image using the first mask data. For example, when the size of the first mask data corresponds to the size of a partial area of ​​the first pyramid image, the masking unit 240 may apply the first mask data to the first pyramid image.

[0077] The feature extractor 300 may receive the masked image IMK and output feature data FD for each masked image IMK. For example, the feature data FD may be the above reference Figure 3 The feature map (eg, FM3) described, the class CL, or the recognition signal REC. For example, the feature data FD may include various forms of data, including features of non-masked regions in the input image IM or the pyramid image PIM.

[0078] The feature extractor 300 can extract feature values ​​from masked images IMK that respectively include different scenarios. For example, the first masked image may include scenarios that are far away from the observer, such as a vanishing point. In this case, the image processor 200 can downsample the input image IM a relatively small number of times for farther objects, thereby including more pixels related to the farther objects. In contrast, the second masked image may include scenarios that are close to the observer. For the efficiency of computational processing and the efficiency of machine learning, the image processor 200 can downsample the input image IM a relatively large number of times. Therefore, the amount of data included in the first masked image and the second masked image can be similar within a small error range. As a result, when machine learning is performed using the first masked image and the second masked image, kernels of substantially the same size can be used to improve learning efficiency.

[0079] The detector 510 may receive feature data FD and, based on the feature data FD, identify information about an object or background included in the input image IM. For example, the detector 510 may detect an object (e.g., another vehicle) included in the input image IM and may detect various information about the object. The various information may include 3D information, such as a 3D bounding box, the shape of the object, the distance to the object, the position of the object, and the like, and 2D information, such as the edges forming the object.

[0080] Figure 5 is a block diagram of an electronic device 3 according to an example embodiment of the inventive concept.

[0081] The electronic device 3 may include a sensor unit 100, an image processor 200, a feature extractor 300, a buffer 410, and a detector 510. The image processor 200 may further include a preprocessor 210, an RoI network 220, a mask generator 230, a masking unit 240, a depth generator 250, and a masking unit 260. The feature extractor 300 may further include an image feature extractor 310 and a depth feature extractor 320. The electronic device 3 may include the above reference Figure 1 and Figure 2 At least part of the configuration of the electronic device 1 described above is omitted in the following. Figures 1 to 4 Description of the process.

[0082] The image processor 200 may receive an input image IM and output a masked image IMK and / or a masked depth map DMK. The masked depth map DMK may be obtained by masking a partial area related to the front elevation view in the depth map.

[0083] The preprocessor 210 may receive an input image IM, downsample the input image IM, and generate and output a pyramid image PIM. Figure 4 The described preprocessor 210 may output a depth map DP based on the input image IM, but according to an embodiment of the inventive concept, the preprocessor 210 may generate and output a pyramid image PIM based on the input image IM instead of generating and outputting the depth map DP.

[0084] The depth generator 250 may receive an input image IM and output a depth map DP. For example, the preprocessor 210 may not output a depth map DP. The depth generator 250 may output a depth map DP. For example, the depth generator 250 may provide the depth map DP to the RoI network 220 and the mask generator 230. For example, the sensor unit 100 may be a stereo camera. In this case, the input image IM may include a left-eye image and a right-eye image. The depth generator 250 may detect parity using the left-eye image and the right-eye image and acquire a depth map DP based on the detected parity.

[0085] The depth generator 250 can output multiple pyramid depth maps PDP based on the input image IM. The relationship between the multiple pyramid depth maps PDP and the depth map DP can be similar to the relationship between the multiple pyramid images PIM and the input image IM. For example, the depth generator 250 can generate a first pyramid depth map by downsampling the horizontal and vertical lengths of the depth map DP at a specific ratio, and can generate a second pyramid depth map by downsampling the first pyramid depth map again at a specific ratio. As another example, the first pyramid depth map can be generated by downsampling the depth map DP once. The second pyramid depth map can be generated by downsampling the depth map DP twice. For example, the depth generator 250 can generate multiple pyramid depth maps PDP of gradually decreasing size compared to the depth map DP.

[0086] The mask generator 230 may provide a plurality of mask data MK to the masking unit 240 and the masking unit 260 based on the depth map DP and the RoI information RID.

[0087] The masking unit 240 may output a masked image IMK based on the plurality of pyramid images PIM and the plurality of mask data MK. The masking unit 260 may output a masked depth map DMK based on the plurality of pyramid depth maps PDP and the plurality of mask data MK.

[0088] The masked depth map DMK may represent a partial area of ​​a depth map DP or a pyramid depth map PDP. For example, a first masked depth map may include a first area, where the first area is part of a first pyramid depth map. Alternatively, the first masked depth map may include a second masked depth map, and the second masked depth map may include a second area, where the second area is part of a second pyramid depth map. In this case, the contexts formed by the first area and the second area may be different from each other.

[0089] The feature extractor 300 may output image feature data IF based on the masked image IMK and output depth feature data DF based on the masked depth map DMK. For example, the image feature data IF and the depth feature data DF may be the same as those in the above reference. Figure 3 The feature map (e.g., FM3) described herein, the class CL, or the recognition signal REC. For example, the image feature extractor 310 may extract feature values ​​of the masked image IMK and provide image feature data IF based on the extracted feature values ​​to the buffer 410. Similarly, the depth feature extractor 320 may extract feature values ​​of the masked depth map DMK and provide depth feature data DF to the buffer 410 based on the extracted feature values.

[0090] The buffer 410 may receive the image feature data IF and the depth feature data DF, perform splicing based on the image feature data IF and the depth feature data DF, and output the spliced ​​data CD. For example, the buffer 410 may splice the image feature data IF and the depth feature data DF into a single feature data. As another example, the buffer 410 may splice the spliced ​​single feature data and the downsampled feature data ( Figure 12 IF_2D) for splicing, which will be referenced below Figure 12 For example, the concatenated data CD may be data in a concatenated form of various feature data.

[0091] The detector 510 may receive the stitched data CD, analyze the feature values ​​of the stitched data CD, and detect objects (e.g., other vehicles, etc.) included in the input image IM. For example, the detector 510 may include various activation layers (e.g., ReLU (Rectified Linear Unit)) that may be implemented in an artificial neural network model. As another example, the detector 510 may include various object recognition models that may recognize objects based on the feature values ​​of the stitched data CD.

[0092] Figure 6 is a block diagram of an electronic device 3 according to an example embodiment of the inventive concept.

[0093] Reference Figure 6, the electronic device 3 may include a sensor unit 100, an image processor 200, a feature extractor 300, a buffer 410 and a detector 510, and the image processor 200 may further include a preprocessor 210, a RoI network 220, a mask generator 230 and a masking unit 240. The feature extractor 300 may further include an image feature extractor 310. In the following, the above references are omitted. Figures 4 and 5 Description of the process.

[0094] According to an exemplary embodiment of the present inventive concept, the sensor unit 100 may include a distance sensor capable of directly acquiring 3D information. For example, the distance sensor may sense distance information and acquire depth values, and may be implemented by a LIDAR, RADAR, or ToF sensor. Alternatively, the sensor unit 100 may include an image sensor capable of capturing 2D images. The sensor unit 100 may output an input image IM, wherein the input image IM is a depth map DP including depth values ​​and / or a 2D image.

[0095] The RoI network 220 may identify at least some areas in the depth map DP as RoIs. For example, the RoI network 220 may identify an area including elements used for vehicle driving (e.g., a front vehicle, a road, or a pedestrian) as an RoI. For example, the RoI network 220 may analyze a plurality of depth values ​​included in the depth map DP and identify an area including a depth value identified as a front vehicle as an RoI.

[0096] The mask generator 230 may output mask data MK based on the RoI information RID including the depth value of the RoI. Figure 7 The mask data MK is described in detail.

[0097] Buffer 410 may receive image feature data IF and output stitched data CD. For example, image feature data IF may include feature values ​​of multiple masked images IMK masked according to distance. For example, image feature data IF may include first image feature data and second image feature data. The first image feature data may include feature values ​​representing objects at a relatively close distance. The second image feature data may include feature values ​​representing objects at a relatively far distance. Buffer 410 may stitch the first image feature data and the second image feature data together and output the stitched data CD to detector 510. Detector 510 may identify objects included in input image IM based on the stitched data CD.

[0098] Figure 7 and Figure 8 The RoI information and mask data according to an exemplary embodiment of the present invention are shown. Figures 4 to 6 's reference numerals.

[0099] Reference Figure 7 and Figure 8 , the RoI network 220 may generate the ROI information RID based on the input image IM and / or the depth map DP. As described above, the depth map DP may be acquired by the pre-processor 210, the sensor unit 100, or the depth generator 250. For example, the RoI information RID may include the depth value of the depth map DP. More specifically, the RoI information RID may include the depth value of the RoI RI. Figure 7 and Figure 8 The RoI information RID may include respective depth values ​​of a third vehicle V3 closest to the sensor unit 100, a second vehicle V2 farther than the third vehicle V3, and a first vehicle V1 farther than the second vehicle V2. In addition, the RoI information RID may include a depth value of the road.

[0100] The mask generator 230 may generate a plurality of mask data, for example, first to fourth mask data MK1 to MK4 based on the RoI information RID. The mask generator 230 may divide the RoI information RID for each specific depth portion and generate a plurality of mask data MK1 to MK4 based on the divided RoI information RID.

[0101] According to example embodiments of the present inventive concepts, the average value of the depth values ​​included in the first mask data MK1 may be greater than the average value of the depth values ​​included in the second mask data MK2. As another example, the depth values ​​included in the first mask data MK1 may be greater than the depth values ​​included in the second mask data MK2. As another example, some depth values ​​included in the first mask data MK1 may be greater than the depth values ​​included in the second mask data MK2.

[0102] According to an example embodiment of the present inventive concepts, the first mask data MK1 may include an object farthest from the sensor unit 100 (e.g., a vanishing point or a horizon). The second mask data MK2 may include an object closer than the object included in the first mask data MK1 (e.g., a first vehicle V1). The third mask data MK3 may include an object closer than the object included in the second mask data MK2 (e.g., a second vehicle V2). The fourth mask data MK4 may include an object closer than the object included in the third mask data MK3 (e.g., a third vehicle V3).

[0103] Reference Figure 8, the RoI network 220 may generate RoI information RID including the depth value of the vehicle and the depth value of the road or including image information of the vehicle and the road. The mask generator 230 may generate the first mask data MK1 so that it does not include a depth value greater than a threshold depth value. In other words, the mask generator 230 may provide mask data MK2 to MK4 in addition to the first mask data MK1. In some cases, the front elevation view captured by the sensor unit 100 may not include a vanishing point. For example, Figure 8 As shown, the building may be located at a specific distance from the sensor unit 100. Therefore, in some cases, there is no image or distance information at a distance greater than the specific distance. Therefore, the mask generator 230 may not generate the first mask data MK1.

[0104] According to an exemplary embodiment of the present invention, the mask generator 230 may generate edge information corresponding to some areas of the input image IM based on the RoI information RID. The plurality of mask data MK1 to MK4 may not include depth values ​​but include edge information. For example, the edge information may indicate the depth of the image. Figure 7 and Figure 8 The outlines (dashed lines) of the mask data MK1 to MK4 shown in FIG. 2 represent boundary information. The masking unit 240 may apply the mask data MK1 to MK4 including edge information to the pyramid image PID and generate a masked image IMK. Figure 9 The generation of the masked image IMK is described in detail.

[0105] Figure 9 The operation of masking a pyramid image and generating a masked image according to an exemplary embodiment of the present invention is shown. Figures 4 to 6 's reference numerals.

[0106] According to example embodiments of the present inventive concepts, the masking unit 240 may apply first to fourth mask data MK1 to MK4 to the first to fourth pyramid images PIM1 to PIM4, respectively. The masking unit 240 may mask partial regions of the first to fourth pyramid images PIM1 to PIM4 based on the first to fourth mask data MK1 to MK4, respectively. In this case, the scenes included in the first to fourth pyramid images PIM1 to PIM4 may be substantially the same. Therefore, the sizes of the first to fourth pyramid images PIM1 to PIM4 may be different from each other. The masking unit 240 may generate first to fourth masked images IMK1 to IMK4 in which the first to fourth pyramid images PIM1 to PIM4 are masked, respectively. The first to fourth masked images IMK1 to IMK4 may indicate different scenes. For example, the first masked image IMK1 may include the scene at the farthest distance. Additionally, the fourth masked image IMK4 may include the scene at the closest distance. The first to fourth masked data MK1 to MK4 may include edge information. The masking unit 240 may represent partial areas of the first to fourth pyramid images PIM1 to PIM4 and mask the remaining areas. For example, the masking unit 240 may generate a first masked image IMK1 including a context inside the edge based on edge information included in the first mask data MK1.

[0107] The first masked image IMK1 includes a first region C1, where the first region C1 is part of the first pyramid image PIM1. The second masked image IMK2 includes a second region C2, where the second region C2 is part of the second pyramid image PIM2. The third masked image IMK3 includes a third region C3, where the third region C3 is part of the third pyramid image PIM3. The fourth masked image IMK4 includes a fourth region C4, where the fourth region C4 is part of the fourth pyramid image PIM4. Each of the plurality of regions C1 to C4 may be a partial region of the pyramid image PIM that is not masked by the plurality of mask data MK1 to MK4.

[0108] According to an exemplary embodiment of the present inventive concept, the feature extractor 300 may extract feature values ​​for the first region C1 from the first pyramid image PIM1 and skip feature extraction for the remaining regions. For example, the first masked image IMK1 may be an image representing the first region C1 of the first pyramid image PIM1. The feature extractor 300 may extract features from the first region C1 and skip extracting features from the remaining regions of the first pyramid image PIM1, even if data for the remaining regions exists.

[0109] Therefore, according to an embodiment of the present inventive concept, an image processing method may include: receiving input data, the input data including image data and distance data corresponding to the image data; generating a plurality of downsampled images (e.g., pyramid images PIM to PIM4) based on the image data, wherein each of the plurality of downsampled images corresponds to a different image resolution; generating a plurality of image masks (e.g., mask data MK1 to MK4) based on the distance data, wherein each of the plurality of image masks corresponds to a different average distance value; generating a plurality of masked images (e.g., masked images IMK1 to IMK4), wherein each of the plurality of masked images is based on one of the plurality of downsampled images and one of the plurality of image masks; generating feature data based on the plurality of masked images; and detecting an object based on the feature data.

[0110] For example, the method can identify an RoI that includes a road on which a vehicle is traveling, and the detected objects can be other vehicles traveling on that road. By downsampling the image data, a higher resolution can be used for areas where increasing the resolution can improve object detection (e.g., for larger distances in the RoI), and a reduced resolution can be used to identify closer objects in the RoI. This can reduce the processing power required to achieve a desired level of real-time object detection.

[0111] Figure 10A 、 Figure 10B and Figure 10C The image processing operation of the preprocessor according to an exemplary embodiment of the present invention is shown. Figures 4 to 6 's reference numerals.

[0112] Reference Figure 10A 、 Figure 10B and Figure 10C, the sensor unit 100 may capture a frontal view of the front and output an input image IM. The input image IM may include a first input image IMa and a second input image IMb. The first input image IMa and the second input image IMb may be obtained by capturing different frontal views of the front. For example, the sensor unit 100 may capture a frontal view of the front that includes a vanishing point. The sensor unit 100 may also generate the first input image IMa. The sensor unit 100 may also capture a frontal view of the front that does not include a vanishing point and generate the second input image IMb. The sky in the first input image IMa and the buildings in the second input image IMb may not be elements used for vehicle driving. Therefore, the RoI network 220 may receive the second input image IMb, delete the data of the non-RoI area (e.g., the area including the building), and generate a corrected image IMc. As another example, the RoI network 220 may receive the first input image IMa, delete the data of the non-RoI area (e.g., the area including the sky), and generate another corrected image. In this case, the computational complexity of the electronic device 3 for image processing can be reduced. In addition, the electronic device 3 can accurately recognize objects used for driving and prevent recognition of unnecessary objects (for example, buildings in the second input image IMb) in advance, thereby enabling driving stability.

[0113] Figure 11A and Figure 11B is a block diagram for describing a feature extractor and a kernel unit according to an exemplary embodiment of the present invention. Figures 4 to 6 and Figure 9 's reference numerals.

[0114] Reference Figure 11A , the electronic device 3 may include a core unit CR, and the core unit CR may include a plurality of cores, such as a first core CR1 to a fourth core CR4. The plurality of cores CR1 to CR4 may include a first sub-feature extractor 301 to a fourth sub-feature extractor 304, respectively. In addition, the core unit CR may include a detector 510. The first sub-feature extractor 301 to the fourth sub-feature extractor 304 may include the above reference Figures 4 to 6 The feature extractor 300 described above may be implemented in a processor such as a CPU or NPU. For example, each of the plurality of cores CR1 to CR4 may be included in a computing processing unit such as a CPU or NPU. The core unit CR may be a unit group including a plurality of cores. For example, each of the plurality of cores CR1 to CR4 may perform arithmetic operations using an artificial neural network. Furthermore, each of the first to fourth sub-feature extractors 301 to 304 and the detector 510 may be an algorithm executed by the core unit CR.

[0115] The first through fourth sub-feature extractors 301 through 304 can each output feature data FD for a plurality of masked images IMK. For example, the first sub-feature extractor 301 can generate and output first feature data FD1 based on the first masked image IMK1. Furthermore, the second sub-feature extractor 302 can generate and output second feature data FD2 based on the second masked image IMK2. The third and fourth sub-feature extractors 303 and 304 can also generate and output third and fourth feature data FD3 and FD4, respectively, in a manner similar to that of the first sub-feature extractor 301.

[0116] The first masked image IMK1 and the second masked image IMK2 may be based on partial regions of the input image IM. For example, the first masked image IMK1 may be an image obtained by masking the region remaining after excluding the first region C1 from the first pyramid image PIM1 using the first mask data MK1. The first pyramid image PIM1 may be obtained by downsampling the input image IM. Furthermore, the second masked image IMK2 may be an image obtained by masking the region remaining after excluding the second region C2 from the second pyramid image PIM2. Masking may be performed using the second mask data MK2. The second pyramid image PIM2 may be obtained by downsampling the first pyramid image PIM1 or the input image IM. Therefore, the first feature data FD1 may include feature values ​​of the first region C1, and the second feature data FD2 may include feature values ​​of the second region C2.

[0117] As described above, the first sub-feature extractor 301 may output first feature data FD1 including feature values ​​of the first region C1 to the detector 510. The second sub-feature extractor 302 may output second feature data FD2 including feature values ​​of the second region C2 to the detector 510. The first region C1 and the second region C2 may each include feature values ​​of an image captured in different contexts. The first core CR1 and the second core CR2 may each process feature values ​​of the different contexts in a distributed manner.

[0118] Reference Figure 11B , the first core CR1 may not receive the first masked image IMK1. Figure 8 As described above, mask generator 230 may not generate first mask data MK1. Therefore, masking unit 240 may not output a first masked image IMK1, where the first masked image IMK1 includes an image region that is farther than a specific distance. For example, the image region that is farther than the specific distance may be first region C1. For example, first region C1 may include a vanishing point. As another example, first region C1 may include the farthest object in the front elevation view.

[0119] In addition, the first core CR1 and the second core CR2 can share the second sub-feature extractor 302. The second sub-feature extractor 302 can output the second feature data FD2 to the detector 510 driven by the first core CR1. As a result, the first core CR1 can detect an object included in the second masked image IMK2 based on the second feature data FD2, wherein the second feature data is feature data about the second masked image IMK2.

[0120] Figure 12 3 is a block diagram of an electronic device 3 according to an exemplary embodiment of the present invention. Figures 4 to 6 's reference numerals.

[0121] Reference Figure 12 , the electronic device 3 may include a feature extractor 300, multiple buffers, multiple downsamplers, a detector 510, and a merging unit 600. An example of the multiple buffers may be the first buffer 411 and the second buffer 412. In addition, an example of the multiple downsamplers may be the first downsampler 421 and the second downsampler 422. The feature extractor 300 may also include multiple image feature extractors and multiple depth feature extractors. An example of the multiple image feature extractors may be the first image feature extractor 311 and the second image feature extractor 312. In addition, an example of the multiple depth feature extractors may be the first depth feature extractor 321 and the second depth feature extractor 322. The detector 510 may also include multiple sub-detectors, such as a first sub-detector 511 and a second sub-detector 512. The first feature data FD1 may include first image feature data IF_1 and first depth feature data DF_1. The second feature data FD2 may include second image feature data IF_2 and second depth feature data DF_2.

[0122] According to example embodiments of the inventive concepts, the first image feature extractor 311 may output first image feature data IF_1 of the first masked image IMK1 , and the second image feature extractor 312 may output second image feature data IF_2 of the second masked image IMK2 .

[0123] The first downsampler 421 may downsample the second image feature data IF_2 according to a specific ratio or a specific value. The first downsampler 421 may then output first downsampled feature data IF_2D. The first buffer 411 may concatenate the first feature data FD1 and the first downsampled feature data IF_2D to output first concatenated data CD1.

[0124] The first sub-detector 511 may receive first concatenated data CD1 based on the first feature data FD1, and the second sub-detector 512 may receive second concatenated data CD2 based on the second feature data FD2. The first sub-detector 511 and the second sub-detector 512 may output detection data IC1 and IC2 to the merging unit 600.

[0125] Merging unit 600 can identify information about an object based on detection data IC1 and IC2. Because partial areas of the plurality of mask data MK overlap, objects included in these partial areas can be repeatedly detected from the plurality of masked images IMK. Detection data IC1 and IC2 may include information about repeatedly detected objects. Therefore, merging unit 600 can remove a portion of the data regarding the repeatedly detected objects and then merge detection data IC1 and IC2. Merging detection data IC1 and IC2 allows merging unit 600 to identify information about an object.

[0126] Figure 13 is a block diagram of an electronic device 3 for describing a sensor fusion operation according to an example embodiment of the inventive concept.

[0127] Reference Figure 13 , the electronic device 3 may include a detector 510, a second sensor unit 520, a sensor fusion unit 530, and a merging unit 600. The detector 510 may further include a plurality of sub-detectors, such as a first sub-detector 511 to a fourth sub-detector 514. The second sensor unit 520 may further include a plurality of distance sensors, such as a first distance sensor 521 to a fourth distance sensor 524. The sensor fusion unit 530 may further include a plurality of sensor fusion units, such as a first sensor fusion unit 531 to a fourth sensor fusion unit 534.

[0128] According to an exemplary embodiment of the present inventive concept, the detector 510 may output a plurality of detection data, for example, first detection data IC1 to fourth detection data IC4. The plurality of distance sensors 521 to 524 may respectively output a plurality of sensing data, for example, first sensing data SD1 to fourth sensing data SD4. The first distance sensor 521 may have a longer effective sensing range than the second distance sensor 522. For example, the first distance sensor 521 may be a long-range RADAR sensor, while the second distance sensor 522 may be a short-range RADAR sensor. The second distance sensor 522 may have a longer effective sensing range than the third distance sensor 523. For example, the third distance sensor 523 may be a LIDAR sensor. The third distance sensor 523 may have a longer effective sensing range than the fourth distance sensor 524. For example, the fourth distance sensor 524 may be a ToF sensor.

[0129] The first sensor fusion unit 531 can fuse the first detection data IC1, the first sensed data SD1, and the second sensed data SD2. The first detection data IC1 can be detected by the first sub-detector 511. The first sensed data SD1 can be output from the first distance sensor 521. The second sensed data SD2 can be output from the second distance sensor 522. For example, the first sensed data SD1 and the second sensed data SD2 can be used to supplement the insufficient depth information in the first detection data IC1. As a result, the electronic device 3 can accurately identify an object. The sensor fusion unit 530 can output a plurality of fused data SF1 to SF4 based on the fused data. The second sensor fusion unit 532 and the third sensor fusion unit 533 are similar to those described above. Therefore, their descriptions are omitted.

[0130] The fourth sensor fusion unit 534 may fuse the fourth detection data IC4 detected by the fourth sub-detector 514 with the fourth sensing data SD4 output from the fourth distance sensor 524. Unlike the first sensor fusion unit 531, the fourth sensor fusion unit 534 may use distance information output from one distance sensor (e.g., the fourth distance sensor 524). For example, the sensor fusion unit 530 may fuse sensing data output from at least one distance sensor with detection data.

[0131] The first detection data IC1 may be generated based on an image including an object at a greater distance than the second detection data IC2. For example, the first detection data IC1 may be based on feature values ​​extracted from the first masked image IMK1. The second detection data IC2 may be based on feature values ​​extracted from the second masked image IMK2. As described above, the first masked image IMK1 may include an object at a relatively greater distance than the second masked image IMK2.

[0132] The merging unit 600 can obtain various information about an object (e.g., another vehicle) included in the input image IM based on the plurality of fused data SF1 to SF4. For example, the merging unit 600 can obtain information about the object's 3D information, distance to the object, speed, type of the object, etc. The merging unit 600 can provide the obtained information to internal or external components of the electronic device 3.

[0133] The merging unit 600 can identify information about an object based on the multiple fused data SF1 to SF4. Because partial areas of the multiple mask data MK overlap, objects included in these partial areas can be repeatedly detected from the multiple masked images IMK. The first detection data IC1 to the fourth detection data IC4 can include information about repeatedly detected objects. The multiple fused data SF1 to SF4 can be generated based on the first detection data IC1 to the fourth detection data IC4. Therefore, the merging unit 600 can remove a portion of the data about the repeatedly detected objects and then merge the multiple fused data SF1 to SF4. The merging unit 600 can identify information about the object by merging the multiple fused data SF1 to SF4.

[0134] Figure 14 is a flowchart of an operating method of an electronic device according to an exemplary embodiment of the present invention. Figures 4 to 6 and the reference numerals of FIG. 11 .

[0135] According to an exemplary embodiment of the present inventive concept, in operation S710, the electronic device 3 may downsample the input image IM and generate a first pyramid image PIM1. In operation S720, the electronic device 3 may downsample the first pyramid image PIM1 and generate a second pyramid image PIM2. This embodiment is not limited thereto. The electronic device 3 may downsample the input image IM and generate a second pyramid image PIM2.

[0136] In operation S730, the electronic device 3 may mask the first pyramid image PIM1 based on the first mask data MK1. For example, the electronic device 3 may mask the region remaining by excluding a partial region (first region) in which the average distance in the first pyramid image PIM1 is a first value.

[0137] In operation S740, the electronic device 3 may mask the second pyramid image PIM2 based on the second mask data MK2. For example, the electronic device 3 may mask the remaining region by excluding a partial region (second region) having an average distance smaller than the first value in the second pyramid image PIM2.

[0138] In operation S750 , the electronic device 3 may acquire a plurality of feature data FD from a plurality of masked images IMK generated based on the masking operation (eg, S730 and S740 ).

[0139] In operation S760, the electronic device 3 may detect objects outside the electronic device 3 based on the plurality of feature data FD. For example, the electronic device 3 may detect objects in the first area based on the first feature data FD1 including feature values ​​of the first area. Furthermore, the electronic device 3 may detect objects in the second area based on the second feature data FD2 including feature values ​​of the second area. For example, the electronic device 3 may identify objects at a relatively long distance based on the first feature data FD1 extracted from the first masked image IMK1. Conversely, the electronic device 3 may identify objects at a relatively close distance based on the second feature data FD2 extracted from the second masked image IMK2.

[0140] Figure 15 is a flowchart of an operating method of an electronic device according to an exemplary embodiment of the present invention. Figures 4 to 6 and Figure 9 's reference numerals.

[0141] According to an exemplary embodiment of the present inventive concept, in operation S721, the sensor unit 100 may acquire a depth map DP related to an object and background of a front view. For example, the sensor unit 100 may acquire a 2D image and acquire the depth map DP based on the 2D image. As another example, the sensor unit 100 may acquire a 3D image and acquire the depth map DP.

[0142] In operation S722, the electronic device 3 may generate a plurality of mask data MK having different average depths based on at least one of the input image IM and the depth map DP. For example, the plurality of mask data MK may include first mask data MK1 and second mask data MK2, where the average depth of the second mask data MK2 is less than the average depth of the first mask data MK1. For example, the depth map DP may be acquired using at least one of a stereo camera, a mono camera, and a range sensor. As a result, a plurality of mask data having different average depths may be generated.

[0143] In operation S731, the electronic device 3 may mask the first pyramid image PIM1 based on the first mask data MK1. The electronic device 3 may generate a first masked image IMK1 by masking the first pyramid image PIM1. In operation S732, the electronic device 3 may obtain first feature data FD1 from the first masked image IMK1. In operation S741, the electronic device 3 may mask the second pyramid image PIM2 based on the second mask data MK2 having an average depth less than the average depth of the first mask data MK1. The electronic device 3 may generate a second masked image IMK2 by masking the second pyramid image PIM2. In operation S742, the electronic device 3 may obtain second feature data FD2 from the second masked image IMK2. For example, the first mask data MK1 and the second mask data MK2 may be resized to correspond to the first pyramid image PIM1 and the second pyramid image PIM2, respectively.

[0144] Figure 16 is a flowchart of an operating method of an electronic device according to an exemplary embodiment of the present invention. Figures 4 to 6 、 Figure 11A 、 Figure 12 and Figure 13 's reference numerals.

[0145] According to an exemplary embodiment of the present inventive concept, in operation S761, the electronic device 3 may receive stitching data CD based on first feature data FD1 and second feature data FD2. The first feature data FD1 may include feature values ​​of the first masked image IMK1. The second feature data FD2 may include feature values ​​of the second masked image IMK2.

[0146] The electronic device 3 may detect an object in the input image IM based on the stitching data CD in operation S762, and output detection data in operation S763. The detection data may include the above reference data. Figure 12 The first detection data IC1 and the second detection data IC2 are described.

[0147] In operation S771, the electronic device 3 may receive first sensing data SD1 from the first distance sensor 521, and in operation S772, receive second sensing data SD2 from the second distance sensor 522, which has a shorter effective sensing distance than the first distance sensor 521. Thereafter, in operation S773, the electronic device 3 may fuse at least one of the plurality of detection data IC1 to IC4 with at least one of the plurality of sensing data SD1 to SD4. For example, the electronic device 3 may fuse the first detection data IC1 with the first sensing data SD1, and fuse the second detection data IC2 with the second sensing data SD2. In operation S774, the electronic device 3 may merge the plurality of fused data SF1 to SF4, and in operation S780, obtain 3D information of an object included in the input image IM based on the merged data.

[0148] Figure 17 is a block diagram of an electronic system according to an example embodiment of the inventive concept.

[0149] Figure 17 The illustrated electronic system may be an application processor 800. The application processor 800 may be implemented as a semiconductor chip by a system on chip (SoC).

[0150] The application processor 800 may include a processor 810 and an operating memory 820. Figure 17 Although not shown in the figure, the application processor 800 may further include one or more intellectual property (IP) modules connected to the system bus. The operating memory 820 may store software such as various programs and instructions associated with the operation of the system employing the application processor 800. In addition, the operating memory 820 may include, for example, an OS 821 and an artificial neural network module 822. The processor 810 may generally control the operating memory 820. The artificial neural network module 822 may execute the above-referenced Figures 1 to 16 Describes image processing operations and object recognition operations.

[0151] While the inventive concept has been particularly shown and described with reference to embodiments thereof, it will be understood that various changes in form and details may be made therein without departing from the spirit and scope of the appended claims.

Claims

1. A driving assistance system comprising: a first sensor unit configured to provide an input image including other vehicles and a background; an image processor configured to generate a plurality of pyramid images by downsampling the input image, identify a depth map including depth values ​​to the other vehicles and the background, identify a region of interest (RoI) based on the input image and output RoI information about the RoI, generate a plurality of mask data having different average depths of the depth values ​​based on the RoI information and the depth map, and output a plurality of masked images based on the plurality of mask data and the plurality of pyramid images, the plurality of masked images representing gradually decreasing average distances corresponding to gradually decreasing sizes of the plurality of pyramid images, respectively; a feature extractor configured to output feature data for each of the plurality of masked images; as well as A detector is configured to detect the other vehicle included in the input image based on the feature data.

2. The driving assistance system according to claim 1, wherein: The first sensor unit includes a stereo camera, The input image includes a left-eye image and a right-eye image, and The image processor is further configured to obtain the depth map based on a comparison of the left-eye image and the right-eye image.

3. The driving assistance system according to claim 2, wherein: The image processor is further configured to mask an area remaining by excluding the RoI from the plurality of pyramid images.

4. The driving assistance system according to claim 1, wherein: The plurality of pyramid images include a first pyramid image and a second pyramid image, and The image processor is further configured to generate the first pyramid image by downsampling the input image, and to generate the second pyramid image by downsampling the first pyramid image.

5. The driving assistance system according to claim 4, wherein: The plurality of mask data include first mask data and second mask data having an average depth smaller than that of the first mask data, The plurality of masked images include a first masked image and a second masked image, and The image processor is further configured to resize the first mask data and the second mask data to correspond to the first pyramid image and the second pyramid image, output the first masked image in which the first mask data is applied to the first pyramid image, and output the second masked image in which the second mask data is applied to the second pyramid image. The driving assistance system according to claim 1 , wherein: The image processor is further configured to mask an area remaining by excluding at least one of the other vehicles, the road, and the pedestrian from the input image, and generate the plurality of pyramid images based on the masked image.

7. The driving assistance system according to claim 1, wherein: the plurality of masked images comprising a first masked image and a second masked image, The feature extractor is further configured to output first feature data generated based on the first masked image and second feature data generated based on the second masked image, and The detector is further configured to receive stitching data generated by stitching the first feature data and the second feature data.

8. The driving assistance system according to claim 7, further comprising: a downsampler configured to downsample the second feature data and output downsampled feature data; as well as The buffer is configured to output the spliced ​​data by splicing the first feature data and the downsampled feature data.

9. The driving assistance system according to claim 1, wherein the detector further comprises a first sub-detector and a second sub-detector, and the driving assistance system further comprises: a second sensor unit including a first distance sensor and a second distance sensor having a shorter sensing distance than the first distance sensor; a first sensor fusion unit configured to fuse first detection data output from the first sub-detector and first sensing data output from the first distance sensor, and output first fused data; as well as a second sensor fusion unit configured to fuse the second detection data output from the second sub-detector and the second sensing data output from the second distance sensor, and output second fused data; Compared with the second detection data, the first detection data is generated based on an image including an object at a farther distance.

10. The driving assistance system according to claim 9, further comprising: a third distance sensor having a shorter sensing distance than that of the second distance sensor, The third distance sensor outputs third sensing data, and The second sensor fusion unit is further configured to fuse the second detection data, the second sensed data, and the third sensed data.

11. The driving assistance system according to claim 9, further comprising: A merging unit is configured to acquire three-dimensional information of the other vehicle based on the first fused data and the second fused data.

12. An electronic device for detecting an object from an input image, the electronic device comprising: The image processor is configured as follows: Generate the first pyramid image by downsampling the input image, generating a second pyramid image smaller in size than the first pyramid image by downsampling the first pyramid image, Acquire a depth map including depth information of an object and a background based on the input image, Identifying a region of interest (RoI) based on the input image and outputting RoI information about the RoI, generating first mask data and second mask data having different average depths based on the RoI information and the depth map, identifying, by masking the first pyramid image based on the first mask data, a partial region in the first pyramid image where the average distance indicates a first value as a first region, By masking the second pyramid image based on the second mask data, a partial region in the second pyramid image where the average distance indicates a second value smaller than the first value is identified as a second region, and outputting images of the first area and the second area; a first core configured to generate first feature data including a feature value of the first area, and detect an object in the first area based on the first feature data; as well as The second core is configured to generate second feature data including a feature value of the second area, and detect an object in the second area based on the second feature data.

13. The electronic device according to claim 12, wherein: The first core and the second core perform arithmetic operations using an artificial neural network.

14. The electronic device according to claim 12, wherein: The image processor is also configured not to output the image including the first area to the first core when the average distance of the images of the first area is greater than a preset distance, and the first core is also configured not to generate the first feature data when the average distance of the images of the first area is greater than the preset distance.

15. The electronic device according to claim 14, wherein The second core is further configured to output the second feature data to the first core, and The first core is further configured to receive the second feature data instead of the first feature data and detect an object in the second area.

16. The electronic device according to claim 12, wherein The first core includes a feature extractor, and The feature extractor is configured to extract features of the first region from the first pyramid image and skip feature extraction operations for remaining regions.

17. The electronic device according to claim 12, wherein: The image processor is further configured to mask a region remaining by excluding the first region from the first pyramid image, mask a region remaining by excluding the second region from the second pyramid image, and output the masked first and second pyramid images.

18. A method for operating an electronic device, the method comprising: generating a first pyramid image by downsampling the input image including the object and the background; generating a second pyramid image smaller in size than the first pyramid image by downsampling the first pyramid image; Acquire a depth map including depth information of the object and the background based on the input image; Identifying a region of interest (RoI) based on the input image, and outputting RoI information about the RoI; generating first mask data and second mask data having different average depths based on the RoI information and the depth map; identifying a first region having an average distance of a first value by masking the first pyramid image based on the first mask data, and identifying a second region having an average distance of a second value smaller than the first value by masking the second pyramid image based on the second mask data; generating first feature data including a feature value of the first area and second feature data including a feature value of the second area; as well as The object is detected based on the first feature data and the second feature data.

19. The operating method according to claim 18, wherein: An average depth of the second mask data is smaller than an average depth of the first mask data.

Images