Vanishing Point Extraction Device, Method for Extracting Vanishing Point, and Autonomous Driving Device

Abstract

A vanishing point extraction device, a method for extracting a vanishing point, and an autonomous driving device are provided. The method for extracting a vanishing point includes: obtaining a straight line including a vanishing point of a first image; processing the first image based on an object included in the first image and the straight line including the vanishing point of the first image to obtain a plurality of sampling points in the first image, such that the plurality of sampling points are determined as pixels whose coordinates in the first image overlap with the coordinates of pixels of the straight line and the coordinates of pixels of the object included in the first image; obtaining at least one matching point corresponding to at least one sampling point among the plurality of sampling points in the first image in a second image, the second image being generated after the first image is generated; and obtaining a vanishing point of the second image based on the at least one matching point of the second image.

Description

[0001] Cross - reference to related applications

[0002] This application claims the benefit of Korean Patent Application No. 10 - 2020 - 0018571, filed on Feb. 14, 2020, with the Korean Intellectual Property Office, the entire disclosure of which is incorporated herein by reference. Technical field

[0003] The present disclosure relates to a vanishing point extraction apparatus and a vanishing point extraction method, and more particularly, to a vanishing point extraction apparatus and a vanishing point extraction method for extracting a vanishing point of a current image based on information about a vanishing point of a previous image and information about an object included in the previous image. Background art

[0004] An image capturing apparatus using an image sensor may be included in various types of electronic devices (such as a smart phone, a PC, a surveillance camera, and a vehicle), or may be used as a single stand - alone electronic device.

[0005] In the case of an autonomous driving vehicle, safe driving may be performed by the following operations: obtaining an image of the surrounding environment of the autonomous driving vehicle through an image sensor, determining the surrounding conditions using the obtained image, and controlling the autonomous driving vehicle according to the determination result. For example, an autonomous driving vehicle may extract a vanishing point from an image through a vanishing point extraction apparatus, estimate the distance between vehicles using the extracted vanishing point, or detect the sway of the vehicle.

[0006] In the case of a vanishing point extraction apparatus according to a conventional method, lanes in an image are detected, and the detected lanes are extended along straight lines to check an intersection point where they cross each other as the vanishing point. Summary of the invention

[0007] The present disclosure provides a vanishing point extraction apparatus and a vanishing point extraction method that can extract a vanishing point of a current image based on a vanishing point of a previous image and information about an object included in the previous image even when lanes are not detected in the current image. Accordingly, even when there are no lanes around and / or even when there are lanes but the lanes are not detected in the image, the vanishing point extraction apparatus according to some example embodiments can extract a vanishing point.

[0008] According to some example embodiments, a method for performing vanishing point extraction may include: obtaining a straight line including the vanishing point of the first image based on processing the first image; obtaining a plurality of sampling points in the first image based on processing the first image according to an object included in the first image and the straight line including the vanishing point of the first image, such that the plurality of sampling points are determined as pixels whose coordinates in the first image overlap with the coordinates of the pixels of the straight line and the coordinates of the pixels of the object included in the first image; obtaining at least one matching point corresponding to at least one of the plurality of sampling points in the first image in a second image, the second image being generated after the first image is generated; and obtaining the vanishing point of the second image based on the at least one matching point of the second image.

[0009] The obtaining of the at least one matching point may include: obtaining a plurality of first templates, each of the first templates corresponding to a respective one of the plurality of sampling points in the first image; obtaining at least one second template in the second image, wherein the at least one second template is determined to be similar to at least one of the plurality of first templates; and obtaining pixels in the second image associated with the at least one second template as the at least one matching point.

[0010] The obtaining of the plurality of first templates may include: obtaining regions having a specific size as the plurality of first templates, wherein the regions respectively include respective ones of the plurality of sampling points.

[0011] The obtaining of the at least one second template may include: obtaining a plurality of search regions in the second image, the plurality of search regions respectively corresponding to respective ones of the plurality of sampling points in the first image; and for each sampling point, obtaining a candidate region similar to the first template corresponding to the sampling point in the search region corresponding to the sampling point as a respective one of the at least one second templates.

[0012] The obtaining of the plurality of search regions may include: identifying points in the second image whose coordinates in the second image are respectively the same as the coordinates of respective ones of the plurality of sampling points in the first image; and obtaining regions in the second image having a specific size and respectively including respective ones of the identified points as the plurality of search regions.

[0013] The size of the plurality of search regions may be larger than the size of the plurality of first templates.

[0014] Obtaining the candidate region similar to the first template corresponding to the sampling point as the separate second template may include: determining a plurality of candidate regions in the search region corresponding to the sampling point; calculating the correlation values between each candidate region in the plurality of candidate regions and the first template corresponding to the sampling point in the plurality of first templates to establish a plurality of correlation values respectively corresponding to the individual candidate regions; and obtaining the candidate region corresponding to the highest correlation value among the plurality of correlation values as the separate second template.

[0015] Obtaining the pixels associated with the at least one second template in the second image as the at least one matching point may include: identifying the pixels corresponding to the center points of the at least one second template as the at least one matching point.

[0016] The straight line including the vanishing point of the first image may be a horizontal line parallel to the horizontal axis of the first image.

[0017] The object included in the first image may be a vehicle or a specific object associated with the vehicle.

[0018] The method may further include: removing the outliers in the at least one matching point, wherein obtaining the vanishing point of the second image further includes: obtaining the vanishing point of the second image based on the at least one matching point from which the outliers have been removed.

[0019] Removing the outliers in the at least one matching point may include: using at least one of a random sample consensus (RANSAC) model, a progressive sample consensus (PROSAC) model, and a stable random sample consensus (StaRSaC) model to remove the outliers.

[0020] Obtaining the vanishing point of the second image may include: correcting the vanishing point of the first image based on the at least one matching point to obtain the vanishing point of the second image.

[0021] Obtaining the vanishing point of the second image may include: obtaining the vanishing point of the second image based on correcting the y - coordinate of the vanishing point of the first image using the y - coordinates of the at least one matching point.

[0022] According to some example embodiments, a vanishing point extraction device may include an image sensor configured to generate a first image and a second image after generating the first image. The vanishing point extraction device may include a memory configured to store the first image, information associated with a vanishing point of the first image, and information associated with at least one object included in the first image. The vanishing point extraction device may include a processing circuit configured to, in response to receiving the second image from the image sensor, perform the following operations: based on the information associated with the vanishing point of the first image, identify a horizontal line including the vanishing point of the first image in the first image; based on processing the first image using the information associated with the at least one object included in the first image, obtain a plurality of sampling points in the first image such that the plurality of sampling points are determined to be pixels whose coordinates in the first image overlap with the coordinates of pixels of the at least one object and the coordinates of pixels of the horizontal line; identify at least one matching point in the second image corresponding to at least one of the plurality of sampling points in the first image; and based on correcting the vanishing point of the first image using the at least one matching point, obtain a vanishing point of the second image.

[0023] The processing circuit may further be configured to: obtain a plurality of first templates of the first image, the plurality of first templates respectively corresponding to individual sampling points among the plurality of sampling points in the first image; and obtain at least one second template of the second image, the at least one second template being similar to at least one of the plurality of first templates.

[0024] The processing circuit may be further configured to: obtain a region having a specific size as the plurality of first templates, wherein each individual region includes an individual sampling point among the plurality of sampling points.

[0025] The processing circuit may further be configured to: obtain a plurality of search regions of the second image, the plurality of search regions respectively corresponding to individual sampling points among the plurality of sampling points in the first image; and for each sampling point, obtain a candidate region similar to the first template corresponding to the sampling point in the search region corresponding to the sampling point as an individual second template among the at least one second template.

[0026] The processing circuit may further be configured to: identify points in the second image, where coordinates of the points in the second image are respectively the same as coordinates of individual points among the multiple sampling points in the first image; and obtain regions in the second image that have a specific size and respectively include individual points among the identified points as the multiple search regions.

[0027] The processing circuit may further be configured to: determine multiple candidate regions in the search region corresponding to the sampling point; calculate a correlation value between each candidate region among the multiple candidate regions and the first template corresponding to the sampling point among the multiple first templates to establish multiple correlation values respectively corresponding to individual candidate regions; and obtain the candidate region corresponding to the highest correlation value among the multiple correlation values as the individual second template.

[0028] The processing circuit may further be configured to: identify the center point of the at least one second template as the at least one matching point.

[0029] The processing circuit may further be configured to: remove outliers from the at least one matching point, and obtain the vanishing point of the second image based on the at least one matching point from which the outliers have been removed.

[0030] The processing circuit may further be configured to: remove the outliers based on at least one of a Random Sample Consensus (RANSAC) model, a Progressive Sample Consensus (PROSAC) model, and a Stable Random Sample Consensus (StaRSaC) model.

[0031] The processing circuit may further be configured to: obtain the vanishing point of the second image by correcting the y coordinate of the vanishing point of the first image based on the y coordinates of the at least one matching point.

[0032] According to some example embodiments, an autonomous driving device configured to be included in a host vehicle may include an image sensor configured to generate a first image and a second image after generating the first image. The autonomous driving device may include a memory configured to store the first image, information associated with a vanishing point of the first image, and information associated with at least one object included in the first image. The autonomous driving device may include a first processing circuit configured to, in response to receiving the second image from the image sensor, perform the following operations: identify a horizontal line including the vanishing point of the first image in the first image based on the information associated with the vanishing point of the first image, obtain a plurality of sampling points in the first image based on processing the first image using the information associated with the at least one object included in the first image, such that the plurality of sampling points are determined to be pixels in the first image where coordinates of the pixels of the at least one object and coordinates of pixels of the horizontal line overlap, identify at least one matching point in the second image corresponding to at least one of the plurality of sampling points in the first image, and obtain a vanishing point of the second image based on correcting the vanishing point of the first image using the at least one matching point. The autonomous driving device may include a second processing circuit configured to: control an operation of the host vehicle based on the information associated with the vanishing point of the second image obtained by the first processing circuit. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Example embodiments of the inventive concept will be more clearly understood from the following detailed description in conjunction with the accompanying drawings, in which:

[0034] Figure 1 is a block diagram showing a vanishing point extraction device according to some example embodiments of the present disclosure;

[0035] Figure 2 is a block diagram showing in detail a vanishing point extraction device according to some example embodiments of the present disclosure;

[0036] Figure 3 is a view showing a host vehicle including a vanishing point extraction device according to some example embodiments of the present disclosure;

[0037] Figure 4 is a view showing a first image according to some example embodiments of the present disclosure;

[0038] Figure 5 is a view showing a straight line including a vanishing point of a first image according to some example embodiments of the present disclosure;

[0039] Figure 6 is a diagram showing a plurality of sampling points of a first image according to some example embodiments of the present disclosure;

[0040] Figure 7 is a diagram showing a first template corresponding to the sampling points according to some example embodiments of the present disclosure;

[0041] Figure 8 is a diagram showing a second image that is the next frame of the first image according to some example embodiments of the present disclosure;

[0042] Figure 9 is a diagram showing a search area of the second image according to some example embodiments of the present disclosure;

[0043] Figure 10 is a diagram for explaining a patchmatch operation for the second image according to some example embodiments of the present disclosure;

[0044] Figure 11 is a diagram showing a second template corresponding to the sampling points according to some example embodiments of the present disclosure;

[0045] Figure 12 is a diagram showing matching points of the second template according to some example embodiments of the present disclosure;

[0046] Figure 13 is a diagram showing matching points of the second image according to some example embodiments of the present disclosure;

[0047] Figure 14 is a diagram showing a plurality of matching points of the second image according to some example embodiments of the present disclosure;

[0048] Figure 15 is a diagram showing a vanishing point of the second image according to some example embodiments of the present disclosure;

[0049] Figure 16 is a diagram showing a plurality of matching points of the second image according to some example embodiments of the present disclosure;

[0050] Figure 17 is a diagram showing a vanishing point of the second image according to some example embodiments of the present disclosure;

[0051] Figure 18 is a diagram showing a plurality of sampling points of a first image according to some example embodiments of the present disclosure;

[0052] Figure 19is a flowchart showing a vanishing point extraction method according to some example embodiments of the present disclosure; and

[0053] Figure 20 is a block diagram showing an autonomous driving device according to some example embodiments of the present disclosure. Detailed Description

[0054] Figure 1 is a block diagram showing a vanishing point extraction device according to some example embodiments of the present disclosure.

[0055] Referring to Figure 1 , the vanishing point extraction device 10 may include an image sensor 100, a memory 200, and a processor 300. In addition, the processor 300 may include a vanishing point extractor 310.

[0056] The vanishing point extraction device 10 is a device that captures an image, analyzes the captured image, and extracts the vanishing point of the image based on the analysis result. The vanishing point extraction device 10 may be implemented as a personal computer (PC), an Internet of Things (IoT) device, or a portable electronic device. The portable electronic device may include various devices such as: a laptop computer, a mobile phone, a smart phone, a tablet PC, a personal digital assistant (PDA), an enterprise digital assistant (EDA), a digital camera, a digital video camera, an audio device, a portable multimedia player (PMP), a personal navigation device (PND), an MP3 player, a handheld game console, an e-book, and a wearable device.

[0057] The image sensor 100 is embedded in the vanishing point extraction device 10 and is configured to receive (e.g., generate, capture, etc.) an image signal of the environment around the vanishing point extraction device 10 and output (e.g., send) the received image signal as an image. Such an image may be understood to be generated, captured, etc. by the image sensor 100. For example, the image sensor 100 may generate an image by converting (e.g., based on) the light received from the front or in various directions from the external environment into electrical energy (e.g., an electrical signal) and output the generated image to the processor 300.

[0058] The memory 200 (e.g., a solid state drive (SSD)) is a storage location (e.g., a non-transitory computer-readable storage medium) for storing data, and can store data generated by the image sensor 100 (e.g., an image generated by the image sensor 100) and various data generated in the processing executed by the processor 300. For example, the memory 200 can store an image obtained by the image sensor 100 (e.g., an image generated by and / or received from the image sensor 100). In addition, the memory 200 can store information about the vanishing point of the image (e.g., information associated with the vanishing point of the image), information about an object included in the image (e.g., information associated with the object included in the image), etc., as described later in connection with the operation of the processor 300.

[0059] The processor 300 can control the overall operation of the vanishing point extraction device 10. The processor 300 can include one or more instances of processing circuitry such as the following, can be included in one or more instances of processing circuitry such as the following, and / or can be implemented by one or more instances of processing circuitry such as the following: hardware including logic circuits; a hardware / software combination such as a processor running software; or a combination thereof. For example, the processing circuitry can more specifically include, but is not limited to, a central processing unit (CPU), an arithmetic logic unit (ALU), a graphics processing unit (GPU), an application processor (AP), a digital signal processor (DSP), a microcomputer, a field programmable gate array (FPGA), and a programmable logic unit, a microprocessor, an application specific integrated circuit (ASIC), a neural network processing unit (NPU), an electronic control unit (ECU), an image signal processor (ISP), etc. In some example embodiments, the processing circuitry can include: a non-transitory computer-readable storage device (e.g., a solid state drive (SSD)) storing an instruction program, and a processor configured to run the instruction program to implement some or all of the functions and / or methods performed by the vanishing point extraction device 10, including some or all of the functions and / or methods performed by the processor 300 (e.g., the functions and / or methods performed by the vanishing point extractor 310).

[0060] The processor 300 can extract a vanishing point through the vanishing point extractor 310, for example, based on processing an image. For example, the processor 300 can receive and / or obtain an image (e.g., an image generated by the image sensor 100) from the image sensor 100, and in response to such reception and / or based on processing the image, extract the vanishing point of the received image through the vanishing point extractor 310 (e.g., based on implementing the function of the vanishing point extractor 310 to process the image). The vanishing point extractor 310 can be implemented in firmware or software, and can be loaded into the memory 200 and run by the processor 300. However, the present disclosure is not limited thereto, and the vanishing point extractor 310 can be implemented in hardware. In this document, the elements obtained or recognized as "of the image" and the elements obtained or recognized as "in the image" can be used interchangeably.

[0061] When receiving the current image (also referred to as the second image IMG2 in this document) from the image sensor 100 (e.g., in response to such reception), the vanishing point extractor 310 can use the previous image (also referred to as the first image IMG1 in this document) to extract the vanishing point of the current image, and the previous image can be an image previously generated by the image sensor 100. Here, the previous image can include the images received from the image sensor 100 before receiving the current image. To reiterate, the image sensor 100 can generate the current image (e.g., the second image IMG2) after generating the previous image (e.g., the first image IMG1). In some example embodiments, the current image (e.g., the second image IMG2) can be the next image generated by the image sensor 100 after the image sensor 100 generates the previous image (e.g., the first image IMG1), and the image sensor 100 does not generate intermediate images between generating the previous image and generating the current image. In some example embodiments, the image sensor 100 can generate one or more intermediate images between generating the previous image and generating the current image. Hereinafter, for ease of description, the current image is referred to as the second image IMG2, and the previous image is referred to as the first image IMG1.

[0062] In some example embodiments, when receiving the second image IMG2 from the image sensor 100 (e.g., in response to the reception), in order to extract the vanishing point of the received second image IMG2, the vanishing point extractor 310 can receive (e.g., access, obtain, etc.) the first image IMG1 and the information Info_IMG1 about the first image IMG1 (e.g., associated with the first image IMG1) from the memory 200.

[0063] In this document, the information Info_IMG1 regarding the first image IMG1 may include: information about the vanishing point extracted from the first image IMG1 (e.g., the vanishing point previously extracted by the vanishing point extractor 310) (e.g., information associated with the vanishing point extracted from the first image IMG1), and information about the objects included in the first image IMG1 (e.g., information associated with the objects included in the first image IMG1). For example, the information about the vanishing point extracted from the first image IMG1 may include the coordinates of the vanishing point of the first image IMG1 (e.g., the coordinates of the vanishing point within the first image IMG1), and the information about the objects included in the first image IMG1 may include the coordinates (e.g., the coordinates within the first image IMG1) of the object regions that are recognized as the result of performing object recognition on the first image IMG1 (e.g., the regions where the objects in the first image IMG1 are located within the first image IMG1).

[0064] Then, the vanishing point extractor 310 obtains (e.g., identifies) a straight line including the vanishing point from the first image IMG1 by (e.g., based on) using the first image IMG1 and the information about the vanishing point of the first image IMG1 (e.g., processing the first image IMG1 based on the information about the vanishing point of the first image IMG1); and the vanishing point extractor 310 may, based on the obtained straight line, obtain (e.g., identify) multiple sampling points from the first image IMG1 by (e.g., based on) using the information about the objects included in the first image IMG1 (e.g., processing the first image IMG1 based on the information about the objects included in the first image IMG1, the information associated with the obtained straight line, and / or the information about the vanishing point of the first image IMG1), where each sampling point intersects both the straight line and the object region simultaneously.

[0065] In addition, the vanishing point extractor 310 may obtain (e.g., identify) at least one matching point corresponding to at least one of the multiple sampling points of the first image IMG1 from the second image IMG2 by (e.g., based on) comparing the first image IMG1 with the second image IMG2 (e.g., based on identifying a region in the second image IMG2 that is similar to the first template of the first image IMG1 and determining the pixel at the center point of the region as the matching point). In addition, the vanishing point extractor 310 may extract the vanishing point of the second image IMG2 based on the obtained at least one matching point. The specific implementation will be described later with reference to Figure 2 describe its specific implementation.

[0066] The vanishing point extraction device 10 according to the inventive concept of the present disclosure may use information on the vanishing point of a previous image and an object identified in the previous image to extract the vanishing point of a current image. That is, even when there is no lane in the current image (e.g., a lane of a road in the environment imaged in the current image and / or a lane marking indicating one or more boundaries of the lane) or when there is a lane in the current image but the lane is not detected, the vanishing point extraction device 10 according to the inventive concept of the present disclosure may extract the vanishing point of the current image. Accordingly, the performance of a device that implements an application and / or service that operates by imaging an external environment (e.g., a vehicle that implements an advanced driver assistance system (ADAS)) may be improved (e.g., the vehicle may have improved performance that at least partially provides autonomous driving and / or provides driving assistance to a vehicle driver).

[0067] For example, the vanishing point extraction device 10 may be included in a vehicle that implements an ADAS, and the vehicle may include a vehicle controller that controls autonomous driving. The vehicle controller may calculate a distance between the vehicle and surrounding vehicles based on the vanishing point information, and / or detect a region of a road on which the vehicle is traveling. The vehicle controller may control the traveling direction and traveling speed of the vehicle based on the calculated distance and / or the detected road region. Even when the vehicle is traveling on a road without a lane or with an unclear lane, the vanishing point extraction device 10 may extract the vanishing point from a real-time captured image and may provide vanishing point information including the extracted vanishing point to the vehicle controller. Accordingly, since the vehicle controller can use the vanishing point information provided in real time, the vehicle controller can control the vehicle using a traveling direction and a traveling speed suitable for real-time changing driving conditions. In other words, the vehicle controller may provide a high-precision autonomous driving function based on the vanishing point information provided from the vanishing point extraction device 10.

[0068] In some example embodiments, in Figure 1 although the vanishing point extraction device 10 is illustrated and described as including the image sensor 100, according to some example embodiments, the vanishing point extraction device 10 and the image sensor 100 may be implemented as separate components, and the vanishing point extraction device 10 may be implemented by a method of receiving an image from an external image sensor 100 and extracting the vanishing point of the received image. In addition, according to some example embodiments, the vanishing point extraction device 10 does not include the image sensor 100 and may be implemented by a method of receiving an image via a communication device (not shown) and extracting the vanishing point of the received image.

[0069] In addition, in Figure 1In [the figure], although the vanishing point extraction device 10 is illustrated and described as including the memory 200, according to some example embodiments, the vanishing point extraction device 10 and the memory 200 may be implemented as separate components, and the vanishing point extraction device 10 may also be implemented by the following method: receiving a previous image and information about the previous image from an external memory 200, and using the received image and information about the image to extract the vanishing point of the current image.

[0070] In addition, in Figure 1 [the figure], although the vanishing point extraction device 10 is illustrated and described as including one memory 200, the vanishing point extraction device 10 may include a plurality of memories 200. In addition, according to some example embodiments, the vanishing point extraction device 10 may store an image captured by the image sensor 100 in one of the plurality of memories, and store information about the image, such as the vanishing point, the recognized object, etc., in other memories among the plurality of memories.

[0071] Figure 2 is a block diagram showing in detail a vanishing point extraction device according to some example embodiments of the present disclosure. Specifically, Figure 2 is showing in detail Figure 1 a view of the vanishing point extraction device 10.

[0072] Referring to Figure 1 and Figure 2 , the vanishing point extraction device 10 may include an image sensor 100, a memory 200, and a processor 300. In some example embodiments, the processor 300 may include and / or may implement a vanishing point extractor 310 and an object detector 320, and the vanishing point extractor 310 may include a sampling point extractor 311, a matching point extractor 313, and a vanishing point corrector 315. It will be understood that the vanishing point extractor 310 (including the sampling point extractor 311, the matching point extractor 313, and / or the vanishing point corrector 315) and the object detector 320, and / or their functions, may be implemented by a processing circuit included in the processor 300 and / or one or more instances of a processing circuit that implements some or all of the processor 300 (such as hardware including logic circuits; a hardware / software combination (such as a processor running software); or a combination thereof). For example, the processing circuit may more specifically include, but is not limited to, a central processing unit (CPU), an arithmetic logic unit (ALU), a graphics processing unit (GPU), an application processor (AP), a digital signal processor (DSP), a microcomputer, a field programmable gate array (FPGA), and a programmable logic unit, a microprocessor, an application specific integrated circuit (ASIC), a neural network processing unit (NPU), an electronic control unit (ECU), an image signal processor (ISP), etc.

[0073] When receiving the second image IMG2 from the image sensor 100 (e.g., in response to such reception), the vanishing point extractor 310 may receive (e.g., access, obtain, etc.) the first image IMG1 and information about the first image IMG1 from the memory 200. Additionally, the vanishing point extractor 310 may obtain (e.g., identify) a plurality of sampling points SP1 to SPn ("n" being a positive integer) from the first image IMG1 through the sampling point extractor 311. As described herein, the obtaining element may be interchangeably referred to as the identifying element.

[0074] Specifically, the sampling point extractor 311 may use the information Info_IMG1 about the first image IMG1 (e.g., process the first image IMG1 based on the information Info_IMG1 about the first image IMG1) to obtain (e.g., identify) a straight line including the vanishing point of the first image IMG1 from the first image IMG1. In some example embodiments, the sampling point extractor 311 may use the coordinates of the vanishing point of the first image IMG1 to obtain the coordinates of the pixels of the first image IMG1 that constitute the straight line including the vanishing point of the first image IMG1. Herein, the straight line including the vanishing point of the first image IMG1 may include the vanishing point of the first image IMG1 (e.g., may intersect and / or overlap with the vanishing point, at least as Figure 5 shown), and may be a horizontal line parallel to the horizontal axis of the first image IMG1. Additionally, the present disclosure is not limited thereto, and the straight line including the vanishing point of the first image IMG1 includes the vanishing point of the first image IMG1 and may be a diagonal line or a vertical line parallel to the vertical axis.

[0075] Furthermore, the sampling point extractor 311 may use the information Info_IMG1 about the first image IMG1 (e.g., based on processing the first image IMG1) to obtain a plurality of sampling points SP1 to SPn that intersect with both the objects and the straight lines included in the first image IMG1 (e.g., points within the first image IMG1) from the first image IMG1. Additionally, the sampling point extractor 311 may send the obtained plurality of sampling points SP1 to SPn to the matching point extractor 313. In some example embodiments, the sampling point extractor 311 may identify the pixels that overlap with the pixel coordinates of the straight line among the coordinates included in the object region included in the first image IMG1, and obtain the identified pixels as the plurality of sampling points SP1 to SPn.

[0076] Moreover, the vanishing point extractor 310 may obtain at least one matching point corresponding to the plurality of sampling points from the second image IMG2 through the matching point extractor 313. The specific operation is as follows.

[0077] First, the matching point extractor 313 can obtain a plurality of first templates corresponding to a plurality of sampling points SP1 to SPn respectively from the first image IMG1. In some example embodiments, the matching point extractor 313 can obtain the following region as a first template: the region has the coordinates of at least one sampling point among the plurality of sampling points SP1 to SPn and a specific (or, predetermined) size. The preset sizes of the plurality of first templates can be set according to the manufacturer or the user. In addition, the sizes of the plurality of first templates can be the same or can have different sizes according to the embodiments.

[0078] Then, the matching point extractor 313 can compare the plurality of first templates of the first image IMG1 with the second image IMG2, identify regions in the second image IMG2 that are similar to at least one of the plurality of first templates, and obtain the at least one identified region from the second image IMG2 as a second template. In addition, the matching point extractor 313 can extract at least one matching point MP1 to MPm ("m" is a positive integer that can be the same as or different from "n") from at least one second template. In some example embodiments, the matching point extractor 313 can obtain the central point (e.g., the center point) of each second template as a matching point. In addition, the matching point extractor 313 can send the at least one obtained matching point MP1 to MPm to the vanishing point corrector 315.

[0079] It will be understood that the regions, templates, etc. determined to be "similar" can refer to corresponding regions in one or more images, and the determined correlation value between the corresponding regions is greater than the correlation value threshold. Such a correlation value threshold can be, for example, equal to or greater than 90%, so that a region in the second image IMG2 can be determined to be "similar" to the corresponding first template of the first image IMG1 in response to the difference between the region and the first template being equal to or less than 10%, and / or at least 90% of the pixels of the region match the pixels of the first template in pattern and / or value (i.e., the match between the region and the first template has at least 90% confidence). It will be understood that in cases where elements are determined to match within a specific (or, predetermined) deviation tolerance (e.g., the tolerance can be 10%), these elements (e.g., images, their limited parts or regions, etc.) can be determined to be "similar".

[0080] In addition, the vanishing point extractor 310 can obtain the vanishing point of the second image IMG2 through the vanishing point corrector 315 using at least one matching point MP1 to MPm.

[0081] Specifically, the vanishing point corrector 315 can correct the vanishing point of the first image IMG1 by using at least one matching point MP1 to MPm, so as to obtain the vanishing point of the second image IMG2. In some example embodiments, the vanishing point corrector 315 can calculate the average value of the coordinates of at least one matching point MP1 to MPm, and correct the y coordinate of the vanishing point of the first image IMG1 by using the y coordinate of the calculated average coordinates, so as to obtain the vanishing point of the second image IMG2. In addition, the vanishing point extractor 310 can store the information about the vanishing point of the second image IMG2 in the memory 200 as the information Info_IMG2 about the second image IMG2.

[0082] The object detector 320 can perform object recognition on the second image IMG2 received from the image sensor 100. In addition, as a result of performing object recognition, the object detector 320 can store the information about the recognized object in the memory 200 as the information about the second image IMG2. The object detector 320 can be implemented in firmware or software, and can be loaded into the memory 200 and run by the processor 300. However, the inventive concept is not limited thereto, and the object detector 320 can be implemented in hardware.

[0083] When receiving the second image IMG2 from the image sensor 100, the processor 300 can extract the vanishing point of the second image IMG2 through the vanishing point extractor 310, and perform object recognition on the second image IMG2 through the object detector 320. According to some example embodiments, the processor 300 can perform the vanishing point extraction operation and the object recognition operation on the second image IMG2 in parallel or sequentially.

[0084] In addition, in Figure 2 , although the object detector 320 is illustrated and described as being included in the processor 300, according to some example embodiments, the object detector 320 can be implemented as a separate processor (not shown) of the vanishing point extraction device 10, or the object detector 320 can be implemented outside the vanishing point extraction device 10. In this case, the processor 300 can only perform the vanishing point extraction operation, and only store the information about the vanishing point of the second image IMG2 in the memory 200.

[0085] Figure 3 is a diagram showing a host vehicle including a vanishing point extraction device according to some example embodiments of the present disclosure. Specifically, Figure 3 is a diagram showing a Figure 1 and / or Figure 2 vanishing point extraction device 10 of the host vehicle 400.

[0086] Refer to Figures 1 to 3, the host vehicle 400 may include a vanishing point extraction device 10 and a vehicle controller 410. The vanishing point extraction device 10 may be disposed at the upper end of the host vehicle 400, and the image sensor 100 may capture the front of the host vehicle 400. In addition, the arrangement position of the vanishing point extraction device 10 is not limited to Figure 3 the example embodiment shown in Figure 3 and may be arranged at various positions of the host vehicle 400 according to the example embodiment shown in

[0087] It will be understood that the vehicle controller 410 (also referred to as a vehicle control circuit) may include one or more instances of a processing circuit such as the following, may be included in one or more more instances of a processing circuit such as the following, and / or may be implemented by one or more instances of a processing circuit such as the following: hardware including logic circuits; a hardware / software combination (such as a processor running software); or a combination thereof. For example, the processing circuit may more specifically include, but is not limited to, a central processing unit (CPU), an arithmetic logic unit (ALU), a graphics processing unit (GPU), an application processor (AP), a digital signal processor (DSP), a microcomputer, a field programmable gate array (FPGA), and a programmable logic unit, a microprocessor, an application specific integrated circuit (ASIC), a neural network processing unit (NPU), an electronic control unit (ECU), an image signal processor (ISP), etc. In some example embodiments, the processing circuit may include: a non-transitory computer-readable storage device (e.g., a solid state drive (SSD)) storing an instruction program; and a processor configured to run the instruction program to implement some or all of the functions and / or methods executed by the vehicle controller 410. The vehicle controller 410 and the vanishing point extraction device 10 may be implemented by the same processing circuit and / or different processing circuits.

[0088] The vehicle controller 410 may control the overall driving of the host vehicle 400. The vehicle controller 410 may determine the surrounding conditions of the host vehicle 400 and control the driving direction or driving speed of the host vehicle 400 based on the determination result. In some example embodiments, the vanishing point extraction device 10 may extract the vanishing point of the captured image in front of the host vehicle 400 obtained through the image sensor 100 and provide information about the extracted vanishing point to the vehicle controller 410. The vehicle controller 410 may calculate the distance between the host vehicle 400 and surrounding vehicles based on the information about the vanishing point received from the vanishing point extraction device 10, or control the driving direction or driving speed of the host vehicle 400. Accordingly, the vehicle controller 410 may generate an output signal through the surrounding environment and / or control one or more devices of the host vehicle 400 to cause the host vehicle 400 to travel and / or proceed along one or more trajectories, routes, etc. Hereinafter, for ease of explanation, in the case where the vanishing point extraction device 10 is assumed to be provided in the host vehicle 400, reference will be made to Figures 4 to 18 describe the specific operations of the vanishing point extraction method.

[0089] Figure 4 is a diagram showing a first image according to some example embodiments of the present disclosure.

[0090] Referring to Figure 4 , an example of the first image IMG1 can be confirmed. The first image IMG1 is an image (e.g., an image generated by the image sensor 100) obtained by capturing in front of the host vehicle 400 (see Figure 3 ) and / or generating an image of at least a part of the surrounding environment adjacent to in front of the host vehicle 400 (e.g., in front of the host vehicle 400), and the first image IMG1 may include an object OB1 identified as a vehicle. In addition, the objects identified in the image may include not only vehicles but also (or) specific objects (or, predetermined objects) related to the vehicle (e.g., associated with the vehicle), and according to some example embodiments, may also include various types of objects.

[0091] When a second image is received by the image sensor 100 (e.g., the image sensor 100 that generates the second image, see Figure 1 ), the vanishing point extractor 310 (see Figure 1 )(specifically, the sampling point extractor 311)(see Figure 2 )) may receive the first image IMG1 and information about the first image IMG1, where the first image IMG1 is the previous frame of the second image IMG2 stored in the memory 200 (e.g., see Figure 1, the most recently generated image generated by the image sensor 100 before generating the second image IMG2). In addition, the vanishing point extractor 310 may use information about the first image IMG1 to identify a pixel region corresponding to the vanishing point VP1 of the first image IMG1 and the object OB1 of the first image IMG1. Herein, at least as Figure 4 shown, according to some example embodiments, the region corresponding to the object OB1 of the first image IMG1 may be implemented in the form of a bounding box.

[0092] Figure 5 is a diagram showing a straight line including the vanishing point of the first image according to some example embodiments of the present disclosure.

[0093] Referring to Figure 5 , the vanishing point extractor 310 (see Figure 1 )(specifically, the sampling point extractor 311 (see Figure 2 )) may obtain a straight line including the vanishing point VP1 of the first image IMG1. Specifically, the vanishing point extractor 310 may use the coordinates of the vanishing point VP1 of the first image IMG1 to obtain the coordinates of the pixels of the first image IMG1 that constitute the straight line including the vanishing point VP1 of the first image IMG1. For example, referring to Figure 5 , the vanishing point extractor 310 may obtain the vanishing point VP1 of the first image IMG1 and obtain a horizontal line HL1 parallel to the horizontal axis of the first image IMG1 as the obtained straight line.

[0094] Figure 6 is a diagram showing a plurality of sampling points of the first image according to some example embodiments of the present disclosure.

[0095] Referring to Figure 6 , the vanishing point extractor 310 (see Figure 1 )(specifically, the sampling point extractor 311 (see Figure 2 )) may obtain a plurality of sampling points SP1 to SP4 that intersect both the horizontal line HL1 of the first image IMG1 and the region corresponding to the object OB1 of the first image IMG1. Specifically, the vanishing point extractor 310 may identify, among the pixels constituting the region of the object OB1 included in the first image IMG1, the pixels in the first image IMG1 whose coordinates overlap with the coordinates of the pixels constituting the horizontal line HL1, and obtain at least some of the identified overlapping pixels as a plurality of sampling points SP1 to SP4 that intersect both the straight line (e.g., the horizontal line HL1) and the object (e.g., the object within the region of the object OB1).

[0096] In addition, the vanishing point extractor 310 may obtain at least some of the overlapping pixels as a plurality of sampling points SP1 to SP4 in various ways. For example, the vanishing point extractor 310 may obtain the plurality of sampling points SP1 to SP4 by randomly selecting a specific (or predetermined) number of pixels among the overlapping pixels. As another example, the vanishing point extractor 310 may obtain the plurality of sampling points SP1 to SP4 by selecting a preset number (e.g., quantity) of pixels at a preset interval among the overlapping pixels. Herein, the preset number may be set according to the manufacturer or the user, and according to some example embodiments, the number may be set according to the size of the region of the object. In addition, the method of obtaining at least some of the overlapping pixels as a plurality of sampling points is not limited to the above examples.

[0097] Figure 7 is a diagram showing a first template corresponding to a sampling point according to some example embodiments of the present disclosure.

[0098] Referring to Figure 7 , the vanishing point extractor 310 (see Figure 1 ), specifically, the matching point extractor 313 (see Figure 2 ), may obtain a separate first template corresponding to (e.g., including) a separate sampling point among the plurality of samples, thereby obtaining a plurality of first templates, each first template corresponding to (e.g., including) a separate sampling point among the plurality of sampling points of the first image. In some example embodiments, as Figure 7 shows, the vanishing point extractor 310 may include the coordinates of the first sampling point SP1 and obtain a region having a specific (or predetermined) size (e.g., area) as the first template TP1. For example, a region having a specific size and with the first sampling point SP1 as the center point of the region, so that the first template TP1 may be understood to be centered on the first sampling point SP1. In addition, although not shown in Figure 7 , the vanishing point extractor 310 may obtain first templates corresponding to the remaining sampling points SP2 to SP4 in Figure 6 (e.g., including the remaining sampling points SP2 to SP4 in Figure 6 , centered on the remaining sampling points SP2 to SP4 in Figure 6 , etc.).

[0099] The preset size (e.g., region) of the plurality of first templates may be set according to the manufacturer or the user, and the plurality of first templates may all have the same size. In addition, the sizes of the plurality of first templates may be the same or may have different sizes according to the embodiments. In addition, in Figure 7 , the first template TP1 is illustrated as a square centered on the sampling point, but the present disclosure is not limited thereto, and the first template TP1 may have various forms.

[0100] Figure 8 shows a second image that is the next frame of the first image according to some example embodiments of the present disclosure.

[0101] Referring to Figure 8 , the second image IMG2 is the next frame of the first image IMG1 (e.g., the next image generated by the image sensor 100 after the generation of the first image IMG1), and is similar to the first image IMG1 and is an image in front of the host vehicle 400 (see Figure 3 ). The host vehicle 400 may shake during operation due to road cracks, and thus, there may be differences between the images continuously captured by the image sensor 100.

[0102] Referring to Figure 8 , due to the shaking of the host vehicle 400, the second image IMG2 may present a downward image compared to the first image IMG1, such that the objects included in the first image IMG1 and the second image IMG2 may be located at different positions in the first image IMG1 and the second image IMG2.

[0103] Figure 9 is a diagram showing a search area of the second image according to some example embodiments of the present disclosure.

[0104] The vanishing point extractor 310 (see Figure 1 ), specifically, the matching point extractor 313 (see Figure 2 ), can identify a plurality of search areas in the second image IMG2, and these search areas respectively correspond to individual sampling points among the plurality of sampling points SP1 to SP4 of the first image (see Figure 6 ).

[0105] Specifically, the vanishing point extractor 310 can identify the following points (e.g., pixels) in the second image IMG2: the coordinates of these points (e.g., pixels) in the second image IMG2 are respectively the same as the coordinates of the individual sampling points among the plurality of sampling points SP1 to SP4 in the first image IMG1. For example, as Figure 9As shown, determine the following points in the second image IMG2: the coordinates of the point in the second image IMG2 are the same as the coordinates of the first sampling point SP1 in the first image IMG1. In addition, the vanishing point extractor 310 can obtain a region that includes one identified point (e.g., centered on one identified point) and has a specific (or, predetermined) size as a search region (e.g., each search region can include a separate sampling point among the multiple sampling points SP1 to SP4 located at the center of the search region). Therefore, each separate search region in the second image IMG2 will be understood to correspond to a separate corresponding sampling point among the multiple sampling points SP1 to SP4 in the first image IMG1. The preset sizes of the multiple search regions can be set according to the manufacturer or the user. In addition, the sizes (e.g., areas) of the multiple search regions can all be larger than the size of at least one of the multiple first templates or all of the first templates. In addition, the sizes of the multiple search regions can be the same or can have different sizes according to the embodiments.

[0106] For example, referring to Figure 9 , the vanishing point extractor 310 can identify a point (e.g., a pixel) in the second image IMG2 whose coordinates in the second image IMG2 are the same as the pixel coordinates of the sampling point SP1 in the first image IMG1, and obtain a region including the identified point as a search region SA1 corresponding to the sampling point SP1 (e.g., centered on the sampling point SP1). In addition, in Figure 9 , the search region SA1 corresponding to the sampling point SP1 is illustrated as a square centered on the sampling point SP1, such that the search region SA1 is understood to be centered on the sampling point SP1, but the present disclosure is not limited thereto, and the search region SA1 can have various forms.

[0107] Figure 10 is a diagram for explaining a patch matching operation for a second image according to some example embodiments of the present disclosure.

[0108] The vanishing point extractor 310 (see Figure 1 ), specifically, the matching point extractor 313 (see Figure 2 ), can identify a region similar to the first template within the search region corresponding to each sampling point. The vanishing point extractor 310 can perform patch matching to identify a region similar to the first template. As described herein, a region can be determined to be "similar" to the first template in response to determining that the pixels of the region match the pixels of the first template within a specific confidence level and / or margin (e.g., at least 90% of the pixels of the region match the pixels of the first template).

[0109] Specifically, referring to Figure 10, the vanishing point extractor 310 may determine, in a search area SA1 corresponding to the sampling point SP1, a plurality of candidate regions CA divided by a search window having the same size (e.g., area) as a first template of the sampling point SP1. In addition, the vanishing point extractor 310 may calculate a correlation value between each of the plurality of candidate regions and the first template TP1 to establish a plurality of correlation values. In addition, the vanishing point extractor 310 may identify, from the plurality of correlation values, a candidate region having the highest correlation value.

[0110] The correlation value may be determined based on implementing one or more different image correlation (e.g., digital image correlation) techniques, where the correlation value (which may be referred to as a "correlation coefficient") indicates the relative similarity between the candidate region and the first template being compared therewith. For example, the correlation value determined based on comparing the candidate region CA with the first template TP1 may be 0.95, which indicates a 95% similarity or a 95% confidence match between the pixels of the candidate region CA and the pixels of the first template TP1. In response to determining that the correlation value determined based on comparing the candidate region CA with the first template TP1 is greater than a specific threshold (or a predetermined threshold), the candidate region CA may be determined to be "similar" to the first template TP1. For example, when the above-mentioned correlation value determined based on comparing the candidate region CA with the first template TP1 is 0.95 and the threshold is 0.90, the candidate region CA may thus be determined to be "similar" to the first template TP1.

[0111] Figure 11 is a diagram showing a second template corresponding to a sampling point according to some example embodiments of the present disclosure.

[0112] Referring to Figure 11 , the vanishing point extractor 310 (see Figure 1 )(specifically, the matching point extractor 313 (see Figure 2)) Calculate the correlation value with the first template TP1 for multiple candidate regions CA divided by the search window in the search region SA1 corresponding to the sampling point SP1 (e.g., by implementing digital image correlation technology between the first template TP1 and each candidate region); compare the resulting correlation values corresponding to the individual candidate regions CA compared with the first template TP1 respectively; and determine the candidate region CA with the highest correlation value among the correlation values of the multiple candidate regions CA as the second template TP'1. Such a process can be repeated for each individual first template TP1. Therefore, for each sampling point SP1 to SP4 (e.g., sampling point SP1), the candidate region CA determined to be similar to the corresponding first template (e.g., TP1) corresponding to the same sampling point in the specific search region SA (e.g., SA1) corresponding to the sampling point can be determined as the second template (e.g., TP'1). Therefore, at least one second template TP'1 can be obtained from the second image IMG2, and in this document, at least one second template TP'1 is determined to be "similar" to at least one first template TP1 of the first image.

[0113] In addition, according to some example embodiments, when the correlation values calculated for the multiple candidate regions CA do not exceed a specific value (or, a predetermined value) (e.g., a threshold correlation value), the vanishing point extractor 310 can determine that there is no second template. Therefore, the number (quantity) of matching points can be less than the number of sampling points. In this document, the preset value can represent a value that can be determined to be difficult to be considered similar due to a very low correlation value with the first template, and the preset value can be set by the manufacturer or the user.

[0114] Figure 12 is a diagram showing the matching points of the second template according to some example embodiments of the present disclosure.

[0115] The vanishing point extractor 310 (see Figure 1 )(specifically, the matching point extractor 313 (see Figure 2 )) can obtain the second template and obtain one of the pixels included in the obtained second template and / or one of the pixels associated with the obtained second template as a matching point. Such a process can be repeated for each second template TP'1 of the second image, so that at least one matching point is obtained as a pixel associated with at least one second template TP'1 in the second image IMG2.

[0116] For example, referring to Figure 12, the vanishing point extractor 310 may obtain a pixel corresponding to the center point among the pixels included in the second template TP'1 (e.g., the pixel at the center point of the second template TP'1) as the matching point MP1 corresponding to the sampling point SP1. In addition, the method of obtaining one of the pixels included in the second template as the matching point is not limited to the above example, and the matching point may be obtained in various ways.

[0117] Figure 13 is a view showing matching points of a second image according to some example embodiments of the present disclosure.

[0118] Referring to Figure 13 , the vanishing point extractor 310 (see Figure 1 )(specifically, the matching point extractor 313 (see Figure 2 )) may obtain the matching point MP1 of the second image IMG2 corresponding to the sampling point SP1 of the first image IMG1. The vanishing point extractor 310 may also perform the above-described matching point extraction operation on the remaining sampling points SP2 to SP4 of the first image IMG1.

[0119] Figure 14 is a view showing multiple matching points of a second image according to some example embodiments of the present disclosure.

[0120] Referring to Figure 14 , it can be confirmed that the vanishing point extractor 310 (see Figure 1 )(specifically, the matching point extractor 313 (see Figure 2 )) obtains multiple matching points MP1 to MP4 respectively corresponding to multiple sampling points SP1 to SP4 of the first image IMG1.

[0121] In addition, the vanishing point extractor 310 (specifically, the vanishing point corrector 315 (see Figure 2 )) may use the multiple matching points MP1 to MP4 to obtain the vanishing point of the second image IMG2.

[0122] Figure 15 is a view showing the vanishing point of a second image according to some example embodiments of the present disclosure.

[0123] Referring to Figure 15 , the vanishing point extractor 310 (specifically, the vanishing point corrector 315 (see Figure 2 )) may obtain the vanishing point VP2 of the second image IMG2 by (e.g., based on) correcting the vanishing point VP1 of the first image IMG1 using the multiple matching points MP1 to MP4 (e.g., adjusting the coordinates of the vanishing point VP1 in the first image IMG1 to new corrected coordinates in the second image IMG2 to establish the vanishing point VP2).

[0124] In some example embodiments, the vanishing point extractor 310 may calculate the average of the coordinates of multiple matching points MP1 to MP4, and use the y - coordinate of the calculated average coordinates to correct (e.g., adjust) the coordinates of the vanishing point VP1 of the first image IMG1 to establish a vanishing point VP2 in the second image IMG2, where the vanishing point VP2 is the point with the corrected coordinates. Thus, the y - coordinate of the vanishing point VP2 can be obtained by correcting (e.g., adjusting) the y - coordinate of the vanishing point VP1 based on the y - coordinates of at least one matching point (e.g., some or all of the matching points MP1 to MP4). In addition, the method of correcting the vanishing point VP1 of the first image IMG1 using the average coordinates may vary according to the type of the line including the vanishing point of the first image IMG1 as referred to above Figure 5 described.

[0125] For example, referring to Figure 5 and Figure 15 , the line including the vanishing point of the first image IMG1 may be a horizontal line parallel to the horizontal axis. In this case, the vanishing point extractor 310 corrects the y - coordinate y1 of the vanishing point VP1 of the first image IMG1 to the y - coordinate y2 of the average coordinates of the multiple matching points MP1 to MP4, thereby obtaining the vanishing point VP2 of the second image IMG2. At this time, the x - coordinate x1 of the vanishing point VP1 of the first image IMG1 may not be corrected. Therefore, the x - coordinate of the vanishing point VP2 of the second image IMG2 may be the same as the x - coordinate x1 of the vanishing point VP1 of the first image IMG1.

[0126] As another example, the line including the vanishing point of the first image IMG1 may be a parallel line parallel to the vertical axis. In this case, the vanishing point extractor 310 corrects the x - coordinate of the vanishing point VP1 of the first image IMG1 to the x - coordinate of the average coordinates of the multiple matching points MP1 to MP4, thereby obtaining the vanishing point VP2 of the second image IMG2. At this time, the y - coordinate of the vanishing point VP1 of the first image IMG1 may not be corrected. Therefore, the y - coordinate of the vanishing point VP2 of the second image IMG2 may be the same as the y - coordinate of the vanishing point VP1 of the first image IMG1.

[0127] As another example, the line including the vanishing point of the first image IMG1 may be a diagonal line. In this case, the coordinate change values (△x, △y) between the coordinates of each sampling point and the coordinates of the corresponding matching point may be calculated, the average of the calculated coordinate change values may be calculated, and the vanishing point VP2 of the second image IMG2 may be obtained by reflecting the average coordinate change value to the vanishing point VP1 of the first image IMG1.

[0128] Figure 16 is a diagram showing multiple matching points of a second image according to some example embodiments of the present disclosure.

[0129] The vanishing point extractor 310 (see Figure 1 ), specifically, the matching point extractor 313 (see Figure 2 ), can obtain a plurality of matching points MP1 to MP4 from the second image IMG2. However, in some cases, the plurality of matching points MP1 to MP4 may also include matching points corresponding to outliers. In this context, an outlier can represent a matching point that significantly deviates from the average value of the plurality of matching points (e.g., greater than 10%, greater than the standard deviation σ, etc.) (e.g., the deviation of the vertical coordinate (e.g., y coordinate) of the outlier matching point MP4 from the average vertical coordinate of the matching points MP1 to MP4 can be greater than one standard deviation σ). For example, referring to Figure 16 , the vanishing point extractor 310 can obtain a plurality of matching points MP1 to MP4 including the matching point MP4 corresponding to the outlier.

[0130] Since the vanishing point extractor 310 uses the average coordinates of the matching points to extract the vanishing point VP2 of the second image IMG2, if the matching points include outliers, the vanishing point extractor 310 may extract the vanishing point VP2 of the second image with errors. Therefore, the vanishing point extractor 310 can additionally perform an operation of removing outliers from the plurality of matching points MP1 to MP4.

[0131] As some example embodiments, the vanishing point extractor 310 can remove the matching point MP4 corresponding to the outlier by applying an outlier removal model to the coordinates of the plurality of matching points MP1 to MP4. Here, the outlier removal model can be at least one of a Random Sample Consensus (RANSAC) model, a Progressive Sample Consensus (PROSAC) model, and a Stable random sample consensus (StaRSaC) model, and the type of the outlier removal model is not limited to the above.

[0132] The above operation of removing outliers can be performed by the matching point extractor 313 in the vanishing point extractor 310. For example, the matching point extractor 313 can obtain at least one matching point that matches a plurality of sampling points and perform an operation of removing outliers from the obtained at least one matching point. In addition, the matching point extractor 313 can send the at least one matching point with outliers removed to the vanishing point corrector 315.

[0133] In addition, according to some example embodiments, the vanishing point corrector 315 of the vanishing point extractor 310 may perform the removal of the above outliers. For example, if at least one matching point including an outlier is received from the matching point extractor 313, the vanishing point corrector 315 may perform an operation of removing the outlier among the received at least one matching point.

[0134] Figure 17 is a diagram showing the vanishing point of a second image according to some example embodiments of the present disclosure. Specifically, Figure 17 is for explaining the operation of obtaining the vanishing point VP2 of the second image IMG2 using Figure 16 a plurality of matching points MP1 to MP3.

[0135] Referring to Figure 17 , the vanishing point extractor 310 (specifically, the vanishing point corrector 315 (see Figure 2 )) may correct the vanishing point VP1 of the first image IMG1 by using the plurality of matching points MP1 to MP3 from which outliers have been removed, to obtain the vanishing point VP2 of the second image IMG2.

[0136] The vanishing point extractor 310 may calculate the average value of the coordinates of the plurality of matching points MP1 to MP3 from which outliers have been removed, and use the y - coordinate of the calculated average coordinate to correct the vanishing point VP1 of the first image IMG1, thereby obtaining the vanishing point VP2 of the second image IMG2.

[0137] Figure 18 is a diagram showing a plurality of sampling points of a first image according to some example embodiments of the present disclosure. Specifically, Figure 18 is a diagram for explaining the operation of extracting the plurality of sampling points SP1 to SP4 of the first image IMG1 when a plurality of objects OB1, OB2, and OB3 are included in the first image IMG1.

[0138] Referring to Figure 18 , the vanishing point extractor 310 (see Figure 1 )(specifically, the sampling point extractor 311 (see Figure 2 )) may obtain a straight line including the vanishing point VP1 of the first image IMG1. For example, referring to Figure 18 , the vanishing point extractor 310 may obtain the vanishing point VP1 of the first image IMG1, and obtain a horizontal line HL1 parallel to the horizontal axis of the first image IMG1.

[0139] Then, the vanishing point extractor 310 (see Figure 1 )(specifically, the sampling point extractor 311 (see Figure 2)(e.g., based on processing the first image), the vanishing point extraction device 10 can obtain a straight line including the vanishing point of the first image from the first image (S110), e.g., as described herein with reference to Figure 18 , the vanishing point extractor 310 can extract a plurality of sampling points SP1 and SP2 among the pixels that intersect both the region corresponding to the first object OB1 and the horizontal line HL1. In addition, the vanishing point extractor 310 can extract a plurality of sampling points SP3 and SP4 among the pixels that intersect both the region corresponding to the second object OB2 and the horizontal line HL1. Further, since there are no pixels that intersect both the region corresponding to the third object OB3 and the horizontal line HL1, the vanishing point extractor 310 cannot extract additional sampling points.

[0140] In this way, even when the first image IMG1 includes a plurality of objects OB1, OB2, and OB3, the vanishing point extractor 310 can extract a plurality of sampling points. In addition, as described above with reference to Figures 7 to 15 , the vanishing point extractor 310 can extract at least one matching point corresponding to the extracted plurality of sampling points from the second image IMG2, and use the extracted at least one matching point to extract the vanishing point VP2 of the second image IMG2.

[0141] Figure 19 is a flowchart showing a vanishing point extraction method according to some example embodiments of the present disclosure. Specifically, Figure 19 is a flowchart showing an example of a vanishing point extraction method implemented by (e.g., implemented by) Figure 1 the vanishing point extraction device 10 of Figure 2 the vanishing point extraction device 10 of Figure 20 the autonomous driving device 500 of Figure 19 the processing circuit that implements any of the devices, the processing circuit included in any of the devices, the processing circuit that includes any of the devices, and / or any of their components, etc. Figure 1 the processor 300 of Figure 2 the processor 300 of Figure 20 the processor 530 of Figure 20 the main processor 550 of

[0142] With reference to Figure 1 and Figure 19 , first, the vanishing point extraction device 10 can (e.g., based on processing the first image) obtain a straight line including the vanishing point of the first image from the first image (S110), e.g., as described herein with reference to Figures 4 - 5As described. In some example embodiments, the straight line including the vanishing point of the first image includes the vanishing point of the first image, and the vanishing point extraction device 10 may obtain a horizontal line parallel to the horizontal axis of the first image.

[0143] In addition, the vanishing point extraction device 10 may obtain a plurality of sampling points that intersect with the straight line and an object included in the first image from the first image (S120), for example, as described herein with reference to Figure 6 As described. Specifically, the vanishing point extraction device 10 may identify pixels in the object region of the first image whose coordinates overlap with the coordinates of the pixels constituting the straight line, and obtain some or all of the identified pixels as a plurality of sampling points that intersect both the straight line and the object included in the first image. Herein, the object included in the first image may include an object identified as a vehicle or a specific object (or, a predetermined object) associated with the vehicle.

[0144] In addition, the vanishing point extraction device 10 may obtain at least one matching point that matches the plurality of sampling points from the second image, which is the next image of the first image (S130), for example, as described herein with reference to Figures 7 - 14 As described. Specifically, the vanishing point extraction device 10 may obtain a plurality of first templates corresponding to the plurality of sampling points respectively (for example, each first template may have a separate region of the first image and use a separate sampling point as the center of the corresponding first template). Each of the plurality of first templates may include the corresponding sampling point and have a specific (or, predetermined) size (for example, area and / or shape). In addition, the vanishing point extraction device 10 may obtain at least one second template that is similar to at least one of the plurality of first templates from the second image by comparing each of the plurality of first templates with the second image (for example, comparing the first template with a candidate region of the second image); for each first template, for example, via performing digital image correlation techniques, determine the corresponding candidate region in the second image with the highest correlation value (for example, the highest similarity to the first template) as the corresponding second template of the second image. In addition, the vanishing point extraction device 10 may obtain at least one matching point from at least one second template, for example, each individual matching point may correspond to (for example, may be determined as) the center point of the corresponding second template.

[0145] In addition, the vanishing point extraction device 10 may obtain the vanishing point of the second image from the second image based on (for example, of the second image) at least one matching point (S140), for example, as described herein with reference to Figure 15As described. Specifically, the vanishing point extraction device 10 can obtain the vanishing point of the second image by correcting the vanishing point of the first image based on at least one matching point. For example, the vanishing point extraction device 10 can calculate the average y coordinate of at least one matching point, and extract the vanishing point of the second image by changing the y coordinate of the vanishing point of the first image to the calculated average y coordinate.

[0146] The method may further include: for example, generating and / or sending an output signal including information based on the determined vanishing point of the second image to, for example, a vehicle controller 410 (e.g., a vehicle control circuit), wherein when the vehicle moves (e.g., travels) through the surrounding environment, the information based on the determined vanishing point of the second image can be used by the vehicle controller 410 to determine changes in the surrounding environment and / or the position of the host vehicle 400. The vehicle controller 410 can control one or more components of the host vehicle 400 (e.g., some or all of the drivers 560 as Figure 20 shown) to control the host vehicle 400 to travel and / or move through the surrounding environment based on the determined vanishing point of the second image. Therefore, even if the boundary markings (e.g., lane markings on the road) of the route that the host vehicle 400 is to follow through the surrounding environment do not exist, Figure 19 the method shown can also be used to determine the specific route that the host vehicle 400 follows through the surrounding environment, and / or at least partially implement autonomous driving of the host vehicle 400 along a specific route (e.g., on the road) when passing through the surrounding environment. Therefore, autonomous driving of the host vehicle 400 through the surrounding environment can be realized and / or improved.

[0147] Figure 20 is a block diagram showing an autonomous driving device according to some example embodiments of the present disclosure.

[0148] Referring to Figure 20 , the autonomous driving device 500 (which may correspond to Figure 3 the host vehicle 400) may include: a sensor 510, a memory 520, a processor 530, a RAM 540, a main processor 550, a driver 560, and a communication interface 570, and the components of the autonomous driving device 500 can be connected to each other and communicate with each other through a bus. At this time, the image sensor 511 included in the sensor 510 can correspond to the image sensor 100 of the above embodiment, and the memory 520 can correspond to the memory 200 of the above embodiment, and the processor 530 can correspond to the processor 300 of the above embodiment and can be referred to as a first processing circuit herein. In addition, the main processor 550 can correspond to Figure 3 the vehicle controller 410 of Figures 1 to 9implemented by the described exemplary embodiments.

[0149] The autonomous driving device 500 can perform real-time analysis on the surrounding environment data of an autonomous vehicle based on a neural network, and perform situation determination and vehicle operation control.

[0150] The neural network can include various neural network systems and / or machine learning systems. For example, an artificial neural network (ANN) system, a convolutional neural network (CNN) system, a deep neural network (DNN) system, a deep learning system, etc. Such machine learning systems can include various learning models, such as a convolutional neural network (CNN), a deconvolutional neural network, a recurrent neural network (RNN) optionally including long short-term memory (LSTM) units and / or gated recurrent units (GRU), a stacked neural network (SNN), a state space dynamic neural network (SSDNN), a deep belief network (DBN), a generative adversarial network (GAN), and / or a restricted Boltzmann machine (RBM). As an additional option or additionally, such machine learning systems can include other forms of machine learning models, such as linear and / or logistic regression, statistical clustering, Bayesian classification, decision trees, dimensionality reduction (such as principal component analysis), and expert systems; and / or combinations thereof, including ensembles such as random forests. Such machine learning models can also be used to provide, for example, at least one of various services and / or applications, such as an image classification service, a user authentication service based on bioinformation or biometric data, an advanced driver assistance system (ADAS) service, a voice assistant service, an automatic speech recognition (ASR) service, etc., and can be executed, run, implemented, processed, etc. by some or all of any system and / or device described herein, including some or all of the autonomous driving device 500 (e.g., the processor 530 and / or the main processor 550).

[0151] Such a model can be implemented using software or hardware and can be a model based on at least one of the following: artificial neural network (ANN) model, multi-layer perceptron (MLP) model, convolutional neural network (CNN) model, deconvolutional neural network, decision tree model, random forest model, Adaboost (adaptive boosting) model, multiple regression analysis model, logistic regression model, recurrent neural network (RNN) optionally including long short-term memory (LSTM) units and / or gated recurrent units (GRU), stacked neural network (SNN), state space dynamic neural network (SSDNN), deep belief network (DBN), generative adversarial network (GAN) and / or restricted Boltzmann machine (RBM)). As an additional option or alternatively, such a model can include other forms of artificial intelligence models, such as linear and / or logistic regression, statistical clustering, Bayesian classification, decision trees, dimensionality reduction (such as principal component analysis) and expert systems, random sample consensus (RANSAC) model; and / or combinations thereof. Examples of such models are not limited to this.

[0152] The sensor 510 can include multiple sensors that receive image signals related to the surrounding environment of the autonomous driving device 500 and output the received image signals as images. For example, the sensor 510 includes image sensors 511 such as charge-coupled device (CCD) and complementary metal oxide semiconductor (CMOS), depth cameras 513, light detection and ranging (LIDAR) sensors 515, radio detection and ranging (RADAR) sensors 517, etc. In addition, the present disclosure is not limited thereto and may include ultrasonic sensors (not shown), infrared sensors (not shown), etc. In some example embodiments, the image sensor 511 can generate a front image of the autonomous driving device 500 and provide the generated front image to the processor 530.

[0153] The memory 520 is a storage location for storing data and can, for example, store various data generated during the operation of the main processor 550 and the processor 530.

[0154] When an image is received from the image sensor 511, based on information about images in the previous order of the received image, the vanishing point of the previous order of images, and objects in the previous order of images, the processor 530 can extract the vanishing point of the received image. The method by which the processor 530 extracts the vanishing point can be substantially the same as the method described above with reference to Figures 1 to 19 and repeated descriptions are omitted.

[0155] The main processor 550 may control the overall operation of the autonomous driving device 500. For example, the main processor 550 may control the functions of the processor 530 by running a program stored in the RAM 540. The RAM 540 may temporarily store programs, data, applications, or instructions.

[0156] In addition, the main processor 550 may control the operation of the autonomous driving device 500 based on the operation results of the processor 530. As some example embodiments, the main processor 550 may receive information about the vanishing point from the processor 530 and control the operation of the driver 560 based on the received vanishing point information. Such control may include: generating an output signal based on the obtained vanishing point of the current image of at least a part of the surrounding environment, based on a previous image of the surrounding environment, for example, as described herein Figures 1 - 19 which causes the driver 560 to drive and / or move the autonomous driving device 500 (which may be a vehicle) through the surrounding environment.

[0157] The driver 560 (e.g., a vehicle driving control device) is configured to drive the autonomous driving device 500 and may include an engine and a motor 561, a steering unit 563 (e.g., a steering device), and a braking unit 565 (e.g., a vehicle brake). In some example embodiments, the driver 560 may adjust the propulsion, braking, speed, direction, etc. of the autonomous driving device 500 using the engine and motor 561, the steering unit 563, and the braking unit 565 under the control of the processor 530. In some example embodiments, the main processor 550 (also referred to herein as a second processing circuit) may control the driver and may thus be configured to control the operation of the host vehicle including the autonomous driving device 500 based on information associated with the vanishing point of a second image obtained by the processor 530 (e.g., a first processing circuit).

[0158] The communication interface 570 may communicate with external devices using wired or wireless communication methods. For example, the communication interface 570 may perform communication using a wired communication method such as Ethernet, or may perform communication using a wireless communication method such as Wi-Fi or Bluetooth.

[0159] Although the inventive concept has been specifically shown and described with reference to some example embodiments of the inventive concept, it will be understood that various changes in form and detail may be made without departing from the spirit and scope of the appended claims.

Claims

1. A method for performing vanishing point extraction, the method comprising: Based on processing the first image, a straight line including the vanishing point of the first image is obtained; Based on processing the first image according to the objects included in the first image and the straight line including the vanishing point of the first image, a plurality of sampling points in the first image are obtained, such that the plurality of sampling points are determined as pixels whose coordinates in the first image overlap with the coordinates of the pixels of the straight line and the coordinates of the pixels of the objects included in the first image; At least one matching point corresponding to at least one of the plurality of sampling points in the first image is obtained in a second image, where the second image is generated after the first image is generated; And Based on the at least one matching point of the second image, the vanishing point of the second image is obtained.

2. The method according to claim 1, wherein, The obtaining of the at least one matching point includes: Obtaining a plurality of first templates, each of the first templates corresponding to a separate sampling point among the plurality of sampling points of the first image; Obtaining at least one second template in the second image, where the at least one second template is determined to be similar to at least one of the plurality of first templates; and Obtaining the pixels in the second image associated with the at least one second template as the at least one matching point.

3. The method according to claim 2, wherein, The obtaining of the plurality of first templates includes: obtaining regions with a specific size as the plurality of first templates, where the regions respectively include separate sampling points among the plurality of sampling points.

4. The method according to claim 2, wherein, The obtaining of the at least one second template includes: Obtaining a plurality of search regions of the second image, where the plurality of search regions respectively correspond to separate sampling points among the plurality of sampling points of the first image; and For each sampling point, obtaining a candidate region similar to the first template corresponding to the sampling point in the search region corresponding to the sampling point as a separate second template among the at least one second template.

5. The method according to claim 4, wherein, The obtaining of the plurality of search regions includes: Identifying points in the second image: the coordinates of the points in the second image are respectively the same as the coordinates of separate points among the plurality of sampling points in the first image; and Obtaining regions in the second image with a specific size and respectively including separate points among the identified points as the plurality of search regions.

6. The method according to claim 4, wherein, The size of the plurality of search regions is larger than the size of the plurality of first templates.

7. The method according to claim 4, wherein, The obtaining of the candidate region similar to the first template corresponding to the sampling point as the separate second template includes: Determining a plurality of candidate regions in the search region corresponding to the sampling point; Calculating the correlation values between each candidate region among the plurality of candidate regions and the first template corresponding to the sampling point to establish a plurality of correlation values respectively corresponding to separate candidate regions; and Obtaining the candidate region corresponding to the highest correlation value among the plurality of correlation values as the separate second template.

8. The method according to claim 2, wherein, Obtaining the pixels associated with the at least one second template in the second image as the at least one matching point includes: identifying the pixels corresponding to the center points of the at least one second template as the at least one matching point.

9. The method according to claim 1, wherein, The straight line including the vanishing point of the first image is a horizontal line parallel to the horizontal axis of the first image.

10. The method according to claim 1, further comprising: Removing the outliers among the at least one matching point, wherein, based on the at least one matching point from which the outliers are removed, the vanishing point of the second image is obtained.

11. The method according to claim 1, wherein, The obtaining of the vanishing point of the second image includes: correcting the vanishing point of the first image based on the at least one matching point to obtain the vanishing point of the second image.

12. The method according to claim 11, wherein, Based on correcting the y coordinate of the vanishing point of the first image using the y coordinates of the at least one matching point, the vanishing point of the second image is obtained.

13. A vanishing point extraction device, comprising: An image sensor configured to generate a first image and a second image after generating the first image; A memory configured to store the first image, information associated with the vanishing point of the first image, and information associated with at least one object included in the first image; and A processing circuit configured to: in response to receiving the second image from the image sensor, perform the following operations: Based on the information associated with the vanishing point of the first image, identify in the first image the horizontal line including the vanishing point of the first image, Based on processing the first image using the information associated with the at least one object included in the first image, obtain in the first image a plurality of sampling points such that the plurality of sampling points are determined to be pixels whose coordinates in the first image overlap with the coordinates of the pixels of the at least one object and the coordinates of the pixels of the horizontal line, Identify in the second image at least one matching point corresponding to at least one of the plurality of sampling points in the first image, and Based on correcting the vanishing point of the first image using the at least one matching point, obtain the vanishing point of the second image.

14. The vanishing point extraction device according to claim 13, wherein, The processing circuit is further configured to: Obtain in the first image a plurality of first templates, the plurality of first templates respectively corresponding to individual sampling points among the plurality of sampling points in the first image, and Obtain in the second image at least one second template, the at least one second template being similar to at least one of the plurality of first templates.

15. The vanishing point extraction device according to claim 14, wherein, The processing circuit is further configured to: obtain a region with a specific size as the plurality of first templates, wherein each individual region includes an individual sampling point among the plurality of sampling points.

16. The vanishing point extraction device according to claim 14, wherein, The processing circuit is further configured to: Obtain a plurality of search regions of the second image, the plurality of search regions respectively corresponding to individual sampling points among the plurality of sampling points in the first image; and For each sampling point, a candidate region similar to a first template corresponding to the sampling point is obtained in a search region corresponding to the sampling point as a separate second template among the at least one second template.

17. The vanishing point extraction device according to claim 16, wherein, The processing circuit is further configured to: Identify points in the second image, the coordinates of which in the second image are the same as the coordinates of a separate point among the plurality of sampling points in the first image; And Obtain regions in the second image having a specific size and each including a separate one of the identified points as the plurality of search regions.

18. The vanishing point extraction device according to claim 16, wherein, The processing circuit is further configured to: Determine a plurality of candidate regions in the search region corresponding to the sampling point; Calculate a correlation value between each candidate region among the plurality of candidate regions and the first template corresponding to the sampling point to establish a plurality of correlation values respectively corresponding to the separate candidate regions; And Obtain the candidate region corresponding to the highest correlation value among the plurality of correlation values as the separate second template.

19. The vanishing point extraction device according to claim 14, wherein, The processing circuit is further configured to: identify the center point of the at least one second template as the at least one matching point.

20. An autonomous driving device configured to be included in a host vehicle, the autonomous driving device comprising: An image sensor configured to generate a first image and, after generating the first image, generate a second image; A memory configured to store the first image, information associated with a vanishing point of the first image, and information associated with at least one object included in the first image; A first processing circuit configured to, in response to receiving the second image from the image sensor, perform the following operations: Based on the information associated with the vanishing point of the first image, identify a horizontal line including the vanishing point of the first image in the first image; Based on processing the first image using the information associated with the at least one object included in the first image, obtain a plurality of sampling points in the first image such that the plurality of sampling points are determined as pixels in the first image where the coordinates overlap with the coordinates of pixels of the at least one object and the coordinates of pixels of the horizontal line; Identify at least one matching point in the second image corresponding to at least one sampling point among the plurality of sampling points in the first image, and Based on correcting the vanishing point of the first image using the at least one matching point, obtain the vanishing point of the second image; And A second processing circuit configured to control the operation of the host vehicle based on the information associated with the vanishing point of the second image obtained by the first processing circuit.

Images