Image processing device, image processing method, and program

Abstract

This image processing device comprises: an object detection unit that detects an object in an image captured from an approximately horizontal direction by an image sensor; a distance estimation unit that, if the detected object is a vehicle, estimates the distance between the vehicle and the image sensor; a virtual line drawing unit that draws a virtual line extending in the lateral direction of the image at a position having a height which is the same as the upper edge of the vehicle in the image; a segmentation execution unit that, if the estimated distance is less than a threshold value, executes segmentation processing for extracting, from the image, a road region on which the vehicle is travelling; and a vanishing point determination unit that determines the height position of a vanishing point of the image on the basis of the virtual line if the estimated distance is at least the threshold value, and determines the height position of the vanishing point on the basis of the extracted road region if the estimated distance is less than the threshold value.

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an image processing method, and a program. [Background technology]

[0002] Conventionally, there is known a technique for extracting a vanishing point from an image captured by a camera mounted on a vehicle. For example, Patent Document 1 discloses a technique for calculating a motion vector between a plurality of images captured by a camera and determining the coordinates of a vanishing point in the image using the calculated motion vector. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2012-123751 Summary of the Invention [Problem to be solved by the invention]

[0004] The technology described in Patent Document 1 calculates a motion vector for each pixel from multiple images captured by a camera using a block matching method, a gradient method, etc. However, such conventional technology may require a large amount of calculation cost in order to improve the accuracy of the vanishing point.

[0005] The present invention has been made in consideration of these circumstances, and one of its objects is to provide an image processing device, an image processing method, and a program that can determine a vanishing point with high accuracy while keeping calculation costs low. [Means for solving the problem]

[0006] The image processing device, image processing method, and program according to the present invention employ the following configuration. (1): An image processing device according to one embodiment of the present invention includes an object detection unit that detects an object in an image captured in an approximately horizontal direction by an image sensor; a distance estimation unit that, if the detected object is a vehicle, estimates the distance between the vehicle and the image sensor; a virtual line drawing unit that draws a virtual line extending laterally in the image at a position at the same height as the top end of the vehicle in the image; a segmentation execution unit that, if the estimated distance is less than a threshold, performs a segmentation process to extract from the image a road area on which the vehicle is traveling; and a vanishing point determination unit that, if the estimated distance is equal to or greater than the threshold, determines the height position of a vanishing point of the image based on the virtual line, and, if the estimated distance is less than the threshold, determines the height position of the vanishing point based on the extracted road area.

[0007] (2): In the above aspect (1), at least the distance estimation unit and the vanishing point determination unit perform processing repeatedly and synchronously for each control cycle, and the distance estimation unit estimates the distance in the current control cycle using the height position of the vanishing point determined by the vanishing point determination unit in the previous control cycle.

[0008] (3): In the above-mentioned aspects (1) or (2), at least the distance estimation unit and the vanishing point determination unit perform processing synchronously and repeatedly for each control cycle, and the distance estimation unit estimates the distance in the current control cycle using the height position of the vanishing point determined by the vanishing point determination unit in the previous control cycle.

[0009] (4): In the aspect (3) above, the trained model is trained to output a vehicle in the image as an area representing the road area.

[0010] (5): In any of the above aspects (1) to (4), if the estimated distance is equal to or greater than the threshold value, the vanishing point determination unit determines the height position of the vanishing point to be a position obtained by shifting the virtual line downward by a predetermined distance.

[0011] (6): In any of the above aspects (1) to (5), when there are multiple vehicles whose estimated distances are equal to or greater than the threshold, the vanishing point determination unit shifts the multiple virtual lines corresponding to the multiple vehicles downward by a predetermined distance, and determines the average position as the height position of the vanishing point.

[0012] (7): In any of the above aspects (1) to (6), the system further includes a bird's-eye view generation unit that generates a bird's-eye view including the detected object from the image based on the determined height position of the vanishing point, and the bird's-eye view generation unit displays the generated bird's-eye view on the display unit of the image processing device.

[0013] (8): Another aspect of the image processing method of the present invention is a method in which a computer detects an object in an image captured in an approximately horizontal direction by an image sensor, and if the detected object is a vehicle, estimates the distance between the vehicle and the image sensor, draws a virtual line extending laterally in the image at a position at the same height as the top of the vehicle in the image, and if the estimated distance is less than a threshold, performs a segmentation process to extract from the image a road area on which the vehicle is traveling, and if the estimated distance is equal to or greater than the threshold, determines the height position of a vanishing point of the image based on the virtual line, and if the estimated distance is less than the threshold, determines the height position of the vanishing point based on the extracted road area.

[0014] (9): Another aspect of the present invention provides a program that causes a computer to detect an object in an image captured in an approximately horizontal direction by an image sensor, and if the detected object is a vehicle, to estimate the distance between the vehicle and the image sensor, and to draw a virtual line extending laterally in the image at a position at the same height as the top of the vehicle in the image, and if the estimated distance is less than a threshold, to perform a segmentation process that extracts from the image the area of the road on which the vehicle is traveling, and if the estimated distance is equal to or greater than the threshold, to determine the height position of the vanishing point of the image based on the virtual line, and if the estimated distance is less than the threshold, to determine the height position of the vanishing point based on the extracted road area. [Effects of the Invention]

[0015] According to aspects (1) to (9), the vanishing point can be determined with high accuracy while keeping the calculation cost low. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a diagram showing an example of a usage environment of a terminal device 100 mounted on a vehicle M. FIG. [Figure 2] 1 is a diagram illustrating an example of the configuration of a terminal device 100. FIG. [Figure 3] 10 is a diagram showing an example of a scene in which the distance estimation unit 120 estimates the distance between the host vehicle M and a nearby vehicle M1. FIG. [Figure 4] 10 is a diagram for explaining a method by which the distance estimation unit 120 estimates the distance in the longitudinal direction between the host vehicle M and a surrounding vehicle M1. FIG. [Figure 5] 10 is a diagram for explaining a method by which the distance estimation unit 120 estimates the distance in the lateral direction between the host vehicle M and a nearby vehicle M1. FIG. [Figure 6] 2A and 2B are diagrams showing an example of a bird's-eye view and a camera image displayed on a display unit 20. FIG. [Figure 7] 10A and 10B are diagrams showing other examples of a bird's-eye view and a camera image displayed on the display unit 20. FIG. [Figure 8]10A and 10B are diagrams for explaining a first method for determining a vanishing point V according to the present embodiment. [Figure 9] 10A and 10B are diagrams for explaining a second method for determining a vanishing point V according to the present embodiment. [Figure 10] 10 is a flowchart illustrating an example of a flow of processing executed by a bird's-eye view application. DETAILED DESCRIPTION OF THE INVENTION

[0017] Hereinafter, embodiments of an image processing device, an image processing method, and a program of the present invention will be described with reference to the drawings. In this embodiment, the image processing device is, for example, a terminal device 100 such as a smartphone having a camera (image sensor) and a display (display unit). However, the present invention is not limited to such a configuration, and the image processing device may be at least a computer device having a computing function, excluding the camera and display, from the configuration of the terminal device 100 described below. In this case, the camera, the display, and the image processing device work together to realize the functions of the present invention.

[0018] [composition] 1 is a diagram showing an example of a usage environment of a terminal device 100 mounted on a host vehicle M. The host vehicle M may be, for example, a two-wheeled, three-wheeled, or four-wheeled vehicle, and its drive source may be an internal combustion engine such as a diesel engine or a gasoline engine, an electric motor, or a combination of these. The electric motor operates using power generated by a generator connected to the internal combustion engine, or discharged power from a secondary battery or a fuel cell.

[0019] 1, the terminal device 100 is installed on the host vehicle M so as to be able to capture an image of the area ahead in the traveling direction of the host vehicle M. The terminal device 100 is held, for example, by an in-vehicle holder (not shown) attached to the dashboard of the host vehicle M, and captures an image of the area ahead of the host vehicle M. As will be described later, a user of the terminal device 100 aligns the position of the terminal device 100 at a predetermined height according to guide lines displayed on the display unit 20. FIG. 1 shows how, as a result of the terminal device 100 being aligned, an image of the vicinity of the upper end of a nearby vehicle M1 of the host vehicle M is captured in a substantially horizontal direction with respect to the road surface.

[0020] Fig. 2 is a diagram showing an example of the configuration of terminal device 100. As shown in Fig. 2, terminal device 100 includes, for example, camera 10, display unit 20, object detection unit 110, distance estimation unit 120, bird's-eye view generation unit 130, virtual line drawing unit 140, segmentation execution unit 150, and vanishing point determination unit 160. The object detection unit 110, distance estimation unit 120, bird's-eye view generation unit 130, virtual line drawing unit 140, segmentation execution unit 150, and vanishing point determination unit 160 are realized by, for example, a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Some or all of these components may be implemented by hardware (including circuitry) such as a large-scale integration (LSI), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a graphics processing unit (GPU), or may be implemented by a combination of software and hardware. The program may be stored in advance in a storage device (a storage device having a non-transitory storage medium) such as a hard disk drive (HDD) or flash memory, or may be stored in a removable storage medium (a non-transitory storage medium) such as a DVD or CD-ROM, and installed by inserting the storage medium into a drive device. In the following description, the functions of the object detection unit 110, the distance estimation unit 120, the bird's-eye view generation unit 130, the virtual line drawing unit 140, the segmentation execution unit 150, and the vanishing point determination unit 160 may be collectively referred to as the "bird's-eye view application." The bird's-eye view application is installed in the terminal device 100 and is activated, for example, when the user of the terminal device 100 starts driving the vehicle M. The camera 10 is, for example, a digital camera using a solid-state imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The display unit 20 is, for example, a display device such as a touch panel or a liquid crystal display.

[0021] [Distance Estimation] The object detection unit 110 detects objects captured in an image captured by the camera 10. More specifically, for example, the object detection unit 110 detects objects using a trained model that has been trained to output information such as the presence, position, and type of an object when an image captured by the camera 10 is input. For example, the object detection unit 110 can use this trained model to detect the presence and position of a vehicle and the presence and position of road markings.

[0022] If the object detected by the object detection unit 110 is a vehicle (in this case, a vehicle means a vehicle with two, three, four, or other wheels), the distance estimation unit 120 estimates the distance between the detected vehicle and the camera 10. FIG. 3 is a diagram showing an example of a scene in which the distance estimation unit 120 estimates the distance between the host vehicle M and a peripheral vehicle M1. In FIG. 3, the symbol hA indicates the height from the road position corresponding to the bottom edge of the display unit 20 to the vanishing point of the image, and the symbol hB indicates the height from the road position corresponding to the bottom edge of the detected peripheral vehicle M1 to the vanishing point of the image. The method for determining the vanishing point will be described later.

[0023] 4 is a diagram illustrating a method by which distance estimation unit 120 estimates the longitudinal distance between host vehicle M and peripheral vehicle M1. In FIG. 4, symbol IS indicates an image sensor included in camera 10, symbol D indicates a display (an end of camera 10) included in camera 10, symbol O indicates a center of the image sensor, symbol A indicates a position on display D corresponding to road position F shown on the bottom of display D, symbol B indicates a position on display D corresponding to road position G of the rear end of peripheral vehicle M1, symbol C indicates an intersection between the imaging direction of the image sensor and display D, symbol H indicates the height of camera 10 relative to the road surface, symbol DA indicates the distance from the position of image sensor IS to road position F shown on the bottom of display D, and symbol DB indicates the distance from the position of image sensor IS to road position G of the rear end of peripheral vehicle M1.

[0024] In FIG. 4, triangles OAC and OEF are similar to each other, and triangles OBC and OEG are similar to each other. That is, since L:hA = DA:H and L:hB = DB:H hold true for distance, transformations yield DA = L×H / hA and DB = L×H / hB. Therefore, the distance estimation unit 120 can calculate the distance to the road position G of the rear end of the surrounding vehicle M1 using the formula DB = DA×hA / hB. Here, the heights hA and hB to the vanishing point are calculated in advance based on the image captured by the camera 10, and the distance DA, which is independent of the position of the surrounding vehicle M1, can be calculated in advance based on the installation position of the terminal device 100. The above calculation can be performed using only the height information of the vanishing point, without requiring all coordinate information of the vanishing point.

[0025] If the object detected by the object detection unit 110 is a vehicle, the distance estimation unit 120 further estimates the lateral distance to the detected vehicle. Fig. 5 is a diagram for explaining a method for estimating the lateral distance to the peripheral vehicle M1 by the distance estimation unit 120. In Fig. 5, the symbol V indicates the vanishing point of the image, the symbol Wb indicates the number of pixels in the lateral direction based on the vanishing point V of the peripheral vehicle M1, and the symbol Wa indicates the number of pixels when the number of pixels Wb is moved to the bottom edge of the display D.

[0026] In FIG. 5, triangle VT'T and triangle VS'S are similar to each other. That is, since Wa:hA == Wb:hB holds true for distance, transformation yields Wa = Wb × hA / hB. Assuming that the total number of pixels Wsc at the bottom edge of display D and the width Wrd of the road on which host vehicle M is traveling are known, distance estimation unit 120 can calculate the actual lateral distance W corresponding to the number of pixels Wa using the formula W = Wrd × Wa / Wsc. In this way, distance estimation unit 120 calculates the longitudinal distance and the lateral distance between host vehicle M and peripheral vehicle M1.

[0027] [Generate a bird's-eye view] The bird's-eye view generating unit 130 generates a bird's-eye view showing the surrounding situation of the host vehicle M, based on the vertical distance and the horizontal distance between the host vehicle M and the surrounding vehicle M1 estimated by the distance estimating unit 120. The bird's-eye view generating unit 130 displays the generated bird's-eye view on the display unit 20 together with the camera image captured by the camera 10.

[0028] FIG. 6 is a diagram showing an example of a bird's-eye view and a camera image displayed on the display unit 20. The left part of FIG. 6 shows a bird's-eye view, and the right part of FIG. 6 shows a camera image. In FIG. 6, the symbol GL indicates guide lines that prompt the user to set the position of the terminal device 100 when the bird's-eye view app is launched (or when the vehicle M starts moving). The user places the terminal device 100 inside the vehicle so that the intersection of the displayed guide lines coincides with the vanishing point (i.e., the point where the road disappears from view). The intersection point set at this time is read into the bird's-eye view app as the initial value of the vanishing point. FIG. 6 shows an example in which only the vehicle M is displayed on the bird's-eye view immediately after the bird's-eye view app is launched.

[0029] Fig. 7 is a diagram showing another example of a bird's-eye view and a camera image displayed on the display unit 20. Fig. 7 shows an example in which the distance estimation unit 120 estimates the vertical distances and horizontal distances relative to the host vehicle M for four surrounding vehicles and one motorcycle detected by the object detection unit 110, and the bird's-eye view generation unit 130 generates a bird's-eye view based on the estimated distances. As shown in the left part of Fig. 7, by referring to the bird's-eye view, the driver of the host vehicle M can more reliably recognize the surrounding vehicles of the host vehicle M and use this information in his or her driving.

[0030] [Determine vanishing point] As described above, the vanishing point of the image captured by the camera 10 is utilized to estimate the distance between the vehicle M and the surrounding vehicles. Immediately after launching the bird's-eye view app, the user determines the vanishing point using guide lines, but once the vehicle M starts traveling, the vanishing point of the image may change depending on, for example, the road surface conditions. Therefore, in order to accurately estimate the distance between the vehicle M and the surrounding vehicles and generate a bird's-eye view, it is necessary to accurately determine the vanishing point. A method for determining the vanishing point in this embodiment will be described below.

[0031] FIG. 8 is a diagram illustrating a first method for determining a vanishing point V according to this embodiment. In FIG. 8, the symbol VR indicates a virtual road area in the camera image, and the symbol TR indicates a target area as a partial area of the virtual road area. More specifically, the virtual road area VR is a rectangular area ranging from the bottom edge of the camera image to the height position of the vanishing point, and the target area TR is an area corresponding to a predetermined percentage of the range from the top edge of the virtual road area VR. Furthermore, the symbol PV indicates the vanishing point calculated in the previous control cycle, and the symbol CV indicates the vanishing point calculated in the current control cycle.

[0032] First, when a nearby vehicle is detected by the object detection unit 110, the virtual line drawing unit 140 sets a bounding box BB surrounding the detected nearby vehicle. In the case of Fig. 8, as an example, the virtual line drawing unit 140 sets a bounding box BB1 for nearby vehicle M1 and a bounding box BB2 for nearby vehicle M2.

[0033] The virtual line drawing unit 140 further draws a virtual line VL extending in the horizontal direction of the image at a position at the same height as the upper end of the surrounding vehicle for which the bounding box BB is set (the upper end of the bounding box BB). In the case of Fig. 8, the virtual line drawing unit 140 draws a virtual line VL1 corresponding to the surrounding vehicle M1 and a virtual line VL2 corresponding to the surrounding vehicle M2.

[0034] Next, the vanishing point determination unit 160 determines whether or not there are a predetermined number of surrounding vehicles in the target area TR (in other words, whether or not there are a predetermined number of surrounding vehicles whose longitudinal distances estimated by the distance estimation unit 120 are equal to or greater than a predetermined distance). If it is determined that there are a predetermined number of surrounding vehicles in the target area TR, the vanishing point determination unit 160 obtains a new virtual line VL' by shifting each virtual line VL downward by a predetermined distance. In the case of FIG. 8, the virtual line drawing unit 140 obtains a virtual line VL1' by shifting downward by a predetermined distance the virtual line VL1 corresponding to the surrounding vehicle M1, and obtains a virtual line VL2' by shifting downward by a predetermined distance the virtual line VL2 corresponding to the surrounding vehicle M2.

[0035] Next, the vanishing point determination unit 160 calculates the average value of the new virtual lines VL' to obtain an average virtual line AVL. The vanishing point determination unit 160 determines the weighted average value between the previous vanishing point PV and the average virtual line AVL as the current vanishing point CV. In this manner, in this embodiment, the new virtual line VL' obtained by shifting the virtual line VL drawn on the surrounding vehicles present in the target area TR downward by a fixed distance is determined as the height position of the vanishing point. Furthermore, the average value of the new virtual lines VL' for multiple surrounding vehicles present in the target area TR is calculated, and the weighted average value with the previous vanishing point PV is obtained, thereby determining the current vanishing point CV with high accuracy. Through the above processing, the vanishing point determination unit 160 can calculate the vanishing point with high accuracy through simpler calculations without using techniques with high computational costs such as machine learning.

[0036] The method of determining the vanishing point V described with reference to FIG. 8 is executed when a predetermined number or more of peripheral vehicles are present in the target area TR of the virtual road area VR. However, depending on the road traffic conditions, there may be cases where there are no predetermined number or more of peripheral vehicles present in the target area TR, or where the field of view of the camera 10 is obstructed due to the presence of peripheral vehicles in the near-forward area of the host vehicle M (i.e., the area excluding the target area TR from the virtual road area VR). In such cases, it may be impossible to determine the vanishing point V using the above-described method. Therefore, when there are no predetermined number or more of peripheral vehicles present in the target area TR, or when peripheral vehicles are present in the near-forward area of the host vehicle M and obstruct the field of view of the camera 10, the vanishing point determination unit 160 determines the vanishing point V using a second determination method described below. Note that whether the field of view of the camera 10 is obstructed can be determined, for example, based on whether the proportion of the area occupied by peripheral vehicles in the near-forward area of the host vehicle M shown in the camera image is equal to or greater than a predetermined value.

[0037] Fig. 9 is a diagram for explaining a second method for determining a vanishing point V according to this embodiment. The upper part of Fig. 9 shows a camera image, and the lower part of Fig. 9 shows a road area extracted by inputting the camera image into a trained model that has been trained to output a road area in an image when the image is input.

[0038] When there are not a predetermined number of surrounding vehicles in the target region TR or when the view of the camera 10 is obstructed, the segmentation execution unit 150 extracts the road region by inputting the camera image into a trained model that has been trained to output the road region in the image when the image is input. Such a trained model can be generated, for example, using a semantic segmentation technique based on training data in which pixels representing the road region are labeled. Note that the training data used in this case is data in which pixels representing vehicles are not labeled (training data that does not include vehicle classes). Using such training data, the road region can be extracted as a simple shape.

[0039] When the road area is extracted by the segmentation execution unit 150, the vanishing point determination unit 160 fits left and right road boundary lines (straight lines) to the boundaries of the extracted road area and determines the left and right road boundary lines using, for example, the least squares method. The vanishing point determination unit 160 determines the intersection of the determined left and right road boundary lines as the vanishing point V. In this way, the vanishing point determination unit 160 determines the vanishing point V using either the first determination method or the second determination method depending on whether a predetermined number of or more surrounding vehicles are present in the target area TR and whether the view of the camera 10 is obstructed. This makes it possible to determine the vanishing point V using the optimal method depending on the traffic conditions on the road.

[0040] Next, the flow of processing executed by the bird's-eye view application will be described with reference to Fig. 10. Fig. 10 is a flowchart showing an example of the flow of processing executed by the bird's-eye view application. The processing of this flowchart is executed, for example, when the bird's-eye view application is started.

[0041] First, when the bird's-eye view application is started, it receives an initial value setting of the vanishing point from the user (step S100). Next, the bird's-eye view application acquires a camera image captured by the camera 10 (step S102).

[0042] Next, the bird's-eye view application detects surrounding vehicles in the camera image and sets a bounding box BB for the detected surrounding vehicles (step S104). Next, the bird's-eye view application selects a bounding box included in the target region TR of the virtual road region VR (step S106). Next, the bird's-eye view application determines whether there are a predetermined number or more of bounding boxes (i.e., a predetermined number of surrounding vehicles) contained in the target area TR of the virtual road area VR and whether the view of the camera 10 is not obstructed (step S108).

[0043] If it is determined that the target region TR of the virtual road region VR contains a predetermined number of bounding boxes or more and that the field of view of the camera 10 is not obstructed, the bird's-eye view application draws a virtual line VL extending in the horizontal direction of the image at a position at the same height as the upper end of the selected bounding box (step S110). Next, the bird's-eye view application obtains a new virtual line VL' by shifting the virtual line VL downward by a certain distance for each of the multiple surrounding vehicles, calculates an average virtual line AVL of the new virtual lines VL', and determines the weighted average value between the previous vanishing point PV and the average virtual line AVL as the current vanishing point CV (step S112).

[0044] On the other hand, if it is determined that the target region TR of the virtual road region VR does not contain a predetermined number of bounding boxes or that the view of the camera 10 is obstructed, the bird's-eye view application extracts the road region by performing segmentation, which inputs the camera image into a trained model (step S114). Next, the bird's-eye view application fits the left and right road boundary lines to the extracted road region and determines the intersection point as the current vanishing point CV (step S116).

[0045] When the current vanishing point CV is determined, the bird's-eye view application estimates the distance between the vehicle M and the surrounding vehicles based on the current vanishing point CV, generates a bird's-eye view, and displays the generated bird's-eye view together with the camera image (step S118). After that, the process of this flowchart returns to step S102, and the current vanishing point CV is used as the previous vanishing point PV in the next control cycle. This process is repeated, for example, until the operation of the bird's-eye view application is stopped or the vehicle M comes to a stop.

[0046] According to the present embodiment described above, the vanishing point determination unit 160 determines the vanishing point by selecting the most appropriate determination method from the first determination method and the second determination method depending on whether or not a predetermined number of surrounding vehicles or more exist in the target area TR of the virtual road area VR, and whether or not the field of view of the camera 10 is obstructed. This makes it possible to determine the vanishing point with high accuracy while keeping calculation costs low.

[0047] The above-described embodiment can be expressed as follows. a storage device storing a program; a hardware processor; The hardware processor executes the program stored in the storage device, Detecting an object in an image captured in a substantially horizontal direction by an image sensor; If the detected object is a vehicle, estimating a distance between the vehicle and the image sensor; drawing a virtual line extending in a lateral direction of the image at a position at the same height as the top end of the vehicle in the image; If the estimated distance is less than a threshold, a segmentation process is performed to extract from the image a road area on which the vehicle is traveling; If the estimated distance is equal to or greater than the threshold, a height position of a vanishing point of the image is determined based on the virtual line, and if the estimated distance is less than the threshold, a height position of the vanishing point is determined based on the extracted road area. The image processing device is configured as follows.

[0048] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0049] 10 Camera 20 Display section 100 Terminal Device 110 Object detection unit 120 Distance estimation unit 130 Bird's-eye view generation unit 140 Virtual Line Drawing Section 150 Segmentation Execution Unit 160 Vanishing point determination part

Claims

1. an object detection unit that detects an object in an image captured in a substantially horizontal direction by an image sensor; a distance estimation unit that estimates a distance between the vehicle and the image sensor when the detected object is a vehicle; a virtual line drawing unit that draws a virtual line extending in a lateral direction of the image at a position at the same height as an upper end of the vehicle in the image; a segmentation execution unit that executes a segmentation process to extract a road area on which the vehicle is traveling from the image when the number of vehicles for which the estimated distance is less than a threshold is less than a predetermined number or when the field of view of the image sensor is obstructed; and a vanishing point determination unit that, when there are at least the predetermined number of vehicles whose estimated distance is equal to or greater than the threshold and the field of view of the image sensor is not obstructed, determines the height position of the vanishing point of the image based on the virtual line, and determines the horizontal position of the vanishing point based on the virtual line and an initial value or a previous value of the vanishing point, and, when there are less than the predetermined number of vehicles whose estimated distance is equal to or greater than the threshold or the field of view of the image sensor is obstructed, determines the height position and horizontal position of the vanishing point based on the extracted road area, Image processing device.

2. At least the distance estimation unit and the vanishing point determination unit perform processing repeatedly and synchronously for each control cycle, the distance estimation unit estimates the distance in the current control cycle using the height position of the vanishing point determined by the vanishing point determination unit in the previous control cycle; The image processing device according to claim 1 .

3. the segmentation execution unit extracts the road area by inputting an image acquired by an image sensor into a trained model that has been trained to output the road area in the image when the image is input; 3. The image processing device according to claim 1 or 2.

4. The trained model is trained to output a vehicle in the image as an area representing the road area. The image processing device according to claim 3 .

5. When the estimated distance is equal to or greater than the threshold, the vanishing point determination unit determines a position obtained by shifting the virtual line downward by a predetermined distance as the height position of the vanishing point. The image processing device according to claim 1 .

6. the vanishing point determination unit, when there are a plurality of vehicles whose estimated distances are equal to or greater than the threshold, shifts the plurality of virtual lines corresponding to the plurality of vehicles downward by a predetermined distance, and determines an average position of height positions of the plurality of virtual lines as the height position of the vanishing point. The image processing device according to claim 1 .

7. a bird's-eye view generating unit that generates a bird's-eye view including the detected object from the image based at least on the determined height position of the vanishing point; the bird's-eye view generation unit causes the generated bird's-eye view to be displayed on a display unit of the image processing device. The image processing device according to claim 1 .

8. The computer Detecting an object in an image captured in a substantially horizontal direction by an image sensor; If the detected object is a vehicle, estimating a distance between the vehicle and the image sensor; a virtual line extending in a lateral direction of the image is drawn at a position at the same height as the top end of the vehicle in the image; If the number of vehicles whose estimated distance is equal to or greater than a threshold is less than a predetermined number or if the view of the image sensor is obstructed, a segmentation process is performed to extract from the image a road area on which the vehicle is traveling; If there are at least the predetermined number of vehicles whose estimated distance is equal to or greater than the threshold and the field of view of the image sensor is not obstructed, the height position of the vanishing point of the image is determined based on the virtual line, and the horizontal position of the vanishing point is determined based on the virtual line and the initial value or previous value of the vanishing point, whereas if there are less than the predetermined number of vehicles whose estimated distance is equal to or greater than the threshold or the field of view of the image sensor is obstructed, the height position and horizontal position of the vanishing point are determined based on the extracted road area. Image processing methods.

9. On the computer, Detecting an object in an image captured in a substantially horizontal direction by an image sensor; For the image, if the detected object is a vehicle, estimating a distance between the vehicle and the image sensor; a virtual line extending in a lateral direction of the image is drawn at a position at the same height as the top end of the vehicle in the image; If the number of vehicles whose estimated distance is equal to or greater than a threshold is less than a predetermined number or if the view of the image sensor is obstructed, a segmentation process is performed to extract from the image a road area on which the vehicle is traveling; If there are at least the predetermined number of vehicles whose estimated distance is equal to or greater than the threshold and the field of view of the image sensor is not obstructed, the height position of the vanishing point of the image is determined based on the virtual line, and the horizontal position of the vanishing point is determined based on the virtual line and the initial value or previous value of the vanishing point, whereas if there are less than the predetermined number of vehicles whose estimated distance is equal to or greater than the threshold or the field of view of the image sensor is obstructed, the height position and horizontal position of the vanishing point are determined based on the extracted road area. program.

Images