Image processing device, image processing method, and program

Abstract

To provide an image processing device, an image processing method, and a program configured to perform more appropriate calibration on images captured by an imaging device mounted on a mobile body.SOLUTION: A driving assistance apparatus includes: an acquisition unit configured to acquire a first image obtained by imaging a mobile body from an imaging device mounted on the mobile body in a first direction and a second image obtained by imaging the mobile body in a second direction different from than the first direction; an extraction unit configured to extract feature points from the first and the second images acquired by the acquisition unit; a first detection unit configured to detect a road area included in the first image; a second detection unit configured to detect a road area included in the second image; a first feature point extraction unit configured to extract a feature point in the road area detected by the first detection unit, out of the feature points extracted, as a first feature point; a second feature point extraction unit configured to extract a feature point in the road area detected by the second detection unit, out of the feature points extracted by the extraction unit, as a second feature point; and a calibration unit configured to calibrate the first and the second images based on the first and the second feature points.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an image processing method, and a program. [Background technology]

[0002] In recent years, efforts to provide access to sustainable transportation systems that take into consideration vulnerable traffic participants have been gaining momentum. Toward this goal, efforts are being focused on research and development into driver assistance technologies to further improve traffic safety and convenience. In this context, a technology is known that acquires an image from a camera mounted on a moving vehicle, extracts feature points from an extraction area of ​​the acquired image that is set or changed based on the external environment of the moving vehicle in the camera's shooting direction, and estimates the camera's posture from the extracted feature points (see, for example, Patent Document 1). Another known technology estimates a road surface range using feature points captured in an image from an on-board camera other than the on-board camera performing the calibration, and performs calibration using only feature points that are present in the estimated road surface range among the feature points captured in the image from the on-board camera performing the calibration (see, for example, Patent Document 2). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-33605 [Patent Document 2] Japanese Patent Application Publication No. 2019-28665 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in driving assistance technology, when processing images captured in multiple directions from an imaging device mounted on a moving body, there has been a problem in that it may not be possible to properly calibrate the images captured by the imaging device due to factors such as misalignment of the imaging device on the moving body or variations in the product performance of the imaging device.

[0005] In order to solve the above-mentioned problems, one object of the present application is to provide an image processing device, an image processing method, and a program that can perform more appropriate calibration on images captured by an imaging device mounted on a moving object, thereby contributing to the development of sustainable transportation systems. [Means for solving the problem]

[0006] The image processing device, image processing method, and program according to the present invention employ the following configuration. (1): An image processing device according to one embodiment of the present invention is an image processing device including: an acquisition unit that acquires a first image captured in a first direction of the moving body from an imaging device mounted on the moving body, and a second image captured in a second direction different from the first direction; an extraction unit that extracts feature points from the first image and the second image acquired by the acquisition unit; a first detection unit that detects a road area included in the first image; a second detection unit that detects a road area included in the second image; a first feature point extraction unit that extracts, from the feature points extracted by the extraction unit, feature points of the road area detected by the first detection unit as first feature points; a second feature point extraction unit that extracts, from the feature points extracted by the extraction unit, feature points of the road area detected by the second detection unit as second feature points; and a calibration unit that calibrates the first image and the second image based on the first feature points and the second feature points.

[0007] (2): In the above aspect (1), the imaging device includes a first imaging unit that captures the first image and a second imaging unit that captures the second image, and the calibration unit derives the relative angle between the first imaging unit and the second imaging unit based on the calibration results of the first image and the second image.

[0008] (3): In the above aspect (1), the calibration unit calibrates the first image and the second image when the first feature point extraction unit and the second feature point extraction unit each extract feature points included in the road area for a predetermined number of consecutive frames or more.

[0009] (4): In the above aspect (2), the calibration unit derives the relative angle based on a normal vector of the road area included in the first image relative to the road surface and a normal vector of the road area included in the second image relative to the road surface, and converts the coordinate system of the second image to the coordinate system of the first image based on the derived relative angle, thereby calibrating the first image and the second image.

[0010] (5): In the above aspect (3), the first feature point extraction unit extracts, as first feature points, feature points of a first road area along which the moving body moves and feature points of a second road area intersecting with the first road area from the image frames of the first image acquired by the acquisition unit at predetermined time intervals; the second feature point extraction unit extracts, as second feature points, feature points of the first road area and the second road area from the image frames of the second image acquired by the acquisition unit at predetermined time intervals; and the calibration unit calibrates the first image and the second image when the first feature points and the second feature points are extracted from the first image and the second image for a predetermined number of consecutive frames or more.

[0011] (6): In the aspect (1) above, the road area includes a first road area along which the moving body moves and a second road area intersecting the first road area.

[0012] (7): In the above aspect (6), the calibration unit calibrates the pitch angles of the first imaging unit that captures the first image and the second imaging unit that captures the second image based on the feature points of the first road area, and calibrates the roll angles of the first imaging unit and the second imaging unit based on the feature points of the second road area.

[0013] (8): Another aspect of the present invention is an image processing method in which a computer acquires a first image captured in a first direction of a moving body from an imaging device mounted on the moving body, and a second image captured in a second direction different from the first direction, extracts feature points from the acquired first image and second image, detects a road area contained in the first image, detects a road area contained in the second image, extracts, from the extracted feature points, feature points of the detected road area contained in the first image as first feature points, extracts, from the extracted feature points, feature points of the detected road area contained in the second image as second feature points, and calibrates the first image and the second image based on the first feature points and the second feature points.

[0014] (9): Another aspect of the present invention provides a program that causes a computer to acquire a first image captured in a first direction of a moving body using an imaging device mounted on the moving body, and a second image captured in a second direction different from the first direction, extract feature points from the acquired first and second images, detect a road area contained in the first image, detect a road area contained in the second image, extract, from the extracted feature points, feature points of the detected road area contained in the first image as first feature points, extract, from the extracted feature points, feature points of the detected road area contained in the second image as second feature points, and calibrate the first image and the second image based on the first feature points and the second feature points. [Effects of the Invention]

[0015] According to the above aspects (1) to (9), it is possible to perform more appropriate calibration on an image captured by an imaging device mounted on a moving object. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a diagram illustrating an example of a functional configuration of a driving assistance device 1 including an image processing device according to an embodiment. [Figure 2]1 is a diagram for explaining the installation positions and imaging directions of the cameras of the imaging device 10 relative to the vehicle M. FIG. [Figure 3] 1 is a diagram showing an example of a front image IM10 captured by the front camera 12 and a rear image IM20 captured by the first rear camera 14. FIG. [Figure 4] 10 is a diagram showing an example of a rear image IM30 captured by a second rear camera 16. FIG. [Figure 5] 10 is a flowchart illustrating an example of processing in a first calibration pattern. [Figure 6] 10 is a flowchart illustrating an example of processing in a second calibration pattern. [Figure 7] 10A and 10B are diagrams for explaining coordinate conversion processing in the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0017] Hereinafter, with reference to the drawings, embodiments of an image processing device, an image processing method, and a program of the present invention will be described. In the following example, an image processing device mounted on a mobile body will be described. The mobile body may include any mobile body on which a person (a passenger such as a driver) rides, such as a three-wheeled or four-wheeled vehicle, a two-wheeled vehicle, or a micromobility vehicle. The mobile body may also be equipped with a driving assistance device that assists the passenger (driver) of the mobile body in driving based on images processed by the image processing device. In the following description, the mobile body is assumed to be a four-wheeled vehicle (hereinafter referred to as "vehicle M") equipped with a driving assistance device. Vehicle M may be any of an automobile powered by an internal combustion engine such as a diesel engine or a gasoline engine, an electric automobile powered by an electric motor, or a hybrid automobile equipped with both an internal combustion engine and an electric motor. In the following description, the forward direction of the vehicle M is the plus X direction, the rearward direction of the vehicle M is the minus X direction, the width direction of the vehicle M is the right direction based on the plus X direction, the left direction is the minus Y direction, and the height direction of the vehicle M, which is the direction perpendicular to the X direction and Y direction, is the plus Z direction.

[0018] FIG. 1 is a diagram illustrating an example of the functional configuration of a driving assistance device 1 including an image processing device according to an embodiment. The driving assistance device 1 illustrated in FIG. 1 includes, for example, an imaging device 10, a recognition unit 20, a driving assistance unit 30, a notification control unit 40, and an image processing device 100. The recognition unit 20, the driving assistance unit 30, the notification control unit 40, and the image processing device 100 are realized by, for example, a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Some or all of these components may be realized by hardware (including circuitry) such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), a GPU (Graphics Processing Unit), or a SOC (System On Chip), or may be realized by a combination of software and hardware. The program may be stored in advance in a storage device (a storage device having a non-transitory storage medium) such as an HDD (Hard Disk Drive) or a flash memory, or may be stored in a removable storage medium (a non-transitory storage medium) such as a DVD or CD-ROM, and installed by inserting the storage medium into a drive device. Note that the vehicle M of the embodiment is equipped with, in addition to the configuration of the driving assistance device 1 shown in Fig. 1, a configuration for running the vehicle M (for example, driving operators, drive devices such as an engine and a motor, a steering device, a braking device, various vehicle sensors (for example, a position sensor and a speed sensor)), a navigation device for providing route guidance, etc., a display device, a speaker, and various other on-board devices.

[0019] The imaging device 10 captures images of the surroundings of the vehicle M. For example, it is a digital camera using a solid-state imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The imaging device 10 may be a stereo camera. The imaging device 10 may also be a camera used in a drive recorder, for example. In the example of FIG. 1, the imaging device 10 includes a front camera 12, a first rear camera 14, and a second rear camera 16 as multiple imaging units. The front camera 12 is an example of a "first imaging unit." At least one of the first rear camera 14 and the second rear camera 16 is an example of a "second imaging unit."

[0020] The front camera 12 captures an image of a predetermined area in front of the vehicle M. The first rear camera 14 and the second rear camera 16 capture an image of a predetermined area in a direction different from the front (for example, rear of the vehicle M). The front is an example of a "first direction," and the rear is an example of a "second direction." At least one of the front image (an example of a first image) captured by the front camera 12 and the rear image (an example of a second image) captured by the first rear camera 14 or the second rear camera 16 may include an area in the lateral direction (side) of the vehicle M.

[0021] FIG. 2 is a diagram illustrating the installation positions and imaging directions of the cameras of the imaging device 10 relative to the vehicle M. The front camera 12 and the first rear camera 14 of the imaging device 10 are attached, for example, near the top of the front windshield in front of the vehicle M as shown in FIG. 2. In the example of FIG. 2, the front camera 12 and the first rear camera 14 are integrally configured within a predetermined distance. The term "integrally configured" may include, for example, a configuration in which the front camera 12 and the first rear camera 14 are housed in a single housing or a configuration in which they are connected (coupled). In this configuration, for example, when the front camera 12 captures an image of a predetermined angle of view area VA1 centered in a direction A1 ahead of the vehicle M (the positive X-axis direction in the figure), the first rear camera 14 captures an image of a predetermined angle of view area VA2 including the interior of the vehicle, centered in a direction A2 opposite to the direction A1 ahead of the vehicle M. Therefore, if, due to a positional shift caused by some factor during or after installation, front camera 12 captures an image of an angle of view centered in a direction tilted downward by angle θ1 with respect to front direction A1 of vehicle M as the reference, first rear camera 14 captures an image of an angle of view centered in a direction tilted upward by angle θ1 with respect to direction A2 opposite to front direction A1 as the reference. In other words, in the embodiment, if a shift occurs in the imaging direction (angle of view) of one of front camera 12 and first rear camera 14, which are integrally configured, a similar shift occurs in the imaging direction (angle of view) of the other camera in a symmetrical direction with respect to the installation position of imaging device 10 as the center.

[0022] 2, the second rear camera 16 is mounted on the rear of the vehicle body (outside the vehicle), but it may also be mounted on the rear of the vehicle interior. The second rear camera 16 is mounted on a position across the center of the pitch rotation direction (e.g., center of gravity) G of the vehicle M from the mounting position of the front camera 12. The second rear camera 16 captures an image of a predetermined field of view VA3 centered in a direction A3 (the same direction as direction A2) opposite to the front direction A1 of the vehicle M. Because the second rear camera 16 captures the field of view VA3 that does not include the vehicle interior, it can capture a wider range of images of the outside of the vehicle (road shape, etc.) than the first rear camera 14.

[0023] The second rear camera 16 may be electrically connected to the front camera 12 and the first rear camera 14 by a cable or the like, or may be connected via a frame member. In the embodiment, the front camera 12 (and the first rear camera 14) and the second rear camera 16 are separate entities. Therefore, the shooting directions of the respective cameras may be misaligned due to some factor during or after installation, or due to variations in product performance, etc., resulting in unrelated misalignments between them.

[0024] The imaging device 10 is not limited to the configuration shown in FIG. 2 , and may include, for example, multiple imaging units that capture images in front of the vehicle M, or one or three or more imaging units that capture images behind the vehicle M. Furthermore, the imaging device 10 may include a side camera that captures images laterally (to the side) of the vehicle M, in addition to the front camera 12, the first rear camera 14, and the second rear camera 16. Instead of the above-described camera configuration, the imaging device 10 may include a fisheye camera that can capture images of the surroundings, including the front and rear of the vehicle M, at a wide angle (e.g., 360 degrees). Images captured by the fisheye camera may be divided into multiple images (for example, a front image, a rear image, a side image), etc., depending on the shooting direction. Each camera in the imaging device 10 repeatedly captures images at a predetermined interval, and outputs the captured images (camera images) to the image processing device 100.

[0025] Returning to FIG. 1, the image processing device 100 acquires an image captured by the imaging device 10 and performs processing to convert the coordinate system of the image (hereinafter referred to as the camera coordinate system) into a coordinate system different from the camera coordinate system. The camera coordinate system includes a front camera coordinate system corresponding to a front image and a rear camera coordinate system corresponding to a rear image. The coordinate system different from the camera coordinate system is, for example, a coordinate system based on the position of the vehicle M when viewed from above (bird's-eye view coordinate system). The image converted into the bird's-eye view coordinate system is used by the recognition unit 20 to recognize the surrounding situation of the vehicle M and to notify (display) the occupants (driver, etc.) of the vehicle M.

[0026] The image processing device 100 includes, for example, an acquisition unit 110, an extraction unit 120, a first detection unit 130, a second detection unit 132, a first feature point extraction unit 140, a second feature point extraction unit 142, a calibration unit 150, a coordinate conversion unit 160, and a memory unit 170.

[0027] The storage unit 170 may be realized by a storage device such as an HDD or flash memory, or a solid state drive (SSD), an electrically erasable programmable read only memory (EEPROM), a read only memory (ROM), or a random access memory (RAM). The storage unit 170 stores, for example, images acquired by the acquisition unit 110, processing results by the calibration unit 150 and the coordinate conversion unit 160, programs, and various other information. The storage unit 170 may also store map information. The map information is, for example, information representing road shapes in association with location information (latitude and longitude information) using links indicating roads and nodes connected by the links. The map information may also include information such as the curvature and gradient of the road, the number of lanes, the width, and centerlines of the lanes, or lane boundary information such as road dividing lines that separate the lanes. The map information may also include traffic regulation information, location information of branches, junctions, intersections, T-junctions, etc., facility information of buildings, parking lots, etc., POI (Point Of Interest) information, etc. The map information may be updated as needed by the vehicle M or the driving assistance device 1 communicating with other devices.

[0028] The acquisition unit 110 acquires camera images captured by the imaging device 10 at predetermined intervals. The acquisition unit 110 includes, for example, a first acquisition unit 111 and a second acquisition unit 112. The first acquisition unit 111 acquires a front image captured by the front camera 12 (an example of a first image captured in a first direction of a moving object). The second acquisition unit 112 acquires a rear image captured by at least one of the first rear camera 14 and the second rear camera 16 (an example of a second image captured in a second direction different from the first direction).

[0029] The extraction unit 120 extracts feature points included in the forward image and the rearward image acquired by the acquisition unit 110. For example, the extraction unit 120 performs known image analysis processing, such as edge extraction processing, on the forward image and the rearward image, and extracts feature points of objects in real space included in the image (e.g., traffic signals, road signs, traffic participants such as pedestrians and other vehicles, buildings, as well as road areas, road dividing lines, stop lines, etc.) based on the image analysis processing results. In this case, the extraction unit 120 extracts, for example, a sequence of points on the edges of the objects included in the image as feature points (feature point group). Furthermore, the extraction unit 120 may extract feature points using a trained model that has been trained to output, when a forward image or a rearward image is input, the edges of objects (e.g., buildings, road structures, etc.) depicted in the image as a point group. This trained model may be stored in the storage unit 170 in advance or may be acquired from an external device via a communication device (not shown) mounted on the vehicle M. Furthermore, the extraction unit 120 may extract feature points using, for example, a Visual SLAM (Simultaneous Localization and Mapping) technique, which is a technique for determining the self-position in three dimensions from image data captured by the imaging device 10. The method for extracting feature points on an image is not limited to the above example, and other known methods may be used.

[0030] The extraction unit 120 may be configured so that a first extraction unit that extracts feature points from a forward image and a second extraction unit that extracts feature points from a rearward image are provided separately, and further, an extraction unit that extracts feature points from a rearward image captured by the first rearward camera 14 and an extraction unit that extracts feature points from a rearward image captured by the second rearward camera 16 are provided separately.

[0031] The first detection unit 130 detects a road area included in the forward image. The second detection unit 132 detects a road area included in the rearward image. The road area includes, for example, a driving lane area on which the vehicle M is driving (moving) (hereinafter referred to as the "own vehicle road area") and an area of ​​lanes intersecting the own vehicle road area (hereinafter referred to as the "cross road area"). The own vehicle road area may include adjacent lanes and oncoming lanes extending in the same direction in addition to the driving lane of the vehicle M. The cross road area is, for example, a road that is connected to the own vehicle road area within a predetermined angle range including a right angle at an intersection or a T-junction. The own vehicle road area is an example of a "first road area." The cross road area is an example of a "second road area."

[0032] The first feature point extraction unit 140 extracts, as first feature points, feature points of the road area detected by the first detection unit 130 from among the feature points (feature point group) extracted by the extraction unit 120. The second feature point extraction unit 142 extracts, as second feature points, feature points of the road area detected by the second detection unit 132 from among the feature points (feature point group) extracted by the extraction unit 120. The first feature points and the second feature points are used for image calibration in the embodiment.

[0033] The calibration unit 150 calibrates the camera images (front image and rear image) captured by the image capture device 10 based on the first feature points extracted by the first feature point extraction unit 140 and the second feature points extracted by the second feature point extraction unit 142. For example, when the coordinate system of the camera image (camera coordinate system) is misaligned with the reference coordinate system (the amount of misalignment is greater than or equal to a threshold) due to misalignment during installation of the image capture device 10, misalignment due to vibration after installation or contact with an occupant, or variations in camera performance, the calibration unit 150 calibrates the three-dimensional axes of the camera coordinate system so that the camera coordinate system matches the reference coordinate system (the amount of misalignment is less than the threshold). The reference coordinate system is, for example, a coordinate system based on the attitude of the vehicle M (vehicle coordinate system), and calibration of the camera images involves calibrating the camera coordinate systems of the first image and the second image to the vehicle coordinate system. Note that calibrating an image may also be referred to as calibrating the attitude of each camera of the image capture device 10. For example, the calibration unit 150 calibrates at least one of the pitch (e.g., the tilt of each camera in the longitudinal direction of the imaging device 10), the roll direction (the tilt of each camera in the lateral direction), and the yaw direction (the rotation direction of each camera when viewed from above) so that a three-dimensional vehicle coordinate system based on the vehicle M matches a three-dimensional camera coordinate system. Note that the calibration unit 150 may derive parameters for image calibration (e.g., information relating to at least one of the pitch angle, roll angle, and yaw angle).

[0034] The coordinate conversion unit 160 converts the coordinate system (camera coordinate system) of the image acquired by the acquisition unit 110 into a bird's-eye view coordinate system, which is an example of another different coordinate system. In this case, the coordinate conversion unit 160 may perform coordinate conversion on the image calibrated by the calibration unit 150 using reference coordinate conversion parameters stored in advance in the storage unit 170, or may perform coordinate conversion including calibration on the image acquired by the acquisition unit 110 based on the reference coordinate conversion parameters and the above-mentioned image calibration parameters. This allows for more accurate image conversion.

[0035] The recognition unit 20 recognizes the surrounding conditions of the vehicle M based on an image converted into a bird's-eye coordinate system by the coordinate conversion unit 160 (hereinafter referred to as a "bird's-eye image"). For example, the recognition unit 20 recognizes objects present around the vehicle M (within a predetermined distance from the vehicle M) based on the bird's-eye image. For example, object recognition may be performed using a model based on deep learning or deep machine learning, object recognition based on a pattern matching technique, or an object recognition technique that combines these. Furthermore, the recognition unit 20 may recognize objects around the vehicle M by referring to map information stored in the storage unit 170 based on position information of the vehicle M acquired by a position sensor such as a GPS (Global Positioning System) device included in the vehicle sensor.

[0036] Here, the objects include, for example, traffic participants such as other vehicles and pedestrians, lane boundaries (road boundaries) including road dividing lines, road shoulders, curbs, medians, and guardrails, stop lines, obstacles, traffic signals, road signs, toll booths, bridges, etc. The objects may also include features such as buildings and roadside trees around vehicle M. Of the above objects, traffic participants such as other vehicles and pedestrians are recognized based on a bird's-eye view image, while other objects are recognized based on either or both of the bird's-eye view image and map information. The recognition unit 20 may also recognize the type, shape, size, etc. of various objects, as well as the position (relative position to vehicle M) and speed (relative speed to vehicle M) of the objects.

[0037] The driving assistance unit 30 provides driving assistance to an occupant (such as a driver) of the vehicle M based on the recognition result by the recognition unit 20. For example, the driving assistance unit 30 determines whether the vehicle M will deviate from a driving lane (e.g., the region of the vehicle's own road) defined by road dividing lines recognized by the recognition unit 20, and if there is a possibility of deviation, notifies the driver of the vehicle M via the notification control unit 40, or controls the steering of the vehicle M using a steering device (not shown) to prevent the vehicle M from deviating from the driving lane (so that the vehicle M moves toward the center of the driving lane). Furthermore, the driving assistance unit 30 recognizes obstacles such as other vehicles present in the vicinity of the vehicle M (within a predetermined distance), and if it determines that there is a possibility of contact with the obstacle based on the relative position and relative speed of the vehicle M, notifies the occupant via the notification control unit 40, or performs driving control (at least one of speed control and steering control) to avoid contact.

[0038] The notification control unit 40 notifies the occupant (driver) of the vehicle M about the driving assistance based on the control by the driving assistance unit 30. In this case, the notification control unit 40 generates notification information such as a sound (alarm) or an image associated with the notification content to be notified to the occupant, and transmits the generated notification information to the terminal device T to output it.

[0039] Here, the terminal device T is, for example, a portable terminal device such as a smartphone or tablet terminal used by a driver who drives a vehicle M equipped with the driving assistance device 1. The terminal device T, for example, runs an application for receiving driving assistance from the driving assistance device 1. The application receives information transmitted by the driving assistance device 1, and displays an image based on the notification on a display unit of the terminal device T or emits a sound from a speaker of the terminal device T. The terminal device T is an example of a "notification unit." The terminal device T is, for example, detachably attached to the vehicle M and used. For example, a holder for the terminal device T having a detachable part is provided on one or both of the terminal device T and the vehicle M, and the terminal device T is supported by the holder. Note that in the embodiment, if a device (notification unit) such as a navigation device, a display device, or a speaker is installed in the vehicle M, notification information may be output from the above-mentioned installed device instead of (or in addition to) the terminal device T based on an instruction from the driving assistance unit 30.

[0040] [Front and rear images] Here, examples of a forward image and a rearward image in the embodiment will be described with reference to the drawings. FIG. 3 is a diagram showing an example of a forward image IM10 captured by the forward camera 12 and an example of a rearward image IM20 captured by the first rearward camera 14. The forward image IM10 captures an area outside the vehicle (in front of the vehicle M) through the front windshield of the vehicle M. The rearward image IM20 captures an area behind the vehicle M, including the interior of the vehicle, and an area outside the vehicle (to the side or rear of the vehicle M) through the side windshield and rear windshield. FIG. 4 is a diagram showing an example of a rearward image IM30 captured by the second rearward camera 16. The rearward image IM30 captures an area outside the vehicle (behind the vehicle M) through the rear windshield of the vehicle M. The image processing device 100 calibrates these images and performs operations such as recognizing surrounding objects using the calibrated images.

[0041] [Image Calibration] Next, the image calibration process in this embodiment will be specifically described using several calibration patterns. For the sake of convenience, the following description will use the rear image IM30 captured by the second rear camera 16 as the rear image. However, instead of (or in addition to) the rear image IM30, the rear image IM20 captured by the first rear camera 14 may be used, or a side image captured of the side of the vehicle M may be used.

[0042] [First calibration pattern] In the first calibration pattern, the front image and the rear image are calibrated, and then the relative angle between the front camera 12 and the second rear camera 16 (the relative angle between the front image and the rear image) is derived based on the calibration results. Then, based on the derived relative angle, the coordinate system of one image (e.g., the rear image) is transformed into the coordinate system of the other image (e.g., the front image) to integrate both images, and further calibration is performed on the integrated image.

[0043] Fig. 5 is a flowchart showing an example of processing in the first calibration pattern. In the example of Fig. 5, the first acquisition unit 111 acquires a front image IM10 captured by the front camera 12 (step S100). Next, the extraction unit 120 extracts feature points from the front image IM10 (step S102). Next, the first detection unit 130 detects a road area included in the front image IM10 (step S104).

[0044] The processing of step S104 will now be described in detail. The first detection unit 130 detects the vehicle road area and the intersecting road area included in the forward image IM10. For example, the first detection unit 130 acquires point sequences (feature points arranged in the same direction (including the allowable error range) within a predetermined distance) on the left and right sides of the vehicle M included in the feature point group extracted by the extraction unit 120 as road dividing lines, and detects the area divided by the acquired road dividing lines as the vehicle road area. For example, the first detection unit 130 may divide the forward image IM10 into multiple divided areas and detect the vehicle road area for each divided area. Note that, as shown in the forward image IM10 and the rearward image IM20 in FIG. 3, the position where the vehicle road area exists is near the center of the image, which is somewhat easy to predict. Therefore, the first detection unit 130 may detect the vehicle road area based on the feature point group in a partial area of ​​the forward image IM10 that is predicted in advance to have a high probability of containing the vehicle road area (for example, a predetermined area including the center of the image). This reduces the processing load related to detecting the road area of ​​the vehicle.

[0045] The first detection unit 130 also detects point sequence portions that are tangent to the detected host road area at a predetermined angle, and if two (two) point sequences exist parallel to each other within a predetermined distance (including an allowable error range), it determines the two point sequences as road dividing lines and detects the area defined by the road dividing lines as a cross road area. The predetermined angle is, for example, a predetermined angle range (for example, approximately 75 to 105 degrees) that includes 90 degrees (a right angle) with respect to the extension direction of the host road area. The first detection unit 130 may detect a cross road area based on the feature point group, targeting a partial area of ​​the forward image IM10 that is predicted in advance to have a high possibility of containing a cross road area.

[0046] Furthermore, the first detection unit 130 may, for example, implement a function based on AI (Artificial Intelligence) and a function based on a predetermined model in parallel. For example, the function of detecting the host vehicle road area and the intersecting road area may be implemented by executing, in parallel, detection of the host vehicle road area and the intersecting road area by deep learning or the like and detection by a predetermined determination process (for example, a determination process based on pattern matching) on ​​the forward image IM10, and scoring both to comprehensively evaluate them.

[0047] When a specific road structure such as a traffic signal or a crosswalk is detected in the forward image IM10, the first detection unit 130 may perform a process of detecting an intersecting road area within a predetermined distance from the position where the road structure is detected. Furthermore, the first detection unit 130 may refer to map information stored in the storage unit 170 based on the position information of the vehicle M, and perform a process of detecting an intersecting road area when the position of the vehicle M is close to a position where an intersecting road is likely to exist, such as an intersection or a T-junction (within a predetermined distance). The position information of the vehicle M is acquired, for example, by a position sensor mounted on the vehicle M. This allows the detection process to be performed in an area where an intersecting road area is likely to exist, thereby enabling more efficient detection of the intersecting road area. In the example of FIG. 3, the first detection unit 130 detects the host vehicle road area AR10F ahead of the vehicle M from the forward image IM10, and detects intersecting road areas AR20L-1 and AR20R-1.

[0048] Next, the first feature point extraction unit 140 extracts feature points of the detected road area from the feature points of the forward image IM10 (step S106). Specifically, the first feature point extraction unit 140 extracts feature points included in the host road area AR10F and the intersecting road areas AR20L-1 and AR20R-1 detected by the first detection unit 130 from the feature points (group of feature points) extracted by the extraction unit 120.

[0049] Next, the calibration unit 150 calibrates the forward image IM10 based on the extracted feature points of the road area (step S108). For example, the calibration unit 150 estimates a least-squares plane for the feature points included in the host vehicle road area AR10F and the intersecting road areas AR20L-1 and AR20R-1 as the road surface. Next, the calibration unit 150 calibrates the forward image so that the camera coordinate system based on the estimated road surface matches (the deviation amount is less than a threshold value) with the vehicle coordinate system based on the attitude of the vehicle M (in other words, so that the estimated road surface matches the road surface when the attitude of the vehicle M is used as the reference). At this time, the calibration unit 150 may derive the deviation amount with respect to the vehicle coordinate system or may derive image calibration parameters.

[0050] Next, the calibration unit 150 determines whether the calibration has reached a predetermined accuracy (step S110). In the processing of step S110, for example, if the deviation amount between the estimated road surface and the road surface when the attitude of the vehicle M is used as a reference is less than a threshold for a predetermined number of consecutive frames or more, the calibration unit 150 determines that the calibration has reached the predetermined accuracy; otherwise, the calibration unit 150 determines that the calibration has not reached the predetermined accuracy. If it is determined that the predetermined accuracy has not been reached, the process returns to step S100, and the above processing is repeatedly performed for the next image frame until it is determined that the predetermined accuracy has been reached. Note that when processing the next image frame, the processing may be performed after calibrating the image using the image calibration parameters derived in advance in the processing of step S108.

[0051] Furthermore, the image processing device 100 performs the following steps S120 to S130 in parallel with (or before or after) the processing of steps S100 to S110. Specifically, first, the second acquisition unit 112 acquires a rear image IM30 captured by the second rear camera 16 (step S120). Next, the extraction unit 120 extracts feature points from the rear image IM30 (step S122). Next, the second detection unit 132 detects road areas (host road area, intersecting road area) included in the rear image IM30, for example, using a method similar to that used by the first detection unit 130 (step S124). In the example of FIG. 3, the second detection unit 132 detects a host road area AR10R-1 behind the vehicle M from the rear image IM20, and in the example of FIG. 4, detects a host road area AR10R-2 and intersecting road areas AR20L-2 and AR20R-2 behind the vehicle M from the rear image IM30.

[0052] Next, the second feature point extraction unit 142 extracts feature points of the road area (host vehicle road areas AR10R-1 and AR10R-2, and cross road areas AR20L-2 and AR20R-2) from the feature points of the rear image IM30 (step S126). Next, the calibration unit 150 calibrates the rear image IM30 based on the feature points of the extracted road area (step S128). Next, the calibration unit 150 determines whether the calibration has reached a predetermined accuracy (step S130). If it determines that the predetermined accuracy has not been reached, the process returns to step S120, and the above processing is repeated until it determines that the predetermined accuracy has been reached. In the processing of steps S128 and S130, the calibration unit 150 performs the same processing as in steps S108 and S110 described above on the feature points of the road area (host vehicle road areas AR10R-1 and AR10R-2, and cross road areas AR20L-2 and AR20R-2).

[0053] If it is determined in the processes of steps S110 and S130 that the calibration of both the front image IM10 and the rear image IM30 has reached a predetermined accuracy, the calibration unit 150 derives the relative angle between the front camera 12 and the second rear camera 16 from each of the calibrated images (in other words, the relative angle in the coordinate system between the front image IM10 and the rear image IM30) (step S140). For example, the calibration unit 150 calculates the relative angle of the rear image IM30 with respect to the front image IM10 as a reference (the amount of deviation in the coordinate system) based on image calibration parameters for the front image IM10 and image calibration parameters for the rear image IM30. Note that because the front image IM10 and the rear image IM30 were captured in different directions, the calibration unit 150 derives the relative angle after adjusting the orientation of the coordinate system based on a reference coordinate system that is based on the attitude of the same object, such as the vehicle M.

[0054] Next, the calibration unit 150 converts the camera coordinate system (rear camera coordinate system) of the second rear camera 16 into the camera coordinate system (front camera coordinate system) of the front camera 12 (forward image IM10) based on the relative angle (step S150). Next, the calibration unit 150 calibrates the forward image IM10 based on normal vectors of the road surface of the road areas obtained from both camera images (step S160). For example, the calibration unit 150 derives normal vectors of the road surface for the road areas (host road area, intersecting road area) included in each of the forward image IM10 and the rear image IM30. The normal vectors may be derived for each image, or may be derived separately for the host road area and the intersecting road area. When multiple normal vectors are derived, they may be averaged, or the normal vector for the road area with the highest priority may be derived. Then, the calibration unit 150 calibrates the front image IM10 (in other words, the attitude of the front camera 12) so that the derived normal vector matches a normal vector with respect to the road surface when the attitude of the vehicle M is used as a reference. In this case, the calibration unit 150 may derive image calibration parameters.

[0055] In the processing of step S160, the calibration unit 150 may calibrate the pitch direction (pitch angle) of the forward image IM10 (forward camera 12) using a normal vector of the host road area relative to the road surface, and may calibrate the roll direction (roll angle) using a normal vector of the cross road area relative to the road surface. The host road area is an area extending in the up-down direction of the image (forward or backward of the vehicle M), and the cross road area is an area extending in the left-right direction of the image (lateral direction of the vehicle M). Therefore, by calibrating the pitch direction of the vehicle M (or the image capture device 10) using the host road area and calibrating the roll direction of the vehicle M (or the image capture device 10) using the cross road area, more appropriate calibration can be performed in each direction.

[0056] Next, the calibration unit 150 calibrates the rear image IM30 (in other words, the attitude of the second rear camera 16) based on the calibration result (image calibration parameters) of the front image IM10 and the relative angle. This completes the processing of this flowchart.

[0057] According to the processing of the first calibration pattern described above, after performing calibration processing on each of the front image and the rear image, the relative angles of both camera images are derived, and calibration is performed by converting the coordinate system of one image to the coordinate system of the other image based on the derived relative angles. This improves the calibration accuracy of each image, and since more road area information can be obtained from both images, more accurate calibration processing can be performed. Furthermore, according to the processing of the first calibration pattern, even if there are factors such as misalignment of the multiple cameras attached to the vehicle M or product variations in the imaging devices, more appropriate calibration can be performed on the images captured by each imaging device.

[0058] [Second calibration pattern] In the second calibration pattern, feature points of the road area included in each of the forward image IM10 and the backward image IM30 are extracted, and then the forward image IM10 and the backward image IM30 are calibrated by combining the feature points.

[0059] FIG. 6 is a flowchart showing an example of processing in the second calibration pattern. In the example of FIG. 6, the first acquisition unit 111 acquires a forward image IM10 (image frame) captured by the forward camera 12 (step S200). Next, the extraction unit 120 extracts feature points from the forward image IM10 (step S202). Next, the first detection unit 130 detects the host road area included in the forward image IM10 (step S204) and detects a cross road area relative to the host road area (step S206). The host road area and the cross road area can be detected using, for example, the same method as the detection method used in the first calibration pattern described above. Next, the first feature point extraction unit 140 extracts feature points from the host road area from the feature points in the forward image IM10 (step S208) and extracts feature points from the cross road area (step S210).

[0060] Next, the calibration unit 150 determines whether or not feature points of both the vehicle road area and the intersecting road area have been extracted for a predetermined number of consecutive frames or more (step S212). If it is determined that feature points of both areas have not been extracted for a predetermined number of consecutive frames or more, the process returns to step S200, and the above process is repeated until it is determined that feature points of both areas have been extracted for a predetermined number of consecutive frames or more.

[0061] Furthermore, the image processing device 100 performs the following steps S220 to S232 in parallel with (or before or after) the processing of steps S200 to S212. Specifically, first, the second acquisition unit 112 acquires the rear image IM30 captured by the second rear camera 16 (step S220). Next, the extraction unit 120 extracts feature points from the rear image IM30 (step S222). Next, the second detection unit 132 detects the host road area included in the rear image IM30 (step S224) and detects intersecting road areas relative to the host road area (step S226). Next, the second feature point extraction unit 142 extracts feature points from the host road area from the feature points of the rear image IM30 (step S228) and extracts feature points from the intersecting road areas (step S230). Next, the calibration unit 150 determines whether or not feature points of both the vehicle road area and the intersecting road area have been extracted for a predetermined number of consecutive frames or more (step S232). If it determines that feature points of both areas have not been extracted for a predetermined number of consecutive frames or more, the process returns to step S220, and the above process is repeated until it determines that feature points of both areas have been extracted for a predetermined number of consecutive frames or more.

[0062] If it is determined in the processing of steps S212 and S232 that feature points have been extracted from both the vehicle road area and the intersecting road area in both the forward image IM10 and the rearward image IM30, the calibration unit 150 calibrates the forward image and the rearward image using these feature points (step S240).

[0063] Here, the processing of step S240 will be specifically described. For example, for each of the forward image IM10 and the rearward image IM20, the calibration unit 150 detects the movement of feature points between frames based on changes in the positions of feature points over time between the host vehicle road area AR10F, AR10R-1, and AR10R-2 (hereinafter abbreviated as "host vehicle road area AR10") and the cross road area AR20L-1, AR20R-1, AR20L-2, and AR20R-2 (hereinafter abbreviated as "cross road area AR20") included in two image frames at different times, and performs optical flow processing to represent the detected movement as a vector (motion vector). The motion vector includes, for example, information about the direction and amount (amount of displacement) of movement. The time interval (period) between two different image frames for acquiring a motion vector may be the period (or an integer multiple of the period) of the image frames acquired by the acquisition unit 110, or may be variably set based on the speed of the vehicle M, the size of each road area (own road area, intersecting road area), etc.

[0064] Furthermore, instead of using all feature points included in the vehicle road area AR10 and the intersecting road area AR20, the feature points used in the optical flow processing may be thinned out to a predetermined number or less. In this case, the calibration unit 150 may divide the vehicle road area AR10 and the intersecting road area AR20 into multiple divided areas and adjust the number of feature points in each divided area so that it is equal to or greater than a lower limit and equal to or less than an upper limit. Reducing the number of feature points used in the optical flow processing can reduce the processing load.

[0065] Furthermore, the calibration unit 150 sets normal vectors perpendicular to the road surface of the vehicle road area AR10 and the intersecting road area AR20 based on the motion vectors obtained by the optical flow processing and the traveling direction of the vehicle M at the time between two different image frames. For example, the calibration unit 150 extracts multiple motion vectors from the road area, sets the road surface (plane) of the vehicle road and the intersecting road based on the directions of the extracted multiple motion vectors (directions corresponding to the traveling direction of the vehicle M), and derives the normal vector to the set road surface. Note that the calibration unit 150 may derive normal vectors separately for the vehicle road area and the intersecting road area obtained from the forward image IM10 and the rearward image IM30.

[0066] Then, the calibration unit 150 performs calibration so that the deviation between the derived normal vector and a normal vector (reference normal vector) to the road surface based on the attitude of the vehicle M is equal to or less than a threshold value. Note that when there are multiple normal vectors, the calibration unit 150 may perform calibration by comparing an average normal vector obtained by averaging the normal vectors with the reference normal vector, or may perform calibration so that the error (least square error) between the multiple normal vectors and the reference normal vector is equal to or less than a threshold value.

[0067] In the embodiment, the front camera 12 and the second rear camera 16 are configured separately, and therefore the amount of deviation differs between them. Therefore, the calibration unit 150 separates the normal vector derived from the front image IM10 and the normal vector derived from the rear image IM30 and performs calibration so that they match the reference normal vector. The calibration unit 150 may also derive the relative angle between the front image and the rear image (the front camera 12 and the second rear camera 16) based on two image calibration parameters obtained by each calibration.

[0068] Furthermore, similar to the first calibration pattern, the calibration unit 150 may calibrate the pitch direction (pitch angle) of the front image IM10 (front camera 12) using a normal vector to the road surface of the host road area, and may calibrate the roll direction (roll angle) using a normal vector to the road surface of the intersecting road area. This allows for more appropriate calibration in each of the pitch and roll directions.

[0069] According to the processing of the second calibration pattern described above, calibration is performed using information on feature points of the road area obtained from each of the forward image and the rearward image, so that more accurate calibration processing can be performed. Furthermore, according to the processing of the second calibration pattern, even if there are factors such as misalignment of the multiple cameras attached to the vehicle M or product variations in the imaging devices, more appropriate calibration can be performed on the images captured by each imaging device.

[0070] [Coordinate conversion and recognition] Next, the processing of the coordinate conversion unit 160 and the recognition unit 20 in the embodiment will be described. For example, the coordinate conversion unit 160 converts the camera coordinate system of the image acquired by the acquisition unit 110 into a bird's-eye view coordinate system. In this case, the coordinate conversion unit 160 may perform coordinate conversion by adding image calibration parameters (calibration parameters in the pitch direction and roll direction) to predetermined reference coordinate conversion parameters, or may perform coordinate conversion processing on an image calibrated by the calibration unit 150.

[0071] 7 is a diagram illustrating the coordinate conversion process in the embodiment. For example, let each axis of the camera coordinate system (three-dimensional) of the image captured by the imaging device 10 (the front camera 12 in the example of FIG. 7) be [Xc, Yc, Zc], and let a virtual camera coordinate system (vehicle coordinate system) be [Xvc, Yvc, Zvc], which is based on the attitude of the vehicle M and is horizontal to the ground (traveling road surface) facing the traveling direction of the vehicle M. Here, Xvc indicates the traveling direction of the vehicle M, Yvc indicates the lateral direction of the vehicle M, and Zvc indicates the vertical direction of the vehicle M. Also, let θ be the roll angle with respect to the vehicle M, ρ be the pitch angle, and φ be the yaw angle. Then, the respective angles [θ, ρ, φ] indicate the rotation angles around the respective axes of the virtual camera coordinate system [Xvc, Yvc, Zvc]. The roll angle θ and pitch angle ρ at this time are values ​​calibrated by the calibration unit 150. The calibration unit 150 performs coordinate transformation (rotation) from the camera coordinate system to a virtual camera coordinate system that faces the traveling direction and is parallel to the ground, using, for example, the following equation (1).

[0072]

number

[0073]

number

[0074] The recognition unit 20 recognizes the surrounding conditions, such as the positions of objects present around the vehicle M, using the images processed by the image processing device 100. For example, the recognition unit 20 recognizes the positions of objects when an image in a calibrated camera coordinate system is converted into a bird's-eye view coordinate system.

[0075] For example, as shown in FIG. 7, when the ground (road surface) is always flat (the variation angle is within the allowable range), the recognition unit 20 recognizes the azimuth angle and distance in the bird's-eye view coordinate system for an object in the camera coordinate system as seen from the front camera 12. In this case, the recognition unit 20 calculates, for example, the depression angle α (the angle at which the object looks down when expressed in polar coordinates based on the orientation of the virtual camera) using the following equation (5), and calculates the azimuth angle β as "β = tan -1 It is calculated by "(Yvc / Xvc)".

[0076]

number

[0077] In addition to notifying the occupant of information related to driving assistance, the notification control unit 40 may generate an image showing the surrounding situation of the vehicle M after conversion into a bird's-eye view coordinate system and notify the occupant of the generated image via the terminal device T. This allows the occupant to be provided with an accurate view of the surrounding situation in a display format that is easy for the occupant to understand.

[0078] [Variations] In an embodiment, instead of extracting feature points of the entire image as described above and then extracting feature points of the vehicle road area and the intersecting road area from the extracted feature points, it is also possible to first extract the vehicle road area and the intersecting road area contained in the image and then extract feature points contained in each extracted road area.

[0079] In addition, in an embodiment, the calibration unit 150 may calibrate the image when a predetermined area or more of the vehicle road area and the intersecting road area is extracted from the forward image or the rearward image. This allows more accurate acquisition of the normal vector to the road surface from a relatively wide road area, thereby enabling more appropriate calibration.

[0080] In addition, in the embodiment, the image processing device 100 may set priorities for the front camera 12, the first rear camera 14, and the second rear camera 16, and prioritize the calibration of the camera with the highest set priority. For example, if there is a camera that is prone to deviation due to its installation position or the like, the calibration process may be performed preferentially over the other cameras, thereby suppressing deterioration of recognition accuracy using subsequent images. In addition, in the embodiment, the image processing device 100 may perform correction processes such as aberration correction and distortion correction on the front camera 12, the first rear camera 14, and the second rear camera 16. In addition, in the embodiment, an image including a lateral view of the vehicle M (a side image) may be used instead of (or in addition to) a rear image.

[0081] Furthermore, one of the first calibration pattern and the second calibration pattern described above may be combined with part or all of the other. Furthermore, the image processing device 100 may execute a predetermined one of the first calibration pattern and the second calibration pattern described above, or may execute both and integrate (average) the execution results. Furthermore, the calibration unit 150 may select and execute one of the calibration patterns depending on the driving conditions (surrounding road conditions) of the vehicle M and the detection conditions of the road area. For example, if the detected road area is equal to or larger than a predetermined area, the calibration process is performed using the first calibration pattern because feature points of the road area can be detected even if the front image and the rear image are separated. However, if the detected road area is smaller than the predetermined area, the number of feature points for the road area is small, so the calibration process is performed using the second calibration pattern that performs calibration by combining feature points from both the front image and the rear image. This allows for more appropriate calibration process depending on the situation.

[0082] According to the embodiment described above, the image processing device 100 includes an acquisition unit 110 that acquires a first image captured in a first direction of a moving body from an imaging device 10 mounted on a vehicle M (an example of a moving body) and a second image captured in a second direction different from the first direction, an extraction unit 120 that extracts feature points from the first image and the second image captured by the acquisition unit 110, a first detection unit 130 that detects a road area included in the first image, a second detection unit 132 that detects a road area included in the second image, and a detection unit 133 that detects the feature points extracted by the extraction unit 120. Among these, by providing a first feature point extraction unit 140 that extracts feature points of the road area detected by the first detection unit 130 as first feature points, a second feature point extraction unit 142 that extracts feature points of the road area detected by the second detection unit 132 from the feature points extracted by the extraction unit 120 as second feature points, and a calibration unit 150 that calibrates the first image and the second image based on the first feature points and the second feature points, more appropriate calibration can be performed on the image captured by the imaging device 10 mounted on the vehicle M.

[0083] Specifically, according to the embodiment, for example, images in front of and behind the vehicle M are used to calibrate the images based on feature points of the host vehicle road area and the intersecting road area, so that calibration can be performed with higher accuracy using more information. Furthermore, the inclusion of a rear image makes it easier to acquire the intersecting road area in particular. Furthermore, according to the embodiment, with regard to the pitch direction of the vehicle M (image capture device 10), calibration can be performed with higher accuracy by using feature amounts of the host vehicle road area extending in the front-to-rear direction as viewed from the vehicle M. Furthermore, with regard to the roll direction of the vehicle M (image capture device 10), calibration can be performed with higher accuracy by using information obtained from the intersecting road area extending in the left-to-right direction as viewed from the vehicle M (normal vector with respect to the road surface of the intersecting road).

[0084] Furthermore, according to the embodiment, even if there is a misalignment in the installation of the imaging device on the vehicle M or variations in the product performance of the imaging device, calibration can be performed using the front image and rear image captured by the imaging device attached to the vehicle M, thereby making it possible to perform more appropriate conversion when performing coordinate transformation of the image. Furthermore, according to the embodiment, even if, for example, the front camera and rear camera are installed on either side of the pitch rotation center of the vehicle M, or even if they are installed at separate locations so that their imaging ranges do not overlap, appropriate calibration can be performed in both the roll direction and the pitch direction. Therefore, the calibrated images can be used to more accurately recognize the relative position and relative distance to objects around the vehicle M, and this recognition can realize more appropriate driving assistance.

[0085] The above-described embodiment can be expressed as follows. a storage medium for storing computer-readable instructions; a processor connected to the storage medium; The processor executes the computer-readable instructions to: acquiring a first image captured in a first direction of the moving body from an imaging device mounted on the moving body and a second image captured in a second direction different from the first direction; extracting feature points from the acquired first image and second image; Detecting a road area included in the first image; Detecting a road area included in the second image; extracting, from the extracted feature points, feature points of a road area included in the detected first image as first feature points; extracting, from the extracted feature points, feature points of a road area included in the detected second image as second feature points; calibrating the first image and the second image based on the first feature points and the second feature points; Image processing device.

[0086] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0087] 1...driving assistance device, 10...imaging device, 12...front camera, 14...first rear camera, 16...second rear camera, 20...recognition unit, 30...driving assistance unit, 40...notification control unit, 100...image processing device, 110...acquisition unit, 111...first acquisition unit, 112...second acquisition unit, 120...extraction unit, 130...first detection unit, 132...second detection unit, 140...first feature point extraction unit, 142...second feature point extraction unit, 150...calibration unit, 160...coordinate conversion unit, 170...storage unit

Claims

1. an acquisition unit that acquires a first image captured in a first direction of the moving body from a first imaging unit mounted on the moving body, and acquires a second image captured in a second direction different from the first direction from a second imaging unit mounted on the moving body; an extracting unit that extracts feature points from the first image and the second image acquired by the acquiring unit; a first detection unit that detects a road area included in the first image; a second detection unit that detects a road area included in the second image; a first feature point extraction unit that extracts, from among the feature points extracted by the extraction unit, feature points of the road area detected by the first detection unit as first feature points; a second feature point extraction unit that extracts, from among the feature points extracted by the extraction unit, feature points of the road area detected by the second detection unit as second feature points; a calibration unit that calibrates the first image and the second image based on the first feature points and the second feature points; An image processing device comprising:

2. The calibration unit derives a relative angle between the first imaging unit and the second imaging unit based on calibration results of the first image and the second image. The image processing device according to claim 1 .

3. the calibration unit calibrates the first image and the second image when the first feature point extraction unit and the second feature point extraction unit have extracted feature points included in the road area consecutively for a predetermined number of frames or more. The image processing device according to claim 1 .

4. the calibration unit derives the relative angle based on a normal vector of a road area included in the first image with respect to a road surface and a normal vector of a road area included in the second image with respect to a road surface, and converts a coordinate system of the second image into a coordinate system of the first image based on the derived relative angle, thereby calibrating the first image and the second image. The image processing device according to claim 2 .

5. the first feature point extraction unit extracts, as first feature points, feature points of a first road area along which the moving object moves and feature points of a second road area intersecting with the first road area from the image frames of the first image acquired by the acquisition unit at predetermined time intervals; the second feature point extraction unit extracts, as second feature points, feature points of the first road area and the second road area from the image frames of the second image acquired by the acquisition unit at predetermined time intervals; the calibration unit calibrates the first image and the second image when the first feature points and the second feature points are extracted from the first image and the second image continuously for a predetermined number of frames or more. The image processing device according to claim 3 .

6. the road area includes a first road area in which the moving object moves and a second road area intersecting the first road area; The image processing device according to claim 1 .

7. The calibration unit calibrating a pitch angle between a first imaging unit that captures the first image and a second imaging unit that captures the second image based on the feature points of the first road area; calibrating a roll angle between the first imaging unit and the second imaging unit based on the feature points of the second road area; The image processing device according to claim 6 .

8. The computer acquiring a first image captured in a first direction of the moving body from a first imaging unit mounted on the moving body, and acquiring a second image captured in a second direction different from the first direction from a second imaging unit mounted on the moving body; extracting feature points from the acquired first image and second image; Detecting a road area included in the first image; Detecting a road area included in the second image; extracting, from the extracted feature points, feature points of a road area included in the detected first image as first feature points; extracting, from the extracted feature points, feature points of a road area included in the detected second image as second feature points; calibrating the first image and the second image based on the first feature points and the second feature points; Image processing methods.

9. On the computer, acquiring a first image captured in a first direction of the moving body from a first imaging unit mounted on the moving body, and acquiring a second image captured in a second direction different from the first direction from a second imaging unit mounted on the moving body; extracting feature points from the acquired first image and second image; detecting a road area included in the first image; detecting a road area included in the second image; extracting, from the extracted feature points, feature points of a road area included in the detected first image as first feature points; extracting, from the extracted feature points, feature points of a road area included in the detected second image as second feature points; calibrating the first image and the second image based on the first feature points and the second feature points; program.

Images