Image processing apparatus, image processing method, and storage medium

Abstract

Provided are an image processing device, an image processing method, and a storage medium capable of more appropriately correcting an image captured by an imaging device mounted on a moving body. The image capturing apparatus includes: an acquisition unit that acquires a first image in a first direction in which a moving body is captured and a second image in a second direction different from the first direction from an image capturing device mounted on the moving body; an extraction unit that extracts feature points from the first image and the second image acquired by the acquisition unit; a first detection unit that detects a road region included in the first image; a second detection unit that detects a road region included in the second image; a first feature point extraction unit that extracts, as a first feature point, a feature point of the road region detected by the first detection unit among the feature points extracted by the extraction unit; a second feature point extraction unit that extracts, as a second feature point, a feature point of the road region detected by the second detection unit among the feature points extracted by the extraction unit; and a correction unit that corrects the first image and the second image on the basis of the first feature point and the second feature point.

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, an image processing method, and a storage medium. Background Art

[0002] In recent years, efforts to provide a sustainable transportation system that also takes into account people who are particularly vulnerable among traffic participants have been intensifying. In order to achieve this goal, research and development have been carried out to further improve traffic safety and convenience through research and development related to driving support technologies. Related to this, there are known the following techniques: obtaining a captured image from a camera mounted on a moving body, extracting feature points from an extraction region set or changed based on the external environment of the moving body in the shooting direction of the camera in the obtained captured image, and estimating the posture of the camera based on the extracted feature points (for example, refer to Japanese Unexamined Patent Application Publication No. 2021-33605). There is known a technique in which feature points reflected in an image of another in-vehicle camera different from the in-vehicle camera that performs calibration are used to estimate the road surface range, and calibration is performed using only the feature points existing in the estimated road surface range among the feature points reflected in the image of the in-vehicle camera that performs calibration (for example, Japanese Unexamined Patent Application Publication No. 2019-28665). Summary of the Invention

[0003] However, in driving support technologies, there is a problem as follows. When processing images captured in multiple directions by a shooting device mounted on a moving body, it is sometimes impossible to appropriately correct the images captured by the shooting device due to factors such as deviation in the installation of the shooting device on the moving body and variations in the product performance of the shooting device.

[0004] The present application has been completed to solve the above problems, and one of its objects is to provide an image processing apparatus, an image processing method, and a storage medium that can more appropriately correct images captured by a shooting device mounted on a moving body. Then, it further contributes to the development of a sustainable transportation system.

[0005] The image processing apparatus, image processing method, and storage medium of the present invention adopt the following configuration.

[0006] (1): An image processing apparatus according to an aspect of the present invention includes: an acquisition unit that acquires a first image obtained by photographing a first direction of a moving body and a second image obtained by photographing a second direction different from the first direction from a photographing device mounted on the moving body; an extraction unit that extracts feature points from the first image and the second image acquired by the acquisition unit; a first detection unit that detects a road area included in the first image; a second detection unit that detects a road area included in the second image; a first feature point extraction unit that extracts, as first feature points, the feature points of the road area detected by the first detection unit among the feature points extracted by the extraction unit; a second feature point extraction unit that extracts, as second feature points, the feature points of the road area detected by the second detection unit among the feature points extracted by the extraction unit; and a correction unit that corrects the first image and the second image based on the first feature points and the second feature points.

[0007] (2): Based on the above (1) aspect, the photographing device includes a first photographing unit that photographs the first image and a second photographing unit that photographs the second image, and the correction unit derives a relative angle between the first photographing unit and the second photographing unit based on the correction results of the first image and the second image.

[0008] (3): Based on the above (1) aspect, the correction unit corrects the first image and the second image when the feature points included in the road area are continuously extracted by the first feature point extraction unit and the second feature point extraction unit for a predetermined number of frames or more.

[0009] (4): Based on the above (2) aspect, the correction unit derives the relative angle based on a normal vector of the road surface with respect to the road area included in the first image and a normal vector of the road surface with respect to the road area included in the second image, and based on the derived relative angle, transforms the coordinate system of the second image into the coordinate system of the first image, thereby correcting the first image and the second image.

[0010] (5): Based on the solution in (3) above, the first feature point extraction unit extracts, from the image frames of the first image obtained by the acquisition unit at regular intervals, the feature points of the first road area where the moving body moves and the feature points of the second road area intersecting with the first road area as first feature points. The second feature point extraction unit extracts, from the image frames of the second image obtained by the acquisition unit at regular intervals, the feature points of the first road area and the feature points of the second road area as second feature points. When the first feature points and the second feature points are continuously extracted from the first image and the second image for a specified number of frames or more, the correction unit corrects the first image and the second image.

[0011] (6): Based on the solution in (1) above, the road area includes the first road area where the moving body moves and the second road area intersecting with the first road area.

[0012] (7): Based on the solution in (6) above, the correction unit performs the following processing: correcting the pitch angles of the first imaging unit that captures the first image and the second imaging unit that captures the second image based on the feature points of the first road area; and correcting the roll angles of the first imaging unit and the second imaging unit based on the feature points of the second road area.

[0013] (8): The image processing method according to other solutions of the present invention, wherein the image processing method causes a computer to perform the following processing: obtaining a first image obtained by capturing the moving body in a first direction and a second image obtained by capturing a second direction different from the first direction from an imaging device mounted on the moving body; extracting feature points from the obtained first image and second image; detecting the road area included in the first image; detecting the road area included in the second image; extracting, as first feature points, the feature points of the road area included in the detected first image among the extracted feature points; extracting, as second feature points, the feature points of the road area included in the detected second image among the extracted feature points; and correcting the first image and the second image based on the first feature points and the second feature points.

[0014] (9): A storage medium according to other embodiments of the present invention stores a program, wherein the program causes a computer to perform the following processes: obtaining a first image obtained by photographing a first direction of a moving body and a second image obtained by photographing a second direction different from the first direction from a photographing device mounted on the moving body; extracting feature points from the obtained first image and the second image; detecting a road region included in the first image; detecting a road region included in the second image; extracting, as first feature points, the feature points of the road region included in the detected first image among the extracted feature points; extracting, as second feature points, the feature points of the road region included in the detected second image among the extracted feature points; and correcting the first image and the second image based on the first feature points and the second feature points.

[0015] According to the above solutions (1) to (9), it is possible to perform more appropriate correction on images photographed by a photographing device mounted on a moving body. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 FIG. is an example showing a functional structure of a driving support device including an image processing device according to an embodiment.

[0017] Figure 2 FIG. is a diagram for explaining the installation positions and photographing directions of respective cameras of a photographing device with respect to a vehicle.

[0018] Figure 3 FIG. is an example showing a front image photographed by a front camera and a rear image photographed by a first rear camera.

[0019] Figure 4 FIG. is an example showing a rear image photographed by a second rear camera.

[0020] Figure 5 FIG. is a flowchart showing an example of processing in a first correction mode.

[0021] Figure 6 FIG. is a flowchart showing an example of processing in a second correction mode.

[0022] Figure 7 FIG. is a diagram for explaining coordinate transformation processing in an embodiment. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0023] Hereinafter, embodiments of the image processing apparatus, the image processing method, and the storage medium of the present invention will be described with reference to the drawings. In the following examples, an image processing apparatus mounted on a moving body will be described. The moving body may include all moving bodies for people (such as drivers and other passengers) to ride, such as three-wheeled or four-wheeled vehicles, two-wheeled vehicles, and micro-moving bodies. The moving body may also be provided with a driving support device that supports the driving of the passengers (drivers) of the moving body based on the images processed by the image processing apparatus. In the following description, the moving body is a four-wheeled vehicle (hereinafter referred to as "vehicle M") and is provided with a driving support device. Vehicle M may be any one of a motor vehicle powered by an internal combustion engine such as a diesel engine or a gasoline engine, an electric motor vehicle powered by an electric motor, or a hybrid motor vehicle having both an internal combustion engine and an electric motor. In the following description, the front direction of vehicle M is set as the positive X direction, the rear direction of vehicle M is set as the negative X direction, the right direction in the width direction of vehicle M and based on the positive X direction is set as the positive Y direction, the left direction in the width direction of vehicle M and based on the positive X direction is set as the negative Y direction, and the direction orthogonal to the X direction and the Y direction and being the height direction of vehicle M is set as the positive Z direction for description.

[0024] Figure 1 FIG. is an example showing the functional configuration of the driving support device 1 including the image processing apparatus according to the embodiment. Figure 1 The driving support device 1 shown, for example, includes a photographing device 10, an identification unit 20, a driving support unit 30, a notification control unit 40, and an image processing apparatus 100. The identification unit 20, the driving support unit 30, the notification control unit 40, and the image processing apparatus 100 are realized, for example, by a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Some or all of these components may also be realized by hardware (including a circuit unit: circuitry) such as an LSI (Large Scale Integration), an ASIC (ApplicationSpecific Integrated Circuit), an FPGA (Field-Programmable Gate Array), a GPU (GraphicsProcessing Unit), an SOC (System On Chip), or may be realized by the cooperation of software and hardware. The program may be pre-stored in a storage device (a storage device having a non-transitory storage medium) such as an HDD (Hard Disk Drive) or a flash memory, or may be stored in a removable storage medium (a non-transitory storage medium) such as a DVD or a CD-ROM, and installed by mounting the storage medium on a drive device. In addition to Figure 1In addition to the structure of the driving support device 1 shown, the vehicle M in the embodiment further includes a structure for driving the vehicle M (for example, a driving operation member, a driving device such as an engine and a motor, a steering device, a braking device, various vehicle sensors (for example, a position sensor, a speed sensor)), a navigation device for performing route guidance, a display device, a speaker, and various in-vehicle devices.

[0025] The imaging device 10 images the periphery of the vehicle M. For example, it is a digital camera using a solid-state imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The imaging device 10 may also be a stereo camera. The imaging device 10 may also be a camera used in a dash cam or the like. In Figure 1 the example, the imaging device 10 includes a front camera 12, a first rear camera 14, and a second rear camera 16 as a plurality of imaging units. The front camera 12 is an example of the "first imaging unit". At least one of the first rear camera 14 and the second rear camera 16 is an example of the "second imaging unit".

[0026] The front camera 12 images a predetermined area in front of the vehicle M. The first rear camera 14 and the second rear camera 16 image a predetermined area in a direction different from the front (for example, the rear of the vehicle M). The front is an example of the "first direction", and the rear is an example of the "second direction". At least one of the front image (an example of the first image) captured by the front camera 12 and the rear images (an example of the second images) captured by the first rear camera 14 and the second rear camera 16 may include an area in the lateral direction (side) of the vehicle M.

[0027] Figure 2 is a diagram for explaining the installation position and imaging direction of each camera of the imaging device 10 with respect to the vehicle M. The front camera 12 and the first rear camera 14 in the imaging device 10 are installed, for example, as Figure 2 shown near the upper part of the front windshield in front of the vehicle M. In Figure 2In the example, the front camera 12 and the first rear camera 14 are integrally formed within a specified distance. By integrally formed, for example, it may also include the case where the front camera 12 and the first rear camera 14 are stored in one housing, or a structure where they are respectively connected (joined). In this structure, for example, when the front camera 12 captures a specified field-of-view angle region VA1 centered on the front direction A1 (positive X-axis direction in the figure) of the vehicle M, the first rear camera 14 captures a specified field-of-view angle region VA2 centered on the direction A2 opposite to the front direction A1 of the vehicle M and including the interior of the vehicle. Thus, assuming that due to position deviation or the like caused by certain factors during or after installation, the front camera 12 captures a field-of-view angle region centered on a direction tilted downward by an angle θ1 with respect to the front direction A1 of the vehicle M as a reference, the first rear camera 14 captures a field-of-view angle region centered on a direction tilted upward by an angle θ1 with respect to the direction A2 opposite to the front direction A1 as a reference. That is, in the embodiment, when the shooting direction (field-of-view angle) of one of the integrally formed front camera 12 and the first rear camera 14 deviates, the shooting direction (field-of-view angle) of the other camera also deviates in the same way in a direction symmetric with respect to the installation position of the shooting device 10.

[0028] In Figure 2 the example, the second rear camera 16 is installed at the rear of the vehicle body (outside the vehicle), but it may also be installed at the rear inside the vehicle. The second rear camera 16 is provided at a position that sandwiches the center of the pitch rotation direction of the vehicle M (e.g., the center of gravity) G with respect to the installation position of the front camera 12. The second rear camera 16 captures a specified field-of-view angle region VA3 centered on a direction A3 (the same direction as direction A2) opposite to the front direction A1 of the vehicle M. Since the second rear camera 16 captures a field-of-view angle region VA3 that does not include the interior of the vehicle, it can capture a larger range of the external conditions (road shape, etc.) of the vehicle compared to the first rear camera 14.

[0029] The second rear camera 16 may also be electrically connected to the front camera 12 and the first rear camera 14 through a cable or the like, or may be connected via a frame member. In the embodiment, the front camera 12 (and the first rear camera 14) and the second rear camera 16 are separate. Therefore, there are uncorrelated deviations in the shooting directions of the respective cameras due to position deviations caused by certain factors during or after installation or performance variations in the products.

[0030] The shooting device 10 is not limited to Figure 2The structure shown, for example, may also include multiple imaging units for imaging the front of the vehicle M, or may include one or three or more imaging units for imaging the rear of the vehicle M. The imaging device 10 may also include side cameras for imaging the lateral (side) of the vehicle M in addition to the front camera 12, the first rear camera 14, and the second rear camera 16. The imaging device 10 may also include a fisheye camera capable of imaging the surroundings including the front and rear of the vehicle M in a wide angle (e.g., 360 degrees) instead of the above camera structure. The images captured by the fisheye camera may be divided into multiple images (front image, rear image, side image, etc.) according to the imaging direction. Each camera of the imaging device 10 repeatedly captures images at a predetermined cycle, and the imaging device 10 outputs the captured images (camera images) to the image processing device 100.

[0031] Return Figure 1 , the image processing device 100 acquires the images captured by the imaging device 10 and performs processing to transform the coordinate system of the images (hereinafter, the camera coordinate system) into a coordinate system different from the camera coordinate system. The camera coordinate system includes a front camera coordinate system corresponding to the front image and a rear camera coordinate system corresponding to the rear image. The coordinate system different from the camera coordinate system is, for example, a coordinate system (bird's-eye coordinate system) based on the position of the vehicle M when viewed from above. The images transformed into the bird's-eye coordinate system are used for the recognition unit 20 to recognize the surrounding conditions of the vehicle M and for notification (display) to the occupants (driver, etc.) of the vehicle M.

[0032] The image processing device 100 includes, for example, an acquisition unit 110, an extraction unit 120, a first detection unit 130, a second detection unit 132, a first feature point extraction unit 140, a second feature point extraction unit 142, a correction unit 150, a coordinate transformation unit 160, and a storage unit 170.

[0033] The storage unit 170 can also be implemented by a storage device such as an HDD, a flash memory, or an SSD (Solid State Drive), an EEPROM (Electrically Erasable Programmable Read Only Memory), a ROM (Read Only Memory), or a RAM (Random Access Memory). The storage unit 170 stores, for example, the images acquired by the acquisition unit 110, the processing results processed by the correction unit 150 and the coordinate transformation unit 160, programs, and various other information. Map information may also be stored in the storage unit 170. The map information is, for example, information that represents the shape of a road in correspondence with position information (latitude and longitude information) by showing road segments and nodes connected by the road segments. The map information may also include lane boundary information such as the curvature, slope, number of lanes, width, information on the center of a lane, or road markings that divide lanes of the road. The map information may include traffic restriction information, position information of branches, merges, intersections, T-junctions, etc., facility information such as buildings or parking lots, POI (Point Of Interest) information, etc. The map information can be updated at any time by the vehicle M or the driving support device 1 communicating with other devices.

[0034] The acquisition unit 110 acquires the camera images captured by the imaging device 10 at a predetermined cycle. The acquisition unit 110 includes, for example, a first acquisition unit 111 and a second acquisition unit 112. The first acquisition unit 111 acquires a front image (an example of a first image captured in a first direction of the moving body) captured by the front camera 12. The second acquisition unit 112 acquires a rear image (an example of a second image captured in a second direction different from the first direction) captured by at least one of the first rear camera 14 and the second rear camera 16.

[0035] The extraction unit 120 extracts the feature points included in the front image and the rear image acquired by the acquisition unit 110. For example, the extraction unit 120 performs known image analysis processes such as edge extraction processing on the front image and the rear image, and extracts the feature points of the objects in the actual space included in the images (for example, traffic signal machines, road signs, pedestrians, other vehicles, etc., traffic participants, not only buildings, but also road areas, road markings, stop lines, etc.) based on the results of the image analysis process. In this case, the extraction unit 120 extracts, for example, the point sequence on the edge of the object included in the image as the feature points (feature point group). The extraction unit 120 may also use a learned model to extract the feature points. The learned model is learned in such a way that when the front image and the rear image are input, the edges of the objects (for example, buildings, road structures, etc.) reflected in the images are output as a point group. The learned model may be stored in advance in the storage unit 170, or may be acquired from an external device via a communication device (not shown) mounted on the vehicle M. The extraction unit 120 may also use, for example, the method of Visual SLAM (Simultaneous Localization and Mapping), which is a technique for three-dimensionally grasping its own position based on the image data captured by the imaging device 10, to extract the feature points. The method for extracting the feature points on the image is not limited to the above examples, and other known methods may be used.

[0036] The extraction unit 120 may also be provided with different structures for the first extraction unit that extracts feature points from the front image and the second extraction unit that extracts feature points from the rear image, and may also be provided with different structures for the extraction unit that extracts feature points from the rear image captured by the first rear camera 14 and the extraction unit that extracts feature points from the rear image captured by the second rear camera 16.

[0037] The first detection unit 130 detects the road area included in the front image. The second detection unit 132 detects the road area included in the rear image. The road area includes, for example, the driving lane area (hereinafter referred to as "own vehicle road area") in which the vehicle M travels (moves) and the area of the lane that intersects the own vehicle road area (hereinafter referred to as "intersection road area"). The own vehicle road area may also include adjacent lanes and oncoming lanes extending in the same direction in addition to the driving lane of the vehicle M. The intersection road area is, for example, a road connected to the own vehicle road area within a specified angle range including a right angle at an intersection, a T-junction, etc. The own vehicle road area is an example of the "first road area". The intersection road area is an example of the "second road area".

[0038] The first feature point extraction unit 140 extracts, as first feature points, the feature points of the road area detected by the first detection unit 130 from the feature points (feature point groups) extracted by the extraction unit 120. The second feature point extraction unit 142 extracts, as second feature points, the feature points of the road area detected by the second detection unit 132 from the feature points (feature point groups) extracted by the extraction unit 120. The first feature points and the second feature points are used for correcting (calibrating) the images in the embodiment.

[0039] The correction unit 150 corrects the camera images (front image and rear image) captured by the imaging device 10 based on the first feature points extracted by the first feature point extraction unit 140 and the second feature points extracted by the second feature point extraction unit 142. For example, when the coordinate system (camera coordinate system) of the camera image deviates from the reference coordinate system (the deviation amount is above the threshold) due to deviations caused by factors such as the deviation during the installation of the imaging device 10, vibrations after installation, contact with the occupant, and differences in camera performance, the correction unit 150 corrects the three-dimensional axes of the camera coordinate system so as to be consistent with the reference coordinate system (the deviation amount is less than the threshold). The reference coordinate system is, for example, a coordinate system (vehicle coordinate system) based on the attitude of the vehicle M. The correction of the camera image means correcting the camera coordinate systems of the first image and the second image to the vehicle coordinate system. The corrected image can also be described as correcting the attitude of each camera of the imaging device 10. For example, the correction unit 150 corrects at least one of the pitch (e.g., the inclination in the front-rear direction of each camera of the imaging device 10), roll direction (the inclination in the lateral direction of each camera), and yaw direction (the rotation direction when observing each camera from above) so that the three-dimensional vehicle coordinate system based on the vehicle M is consistent with the three-dimensional camera coordinate system. The correction unit 150 can also derive parameters for image correction (e.g., information related to at least one of the pitch angle, roll angle, and yaw angle).

[0040] The coordinate transformation unit 160 transforms the coordinate system (camera coordinate system) of the image acquired by the acquisition unit 110 into a bird's-eye view coordinate system, which is an example of another different coordinate system. In this case, the coordinate transformation unit 160 can either perform coordinate transformation on the image corrected by the correction unit 150 using the reference coordinate transformation parameters pre-stored in the storage unit 170 or the like, or perform coordinate transformation including correction on the image acquired by the acquisition unit 110 based on the reference coordinate transformation parameters and the above-mentioned image correction parameters. Thereby, more accurate image transformation can be performed.

[0041] The recognition unit 20 recognizes the surrounding conditions of the vehicle M based on the image transformed into the bird's-eye coordinate system by the coordinate transformation unit 160 (hereinafter, "bird's-eye image"). For example, the recognition unit 20 recognizes the objects existing in the vicinity of the vehicle M (within a specified distance from the vehicle M) based on the bird's-eye image. In the recognition of objects, for example, object recognition using a model based on deep learning, deep machine learning, etc., object recognition based on a pattern matching method, or an object recognition method obtained by combining these is performed. The recognition unit 20 may also recognize the objects around the vehicle M based on the position information of the vehicle M obtained by a position sensor such as a GPS (Global Positioning System) device included in the vehicle sensor and referring to the map information stored in the storage unit 170.

[0042] Here, the objects include, for example, other vehicles, traffic participants such as pedestrians, driving road boundaries (road boundaries) including road dividing lines, road shoulders, curbstones, median strips, guardrails, etc., stop lines, obstacles, traffic signal machines, road signs, toll booths, bridges, etc. The objects may include structures and ground features such as roadside trees around the vehicle M. Regarding other vehicles, traffic participants such as pedestrians among the above objects, they are recognized based on the bird's-eye image, and regarding the other objects, they are recognized by one or both of the bird's-eye image and the map information. The recognition unit 20 may also recognize the types, shapes, sizes, etc. of various objects, or recognize the positions (relative positions with respect to the vehicle M), speeds (relative speeds with respect to the vehicle M), etc. of the objects.

[0043] The driving support unit 30 performs driving support for the occupants (such as the driver) of the vehicle M based on the recognition result recognized by the recognition unit 20. For example, the driving support unit 30 determines whether the vehicle M has deviated from the driving lane (for example, the own lane area) divided by the road dividing line recognized by the recognition unit 20, and in the case of a possibility of deviation, notifies the driver of the vehicle M via the notification control unit 40, or controls the steering of the vehicle M using a steering device (not shown) in a manner that suppresses the deviation of the driving lane of the vehicle M (in a manner that the vehicle M moves toward the center side of the driving lane). The driving support unit 30 recognizes obstacles such as other vehicles existing in the vicinity (within a specified distance) of the vehicle M, and in the case of determining that there is a possibility of contact with the obstacle based on the relative position and relative speed with respect to the obstacle, notifies the occupants using the notification control unit 40 or performs driving control to avoid contact (at least one of speed control and steering control).

[0044] The notification control unit 40 makes a notification related to driving support to the occupant (driver) of the vehicle M based on the control performed by the driving support unit 30. In this case, the notification control unit 40 generates notification information such as sound (alarm), image, etc. that has a corresponding relationship with the notification content to be notified to the occupant, and causes the generated notification information to be transmitted to the terminal device T for output.

[0045] Here, the terminal device T is, for example, a mobile terminal device such as a smartphone or a tablet terminal used by the driver of the vehicle M equipped with the driving support device 1. In the terminal device T, for example, an application program for receiving driving support performed by the driving support device 1 is executed. The application program receives the information transmitted by the driving support device 1, and causes the display unit of the terminal device T to display an image obtained based on the notification, or causes the speaker of the terminal device T to emit sound. The terminal device T is an example of the "notification unit". The terminal device T can be detachably mounted on the vehicle M and used. For example, a holder for the terminal device T having a mounting / detaching portion is provided on one or both of the terminal device T and the vehicle M, and the terminal device T is supported by the holder. In the embodiment, when the vehicle M is equipped with devices (notification units) such as a navigation device, a display device, and a speaker, the notification information may be output from the above-mentioned mounted devices instead of (or in addition to) the terminal device T based on the instruction of the driving support unit 30.

[0046] [Front image and rear image]

[0047] Here, an example of the front image and the rear image in the embodiment will be described with reference to the drawings. Figure 3 It is a diagram showing an example of the front image IM10 captured by the front camera 12 and the rear image IM20 captured by the first rear camera 14. In the front image IM10, the area outside the vehicle (in front of the vehicle M) is captured across the front windshield of the vehicle M. In the rear image IM20, the rear area including the interior of the vehicle is captured, and the area outside the vehicle (side or rear of the vehicle M) is captured across the side windshield and the rear windshield. Figure 4 It is a diagram showing an example of the rear image IM30 captured by the second rear camera 16. In the rear image IM30, the area outside the vehicle (rear of the vehicle M) is captured across the rear windshield of the vehicle M. The image processing device 100 corrects these images and performs recognition of surrounding objects using the corrected images.

[0048] [Image correction]

[0049] Next, the image correction process in the embodiment will be specifically described by dividing it into several correction modes. Hereinafter, for the sake of convenience of explanation, as the rear image, the rear image IM30 captured by the second rear camera 16 is used. However, instead of the rear image IM30 (or on the basis of the rear image IM30), the rear image IM20 captured by the first rear camera 14 can also be used, or a side image captured by photographing the side of the vehicle M can be used.

[0050] [First correction mode]

[0051] In the first correction mode, after correcting the images in the front image and the rear image respectively, the relative angle between the front camera 12 and the second rear camera 16 (the relative angle between the front image and the rear image) is derived based on the correction results. Then, based on the derived relative angle, the coordinate system of one image (for example, the rear image) is transformed into the coordinate system of the other image (for example, the front image) to synthesize the two images, and the synthesized image is further corrected.

[0052] Figure 5 is a flowchart showing an example of the processing in the first correction mode. In Figure 5 example, the first acquisition unit 111 acquires the front image IM10 captured by the front camera 12 (step S100). Next, the extraction unit 120 extracts the feature points of the front image IM10 (step S102). Next, the first detection unit 130 detects the road area included in the front image IM10 (step S104).

[0053] Here, the processing of step S104 will be specifically described. The first detection unit 130 detects the own vehicle road area and the cross road area included in the front image IM10. For example, the first detection unit 130 acquires the left and right point column portions of the vehicle M (the feature point group arranged within a specified distance along the same direction (including the allowable error range)) included in the feature point group extracted by the extraction unit 120 as the road dividing line, and detects the area divided by the acquired road dividing line as the own vehicle road area. For example, the first detection unit 130 can also divide the front image IM10 into a plurality of divided areas, and detect the own vehicle road area for each divided area. As Figure 3 shown in the front image IM10 and the rear image IM20, on each image, the position where the own vehicle road area exists is near the center of the image, and it is somewhat easy to predict. Therefore, the first detection unit 130 can also use a partial area (for example, a specified area including the center of the image) in the front image IM10 where the possibility of the existence of the own vehicle road area is predicted in advance as the object, and detect the own vehicle road area obtained based on the feature point group. Thereby, the processing burden related to the detection of the own vehicle road area can be reduced.

[0054] The first detection unit 130 detects a point sequence portion that contacts the detected own-vehicle road area at a specified angle. When two (two) point sequences exist in parallel (including the allowable error range) within a specified distance, these two point sequences are regarded as road dividing lines, and the area divided by the road dividing lines is detected as an intersection road area. The so-called specified angle is, for example, a specified angle range including 90 degrees (right angle) with respect to the extension direction of the own-vehicle road area (for example, about 75 to 105 degrees). The first detection unit 130 may also target a partial area in the front image IM10 where the possibility of an intersection road area existing is predicted to be high in advance, and detect the intersection road area obtained based on the feature point group.

[0055] The first detection unit 130 may also implement functions based on AI (Artificial Intelligence) and functions based on a pre-given model in parallel, for example. For example, the functions of detecting the own-vehicle road area and the intersection road area can be achieved by performing the detection of the own-vehicle road area and the intersection road area based on deep learning, etc. and the detection based on a pre-given determination process (for example, a determination process based on pattern matching) on the front image IM10 in parallel, and comprehensively evaluating by scoring both.

[0056] The first detection unit 130 may also perform the detection process of the intersection road area within a range within a specified distance from the position where a specific road structure such as a traffic signal or a crosswalk is detected from the front image IM10. The first detection unit 130 may also perform the detection process of the intersection road area based on the position information of the vehicle M and referring to the map information stored in the storage unit 170 when the position of the vehicle M is close to a position where there is a high possibility of an intersection road such as an intersection or a T-junction (within a specified distance). The position information of the vehicle M is obtained by, for example, a position sensor mounted on the vehicle M. Thus, since the detection process is performed in an area where there is a high possibility of an intersection road area, the intersection road area can be detected more efficiently. In Figure 3 the example, the first detection unit 130 detects the own-vehicle road area AR10F in front of the vehicle M from the front image IM10, and detects the intersection road areas AR20L-1 and AR20R-1.

[0057] Next, the first feature point extraction unit 140 extracts the feature points of the detected road area among the feature points of the front image IM10 (step S106). Specifically, the first feature point extraction unit 140 extracts the feature points included in the own-vehicle road area AR10F and the intersection road areas AR20L-1 and AR20R-1 detected by the first detection unit 130 from the feature points (feature point group) extracted by the extraction unit 120.

[0058] Next, the correction unit 150 corrects the front image IM10 based on the feature points of the extracted road region (step S108). For example, the correction unit 150 estimates the least-squares plane with respect to the set of feature points included in the own vehicle road region AR10F and the intersecting road regions AR20L-1 and AR20R-1 as the road surface. Next, the correction unit 150 corrects the front image in such a manner that the camera coordinate system with the estimated road surface as a reference coincides with the vehicle coordinate system with the attitude of the vehicle M as a reference (the deviation amount is less than the threshold) (in other words, in such a manner that the estimated road surface coincides with the road surface when the attitude of the vehicle M is used as a reference). At this time, the correction unit 150 can derive either the deviation amount with respect to the vehicle coordinate system or the image correction parameters.

[0059] Next, the correction unit 150 determines whether the correction has reached a specified accuracy (step S110). In the process of step S110, for example, when the deviation amount between the estimated road surface and the road surface when the attitude of the vehicle M is used as a reference is less than the threshold for a continuous number of frames or more, the correction unit 150 determines that the correction has reached the specified accuracy, and in other cases, determines that the correction has not reached the specified accuracy. In the case where it is determined that the specified accuracy has not been reached, the process returns to step S100, and the above-described process is repeatedly executed until it is determined that the specified accuracy has been reached for the next image frame. In the case of processing the next image frame, the process may also be performed after correcting the image using the image correction parameters derived in the process of step S108 in advance.

[0060] The image processing device 100 performs the processes of steps S120 to S130 shown below in parallel with (or at the timing before and after) the processes of steps S100 to S110. Specifically, first, the second acquisition unit 112 acquires the rear image IM30 captured by the second rear camera 16 (step S120). Next, the extraction unit 120 extracts the feature points of the rear image IM30 (step S122). Next, the second detection unit 132 detects the road regions (the own vehicle road region, the intersecting road regions) included in the rear image IM30, for example, using the same method as the first detection unit 130 (step S124). In the example of Figure 3 the own vehicle road region AR10R-1 behind the vehicle M is detected from the rear image IM20, and in the example of Figure 4 the own vehicle road region AR10R-2 behind the vehicle M and the intersecting road regions AR20L-2 and AR20R-2 are detected from the rear image IM30.

[0061] Next, the second feature point extraction unit 142 extracts the feature points of the road regions (the own vehicle road regions AR10R-1, AR10R-2, the intersecting road regions AR20L-2, AR20R-2) among the feature points of the rear image IM30 (step S126). Next, the correction unit 150 corrects the rear image IM30 based on the extracted feature points of the road regions (step S128). Next, the correction unit 150 determines whether the correction has reached a specified accuracy (step S130). If it is determined that the specified accuracy has not been reached, the process returns to step S120, and the above-described processing is repeated until it is determined that the specified accuracy has been reached. Regarding the processing of steps S128 and S130, the correction unit 150 performs the same processing as steps S108 and S110 with respect to the feature points of the road regions (the own vehicle road regions AR10R-1, AR10R-2, the intersecting road regions AR20L-2, AR20R-2).

[0062] In the processing of steps S110 and S130, when it is determined that the correction of both the front image IM10 and the rear image IM30 has reached the specified accuracy, the correction unit 150 derives the relative angle between the front camera 12 and the second rear camera 16 from the corrected respective images (in other words, the relative angle in the coordinate systems of the front image IM10 and the rear image IM30) (step S140). For example, the correction unit 150 calculates the relative angle (the deviation amount of the coordinate system) of the rear image IM30 with respect to the front image IM10 based on the image correction parameters for the front image IM10 and the image correction parameters for the rear image IM30. Since the shooting directions of the front image IM10 and the rear image IM30 are different, the correction unit 150 derives the relative angle, for example, after adjusting the orientation of the coordinate system based on a reference coordinate system with the posture of the same object such as the vehicle M as a reference.

[0063] Next, the calibration unit 150 transforms the camera coordinate system (rear camera coordinate system) of the second rear camera 16 into the camera coordinate system (front camera coordinate system) of the front camera 12 (front image IM10) based on the relative angle (step S150). Next, the calibration unit 150 corrects the front image IM10 based on the normal vector of the road surface with respect to the road region obtained from the two camera images (step S160). For example, the calibration unit 150 derives the normal vector of the road surface with respect to the road regions (own vehicle road region, crossroad region) included in the front image IM10 and the rear image IM30, respectively. The normal vector can be derived for each image, or can be derived by separating the own vehicle road region and the crossroad region. In the case of deriving a plurality of normal vectors, averaging can be performed, or the normal vector with respect to the road region with a higher priority can be derived. Further, the calibration unit 150 corrects the front image IM10 (in other words, the attitude of the front camera 12) so that the derived normal vector coincides with the normal vector of the road surface with respect to the attitude of the vehicle M. In this case, the calibration unit 150 may also derive the image correction parameter.

[0064] In the process of step S160, the calibration unit 150 may also use the normal vector of the road surface with respect to the own vehicle road region to correct the pitch direction (pitch angle) of the front image IM10 (front camera 12), and use the normal vector of the road surface with respect to the crossroad region to correct the roll direction (roll angle). The own vehicle road region is a region extending in the vertical direction of the image (front or rear of the vehicle M), and the crossroad region is a region extending in the horizontal direction of the image (lateral direction of the vehicle M). Therefore, by using the own vehicle road region to correct the pitch direction of the vehicle M (or the imaging device 10), and using the crossroad region to correct the roll direction of the vehicle M (or the imaging device 10), more appropriate correction can be performed in each direction.

[0065] Next, the calibration unit 150 corrects the rear image IM30 (in other words, the attitude of the second rear camera 16) based on the correction result (image correction parameter) of the front image IM10 and the relative angle. Thus, the processing of this flowchart ends.

[0066] After the correction process according to the first correction mode described above is performed on the front image and the rear image respectively, the relative angle between the camera images of both sides is derived, and based on the derived relative angle, the coordinate system of one image is transformed into the coordinate system of the other image for correction, so that the correction accuracy of each image can be improved, and more information on the road area can be obtained from both images. Therefore, a higher-precision correction process can be performed. According to the process of the first correction mode, even in the case of factors such as deviations in the installation of multiple cameras installed on the vehicle M and manufacturing variations in the imaging devices, the images captured by each imaging device can be more appropriately corrected.

[0067] [Second Correction Mode]

[0068] In the second correction mode, after feature points of the road area included in the image are extracted from the front image IM10 and the rear image IM30 respectively, the respective feature points are aggregated to perform correction of the front image IM10 and the rear image IM30.

[0069] Figure 6 is a flowchart showing an example of the process in the second correction mode. In Figure 6 this example, the first acquisition unit 111 acquires the front image IM10 (image frame) captured by the front camera 12 (step S200). Next, the extraction unit 120 extracts the feature points of the front image IM10 (step S202). Next, the first detection unit 130 detects the vehicle's own road area included in the front image IM10 (step S204), and detects the crossroad area relative to the vehicle's own road area (step S206). The detection methods for the vehicle's own road area and the crossroad area can be, for example, the same methods as those in the detection method in the above-described first correction mode. Next, the first feature point extraction unit 140 extracts the feature points of the vehicle's own road area among the feature points of the front image IM10 (step S208), and extracts the feature points of the crossroad area (step S210).

[0070] Next, the correction unit 150 determines whether feature points of both the vehicle's own road area and the crossroad area have been continuously extracted for a specified number of frames or more (step S212). If it is determined that feature points of both the vehicle's own road area and the crossroad area have not been continuously extracted for a specified number of frames or more, the process returns to step S200, and the above-described process is repeatedly executed until it is determined that feature points of both the vehicle's own road area and the crossroad area have been continuously extracted for a specified number of frames or more.

[0071] The image processing apparatus 100 performs the processes of steps S220 to S232 shown below in parallel with (or at an earlier or later timing than) the processes of steps S200 to S212. Specifically, first, the second acquisition unit 112 acquires a rear image IM30 captured by the second rear camera 16 (step S220). Next, the extraction unit 120 extracts feature points of the rear image IM30 (step S222). Next, the second detection unit 132 detects the own vehicle road region included in the rear image IM30 (step S224), and detects the intersecting road region with respect to the own vehicle road region (step S226). Next, the second feature point extraction unit 142 extracts the feature points of the own vehicle road region among the feature points of the rear image IM30 (step S228), and extracts the feature points of the intersecting road region (step S230). Next, the correction unit 150 determines whether feature points of both the own vehicle road region and the intersecting road region have been continuously extracted for a specified number of frames or more (step S232). When it is determined that the feature points of both the own vehicle road region and the intersecting road region have not been continuously extracted for a specified number of frames or more, the process returns to step S220, and the above-described processes are repeatedly executed until it is determined that the feature points of both the own vehicle road region and the intersecting road region have been continuously extracted for a specified number of frames or more.

[0072] In the processes of steps S212 and S232, when it is determined that the feature points of both the own vehicle road region and the intersecting road region have been extracted from both the front image IM10 and the rear image IM30, the correction unit 150 corrects the front image and the rear image using these feature points (step S240).

[0073] Here, the process of step S240 will be specifically described. For example, the correction unit 150 performs an optical flow process, which is a process for detecting the movement between frames of feature points based on the change in the positions of the feature points of the own vehicle road regions AR10F, AR10R - 1, AR10R - 2 (hereinafter simply referred to as "own vehicle road region AR10") and the intersecting road regions AR20L - 1, AR20R - 1, AR20L - 2, AR20R - 2 (hereinafter simply referred to as "intersecting road region AR20") included in two image frames at different times for the front image IM10 and the rear image IM20, and representing the detected movement using vectors (motion vectors). The motion vectors include information related to the direction and amount (displacement amount) of the movement, for example. The time interval (period) between the two image frames for which the motion vectors are obtained may be the period of the image frames acquired by the acquisition unit 110 (or an integer multiple of the period), or may be variably set based on the speed of the vehicle M, the sizes of the road regions (own vehicle road region, intersecting road region), etc.

[0074] The feature points used in the optical flow processing can also be used to replace all the feature points included in the own-lane area AR10 and the intersection-lane area AR20, and instead, the feature points are intermittently removed to a number less than or equal to a specified number. In this case, the correction unit 150 can also divide the own-lane area AR10 and the intersection-lane area AR20 into a plurality of divided areas, and adjust each divided area so that the number of feature points is above the lower limit value and below the upper limit value. By reducing the number of feature points used in the optical flow processing, the processing burden can be reduced.

[0075] The correction unit 150 sets a normal vector in a direction perpendicular to the road surface of the own-lane area AR10 and the intersection-lane area AR20 based on the motion vector obtained by the optical flow processing and the traveling direction of the vehicle M in time between two different image frames. For example, the correction unit 150 extracts a plurality of motion vectors from the road area, sets the road surface (plane) of the own-lane and the intersection-lane based on the directions of the extracted plurality of motion vectors (the directions corresponding to the traveling direction of the vehicle M), and derives the normal vector with respect to the set road surface. The correction unit 150 can also derive the normal vector for the own-lane area and the intersection-lane area obtained from the front image IM10 and the rear image IM30, respectively.

[0076] Moreover, the correction unit 150 corrects so that the deviation amount between the derived normal vector and the normal vector of the road surface based on the attitude of the vehicle M (reference normal vector) becomes below the threshold value. When there are a plurality of normal vectors, the correction unit 150 can correct by comparing the average normal vector obtained by averaging the normal vectors with the reference normal vector, or can correct so that the error (least square error) between the plurality of normal vectors and the reference normal vector becomes below the threshold value.

[0077] In the embodiment, since the front camera 12 and the second rear camera 16 are configured separately, the deviation amounts are different in each of them. Therefore, the correction unit 150 corrects so that the normal vector derived from the front image IM10 and the normal vector derived from the rear image IM30 are separately made to coincide with the reference normal vector. The correction unit 150 can also derive the relative angle between the front image and the rear image (the front camera 12 and the second rear camera 16) based on the two image correction parameters obtained by each correction.

[0078] The correction unit 150 can also correct the pitch direction (pitch angle) of the front image IM10 (front camera 12) using the normal vector with respect to the road surface of the own-lane area in the same manner as in the first correction mode, and correct the roll direction (roll angle) using the normal vector with respect to the road surface of the intersection-lane area. Thereby, more appropriate correction can be performed in each of the pitch and roll directions.

[0079] According to the processing of the second correction mode described above, correction is performed using the information of the feature points of the road area obtained from the front image and the rear image respectively. Therefore, a higher-precision correction process can be performed. According to the processing of the second correction mode, even in the case of factors such as deviations in the installation of multiple cameras installed in the vehicle M and unevenness in the products of the photographing devices, the images captured by each photographing device can be more appropriately corrected.

[0080] [Coordinate transformation unit and recognition unit]

[0081] Next, the processing of the coordinate transformation unit 160 and the recognition unit 20 in the embodiment will be described. For example, the coordinate transformation unit 160 transforms the coordinates in the camera coordinate system of the image acquired by the acquisition unit 110 into the bird's-eye coordinate system. In this case, the coordinate transformation unit 160 can perform coordinate transformation by adding image correction parameters (correction parameters in the pitch direction and roll direction) to the pre-determined reference coordinate transformation parameters, or can perform coordinate transformation processing on the image corrected by the correction unit 150.

[0082] Figure 7 is a diagram for explaining the coordinate transformation process in the embodiment. For example, the axes of the camera coordinate system (three-dimensional) of the image captured by the photographing device 10 (the front camera 12 in the Figure 7 example) are set as [Xc, Yc, Zc], and the imaginary camera coordinate system (vehicle coordinate system) parallel to the ground (moving road surface) with the attitude of the vehicle M as the reference and facing the traveling direction of the vehicle M is set as [Xvc, Yvc, Zvc]. Here, Xvc represents the traveling direction of the vehicle M, Yvc represents the lateral direction of the vehicle M, and Zvc represents the up-and-down direction of the vehicle M. When the roll angle with respect to the vehicle M is set as θ, the pitch angle with respect to the vehicle M is set as ρ, and the yaw angle with respect to the vehicle M is set as φ, each angle [θ, ρ, φ] represents the rotation angle around each axis of the imaginary camera coordinate system [Xvc, Yvc, Zvc]. At this time, the roll angle θ and the pitch angle ρ become the values corrected by the correction unit 150. The correction unit 150, for example, uses the following formula (1) to perform coordinate transformation (rotation) from the camera coordinate system to the imaginary camera coordinate system parallel to the ground in the traveling direction.

[0083]

[0084] The rotational degrees of freedom Rx, Ry, and Rz with respect to each axis in formula (1) are derived using the following formulas (2) to (4).

[0085]

[0086] The recognition unit 20 uses the image processed by the image processing device 100 to recognize the surrounding conditions such as the position of an object existing around the vehicle M. For example, the recognition unit 20 recognizes the position of the object when the image in the corrected camera coordinate system is transformed into the bird's-eye view coordinate system.

[0087] For example, as Figure 7 shown, when the ground (road surface) is always flat (the variation angle is within the allowable range), the recognition unit 20 recognizes the azimuth angle and distance of the object in the bird's-eye view coordinate system with respect to the object on the camera coordinate system observed from the front camera 12. In this case, the recognition unit 20, for example, uses the following formula (5) to calculate the depression angle α (the downward viewing angle of the object when expressed in polar coordinates with the orientation of the imaginary camera as the reference), and calculates the azimuth angle β through "β = tan －1 (Yvc / Xvc)".

[0088]

[0089] When the installation position (height hc) of the front camera 12 is preset, the recognition unit 20 uses the depression angle α and the height hc, and calculates the distance D from the vehicle M (more specifically, the position of the front camera 12 mounted on the vehicle M) to the object through "D = hc / tan α". By performing these processes on the objects around the vehicle M, the recognition unit 20 can more appropriately recognize the position of the objects around the vehicle M and the distance to the objects in the bird's-eye coordinate system.

[0090] In addition to notifying the occupant of information related to driving support, the notification control unit 40 can also generate an image representing the surrounding conditions of the vehicle M after being transformed into the bird's-eye view coordinate system, and notify the generated image to the occupant via the terminal device T. Thereby, the accurate surrounding conditions can be displayed to the occupant in a display manner that is easy for the occupant to process.

[0091] [Modification Example]

[0092] In the embodiment, instead of extracting the feature points of the entire image as described above and extracting the feature points of the own vehicle road area and the intersection road area among the extracted feature points, it is also possible to first extract the own vehicle road area and the intersection road area included in the image, and then extract the feature points included in each of the extracted road areas.

[0093] In the embodiment, the correction unit 150 can also correct the image when the own vehicle road area and the intersection road area of a specified area or more are extracted from the front image or the rear image. Thereby, the normal vector with respect to the road surface can be obtained more accurately from a relatively wide road area, and thus more appropriate correction can be performed.

[0094] In an embodiment, the image processing apparatus 100 may also set priorities for the front camera 12, the first rear camera 14, and the second rear camera 16, and preferentially perform calibration of the camera with the higher set priority. For example, in the case of a camera that is likely to deviate due to a pre-set position or the like, calibration processing is preferentially performed over other cameras, thereby suppressing deterioration in the recognition accuracy of the images used later. In the embodiment, the image processing apparatus 100 may also perform calibration processing such as aberration correction and distortion correction on the front camera 12, the first rear camera 14, and the second rear camera 16. In the embodiment, an image (side image) including the lateral direction of the vehicle M may be used instead of (or on the basis of) the rear image.

[0095] One of the first calibration mode and the second calibration mode described above may also be combined with a part or all of the other. The image processing apparatus 100 may either execute a predetermined one of the first calibration mode and the second calibration mode described above, or execute both and synthesize (average) the execution results. The calibration unit 150 may also select and execute which calibration mode according to the driving condition (surrounding road condition) of the vehicle M and the detection condition of the road area. For example, when the detected road area is equal to or larger than a specified area, even if the front image and the rear image are separated, the feature points of the road area can be detected, so calibration processing is performed in the first calibration mode. When the detected road area is smaller than the specified area, since there are fewer feature points with respect to the road area, calibration processing is performed in the second calibration mode in which the feature points of both the front image and the rear image are combined. Thereby, more appropriate calibration processing can be performed according to the situation.

[0096] According to the embodiment described above, in the image processing apparatus 100, there are provided: an acquisition unit 110 that acquires a first image obtained by photographing a first direction of the moving body from a photographing device 10 mounted on a vehicle M (an example of a moving body) and a second image obtained by photographing a second direction different from the first direction; an extraction unit 120 that extracts feature points from the first image and the second image acquired by the acquisition unit 110; a first detection unit 130 that detects a road region included in the first image; a second detection unit 132 that detects a road region included in the second image; a first feature point extraction unit 140 that extracts, as first feature points, the feature points of the road region detected by the first detection unit 130 among the feature points extracted by the extraction unit 120; a second feature point extraction unit 142 that extracts, as second feature points, the feature points of the road region detected by the second detection unit 132 among the feature points extracted by the extraction unit 120; and a correction unit 150 that corrects the first image and the second image based on the first feature points and the second feature points, so that the images photographed by the photographing device 10 mounted on the vehicle M can be corrected more appropriately.

[0097] Specifically, according to the embodiment, for example, by using the images of the front and rear of the vehicle M and based on the feature points of the own vehicle road region and the crossroad region, the image can be corrected, so that higher-precision correction can be performed using more information. By including the rear image, it is particularly easy to obtain the crossroad region. According to the embodiment, regarding the pitch direction of the vehicle M (the photographing device 10), by using the feature amount of the own vehicle road region extending in the front-rear direction when observed from the vehicle M, higher-precision correction in the pitch direction can be performed. Regarding the roll direction of the vehicle M (the photographing device 10), by using the information obtained from the crossroad region extending in the left-right direction when observed from the vehicle M (the normal vector to the road surface of the crossroad), higher-precision correction in the roll direction can be performed.

[0098] According to the embodiment, even when there are deviations in the installation of the photographing device on the vehicle M and variations in the product performance of the photographing device, by using the front image and the rear image photographed by the photographing device mounted on the vehicle M for correction, more appropriate transformation can be performed when performing coordinate transformation of the image. According to the embodiment, for example, even when a front camera and a rear camera are provided in a manner sandwiching the pitch rotation center of the vehicle M and when they are provided at separated positions so that there is no overlap in the photographing range, appropriate correction related to both the roll direction and the pitch direction can be performed. Thus, the relative position and relative distance of the objects around the vehicle M can be more accurately recognized using the corrected images, and more appropriate driving support can be achieved by recognizing these.

[0099] The embodiments described above can be represented as follows.

[0100] An image processing apparatus, comprising:

[0101] A storage medium (storage medium) storing computer-readable instructions; and

[0102] A processor connected to the storage medium,

[0103] The processor executes the following processing by executing computer-readable instructions: (the processor executing the computer-readable instructions to:)

[0104] Obtain a first image obtained by photographing the mobile body in a first direction and a second image obtained by photographing a second direction different from the first direction from a photographing device mounted on the mobile body;

[0105] Extract feature points from the obtained first image and the second image;

[0106] Detect a road area included in the first image;

[0107] Detect a road area included in the second image;

[0108] Extract, as first feature points, the feature points of the road area included in the detected first image among the extracted feature points;

[0109] Extract, as second feature points, the feature points of the road area included in the detected second image among the extracted feature points; and

[0110] Correct the first image and the second image based on the first feature points and the second feature points.

[0111] As described above, specific embodiments of the present invention have been described using embodiments, but the present invention is not limited to such embodiments at all, and various modifications and substitutions can be made without departing from the gist of the present invention.

Claims

1. An image processing device, wherein: The image processing device comprises: An acquisition unit that acquires, from a camera mounted on the moving object, a first image obtained by photographing the moving object in a first direction and a second image obtained by photographing the moving object in a second direction different from the first direction; an extraction unit that extracts feature points from the first image and the second image acquired by the acquisition unit; A first detection unit, configured to detect a road area included in the first image; a second detection unit configured to detect a road area included in the second image; a first feature point extraction unit that extracts, from among the feature points extracted by the extraction unit, feature points of the road area detected by the first detection unit as first feature points; a second feature point extraction unit that extracts, from among the feature points extracted by the extraction unit, feature points of the road area detected by the second detection unit as second feature points; as well as A correction unit corrects the first image and the second image based on the first feature point and the second feature point.

2. The image processing device according to claim 1, wherein: The imaging device includes a first imaging unit that captures the first image and a second imaging unit that captures the second image. The correction unit derives a relative angle between the first imaging unit and the second imaging unit based on a correction result of the first image and the second image.

3. The image processing device according to claim 1, wherein: The correction unit corrects the first image and the second image when the first feature point extraction unit and the second feature point extraction unit each continuously extract feature points included in the road area for a predetermined number of frames or more.

4. The image processing device according to claim 2, wherein: The correction unit derives the relative angle based on the normal vector of the road surface in the road area included in the first image and the normal vector of the road surface in the road area included in the second image, and based on the derived relative angle, transforms the coordinate system of the second image into the coordinate system of the first image, thereby correcting the first image and the second image.

5. The image processing device according to claim 3, wherein: The first feature point extraction unit extracts, from the image frames of the first image acquired by the acquisition unit at predetermined intervals, feature points of a first road area where the moving body moves and feature points of a second road area intersecting the first road area as first feature points. The second feature point extraction unit extracts feature points of the first road area and feature points of the second road area as second feature points from the image frames of the second image acquired by the acquisition unit at predetermined intervals. The correction unit corrects the first image and the second image when the first feature points and the second feature points are continuously extracted from the first image and the second image for a predetermined number of frames or more.

6. The image processing device according to claim 1, wherein: The road area includes a first road area on which the moving body moves and a second road area intersecting the first road area.

7. The image processing device according to claim 6, wherein: The correction unit performs the following processing: Correcting the pitch angles of a first imaging unit that captures the first image and a second imaging unit that captures the second image based on the feature points of the first road area; and The roll angles of the first imaging unit and the second imaging unit are corrected based on the feature points of the second road area.

8. An image processing method, wherein: The image processing method enables the computer to perform the following processing: Acquire, from a photographing device mounted on a mobile body, a first image obtained by photographing the mobile body in a first direction and a second image obtained by photographing the mobile body in a second direction different from the first direction; Extracting feature points from the first image and the second image obtained; Detecting a road area included in the first image; Detecting a road area included in the second image; extracting, from among the extracted feature points, feature points of a road area detected and included in the first image as first feature points; extracting, from among the extracted feature points, feature points of a road area detected and included in the second image as second feature points; and The first image and the second image are corrected based on the first feature points and the second feature points.

9. A storage medium storing a program, wherein: The program causes the computer to execute the following processing: Acquire, from a photographing device mounted on a mobile body, a first image obtained by photographing the mobile body in a first direction and a second image obtained by photographing the mobile body in a second direction different from the first direction; Extracting feature points from the first image and the second image obtained; Detecting a road area included in the first image; Detecting a road area included in the second image; extracting, from among the extracted feature points, feature points of a road area detected and included in the first image as first feature points; extracting, from among the extracted feature points, feature points of a road area detected and included in the second image as second feature points; and The first image and the second image are corrected based on the first feature points and the second feature points.

Images