Image processing apparatus, image processing method, and storage medium

Abstract

Provided are an image processing device, an image processing method, and a storage medium capable of more appropriately correcting an image captured by an imaging device mounted on a moving body. The image processing apparatus includes: an acquisition unit configured to acquire an image of a periphery of a moving body from an imaging device mounted on the moving body; an extraction unit that extracts feature points from the image acquired by the acquisition unit; a first detection unit that detects, from the image, a movement road region in which the moving body moves; a second detection unit that detects, from the image, an intersection road region that intersects the moving road region; a first feature point extraction unit that extracts, as a first feature point, a feature point of the moving road region detected by the first detection unit from among the feature points extracted by the extraction unit; a second feature point extraction unit that extracts, as a second feature point, a feature point of the intersection road region detected by the second detection unit from among the feature points; and a correction unit that corrects the image on the basis of the first feature point and the second feature point.

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, an image processing method, and a storage medium. Background Art

[0002] In recent years, efforts have been actively made to provide a sustainable transportation system that also takes into account people who are particularly vulnerable among traffic participants. In order to achieve this goal, research and development have been carried out to further improve traffic safety and convenience through research and development related to driving support technology. Related to this, there is known the following technology: obtaining a captured image from a camera mounted on a moving body, extracting feature points from an extraction region set or changed based on the external environment of the moving body in the shooting direction of the camera in the obtained captured image, and estimating the attitude of the camera based on the extracted feature points (for example, refer to Japanese Unexamined Patent Application Publication No. 2021-33605). Summary of the Invention

[0003] However, in driving support technology, there is a problem that appropriate correction of a captured image may not be possible due to factors such as deviation in the installation of a shooting device on a moving body, product variations in the shooting device, and the like.

[0004] The present application has been completed to solve the above problems, and one of its purposes is to provide an image processing apparatus, an image processing method, and a storage medium that can more appropriately correct an image captured by a shooting device mounted on a moving body. Then, it further contributes to the development of a sustainable transportation system.

[0005] The image processing apparatus, image processing method, and storage medium of the present invention adopt the following configuration.

[0006] (1): An image processing apparatus according to one aspect of the present invention includes: an acquisition unit that acquires an image of the periphery of the moving body from a shooting device mounted on the moving body; an extraction unit that extracts feature points from the image acquired by the acquisition unit; a first detection unit that detects a moving road area on which the moving body moves from the image; a second detection unit that detects an intersection road area intersecting the moving road area from the image; a first feature point extraction unit that extracts, as first feature points, the feature points of the moving road area detected by the first detection unit among the feature points extracted by the extraction unit; a second feature point extraction unit that extracts, as second feature points, the feature points of the intersection road area detected by the second detection unit among the feature points; and a correction unit that corrects the image based on the first feature points and the second feature points.

[0007] (2): Based on the solution in (1) above, the acquisition unit acquires an image obtained by photographing at least the front of the moving body using the photographing device and an image in a direction different from the front.

[0008] (3): Based on the solution in (1) above, the correction unit corrects the image in the pitch direction of the moving body based on the first feature point, and corrects the image in the roll direction of the moving body based on the second feature point.

[0009] (4): Based on the solution in (1) above, when the correction unit continuously extracts the first feature point of the moving road area and the second feature point of the cross road area from the image frames acquired by the acquisition unit at a predetermined period for a predetermined number of frames or more, the correction unit corrects the coordinate system of the image.

[0010] (5): Based on the solution in (1) above, the correction unit corrects the coordinate transformation parameters for transforming the coordinate system of the image based on the front of the moving body included in the image to the bird's-eye coordinate system for observing the moving body from above.

[0011] (6): Based on the solution in (1) above, in the photographing device, the photographing unit for photographing the front of the moving body and the photographing unit for photographing a direction different from the front are integrally formed.

[0012] (7): The image processing method according to other solutions of the present invention, wherein the image processing method causes a computer to perform the following processing: acquiring an image of the periphery of the moving body from a photographing device mounted on the moving body; extracting feature points from the acquired image; detecting a moving road area on which the moving body moves from the image; detecting a cross road area intersecting with the moving road area from the image; extracting the feature points of the moving road area among the extracted feature points as the first feature points, and extracting the feature points of the cross road area among the feature points as the second feature points; and correcting the image based on the first feature points and the second feature points.

[0013] (8): A storage medium according to other aspects of the present invention stores a program, where the program causes a computer to perform the following processes: obtaining an image of the periphery of the moving body from an imaging device mounted on the moving body; extracting feature points from the obtained image; detecting a moving road area on which the moving body moves from the image; detecting an intersecting road area intersecting with the moving road area from the image; extracting the feature points of the moving road area among the extracted feature points as first feature points; extracting the feature points of the intersecting road area among the feature points as second feature points; and correcting the image based on the first feature points and the second feature points.

[0014] According to the above aspects (1) to (8), it is possible to perform more appropriate correction on an image captured by an imaging device mounted on a moving body. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a diagram showing an example of the functional structure of a driving support device including an image processing device according to an embodiment.

[0016] Figure 2A It is a diagram showing an example of the installation position of the imaging device.

[0017] Figure 2B It is a diagram showing an example of the structure of the imaging device.

[0018] Figure 3 It is a diagram for explaining the shooting directions of the front camera and the rear camera.

[0019] Figure 4 It is a diagram showing an example of an image captured by the imaging device.

[0020] Figure 5 It is a diagram for explaining the coordinate transformation process in the embodiment.

[0021] Figure 6 It is a flowchart showing an example of the process flow executed by the image processing device according to the embodiment. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0022] Hereinafter, embodiments of the image processing apparatus, the image processing method, and the storage medium of the present invention will be described with reference to the accompanying drawings. In the following examples, an image processing apparatus mounted on a moving body will be described. The moving body may include all moving bodies for a person (such as a driver or other passengers) to ride, such as a three-wheeled or four-wheeled vehicle, a two-wheeled vehicle, or a micro-moving body. The moving body may also be provided with a driving support device that supports the driving of the passengers (drivers) of the moving body based on the images processed by the image processing apparatus. In the following description, the moving body is a four-wheeled vehicle (hereinafter referred to as "vehicle M") and is provided with a driving support device. Vehicle M may be any one of a motor vehicle powered by an internal combustion engine such as a diesel engine or a gasoline engine, an electric motor vehicle powered by an electric motor, or a hybrid motor vehicle having both an internal combustion engine and an electric motor. In the following description, the front direction of vehicle M is set as the positive X direction, the rear direction of vehicle M is set as the negative X direction, the right direction in the width direction of vehicle M and based on the positive X direction is set as the positive Y direction, the left direction in the width direction of vehicle M and based on the positive X direction is set as the negative Y direction, and the direction orthogonal to the X direction and the Y direction and being the height direction of vehicle M is set as the positive Z direction for description.

[0023] [Structure]

[0024] Figure 1 It is a diagram showing an example of the functional structure of the driving support device 1 including the image processing apparatus of the embodiment. Figure 1The driving support device 1 shown, for example, includes a photographing device 10, an identification unit 20, a driving support unit 30, a notification control unit 40, and an image processing device 100. The identification unit 20, the driving support unit 30, the notification control unit 40, and the image processing device 100 are realized, for example, by a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Some or all of these components may also be realized by hardware (including a circuitry unit: circuitry) such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), a GPU (Graphics Processing Unit), an SOC (System On Chip), or may be realized by the cooperation of software and hardware. The program may be pre-stored in a storage device (a storage device having a non-transitory storage medium) such as an HDD (Hard Disk Drive) or a flash memory, or may be stored in a removable storage medium (a non-transitory storage medium) such as a DVD or a CD-ROM, and installed by mounting the storage medium on a drive device. In addition to Figure 1 the structure, the vehicle M in the embodiment further includes a structure for driving the vehicle M (for example, a driving operation member, a driving device such as an engine or a motor, a steering device, a braking device, a vehicle sensor), etc.

[0025] The photographing device 10 photographs the periphery of the vehicle M. For example, it is a digital camera using a solid-state imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The photographing device 10 may also be a stereo camera. The photographing device 10 may also be a camera used in, for example, a dash cam. In Figure 1In the example, the imaging device 10 includes a front camera 12 and a rear camera 14 as a plurality of imaging units. The front camera 12 images a predetermined area in front of the vehicle M. The rear camera 14 images a predetermined area in a direction different from the front (for example, the rear of the vehicle M). At least one of the front image captured by the front camera 12 and the rear image captured by the rear camera 14 may include a lateral area of the vehicle M. The imaging device 10 may also be provided with a side camera that images the lateral side of the vehicle M in addition to the front camera 12 and the rear camera 14. The imaging device 10 may also be a fisheye camera that can image the periphery including the front and rear of the vehicle M in a wide angle (for example, 360 degrees) instead of the above cameras. Each camera of the imaging device 10 repeatedly images at a predetermined cycle, and the imaging device 10 outputs the captured image to the image processing device 100.

[0026] The image processing device 100 acquires the image captured by the imaging device 10 and performs a process of transforming the coordinate system of the image (hereinafter, the camera coordinate system) into a coordinate system different from the camera coordinate system. The camera coordinate system includes a front camera coordinate system corresponding to the front image and a rear camera coordinate system corresponding to the rear image. The coordinate system different from the camera coordinate system is, for example, a coordinate system (vehicle coordinate system, bird's-eye coordinate system) based on the position of the vehicle M when observing the vehicle M from above.

[0027] The image processing device 100 includes, for example, an acquisition unit 110, an extraction unit 120, a first detection unit 130, a second detection unit 132, a first feature point extraction unit 140, a second feature point extraction unit 142, a correction unit 150, a coordinate transformation unit 160, and a storage unit 170.

[0028] The storage unit 170 can also be implemented by a storage device such as an HDD, a flash memory, or an SSD (Solid State Drive), an EEPROM (Electrically Erasable Programmable Read Only Memory), a ROM (Read Only Memory), or a RAM (Random Access Memory), etc. The storage unit 170 stores, for example, the images acquired by the acquisition unit 110, the processing results processed by the correction unit 150 and the coordinate transformation unit 160, programs, and various other information. Map information may also be stored in the storage unit 170. The map information is, for example, information that represents the shape of a road by showing road segments and nodes connected by the road segments and establishing a correspondence relationship with position information (latitude and longitude information). The map information may also include lane boundary information such as the curvature, slope, number of lanes, width, information on the center of a lane, or road dividing lines that divide lanes of the road. The map information may include traffic restriction information, position information where there are branches, merges, intersections, T-junctions, etc., facility information such as buildings or parking lots, POI (Point Of Interest) information, etc. The map information can be updated at any time by the vehicle M or the driving support device 1 communicating with other devices.

[0029] The acquisition unit 110 acquires the image frames of the front image and the rear image captured by the imaging device 10 at a predetermined period. The extraction unit 120 extracts the feature points included in the front image and the rear image acquired by the acquisition unit 110. For example, the extraction unit 120 performs known image analysis processes such as edge extraction processing on the front image and the rear image, and extracts the feature points of the objects (such as traffic signal machines, road signs, pedestrians, other vehicles, etc., traffic participants in the actual space, including not only buildings but also road dividing lines, stop lines, etc.) included in the image based on the results of the image analysis processing. In this case, the extraction unit 120 extracts, for example, a point sequence on the edge of the object included in the image as the feature points (feature point group). The method for extracting the feature points on the image is not limited to the above example, and other known methods can be used. The extraction unit 120 can also use, for example, a learned model to extract the feature points. The learned model is learned in such a way that when the front image and the rear image are input, the edges of the objects (such as buildings, road structures, etc.) reflected in the image are output as a point group. The learned model can be either pre-stored in the storage unit 170 or obtained from an external device via a communication device (not shown) mounted on the vehicle M. The extraction unit 120 can also use, for example, the method of Visual SLAM (Simultaneous Localization and Mapping), which is a technique for three-dimensionally grasping its own position based on the image data captured by the imaging device 10, to extract the feature points.

[0030] The first detection unit 130 detects the own vehicle road area (an example of the moving road area) on which the vehicle M travels (moves) from the front image and the rear image. The second detection unit 132 detects the cross road area that intersects the own vehicle road from the front image and the rear image. The cross road is, for example, a road that is connected to the own vehicle road at a predetermined angle range including a right angle at an intersection, a T-junction, etc. Details of the processing of the first detection unit 130 and the second detection unit 132 will be described later.

[0031] The first feature point extraction unit 140 extracts the feature points of the own vehicle road area detected by the first detection unit 130 from the feature points (feature point group) extracted by the extraction unit 120 as the first feature points. The second feature point extraction unit 142 extracts the feature points of the cross road area detected by the second detection unit 132 from the feature points (feature point group) extracted by the extraction unit 120 as the second feature points. The first feature points and the second feature points are the feature points required for image correction (calibration) in the embodiment. Details of the processing of the first feature point extraction unit 140 and the second feature point extraction unit 142 will be described later.

[0032] The correction unit 150 corrects the image captured by the imaging device 10 based on the first feature points extracted by the first feature point extraction unit 140 and the second feature points extracted by the second feature point extraction unit 142. For example, the correction unit 150 corrects the coordinate transformation parameters for transforming the coordinate system of the captured image from the camera coordinate system to a coordinate system different from the camera coordinate system (bird's-eye coordinate system). Thereby, more appropriate coordinate transformation can be performed. Details of the function of the correction unit 150 will be described later. The correction unit 150 may also store information related to the corrected coordinate transformation parameters in the storage unit 170.

[0033] The coordinate transformation unit 160 transforms the coordinate system (camera coordinate system) of the image acquired by the acquisition unit 110 into another coordinate system. For example, the coordinate transformation unit 160 transforms the camera coordinate system into a bird's-eye coordinate system used in the recognition of the surrounding conditions of the vehicle M by the recognition unit 20. In this case, the coordinate transformation unit 160 uses the coordinate transformation parameters corrected by the correction unit 150 for the reference coordinate transformation parameters pre-stored in the storage unit 170 or the like to transform the camera coordinate system into the bird's-eye coordinates.

[0034] The recognition unit 20 recognizes the surrounding conditions of the vehicle M based on the image (hereinafter, "bird's-eye image") transformed into the bird's-eye coordinate system by the coordinate transformation unit 160. For example, the recognition unit 20 recognizes the objects existing in the vicinity of the vehicle M (within a specified distance from the vehicle M). The objects include, for example, other vehicles, pedestrians, and other traffic participants. The recognition unit 20 recognizes the position (relative position when observed from the vehicle M), speed (relative speed when observed from the vehicle M), type, shape, size, etc. of the object. In the recognition of the object, for example, object recognition using a model based on deep learning, deep machine learning, etc., object recognition based on a pattern matching method, or an object recognition method obtained by combining these is performed.

[0035] The driving support unit 30 performs driving support for the occupants of the vehicle M based on the recognition result recognized by the recognition unit 20. For example, the driving support unit 30 determines whether the vehicle M has deviated from the driving lane of the vehicle divided by the road division lines recognized by the recognition unit 20, and when there is a possibility of deviation, notifies the driver of the vehicle M via the notification control unit 40, or controls the steering of the vehicle M in a manner that suppresses the deviation from the driving lane of the vehicle M (in a manner that the vehicle M moves toward the center side of the driving lane) using a steering device (not shown). The driving support unit 30 recognizes obstacles such as other vehicles existing in the vicinity of the vehicle M (within a specified distance), and when it is determined based on the relative position and relative speed with the obstacle that there is a possibility of contact with the obstacle, notifies the occupants via the notification control unit 40 or performs driving control to avoid contact (at least one of speed control and steering control).

[0036] The notification control unit 40 makes a notification related to driving support to the occupant (driver) of the vehicle M based on the control performed by the driving support unit 30. In this case, the notification control unit 40 generates notification information such as sound (alarm), image, etc. that corresponds to the content of the notification to be notified to the occupant, and causes the generated notification information to be transmitted to the terminal device T for output.

[0037] Here, the terminal device T is, for example, a mobile terminal device such as a smartphone or a tablet terminal used by the driver of the vehicle M equipped with the driving support device 1. In the terminal device T, for example, an application program for receiving driving support performed by the driving support device 1 is executed. The application program causes the display unit of the terminal device T to display an image obtained based on the information and notification transmitted by the driving support device 1, or causes the speaker of the terminal device T to emit sound. The terminal device T is an example of the "notification unit". The terminal device T can be detachably mounted on the vehicle M and used, for example. For example, a holder for the terminal device T having a mounting / demounting portion is provided on one or both of the terminal device T and the vehicle M, and the terminal device T is supported by the holder. In the embodiment, when devices such as a navigation device, a display device, and a speaker are mounted on the vehicle M, the notification information can also be output from the mounted devices instead of the terminal device T based on the instruction of the driving support unit 30.

[0038] [Processing up to feature point extraction]

[0039] Next, a specific example of the processing up to extracting feature points from the images (front image, rear image) acquired by the acquisition unit 110 will be described. Figure 2A FIG. is an example of the imaging device 10. The imaging device 10 of the embodiment is, for example, installed near the interior rearview mirror RM at the upper part of the front windshield (in the Figure 2A example, 2B below the interior rearview mirror RM). In the Figure 2A example, the imaging device 10 is installed at the lower part of the interior rearview mirror RM, but the position is not limited thereto. For example, it can also be installed on the right side or left side of the interior rearview mirror RM. Figure 2B FIG. is an example of the structure of the imaging device 10. The imaging device 10 includes, for example, a mounting portion AT that can be detachably mounted on the vehicle M. The mounting portion AT is, for example, any member such as a suction cup, a seal, a bracket, or other supporting members. In addition, in the Figure 2B example, a front camera 12 and a rear camera 14 are provided in the imaging device 10 so as to photograph in different directions.

[0040] Figure 3 FIG. is a diagram for explaining the photographing directions of the front camera 12 and the rear camera 14. As described above in the Figure 2B , Figure 3As shown, in the imaging device 10, the front camera 12 and the rear camera 14 are integrally formed within a specified distance. The so-called integral formation may also include, for example, the case where the front camera 12 and the rear camera 14 are stored in one housing, or a structure where they are each connected (joined). In this structure, for example, when the front camera 12 captures a specified field-of-view angle region VA1 centered on the front direction A1 (the positive X-axis direction in the figure) of the vehicle M, the rear camera 14 captures a specified field-of-view angle region VA2 centered on the direction A2 opposite to the front direction A1 of the vehicle M. Therefore, assuming that due to a setting deviation or the like caused by some factor during or after installation, the front camera 12 captures a field-of-view angle region centered on a direction that is inclined downward by an angle θ1 with respect to the front direction A1 of the vehicle M as a reference, the rear camera 14 captures a field-of-view angle region centered on a direction that is inclined upward by an angle θ1 with respect to the direction A2 opposite to the front direction A1 as a reference. In the embodiment, when the shooting direction (field-of-view angle) of one of the integrally formed front camera 12 and rear camera 14 deviates, the shooting direction (field-of-view angle) of the other camera also deviates in the same way in a symmetric direction centered on the installation position of the imaging device 10.

[0041] Figure 4 FIG. is an example of an image captured by the imaging device 10. In Figure 4 this example, a front image IM10 captured by the front camera 12 and a rear image IM20 of the vehicle M captured by the rear camera 14 are shown. In the front image IM10, a region outside the vehicle (in front of the vehicle M) is captured across the front windshield of the vehicle M. In the rear image IM20, a rear region including the interior of the vehicle is captured, and a region outside the vehicle (the side or rear of the vehicle M) is captured across the side windshield and the rear windshield.

[0042] The extraction unit 120 extracts a plurality of feature points (feature point groups) from Figure 4 the front image IM10 and the rear image IM20 shown. For example, the extraction unit 120 extracts feature points for each image frame of the front image IM10 and the rear image IM20 acquired at a specified period.

[0043] The first detection unit 130 detects the own vehicle road region in the images from the front image IM10 and the rear image IM20. For example, the first detection unit 130 obtains the left and right road dividing lines of the vehicle M based on the point sequence part included in the feature point group extracted by the extraction unit 120, and detects the region divided by the obtained road dividing lines as the own vehicle road region. For example, the first detection unit 130 may also divide the front image IM10 and the rear image IM20 into a plurality of divided regions, and detect the own vehicle road region for each divided region. AsFigure 4 As shown by the front image IM10 and the rear image IM20, on each image, the position where the own vehicle lane area exists is near the center of the image and is somewhat easy to predict. Therefore, the first detection unit 130 can also target partial areas (for example, a specified area including the center of the image) in the front image IM10 and the rear image IM20 that are predicted in advance to have a high possibility of the existence of the own vehicle lane area, and detect the own vehicle lane area obtained based on the feature point group. Thereby, the processing burden related to the detection of the own vehicle lane area can be reduced.

[0044] The first detection unit 130 can also implement functions based on AI (Artificial Intelligence) and functions based on a pre-given model in parallel, for example. For example, the function of "detecting the own vehicle lane area" can be achieved by parallelly performing the detection of the own vehicle lane area of the vehicle based on deep learning, etc. and the detection of the own vehicle lane area based on a pre-given determination process (for example, a determination process based on pattern matching) on the front image IM10 and the rear image IM20, and comprehensively evaluating by scoring both.

[0045] In Figure 4 this example, the first detection unit 130 detects the own vehicle lane area AR10F in front of the vehicle M from the front image IM10, and detects the own vehicle lane area AR10R behind the vehicle M from the rear image IM20.

[0046] The second detection unit 132 detects the crossroad area in the image from the front image IM10 and the rear image IM20. For example, the second detection unit 132 detects a point sequence that contacts the own vehicle lane area detected by the first detection unit 130 at a specified angle based on the point sequence part included in the feature point group extracted by the extraction unit 120. The so-called specified angle is, for example, a specified angle range (for example, about 75 to 105 degrees) including 90 degrees (right angle) relative to the extension direction of the own vehicle lane area. Moreover, when the two point sequences exist parallel (including the allowable error range) within a specified distance, the second detection unit 132 regards the two point sequences as road dividing lines, and detects the area divided by the road dividing lines as the crossroad area.

[0047] For example, the second detection unit 132 can also divide the front image IM10 and the rear image IM20 into a plurality of divided areas, and detect the crossroad area for each divided area. The second detection unit 132 can also target partial areas in the front image IM10 and the rear image IM20 that are predicted in advance to have a high possibility of the existence of the crossroad area, and detect the crossroad area obtained based on the feature point group. Thereby, the processing burden related to the detection of the crossroad area can be reduced.

[0048] The second detection unit 132 can also implement functions based on AI and functions based on a pre-given model in parallel. For example, the function of "detecting an intersection road area" can be achieved by performing the detection of the intersection road area based on deep learning, etc., and the detection of the intersection road area based on a pre-given determination process (for example, a determination process based on pattern matching) on the front image IM10 and the rear image IM20 in parallel, and comprehensively evaluating by scoring both.

[0049] The second detection unit 132 can also perform the detection process of the intersection road area within a range within a specified distance from the position where a specific road structure such as a traffic signal or a crosswalk is detected from the front image IM10 or the rear image IM20. The second detection unit 132 can also perform the detection process of the intersection road area based on the position information of the vehicle M and referring to the map information stored in the storage unit 170 when the position of the vehicle M is close to a position where there is a high possibility of an intersection road such as an intersection or a T-junction (within a specified distance). The position information of the vehicle M is obtained, for example, by a position sensor (not shown) mounted on the vehicle M. The position sensor obtains position information (longitude and latitude information) from a GPS (Global Positioning System) device, for example. The position sensor can also use a GNSS (Global Navigation Satellite System) receiver of a navigation device (not shown) mounted on the vehicle M to obtain position information. Thus, since the detection process is performed in an area where there is a high possibility of an intersection road area, the intersection road area can be detected more efficiently.

[0050] In Figure 4 this example, the second detection unit 132 detects the intersection road areas AR20L and AR20R from the front image IM10. The second detection unit 132 can also distinguish and identify the intersection road area AR20R connected to the vehicle's own road area AR10F on the right side and the intersection road area AR20L connected to the vehicle's own road area AR10F on the left side. If the intersection road area is included in the rear image IM20, the second detection unit 132 also detects this area.

[0051] The first feature point extraction unit 140 extracts the feature points of the vehicle's own road areas AR10F and AR10R detected by the first detection unit 130 from the feature points (feature point group) extracted by the extraction unit 120. The second feature point extraction unit 142 extracts the feature points of the intersection road areas AR20L and AR20R detected by the second detection unit 132 from the feature points extracted by the extraction unit 120.

[0052] [Calibration Unit]

[0053] Next, the image calibration (calibration) process performed by the calibration unit 150 will be specifically described. For example, the calibration unit 150 performs image calibration when at least one of the front image IM10 and the rear image IM20 continuously extracts feature points of the host vehicle road area and the crossroad area from frames of images at different times for a specified number of frames or more. The specified number of frames can be a fixed number or can be variably set according to driving conditions such as the road shape of the driving lane of the vehicle M, the speed of the vehicle M, and the traveling direction.

[0054] For example, the calibration unit 150 performs an optical flow process, which is based on the change in the positions of the feature points of the host vehicle road area (for example, Figure 4 the host vehicle road areas AR10F and AR10R shown) included in two image frames at different times of the front image IM10 and the rear image IM20 over time, to detect the movement of the feature points between the frames and represents the detected movement using vectors (motion vectors). The motion vectors include information related to the direction and magnitude (displacement amount) of the motion, for example. The time interval (period) between two different image frames for obtaining the motion vectors can be the period of the image frames obtained by the acquisition unit 110 (or an integer multiple of the period) or can be variably set based on the speed of the vehicle M, the size of the host vehicle road area, etc.

[0055] The feature points used in the optical flow process can also be feature points with a reduced number removed at intervals instead of using all the feature points included in the host vehicle road areas AR10F and AR10R to be less than a specified number. In this case, the calibration unit 150 can also divide the host vehicle road areas AR10F and AR10R into multiple divided areas and adjust each divided area so that the number of feature points is above the lower limit value and below the upper limit value. By reducing the number of feature points used in the optical flow process, the processing burden can be reduced.

[0056] The calibration unit 150 sets a normal vector in the direction perpendicular to the road surface of the host vehicle road based on the motion vectors obtained by the optical flow process and the traveling direction of the vehicle M in time between two different image frames. For example, the calibration unit 150 extracts multiple motion vectors from the host vehicle road area AR10F obtained from the front image, sets the road surface (plane) of the host vehicle road based on the directions of the extracted multiple motion vectors (the direction corresponding to the traveling direction of the vehicle M), and sets the normal vector with respect to the set road surface. The calibration unit 150 also similarly sets the normal vector with respect to the road surface of the host vehicle road for the host vehicle road area AR10R obtained from the rear image.

[0057] Then, the calibration unit 150 calibrates the image in such a manner that the deviation amount between the normal vectors of the own-lane regions AR10F and AR10R in the camera coordinate system and the normal vector of the actual road surface (i.e., the reference normal vector in the direction perpendicular to the horizontal plane) becomes below the threshold value. In the embodiment, since the front camera 12 and the rear camera 14 are integrally formed, the deviation directions in the front image and the rear image are opposite. That is, assuming that the normal vector of the own-lane region AR10F in the front image deviates to the right with respect to the reference normal vector, the normal vector of the own-lane region AR10R in the rear image deviates to the left with respect to the reference normal vector. Therefore, the calibration unit 150 performs calibration corresponding to each of the front image and the rear image.

[0058] The own-lane regions AR10F and AR10R are regions extending along the vertical direction (front or rear of the vehicle M) of the image. Therefore, the calibration unit 150 mainly uses the own-lane regions AR10F and AR10R to correct the pitching direction of the vehicle M (or the imaging device 10). Thereby, more appropriate correction of the pitching direction can be performed.

[0059] In addition, similarly, the calibration unit 150 performs an optical flow process for extracting a motion vector based on the change in the position of the feature points of the intersection road regions (e.g., Figure 4 the intersection road regions AR20L and AR20R shown) included in two image frames with different times in the front image IM10 and the rear image IM20. In this case, the calibration unit 150 may either remove the feature point intervals used in the optical flow process or divide the intersection road regions AR20L and AR20R into a plurality of divided regions and adjust each divided region so that the number of feature points is above the lower limit value and below the upper limit value.

[0060] As described above, the calibration unit 150 sets the normal vector with respect to the road surface of the intersection road based on the motion vector and the traveling direction of the vehicle M in terms of time between two different image frames, and calibrates the image in such a manner that the deviation amount between the set normal vector and the reference normal vector becomes below the threshold value. Since the intersection road regions AR20L and AR20R are regions extending along the left-right direction (lateral direction of the vehicle M) of the image, the calibration unit 150 mainly uses the intersection road regions AR20L and AR20R to correct the roll direction of the vehicle M (or the imaging device 10 mounted on the vehicle M). Thereby, more appropriate correction of the roll direction can be performed.

[0061] As described above, the correction unit 150 performs pitch - direction calibration based on the feature points included in the own - lane area and roll - direction calibration based on the feature points included in the cross - road area. When there are multiple normal vectors, the correction unit 150 can perform calibration by comparing the average of the normal vectors with the reference normal vector, or can perform calibration in such a way that the error (least - square error) between the multiple normal vectors and the reference normal vector is below a threshold value.

[0062] The correction unit 150 can either acquire information related to calibration as correction parameters or correct the coordinate transformation parameters used when the coordinate transformation unit 160 transforms the coordinate system of the image from the camera coordinate system to a different coordinate system (for example, the bird's - eye coordinate system). Thereby, more accurate coordinate transformation can be performed during the coordinate transformation of the image.

[0063] [Coordinate Transformation Unit]

[0064] Next, the processing of the coordinate transformation unit 160 will be described. For example, the coordinate transformation unit 160 transforms the camera - coordinate - system coordinates of the image acquired by the acquisition unit 110 into the bird's - eye coordinate system. In this case, the coordinate transformation unit 160 adds the parameters (pitch - direction and roll - direction correction parameters) required for correction by the correction unit 150 to the pre - determined reference coordinate transformation parameters to perform the coordinate transformation.

[0065] Figure 5 is a diagram for explaining the coordinate transformation process in the embodiment. For example, the three - dimensional axes of the camera coordinate system captured by the imaging device 10 (the front camera 12 in the Figure 5 example) are set as [Xc, Yc, Zc], and the imaginary camera coordinate system parallel to the ground in the traveling direction of the vehicle M is set as [Xvc, Yvc, Zvc]. Here, Xvc represents the traveling direction of the vehicle M, Yvc represents the lateral direction of the vehicle M, and Zvc represents the up - and - down direction of the vehicle M. When the roll angle with respect to the vehicle M is θ, the pitch angle with respect to the vehicle M is ρ, and the yaw angle with respect to the vehicle M is φ, the respective angles [θ, ρ, φ] represent the rotation angles around the respective axes of the imaginary camera coordinate system [Xvc, Yvc, Zvc]. The roll angle θ and pitch angle ρ at this time are the values calibrated by the correction unit 150. The correction unit 150, for example, uses the following formula (1) to perform coordinate transformation (rotation) from the camera coordinate system to the imaginary camera coordinate system parallel to the ground in the traveling direction.

[0066]

[0067] The rotational degrees of freedom Rx, Ry, Rz with respect to each axis in formula (1) are derived using the following formulas (2) - (4).

[0068]

[0069] The recognition unit 20 uses the image processed by the image processing device 100 (the image after the calibration process) to recognize the surrounding conditions such as the position of an object existing around the vehicle M. For example, the recognition unit 20 recognizes the position of the object when the image in the camera coordinate system after the calibration process is transformed into the bird's-eye view coordinate system.

[0070] For example, as Figure 5 shown, when the ground (road surface) is always flat (the variation angle is within the allowable range), the recognition unit 20 recognizes the azimuth angle and distance of the object in the bird's-eye view coordinate system with respect to the object on the camera coordinate system observed from the front camera 12. In this case, the recognition unit 20, for example, uses the following formula (5) to calculate the depression angle α (the downward viewing angle of the object when using polar coordinate representation with the orientation of the imaginary camera as the reference), and calculates the azimuth angle β by "β = tan －1 (Yvc / Xvc)".

[0071]

[0072] When the installation position (height hc) of the front camera 12 is preset, the recognition unit 20 uses the depression angle α and the height hc, and calculates the distance D from the vehicle M (more specifically, the position of the front camera 12 mounted on the vehicle M) to the object by "D = hc / tan α". By performing these processes on the objects around the vehicle M, it is possible to more appropriately recognize the position of the objects around the vehicle M and the distance to the objects in the bird's-eye coordinate system.

[0073] The notification control unit 40 may also generate an image of the surrounding conditions of the vehicle M after being transformed into the bird's-eye view coordinate system, and notify the occupant of the generated image via the terminal device T. Thereby, it is possible to display the accurate surrounding conditions to the occupant in a display manner that is easy for the occupant to handle.

[0074] As described above, in the embodiment, image correction (for example, calibration of coordinate transformation parameters from the camera coordinate system to the bird's-eye coordinate system) is performed based on the host vehicle road region included in both the front image IM10 and the rear image IM20, so that higher-precision correction can be achieved. According to the embodiment, for example, by performing calibration based on the image information of the front camera 12 and the rear camera 14 actually mounted on the vehicle M, misrecognition of the images in the pitch direction and the roll direction caused by mounting deviation and variation between products can be suppressed. Thus, more appropriate object recognition and driving support can be performed. In particular, when calibrating the camera image based on the road region, since the lateral region of the host vehicle lane region is narrow, there is a possibility that the calibration in the roll direction cannot be appropriately performed. Therefore, in the embodiment, by using the cross-road region that intersects the host vehicle road, more appropriate calibration in the roll direction can also be performed.

[0075] [Processing Flow]

[0076] Figure 6 It is a flowchart showing an example of the processing flow executed by the image processing apparatus 100 of the embodiment. Figure 6 The processing can be executed, for example, at a specified timing such as the start of driving after the imaging device 10 is mounted on the vehicle M or at a specified cycle. In Figure 6 this example, the acquisition unit 110 acquires camera images (front image and rear image) from the imaging device 10 (front camera 12 and rear camera 14) (step S100). Next, the extraction unit 120 extracts feature points of the acquired camera images (step S110). Next, the first detection unit 130 detects the host vehicle road region from the camera images (step S120). Next, the second detection unit 132 detects the cross-road region from the camera images (step S130).

[0077] Next, the first feature point extraction unit 140 extracts the feature points within the host vehicle road region among the feature points extracted by the processing of step S110 (step S140). Next, the second feature point extraction unit 142 extracts the feature points within the host vehicle road region among the feature points extracted by the processing of step S110 (step S150).

[0078] Next, the correction unit 150 determines whether feature points of both the own-lane region and the cross-road region are continuously extracted for a specified number of frames or more (step S160). If it is determined that the feature points of both the own-lane region and the cross-road region are not continuously extracted for a specified number of frames or more, the process returns to step S100. If it is determined that the feature points of both the own-lane region and the cross-road region are continuously extracted for a specified number of frames or more, the correction unit 150 performs correction of the image in the pitch direction and the roll direction based on the extracted feature points (step S170). Next, the coordinate transformation unit 160 performs coordinate transformation of the image based on the correction result (step S180). Thus, the processing of this flowchart is executed.

[0079] In Figure 6 the example of, the processing of steps S110 to S130 can be executed in an order different from the order shown in Figure 6 or can be executed in parallel using a multi-core processor or the like. The same applies to the processing of steps S140 and S150.

[0080] In Figure 6 the example of, in the processing of step S160, correction is performed at the stage where feature points are continuously extracted from both the own-lane region and the cross-road region for a specified number of frames or more, but the correction unit 150 can also perform only pitch direction correction of the vehicle M when feature points are continuously extracted from the own-lane region for a specified number of frames or more. The correction unit 150 can also perform only roll direction correction when feature points are continuously extracted from the cross-road region for a specified number of frames or more.

[0081] <Variation>

[0082] In the embodiment, instead of extracting feature points of the entire image as described above and extracting the feature points of the own-lane region and the cross-road region among the extracted feature points, the own-vehicle lane region and the cross-road region included in the image can be extracted first, and then feature points can be extracted from each of the extracted road regions.

[0083] In the embodiment, the correction unit 150 can also perform image correction when the own-lane region and the cross-road region of a specified area or more are extracted from the front image or the rear image. Thus, the normal vector with respect to the road surface can be obtained more accurately from a relatively wide road region, and thus more appropriate correction can be performed.

[0084] In an embodiment, the image processing device 100 may also perform correction processes such as aberration correction and distortion correction on the front camera 12 and the rear camera 14. In an embodiment, an image including the lateral direction of the vehicle M (a side image, a side view image) may be used instead of (or in addition to) the rear image.

[0085] According to the embodiment described above, the image processing device 100 includes: an acquisition unit 110 that acquires an image of the periphery of the vehicle M from a photographing device mounted on the vehicle M (an example of a moving body); an extraction unit 120 that extracts feature points from the image acquired by the acquisition unit 110; a first detection unit 130 that detects a moving road area on which the vehicle M moves from the image; a second detection unit 132 that detects an intersecting road area intersecting the moving road area from the image; a first feature point extraction unit 140 that extracts, as first feature points, the feature points of the moving road area detected by the first detection unit 130 among the feature points extracted by the extraction unit 120; a second feature point extraction unit 142 that extracts, as second feature points, the feature points of the intersecting road area detected by the second detection unit 132 among the feature points; and a correction unit 150 that corrects the image based on the first feature points and the second feature points, so that the image captured by the photographing device 10 mounted on the vehicle M can be corrected more appropriately.

[0086] Specifically, according to the embodiment, for example, the front and rear images of the vehicle M are used and the image is corrected based on the feature points of the own vehicle road area and the intersecting road area, so that higher-precision correction can be performed using more information. By including the rear image, it is particularly easy to obtain the intersecting road area. According to the embodiment, regarding the roll direction of the vehicle M, by using the information (the normal vector to the road surface of the own vehicle road) obtained from the own vehicle road area extending in the front-rear direction when viewed from the vehicle M, higher-precision correction in the roll direction can be performed. Regarding the pitch direction of the vehicle M, by using the information (the normal vector to the road surface of the intersecting road) obtained from the intersecting road area extending in the left-right direction when viewed from the vehicle M, higher-precision correction in the pitch direction can be performed.

[0087] According to the embodiment, even when there are deviations in the installation of the photographing device on the vehicle M and variations in the products of the photographing device, by performing calibration using the front image and the rear image captured by the photographing device mounted on the vehicle M, more appropriate conversion can be performed when performing coordinate transformation of the image, and the relative position and relative distance from the objects around the vehicle M can be identified more accurately. Therefore, more appropriate driving support can be executed by identifying these.

[0088] The embodiments described above can be represented as follows.

[0089] An image processing apparatus includes:

[0090] A storage medium storing computer-readable instructions; and

[0091] A processor connected to the storage medium,

[0092] The processor executes the following processing by executing the computer-readable instructions: (the processor executing the computer-readable instructions to:)

[0093] Obtain an image of the periphery of the mobile body from an imaging device mounted on the mobile body;

[0094] Extract feature points from the obtained image;

[0095] Detect a moving road area on which the mobile body moves from the image;

[0096] Detect an intersecting road area intersecting the moving road area from the image;

[0097] Extract the feature points of the moving road area among the extracted feature points as first feature points, and extract the feature points of the intersecting road area among the feature points as second feature points; and

[0098] Correct the image based on the first feature points and the second feature points.

[0099] As described above, specific embodiments of the present invention have been described using embodiments, but the present invention is in no way limited to such embodiments, and various modifications and substitutions can be made without departing from the gist of the present invention.

Claims

1. An image processing device, wherein: The image processing device comprises: an acquisition unit that acquires an image of the periphery of the moving object from a camera mounted on the moving object; an extraction unit that extracts feature points from the image acquired by the acquisition unit; a first detection unit configured to detect, from the image, a moving road area on which the moving object moves; a second detection unit configured to detect an intersecting road area intersecting with the moving road area from the image; a first feature point extraction unit that extracts, from among the feature points extracted by the extraction unit, feature points of the moving road area detected by the first detection unit as first feature points; a second feature point extraction unit that extracts, from among the feature points, feature points of an intersecting road area detected by the second detection unit as second feature points; as well as A correction unit corrects the image based on the first feature point and the second feature point.

2. The image processing device according to claim 1, wherein: The acquisition unit acquires an image obtained by capturing at least the front of the moving object using the imaging device and an image in a direction different from the front.

3. The image processing device according to claim 1, wherein: The correction unit corrects the image of the moving object in the pitch direction based on the first feature points, and corrects the image of the moving object in the roll direction based on the second feature points.

4. The image processing device according to claim 1, wherein: The correction unit corrects the coordinate system of the image when first feature points of the moving road region and second feature points of the intersecting road region are continuously extracted for a predetermined number of frames or more from the image frames acquired by the acquisition unit at a predetermined cycle.

5. The image processing device according to claim 1, wherein: The correction unit corrects coordinate conversion parameters for converting from a coordinate system of an image based on a front of the moving object included in the image to a bird's-eye view coordinate system in which the moving object is viewed from above.

6. The image processing device according to claim 1, wherein: In the imaging device, the imaging unit that captures the front of the moving object and the imaging unit that captures the direction different from the front are integrally configured.

7. An image processing method, wherein: The image processing method enables the computer to perform the following processing: acquiring an image of the periphery of the moving object from a camera mounted on the moving object; Extracting feature points from the acquired image; detecting a moving road area on which the moving object moves from the image; detecting an intersecting road area intersecting with the moving road area from the image; Extracting a feature point of the moving road area from among the extracted feature points as a first feature point; extracting the feature points of the intersection road area among the feature points as second feature points; and The image is corrected based on the first feature points and the second feature points.

8. A storage medium storing a program, wherein: The program causes the computer to execute the following processing: acquiring an image of the periphery of the moving object from a camera mounted on the moving object; Extracting feature points from the acquired image; detecting a moving road area on which the moving object moves from the image; detecting an intersecting road area intersecting with the moving road area from the image; Extracting a feature point of the moving road area from among the extracted feature points as a first feature point; extracting the feature points of the intersection road area among the feature points as second feature points; as well as The image is corrected based on the first feature points and the second feature points.

Images