Image processing device, image processing method, and program

Abstract

To provide an image processing device, an image processing method, and a program configured to perform more appropriate calibration on images captured by an imaging device mounted on a mobile body.SOLUTION: In a driving assistance apparatus, an image processing device includes: an acquisition unit configured to acquire an image of the surroundings of a mobile body from an imaging apparatus mounted on the mobile body; an extraction unit configured to extract feature points from the image acquired by the acquisition unit; a first detection unit configured to detect a moving road area in which the mobile body moves, from the image; a second detection unit configured to detect a cross road area that crosses the moving road area, from the image; a first feature point extraction unit configured to extract a feature point of a moving road area detected by the first detection unit, as a first feature point, out of the feature points extracted by the extraction unit; a second feature point extraction unit configured to extract a feature point of a cross road area detected by the second detection unit, as a second feature point, out of the feature points; and a calibration unit configured to calibrate the image based on the first and the second feature points.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an image processing method, and a program. [Background technology]

[0002] In recent years, efforts to provide access to sustainable transportation systems that take into consideration vulnerable traffic participants have been gaining momentum. To achieve this, efforts are being focused on research and development to further improve traffic safety and convenience through research and development of driving assistance technologies. In this context, a technology is known that acquires an image from a camera mounted on a moving object, extracts feature points from an extraction area of ​​the acquired image that is set or changed based on the external environment of the moving object in the camera's shooting direction, and estimates the camera's posture from the extracted feature points (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-33605 Summary of the Invention [Problem to be solved by the invention]

[0004] However, driving assistance technology has had a problem in that it may not be possible to perform appropriate calibration on captured images due to factors such as misalignment of the imaging device attached to the moving body or product variations in the imaging device.

[0005] In order to solve the above-mentioned problems, one object of the present application is to provide an image processing device, an image processing method, and a program that can perform more appropriate calibration on images captured by an imaging device mounted on a moving object, thereby contributing to the development of sustainable transportation systems. [Means for solving the problem]

[0006] The image processing device, image processing method, and program according to the present invention employ the following configuration. (1): An image processing device according to one embodiment of the present invention is an image processing device comprising: an acquisition unit that acquires an image of the area around the moving body from an imaging device mounted on the moving body; an extraction unit that extracts feature points from the image acquired by the acquisition unit; a first detection unit that detects a moving road area along which the moving body moves from the image; a second detection unit that detects a cross road area that intersects with the moving road area from the image; a first feature point extraction unit that extracts, from the feature points extracted by the extraction unit, feature points of the moving road area detected by the first detection unit as first feature points; a second feature point extraction unit that extracts, from the feature points extracted by the extraction unit, feature points of the cross road area detected by the second detection unit as second feature points; and a calibration unit that calibrates the image based on the first feature points and the second feature points.

[0007] (2) In the above aspect (1), the acquisition unit acquires, by the imaging device, at least an image of a view ahead of the moving body and an image of a direction different from the view ahead.

[0008] (3): In the above aspect (1), the calibration unit calibrates the image of the moving body in the pitch direction based on the first feature point, and calibrates the image of the moving body in the roll direction based on the second feature point.

[0009] (4): In the above aspect (1), the calibration unit calibrates the coordinate system of the image when a first feature point of the intersecting road area and a second feature point of the travel road area are extracted for a predetermined number of consecutive frames or more from image frames acquired by the acquisition unit at a predetermined period.

[0010] (5): In the aspect (1) above, the calibration unit calibrates coordinate transformation parameters that convert from an image coordinate system based on the front of the moving body contained in the image to a bird's-eye view coordinate system that views the moving body from above.

[0011] (6): In the above aspect (1), the imaging device is configured to integrally include an imaging unit that images the area in front of the moving body and an imaging unit that images a direction different from the area in front.

[0012] (7): Another aspect of the image processing method of the present invention is an image processing method in which a computer acquires an image of the surroundings of a moving body from an imaging device mounted on the moving body, extracts feature points from the acquired image, detects a moving road area along which the moving body moves from the image, detects a cross road area that intersects with the moving road area from the image, extracts, from the extracted feature points, feature points of the moving road area as first feature points, extracts, from the extracted feature points, feature points of the cross road area as second feature points, and calibrates the image based on the first feature points and the second feature points.

[0013] (8): Another aspect of the present invention provides a program that causes a computer to acquire an image of the area around a moving body from an imaging device mounted on the moving body, extract feature points from the acquired image, detect a moving road area along which the moving body moves from the image, detect a cross road area that intersects with the moving road area from the image, extract feature points of the moving road area from the extracted feature points as first feature points, extract feature points of the cross road area from the extracted feature points as second feature points, and calibrate the image based on the first feature points and the second feature points. [Effects of the Invention]

[0014] According to the above aspects (1) to (8), it is possible to perform more appropriate calibration on an image captured by an imaging device mounted on a moving body. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a diagram illustrating an example of a functional configuration of a driving assistance device 1 including an image processing device according to an embodiment. [Figure 2] 1 is a diagram illustrating an example of an imaging device 10. FIG. [Figure 3]1 is a diagram for explaining the shooting directions of the front camera 12 and the rear camera 14. FIG. [Figure 4] 1 is a diagram showing an example of an image captured by the imaging device 10. FIG. [Figure 5] 10A and 10B are diagrams for explaining coordinate conversion processing in the embodiment. [Figure 6] 1 is a flowchart showing an example of the flow of processing executed by the image processing apparatus 100 of the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0016] Hereinafter, with reference to the drawings, embodiments of an image processing device, an image processing method, and a program of the present invention will be described. In the following example, an image processing device mounted on a mobile body will be described. The mobile body may include any mobile body on which a person (a passenger such as a driver) rides, such as a three-wheeled or four-wheeled vehicle, a two-wheeled vehicle, or a micromobility vehicle. The mobile body may also be equipped with a driving assistance device that assists the passenger (driver) of the mobile body in driving based on images processed by the image processing device. In the following description, the mobile body is assumed to be a four-wheeled vehicle (hereinafter referred to as "vehicle M") equipped with a driving assistance device. Vehicle M may be any of an automobile powered by an internal combustion engine such as a diesel engine or a gasoline engine, an electric automobile powered by an electric motor, or a hybrid automobile equipped with both an internal combustion engine and an electric motor. In the following description, the forward direction of the vehicle M is the plus X direction, the rearward direction of the vehicle M is the minus X direction, the width direction of the vehicle M is the right direction based on the plus X direction, the left direction is the minus Y direction, and the height direction of the vehicle M, which is the direction perpendicular to the X direction and Y direction, is the plus Z direction.

[0017] [composition] FIG. 1 is a diagram illustrating an example of the functional configuration of a driving assistance device 1 including an image processing device according to an embodiment. The driving assistance device 1 illustrated in FIG. 1 includes, for example, an imaging device 10, a recognition unit 20, a driving assistance unit 30, a notification control unit 40, and an image processing device 100. The recognition unit 20, the driving assistance unit 30, the notification control unit 40, and the image processing device 100 are realized by, for example, a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Some or all of these components may be realized by hardware (including circuitry) such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), a GPU (Graphics Processing Unit), or a SOC (System On Chip), or may be realized by a combination of software and hardware. The program may be stored in advance in a storage device (a storage device having a non-transitory storage medium) such as an HDD (Hard Disk Drive) or a flash memory, or may be stored in a removable storage medium (a non-transitory storage medium) such as a DVD or CD-ROM, and installed by inserting the storage medium into a drive device. Note that the vehicle M of the embodiment is equipped with components for running the vehicle M (for example, driving operators, drive devices such as an engine and a motor, a steering device, a braking device, and vehicle sensors) in addition to the components of FIG.

[0018] The imaging device 10 captures images of the periphery of the vehicle M. For example, the imaging device 10 is a digital camera using a solid-state imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The imaging device 10 may be a stereo camera. The imaging device 10 may also be a camera used in a drive recorder, for example. In the example of FIG. 1 , the imaging device 10 includes a front camera 12 and a rear camera 14 as multiple imaging units. The front camera 12 captures an image of a predetermined area in front of the vehicle M. The rear camera 14 captures an image of a predetermined area in a direction different from the front (for example, rear of the vehicle M). At least one of the front image captured by the front camera 12 and the rear image captured by the rear camera 14 may include an area in the lateral direction of the vehicle M. The imaging device 10 may also be provided with a side camera that captures an image of the lateral direction of the vehicle M in addition to the front camera 12 and the rear camera 14. Furthermore, instead of the above-described cameras, the imaging device 10 may be a fisheye camera capable of capturing images of the surroundings including the front and rear of the vehicle M at a wide angle (for example, 360 degrees). In the imaging device 10, each camera repeatedly captures images at a predetermined cycle and outputs the captured images to the image processing device 100.

[0019] The image processing device 100 acquires an image captured by the imaging device 10 and performs processing to convert the coordinate system of the image (hereinafter referred to as the camera coordinate system) into a coordinate system different from the camera coordinate system. The camera coordinate system includes a front camera coordinate system corresponding to a front image and a rear camera coordinate system corresponding to a rear image. The coordinate system different from the camera coordinate system is, for example, a coordinate system (vehicle coordinate system, bird's-eye view coordinate system) based on the position of the vehicle M when viewed from above.

[0020] The image processing device 100 includes, for example, an acquisition unit 110, an extraction unit 120, a first detection unit 130, a second detection unit 132, a first feature point extraction unit 140, a second feature point extraction unit 142, a calibration unit 150, a coordinate conversion unit 160, and a memory unit 170.

[0021] The storage unit 170 may be realized by a storage device such as an HDD or flash memory, or a solid state drive (SSD), an electrically erasable programmable read only memory (EEPROM), a read only memory (ROM), or a random access memory (RAM). The storage unit 170 stores, for example, images acquired by the acquisition unit 110, processing results by the calibration unit 150 and the coordinate conversion unit 160, programs, and various other information. The storage unit 170 may also store map information. The map information is, for example, information representing road shapes in association with location information (latitude and longitude information) using links indicating roads and nodes connected by the links. The map information may also include information such as the curvature and gradient of the road, the number of lanes, the width, and centerlines of the lanes, or lane boundary information such as road dividing lines that separate the lanes. The map information may also include traffic regulation information, location information of branches, junctions, intersections, T-junctions, etc., facility information of buildings, parking lots, etc., POI (Point Of Interest) information, etc. The map information may be updated as needed by the vehicle M or the driving assistance device 1 communicating with other devices.

[0022] The acquisition unit 110 acquires image frames of forward and rearward images captured by the imaging device 10 at a predetermined interval. The extraction unit 120 extracts feature points included in the forward and rearward images acquired by the acquisition unit 110. For example, the extraction unit 120 performs known image analysis processing, such as edge extraction processing, on the forward and rearward images, and extracts feature points of real-space objects included in the images (e.g., traffic signals, road signs, traffic participants such as pedestrians and other vehicles, buildings, as well as road dividing lines and stop lines) based on the image analysis processing results. In this case, the extraction unit 120 extracts, for example, a sequence of points on the edges of the objects included in the images as feature points (a feature point group). The method for extracting feature points on an image is not limited to the above example, and other known methods may be used. Furthermore, the extraction unit 120 may extract feature points using, for example, a trained model that has been trained to output, when a forward or rearward image is input, the edges of objects (e.g., buildings, road structures, etc.) depicted in the images as a point group. The trained model may be stored in advance in the storage unit 170, or may be acquired from an external device via a communication device (not shown) mounted on the vehicle M. In addition, the extraction unit 120 may extract feature points using, for example, a Visual SLAM (Simultaneous Localization and Mapping) technique, which is a technology for determining the vehicle's own position in three dimensions from image data captured by the imaging device 10.

[0023] The first detection unit 130 detects the vehicle road area (an example of a travel road area) on which the vehicle M travels (moves) from the forward image and the rearward image. The second detection unit 132 detects the cross road area that intersects with the vehicle road from the forward image and the rearward image. The cross road is, for example, a road that connects to the vehicle road within a predetermined angle range including a right angle at an intersection or a T-junction. The processing of the first detection unit 130 and the second detection unit 132 will be described in detail later.

[0024] The first feature point extraction unit 140 extracts, as first feature points, feature points of the vehicle road area detected by the first detection unit 130 from among the feature points (feature point group) extracted by the extraction unit 120. The second feature point extraction unit 142 extracts, as second feature points, feature points of the intersecting road area detected by the second detection unit 132 from among the feature points (feature point group) extracted by the extraction unit 120. The first feature points and second feature points are feature points necessary for image calibration in this embodiment. Details of the processing by the first feature point extraction unit 140 and the second feature point extraction unit 142 will be described later.

[0025] The calibration unit 150 calibrates the image captured by the imaging device 10 based on the first feature points extracted by the first feature point extraction unit 140 and the second feature points extracted by the second feature point extraction unit 142. For example, the calibration unit 150 calibrates coordinate transformation parameters that convert the coordinate system of the captured image from the camera coordinate system to a coordinate system (bird's-eye view coordinate system) different from the camera coordinate system. This enables more appropriate coordinate transformation. The function of the calibration unit 150 will be described in detail later. The calibration unit 150 may store information related to the calibrated coordinate transformation parameters in the storage unit 170.

[0026] The coordinate conversion unit 160 converts the coordinate system (camera coordinate system) of the image acquired by the acquisition unit 110 into another coordinate system. For example, the coordinate conversion unit 160 converts the camera coordinate system into a bird's-eye view coordinate system used by the recognition unit 20 to recognize the surrounding conditions of the vehicle M. In this case, the coordinate conversion unit 160 converts the image into bird's-eye view coordinates using coordinate conversion parameters calibrated by the calibration unit 150 with respect to reference coordinate conversion parameters previously stored in the storage unit 170 or the like.

[0027] The recognition unit 20 recognizes the surrounding situation of the vehicle M based on an image converted into a bird's-eye coordinate system by the coordinate conversion unit 160 (hereinafter referred to as a "bird's-eye image"). For example, the recognition unit 20 recognizes objects present around the vehicle M (within a predetermined distance from the vehicle M) based on the bird's-eye image. The objects include, for example, other vehicles, pedestrians, and other traffic participants. The recognition unit 20 recognizes the position (relative position as seen from the vehicle M), speed (relative speed as seen from the vehicle M), type, shape, size, etc. of the object. For object recognition, for example, object recognition using a model based on deep learning or deep machine learning, object recognition based on a pattern matching method, or an object recognition method combining these is performed.

[0028] The driving assistance unit 30 provides driving assistance to the occupants of the vehicle M based on the recognition result by the recognition unit 20. For example, the driving assistance unit 30 determines whether the vehicle will deviate from the vehicle's driving lane, which is defined by the road dividing lines recognized by the recognition unit 20, and if there is a possibility of deviation, notifies the driver of the vehicle M via the notification control unit 40, or controls the steering of the vehicle M using a steering device (not shown) to prevent the vehicle M from deviating from the driving lane (so that the vehicle M moves toward the center of the driving lane). Furthermore, the driving assistance unit 30 recognizes obstacles, such as other vehicles, present around the vehicle M (within a predetermined distance), and if it determines that there is a possibility of contact with the obstacle based on the relative position and relative speed of the obstacle, notifies the occupants via the notification control unit 40, or performs driving control (at least one of speed control and steering control) to avoid contact.

[0029] The notification control unit 40 notifies the occupant (driver) of the vehicle M about the driving assistance based on the control by the driving assistance unit 30. In this case, the notification control unit 40 generates notification information such as a sound (alarm) or an image associated with the notification content to be notified to the occupant, and transmits the generated notification information to the terminal device T to output it.

[0030] Here, the terminal device T is, for example, a portable terminal device such as a smartphone or tablet terminal used by a driver who drives a vehicle M equipped with the driving assistance device 1. The terminal device T, for example, runs an application for receiving driving assistance from the driving assistance device 1. The application displays an image based on information or notification transmitted by the driving assistance device 1 on a display unit of the terminal device T, or emits sound from a speaker of the terminal device T. The terminal device T is an example of a "notification unit." The terminal device T is, for example, detachably attached to the vehicle M and used. For example, a holder for the terminal device T having a detachable part is provided on one or both of the terminal device T and the vehicle M, and the terminal device T is supported by the holder. Note that in the embodiment, if the vehicle M is equipped with devices such as a navigation device, a display device, or a speaker, notification information may be output from the installed device instead of the terminal device T based on an instruction from the driving assistance unit 30.

[0031] [Processing up to feature point extraction] Next, a specific example of the process of extracting feature points from images (front image, rear image) acquired by the acquisition unit 110 will be described. FIG. 2 is a diagram showing an example of the image capture device 10. The image capture device 10 of the embodiment is attached, for example, near the rearview mirror RM on the top of the front windshield as shown in FIG. 2. For example, the image capture device 10 includes an attachment part AT that can be detachably attached to the vehicle M. The attachment part AT is any member, for example, a suction cup, a sticker, a bracket, or other support member. In the example of FIG. 2, the image capture device 10 is attached to the bottom of the rearview mirror RM, but the position is not limited thereto and may be attached to the right or left side of the rearview mirror RM, for example.

[0032] FIG. 3 is a diagram for explaining the imaging directions of the front camera 12 and the rear camera 14. As shown in FIGS. 2 and 3 described above, the imaging device 10 has the front camera 12 and the rear camera 14 integrally configured within a predetermined distance. The term "integrally configured" may include, for example, a configuration in which the front camera 12 and the rear camera 14 are housed in a single housing, or a configuration in which the front camera 12 and the rear camera 14 are connected (coupled) to each other. In this configuration, for example, when the front camera 12 captures an image of a predetermined angle of view area VA1 centered in a front direction A1 of the vehicle M (the positive X-axis direction in the figure), the rear camera 14 captures an image of a predetermined angle of view area VA2 centered in a direction A2 opposite to the front direction A1 of the vehicle M. Therefore, if front camera 12 captures an image of a field of view centered in a direction tilted downward by angle θ1 with respect to front direction A1 of vehicle M as the reference due to installation misalignment caused by any factor during or after installation, rear camera 14 captures an image of a field of view centered in a direction tilted upward by angle θ1 with respect to direction A2 opposite to front direction A1 as the reference. In the embodiment, if a misalignment occurs in the imaging direction (field of view) of one of front camera 12 and rear camera 14, which are integrally configured, a similar misalignment occurs in the imaging direction (field of view) of the other camera in a symmetrical direction with respect to the installation position of imaging device 10 as the center.

[0033] Fig. 4 is a diagram showing an example of an image captured by the imaging device 10. The example of Fig. 4 shows a forward image IM10 captured by the forward camera 12 and a rearward image IM20 of the vehicle M captured by the rearward camera 14. The forward image IM10 captures an area outside the vehicle (in front of the vehicle M) through the front windshield of the vehicle M. The rearward image IM20 captures an area behind the vehicle M, including the interior of the vehicle, and an area outside the vehicle (to the side or rear of the vehicle M) through the side windshield and rear windshield.

[0034] The extraction unit 120 extracts a plurality of feature points (a group of feature points) from the front image IM10 and the rear image IM20 shown in Fig. 4. For example, the extraction unit 120 extracts feature points for each image frame of the front image IM10 and the rear image IM20 acquired at a predetermined interval.

[0035] The first detection unit 130 detects the vehicle road area in the image from the forward image IM10 and the rearward image IM20. For example, the first detection unit 130 acquires road dividing lines on the left and right sides of the vehicle M based on the point sequence portion included in the feature point cloud extracted by the extraction unit 120, and detects the area divided by the acquired road dividing lines as the vehicle road area. For example, the first detection unit 130 may divide the forward image IM10 and the rearward image IM20 into multiple divided areas and detect the vehicle road area for each divided area. As shown in the forward image IM10 and the rearward image IM20 in FIG. 4, the vehicle road area is located near the center of each image, which is relatively easy to predict. Therefore, the first detection unit 130 may detect the vehicle road area based on the feature point cloud by targeting a partial area of ​​the forward image IM10 and the rearward image IM20 that is predicted in advance to be likely to contain the vehicle road area (for example, a predetermined area including the center of the image). This reduces the processing load related to the detection of the vehicle road area.

[0036] Furthermore, the first detection unit 130 may, for example, implement a function based on AI (Artificial Intelligence) and a function based on a predetermined model in parallel. For example, the function of "detecting the host vehicle road area" may be implemented by detecting the host vehicle road area using deep learning or the like and detecting the host vehicle road area using a predetermined determination process (for example, a determination process based on pattern matching) in parallel for the front image IM10 and the rear image IM20, and scoring both to comprehensively evaluate them.

[0037] In the example of FIG. 4, the first detection unit 130 detects a host road area AR10F ahead of the vehicle M from the forward image IM10, and detects a host road area AR10R behind the vehicle M from the rearward image IM20.

[0038] The second detection unit 132 detects a cross road area in the image from the forward image IM10 and the rearward image IM20. For example, the second detection unit 132 detects a point sequence that is tangent to the host road area detected by the first detection unit 130 at a predetermined angle, based on a point sequence portion included in the feature point group extracted by the extraction unit 120. The predetermined angle is, for example, a predetermined angle range (e.g., approximately 75 to 105 degrees) that includes 90 degrees (a right angle) with respect to the extension direction of the host road area. Then, if two point sequences exist in parallel within a predetermined distance (including an allowable error range), the second detection unit 132 determines that the two point sequences are road dividing lines, and detects the area divided by the road dividing lines as a cross road area.

[0039] For example, the second detection unit 132 may divide the front image IM10 and the rear image IM20 into a plurality of divided areas and detect the cross road area for each divided area. Alternatively, the second detection unit 132 may detect the cross road area based on the feature point group for a partial area of ​​the front image IM10 and the rear image IM20 that is predicted in advance to have a high possibility of containing the cross road area. This reduces the processing load related to the detection of the cross road area.

[0040] Furthermore, the second detection unit 132 may, for example, implement a function based on AI and a function based on a predetermined model in parallel. For example, the function of "detecting an intersecting road area" may be implemented by executing, in parallel, detection of an intersecting road area by deep learning or the like and detection of both intersecting road areas by a predetermined determination process (for example, a determination process based on pattern matching) on ​​the front image IM10 and the rear image IM20, and scoring and comprehensively evaluating both.

[0041] When a specific road structure such as a traffic signal or a crosswalk is detected in the forward image IM10 or the rearward image IM20, the second detection unit 132 may perform a crossroad area detection process within a predetermined distance from the position where the road structure is detected. Furthermore, the second detection unit 132 may refer to map information stored in the storage unit 170 based on the position information of the vehicle M, and perform a crossroad area detection process when the position of the vehicle M is close to a position where a crossroad is likely to exist, such as an intersection or a T-junction (within a predetermined distance). The position information of the vehicle M is acquired, for example, by a position sensor (not shown) mounted on the vehicle M. The position sensor acquires position information (longitude and latitude information) from, for example, a GPS (Global Positioning System) device. Furthermore, the position sensor may acquire position information using a GNSS (Global Navigation Satellite System) receiver of a navigation device (not shown) mounted on the vehicle M. This allows the detection process to be performed in an area where a crossroad area is likely to exist, thereby making it possible to more efficiently detect the crossroad area.

[0042] 4, the second detection unit 132 detects intersecting road areas AR20L and AR20R from the forward image IM10. The second detection unit 132 may also distinguish between an intersecting road area AR20R that connects to the host vehicle road area AR10F on the right side and an intersecting road area AR20L that connects to the host vehicle road area AR10F on the left side. If an intersecting road area is included in the rearward image IM20, the second detection unit 132 also detects that area.

[0043] The first feature point extraction unit 140 extracts the feature points of the own vehicle road areas AR10F, AR10R detected by the first detection unit 130 from the feature points (group of feature points) extracted by the extraction unit 120. In addition, the second feature point extraction unit 142 extracts the feature points of the cross road areas AR20L, AR20R detected by the second detection unit 132 from the feature points extracted by the extraction unit 120.

[0044] [Calibration section] Next, the image calibration process performed by the calibration unit 150 will be described in detail. For example, the calibration unit 150 performs image calibration when feature points of the vehicle road area and the intersecting road area are extracted consecutively for a predetermined number of frames or more from image frames taken at different times for at least one of the forward image IM10 and the rearward image IM20. The predetermined number of frames may be a fixed number, or may be variably set depending on the road shape of the lane in which the vehicle M is traveling, the speed of the vehicle M, the traveling direction, and other driving conditions.

[0045] For example, the calibration unit 150 detects the movement of feature points between frames based on changes in the positions of feature points over time in the vehicle road area (e.g., the vehicle road areas AR10F and AR10R shown in FIG. 4) included in two image frames of the forward image IM10 and the rearward image IM20 captured at different times, and performs optical flow processing to represent the detected movement as a vector (motion vector). The motion vector includes information on, for example, the direction and amount of movement (amount of displacement). The time interval (period) between the two different image frames for acquiring the motion vector may be the period of the image frames acquired by the acquisition unit 110 (or an integer multiple of the period), or may be variably set based on the speed of the vehicle M, the size of the vehicle road area, etc.

[0046] Furthermore, instead of using all feature points included in the vehicle road areas AR10F, AR10R, the feature points used in the optical flow processing may be thinned out to a predetermined number or less. In this case, the calibration unit 150 may divide the vehicle road areas AR10F, AR10R into multiple divided areas and adjust the number of feature points in each divided area so that it is equal to or greater than a lower limit and equal to or less than an upper limit. Reducing the number of feature points used in the optical flow processing can reduce the processing load.

[0047] The calibration unit 150 also sets a normal vector perpendicular to the road surface of the host road based on the motion vector obtained by the optical flow processing and the traveling direction of the vehicle M between two different image frames. For example, the calibration unit 150 extracts multiple motion vectors from the host road area AR10F acquired from the forward image, sets the road surface (plane) of the host road based on the directions of the extracted multiple motion vectors (directions corresponding to the traveling direction of the vehicle M), and sets a normal vector to the set road surface. The calibration unit 150 also similarly sets a normal vector to the road surface of the host road for the host road area AR10R acquired from the rearward image.

[0048] The calibration unit 150 then calibrates the images so that the amount of deviation between the normal vectors of the host vehicle road areas AR10F and AR10R in the camera coordinate system and the normal vector of the actual road surface (i.e., the reference normal vector perpendicular to the horizontal plane) is equal to or less than a threshold. In this embodiment, since the front camera 12 and the rear camera 14 are configured integrally, the direction of deviation is opposite between the front image and the rear image. In other words, if the normal vector of the host vehicle road area AR10F in the front image is deviated to the right with respect to the reference normal vector, the normal vector of the host vehicle road area AR10R in the rear image will be deviated to the left with respect to the reference normal vector. Therefore, the calibration unit 150 performs calibration corresponding to each of the front image and the rear image.

[0049] Furthermore, since the vehicle road areas AR10F and AR10R are areas that extend up and down in the image (forward or backward of the vehicle M), the calibration unit 150 mainly calibrates the vehicle M (or the imaging device 10) in the pitch direction using the vehicle road areas AR10F and AR10R. This allows for more appropriate calibration in the pitch direction.

[0050] Similarly, the calibration unit 150 performs optical flow processing to extract motion vectors based on changes in the positions of feature points over time in crossroad areas included in two image frames taken at different times, the forward image IM10 and the backward image IM20 (for example, the crossroad areas AR20L and AR20R shown in FIG. 4). In this case, the calibration unit 150 may thin out the feature points used in the optical flow processing, or may divide the crossroad areas AR20L and AR20R into a plurality of divided areas and adjust the number of feature points in each divided area so that it is equal to or greater than a lower limit and equal to or less than an upper limit.

[0051] Furthermore, as described above, the calibration unit 150 sets a normal vector with respect to the road surface of the intersecting road based on the motion vector and the traveling direction of the vehicle M at the time between two different image frames, and calibrates the image so that the deviation amount between the set normal vector and the reference normal vector is equal to or less than a threshold. Note that, since the intersecting road areas AR20L and AR20R are areas extending to the left and right of the image (the lateral direction of the vehicle M), the calibration unit 150 mainly calibrates the roll direction of the vehicle M (or the imaging device 10 mounted on the vehicle M) using the intersecting road areas AR20L and AR20R. This allows for more appropriate calibration in the roll direction.

[0052] As described above, the calibration unit 150 performs calibration in the pitch direction based on the feature points included in the vehicle road area, and performs calibration in the roll direction based on the feature points included in the intersecting road area. Note that when there are multiple normal vectors, the calibration unit 150 may perform calibration by comparing the average of the normal vectors with a reference normal vector, or may perform calibration so that the error (least square error) between the multiple normal vectors and the reference normal vector is equal to or smaller than a threshold value.

[0053] Furthermore, the calibration unit 150 may acquire information related to the calibration as calibration parameters, and may calibrate coordinate transformation parameters used when converting the coordinate system of an image from the camera coordinate system to a different coordinate system (for example, a bird's-eye view coordinate system) in the coordinate transformation unit 160. This allows for more accurate coordinate transformation when converting the coordinates of an image.

[0054] [Coordinate conversion section] Next, a description will be given of the processing of the coordinate conversion unit 160. For example, the coordinate conversion unit 160 converts the camera coordinate system of the image acquired by the acquisition unit 110 into a bird's-eye view coordinate system. In this case, the coordinate conversion unit 160 performs coordinate conversion by adding parameters required for calibration by the calibration unit 150 (calibration parameters in the pitch direction and roll direction) to predetermined reference coordinate conversion parameters.

[0055] FIG. 5 is a diagram illustrating coordinate conversion processing in an embodiment. For example, the three-dimensional axes of a camera coordinate system captured by the imaging device 10 (the front camera 12 in the example of FIG. 5) are defined as [Xc, Yc, Zc], and a virtual camera coordinate system parallel to the ground (travel surface) facing the traveling direction of the vehicle M is defined as [Xvc, Yvc, Zvc]. Here, Xvc indicates the traveling direction of the vehicle M, Yvc indicates the lateral direction of the vehicle M, and Zvc indicates the vertical direction of the vehicle M. Furthermore, if the roll angle relative to the vehicle M is defined as θ, the pitch angle is defined as ρ, and the yaw angle is defined as φ, then the respective angles [θ, ρ, φ] indicate the rotation angles around the respective axes of the virtual camera coordinate system [Xvc, Yvc, Zvc]. The roll angle θ and the pitch angle ρ at this time are values ​​calibrated by the calibration unit 150. The calibration unit 150 performs coordinate conversion (rotation) from the camera coordinate system to a virtual camera coordinate system parallel to the ground facing the traveling direction, for example, using the following equation (1):

[0056]

number

[0057]

number

[0058] The recognition unit 20 recognizes the surrounding conditions such as the positions of objects present around the vehicle M using the image (calibrated image) processed by the image processing device 100. For example, the recognition unit 20 recognizes the positions of objects when the calibrated image in the camera coordinate system is converted into the bird's-eye view coordinate system.

[0059] For example, as shown in FIG. 5, when the ground (road surface) is always flat (the variation angle is within the allowable range), the recognition unit 20 recognizes the azimuth angle and distance in the bird's-eye view coordinate system for an object in the camera coordinate system as seen from the front camera 12. In this case, the recognition unit 20 calculates, for example, the depression angle α (the angle at which the object is viewed downward when expressed in polar coordinates based on the orientation of the virtual camera) using the following equation (5), and calculates the azimuth angle β as "β = tan -1 It is calculated by "(Yvc / Xvc)".

[0060]

number

[0061] The notification control unit 40 may generate an image of the surrounding situation of the vehicle M after conversion into a bird's-eye view coordinate system, and notify the occupant of the generated image via the terminal device T. This allows the occupant to be provided with an accurate view of the surrounding situation in a display format that is easy for the occupant to understand.

[0062] As described above, in the embodiment, image calibration (e.g., calibration of coordinate transformation parameters from a camera coordinate system to a bird's-eye view coordinate system) is performed based on the host road area included in both the forward image IM10 and the rearward image IM20, thereby achieving more accurate calibration. Furthermore, according to the embodiment, calibration is performed based on image information from the forward camera 12 and the rearward camera 14 actually mounted on the vehicle M, for example, thereby suppressing erroneous recognition of images in the pitch direction and roll direction due to misalignment or product-to-product variations. Therefore, more appropriate object recognition and driving assistance can be performed. In particular, when calibrating camera images based on the road area, the host lane area has a narrow lateral area, so calibration in the roll direction may not be performed appropriately. Therefore, in the embodiment, more appropriate calibration in the roll direction can also be performed by using the cross road area that intersects the host road.

[0063] [Processing flow] FIG. 6 is a flowchart showing an example of the flow of processing executed by the image processing device 100 of the embodiment. The processing of FIG. 6 may be executed at a predetermined timing, such as when the imaging device 10 starts traveling after being attached to the vehicle M, or at a predetermined interval. In the example of FIG. 6, the acquisition unit 110 acquires camera images (front image and rear image) from the imaging device 10 (front camera 12 and rear camera 14) (step S100). Next, the extraction unit 120 extracts feature points from the acquired camera images (step S110). Next, the first detection unit 130 extracts the host road area from the camera images (step S120). Next, the second detection unit 132 extracts the intersecting road area from the camera images (step S130).

[0064] Next, the first feature point extraction unit 140 extracts feature points within the host road area from among the feature points extracted by the processing of step S110 (step S140). Next, the second feature point extraction unit 142 extracts feature points within the host road area from among the feature points extracted by the processing of step S110 (step S150).

[0065] Next, the calibration unit 150 determines whether feature points of both the vehicle road area and the intersecting road area have been extracted for a predetermined number of consecutive frames or more (step S160). If it is determined that feature points of both areas have not been extracted for a predetermined number of consecutive frames or more, the process returns to step S100. If it is determined that feature points of both areas have been extracted for a predetermined number of consecutive frames or more, the calibration unit 150 calibrates the pitch and roll directions of the image based on the extracted feature points (step S170). Next, the coordinate transformation unit 160 transforms the coordinates of the image based on the calibration results (step S180). This completes the process of this flowchart.

[0066] In the example of Fig. 6, the processes of steps S110 to S130 may be executed in an order different from that shown in Fig. 6, and may be executed in parallel by a multiprocessor, etc. The same applies to the processes of steps S140 and S150.

[0067] 6, in the process of step S160, calibration is performed when feature points are extracted from both the own vehicle road area and the intersecting road area for a predetermined number of consecutive frames or more, but the calibration unit 150 may perform calibration only in the pitch direction of the vehicle M when feature points are extracted from the own vehicle road area for a predetermined number of consecutive frames or more. Also, the calibration unit 150 may perform calibration only in the roll direction when feature points are extracted from the intersecting road area for a predetermined number of consecutive frames or more.

[0068] <Modification> In an embodiment, instead of extracting feature points from the entire image as described above and then extracting feature points from the vehicle road area and the intersecting road area from the extracted feature points, the vehicle road area and the intersecting road area contained in the image may be extracted first, and feature points may be extracted from each extracted road area.

[0069] In addition, in an embodiment, the calibration unit 150 may calibrate the image when a predetermined area or more of the vehicle road area and the intersecting road area is extracted from the forward image or the rearward image. This allows more accurate acquisition of the normal vector to the road surface from a relatively wide road area, thereby enabling more appropriate calibration.

[0070] In addition, in the embodiment, the image processing device 100 may perform correction processing such as aberration correction and distortion correction on the front camera 12 and the rear camera 14. In the embodiment, an image including the lateral direction of the vehicle M (a lateral image, a side image) may be used instead of (or in addition to) the rear image.

[0071] According to the embodiment described above, the image processing device 100 is equipped with an acquisition unit 110 that acquires images of the surroundings of the vehicle M from an imaging device mounted on the vehicle M (an example of a moving body), an extraction unit 120 that extracts feature points from the images acquired by the acquisition unit 110, a first detection unit 130 that detects the travel road area along which the vehicle M moves from the image, a second detection unit 132 that detects the cross road area that intersects with the travel road area from the image, a first feature point extraction unit 140 that extracts the feature points of the travel road area detected by the first detection unit 130 from the feature points extracted by the extraction unit 120 as first feature points, a second feature point extraction unit 142 that extracts the feature points of the cross road area detected by the second detection unit 132 from the above feature points as second feature points, and a calibration unit 150 that calibrates the image based on the first feature points and the second feature points, thereby enabling more appropriate calibration to be performed on the image captured by the imaging device 10 mounted on the vehicle M.

[0072] Specifically, according to the embodiment, for example, images in front of and behind the vehicle M are used to calibrate the images based on feature points of the host vehicle road area and the intersecting road area, so that calibration can be performed with higher accuracy using more information. Furthermore, the inclusion of a rear image makes it easier to acquire the intersecting road area in particular. Furthermore, according to the embodiment, with regard to the roll direction of the vehicle M, calibration can be performed with higher accuracy by using information obtained from the host vehicle road area extending in the front-rear direction as viewed from the vehicle M (normal vector to the road surface of the host vehicle road). Furthermore, with regard to the pitch direction of the vehicle M, calibration can be performed with higher accuracy by using information obtained from the intersecting road area extending in the left-right direction as viewed from the vehicle M (normal vector to the road surface of the intersecting road).

[0073] Furthermore, according to the embodiment, even if there is a misalignment in the installation of the imaging device on the vehicle M or product variations in the imaging device, by performing calibration using the forward image and rearward image captured by the imaging device attached to the vehicle M, more appropriate conversion can be performed when performing coordinate conversion of the image, and the relative position and relative distance to objects around the vehicle M can be recognized more accurately. Therefore, this recognition makes it possible to perform more appropriate driving assistance.

[0074] The above-described embodiment can be expressed as follows. a storage medium for storing computer-readable instructions; a processor connected to the storage medium; The processor executes the computer-readable instructions to: Acquiring an image of the surroundings of the moving body from an imaging device mounted on the moving body; Extracting feature points from the acquired image; detecting a travel road area where the moving object is traveling from the image; detecting an intersecting road area that intersects with the travel road area from the image; extracting a feature point of the travel road area from the extracted feature points as a first feature point; extracting, from the feature points, feature points in the intersecting road area as second feature points; calibrating the image based on the first feature points and the second feature points; Image processing device.

[0075] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0076] 1...driving assistance device, 10...imaging device, 12...front camera, 14...rear camera, 20...recognition unit, 30...driving assistance unit, 40...notification control unit, 100...image processing device, 110...acquisition unit, 120...extraction unit, 130...first detection unit, 132...second detection unit, 140...first feature point extraction unit, 142...second feature point extraction unit, 150...calibration unit, 160...coordinate conversion unit, 170...storage unit

Claims

1. an acquisition unit that acquires an image of the surroundings of the moving object from an imaging device mounted on the moving object; an extraction unit that extracts feature points from the image acquired by the acquisition unit; a first detection unit that detects a travel road area along which the moving object moves from the image; a second detection unit that detects an intersecting road area that intersects with the travel road area from the image; a first feature point extraction unit that extracts, from among the feature points extracted by the extraction unit, feature points of the travel road area detected by the first detection unit as first feature points; a second feature point extraction unit that extracts, from the feature points, feature points of the intersecting road area detected by the second detection unit as second feature points; a calibration unit that calibrates the image based on the first feature points and the second feature points; An image processing device comprising:

2. the acquisition unit acquires, by the imaging device, at least an image of a front view of the moving object and an image of a direction different from the front view; The image processing device according to claim 1 .

3. the calibration unit calibrates an image of the moving object in a pitch direction based on the first feature points, and calibrates an image of the moving object in a roll direction based on the second feature points; The image processing device according to claim 1 .

4. the calibration unit calibrates a coordinate system of the image when the first feature point and the second feature point are extracted consecutively for a predetermined number of frames or more from image frames acquired by the acquisition unit at a predetermined cycle; The image processing device according to claim 1 .

5. the calibration unit calibrates coordinate transformation parameters for transforming an image coordinate system based on a point in front of the moving object included in the image into a bird's-eye view coordinate system viewed from above the moving object; The image processing device according to claim 1 .

6. The imaging device is configured such that an imaging unit that images a front view of the moving body and an imaging unit that images a direction different from the front view are integrally formed. The image processing device according to claim 1 .

7. The computer Acquiring an image of the surroundings of the moving body from an imaging device mounted on the moving body; Extracting feature points from the acquired image; detecting a travel road area where the moving object is traveling from the image; detecting an intersecting road area that intersects with the travel road area from the image; extracting a feature point of the travel road area from the extracted feature points as a first feature point; extracting, from the feature points, feature points in the intersecting road area as second feature points; calibrating the image based on the first feature points and the second feature points; Image processing methods.

8. On the computer, Acquiring an image of the surroundings of the moving body from an imaging device mounted on the moving body; extracting feature points from the acquired image; detecting a travel road area where the moving object is traveling from the image; detecting an intersecting road area that intersects with the travel road area from the image; extracting a feature point of the travel road area from the extracted feature points as a first feature point; extracting, from the feature points, feature points in the intersecting road area as second feature points; calibrating the image based on the first feature points and the second feature points; program.

Images