IMAGE PROCESSING APPARATUS, IMAGE PROCESSING METHOD, AND IMAGE PROCESSING PROGRAM

Abstract

In one embodiment, an image processing device according to the present invention comprises a determination unit (30) and an image generation unit (37). The determination unit (30) uses position information, which includes a plurality of detection points at the periphery of a moving body, and host position information of the moving body to determine a boundary region in a region where spatially-adjacent peripheral images, from among a plurality of peripheral images of the moving body, overlap. The image generation unit uses the boundary region to generate a synthesized image by using the spatially adjacent peripheral images.

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an image processing method, and an image processing program. [Background technology]

[0002] A technique has been disclosed for generating a composite image from an arbitrary viewpoint using a projected image obtained by projecting a captured image of the periphery of a moving object onto a virtual projection surface. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2013-207637 [Patent Document 2] Special Publication No. 2014-531078 [Patent Document 3] Japanese Patent Application Laid-Open No. 2009-232310 [Patent Document 4] Japanese Patent Application Publication No. 2019-140518 [Patent Document 5] Japanese Patent Application Laid-Open No. 2019-153138 [Non-patent literature]

[0004] [Non-Patent Document 1] "Environmental Recognition of Mobile Robots - Map Construction and Self-Localization" Systems / Control / Information (Journal of the Society of Systems Information Engineers), Vol. 60, No. 12, pp. 509-514, 2016 Summary of the Invention [Problem to be solved by the invention]

[0005] However, when a composite image made by superimposing multiple images is projected and displayed on a projection surface, problems may occur in the projected composite image, such as double parking space lines on a horizontal surface.

[0006] In one aspect, the present invention aims to provide an image processing device, an image processing method, and an image processing program that eliminate problems that occur when projecting and displaying a composite image formed by superimposing multiple images onto a projection surface. [Means for solving the problem]

[0007] In one aspect, the image processing device disclosed in the present application includes a determination unit that uses position information including a plurality of detection points around a moving body and self-position information of the moving body to determine a boundary area in an overlapping area of ​​spatially adjacent peripheral images of the moving body, and an image generation unit that uses the boundary area to generate a composite image using the spatially adjacent peripheral images. [Effects of the Invention]

[0008] According to one aspect of the image processing device disclosed in the present application, it is possible to eliminate problems that occur when a composite image obtained by superimposing a plurality of images is projected and displayed on a projection surface. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing the overall configuration of an image processing system according to the first embodiment. [Figure 2] FIG. 2 is a diagram showing the hardware configuration of the image processing apparatus according to the first embodiment. [Figure 3] FIG. 3 is a diagram showing the functional configuration of the image processing apparatus according to the first embodiment. [Figure 4] FIG. 4 is a schematic diagram of environment map information according to the first embodiment. [Figure 5] FIG. 5 is an explanatory diagram of an asymptotic curve according to the first embodiment. [Figure 6] FIG. 6 is a schematic diagram showing an example of a reference projection plane according to the first embodiment. [Figure 7] FIG. 7 is a schematic diagram showing an example of a projection shape determined by the shape determination unit according to the first embodiment. [Figure 8]FIG. 8 is a schematic diagram illustrating an example of a functional configuration of a determination unit according to the first embodiment. [Figure 9] FIG. 9 is a diagram for explaining the boundary angle and the boundary width. [Figure 10] FIG. 10 is a conceptual diagram showing the process of projecting a composite image onto a reference projection plane. [Figure 11] FIG. 11 is a diagram schematically illustrating an example of the composite image shown in FIG. [Figure 12] FIG. 12 is a conceptual diagram showing the process of projecting a composite image when the projection shape determination process is executed on the reference projection plane shown in FIG. [Figure 13] FIG. 13 is a diagram schematically illustrating an example of the composite image shown in FIG. [Figure 14] FIG. 14 is a conceptual diagram showing the process of projecting a composite image onto a projection shape when the moving object moves further toward a parking space line from the state shown in FIGS. [Figure 15] FIG. 15 is a diagram schematically illustrating an example of the composite image shown in FIG. [Figure 16] FIG. 16 is a conceptual diagram showing an example in which the projection processing shown in FIG. 14 is corrected by boundary area adaptive control. [Figure 17] FIG. 17 is a diagram schematically illustrating an example of the composite image shown in FIG. [Figure 18] FIG. 18 is a flowchart showing an example of the flow of image processing executed by the image processing apparatus according to the first embodiment. [Figure 19] FIG. 19 is a diagram for explaining a comparative example in which boundary region adaptive control is not executed. [Figure 20] FIG. 20 is a diagram showing an example of a composite image obtained by image processing including boundary area adaptation control. [Figure 21] FIG. 21 is a diagram for explaining a comparative example in which boundary region adaptive control is not executed. [Figure 22] FIG. 22 is a diagram showing an example of a composite image obtained by image processing including boundary area adaptation control. [Figure 23]FIG. 23 is a diagram for explaining a comparative example in which boundary region adaptive control is not executed. [Figure 24] FIG. 24 is a diagram showing an example of a composite image obtained by image processing including boundary area adaptation control. [Figure 25] FIG. 25 is a diagram illustrating an example of a functional configuration of an image processing apparatus according to the second embodiment. [Figure 26] FIG. 26 is a flowchart showing an example of the flow of image processing executed by the image processing apparatus according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, with reference to the accompanying drawings, embodiments of the image processing device, image processing method, and image processing program disclosed herein will be described in detail. Note that the following embodiments do not limit the disclosed technology. Furthermore, each embodiment can be appropriately combined within a range that does not cause contradiction in the processing content.

[0011] (First embodiment) 1 is a diagram showing an example of the overall configuration of an image processing system 1 according to this embodiment. The image processing system 1 includes an image processing device 10, an imaging unit 12, a detection unit 14, and a display unit 16. The image processing device 10, the imaging unit 12, the detection unit 14, and the display unit 16 are connected to each other so as to be able to exchange data or signals.

[0012] In this embodiment, an example will be described in which the image processing device 10, the photographing unit 12, the detection unit 14, and the display unit 16 are mounted on a moving object 2. Note that the image processing device 10 according to the first embodiment is an example that uses Visual SLAM.

[0013] The moving body 2 is an object that can move. The moving body 2 is, for example, a vehicle, a flyable object (a manned airplane, an unmanned airplane (e.g., a UAV (Unmanned Aerial Vehicle), a drone)), a robot, etc. The moving body 2 is, for example, a moving body that moves through human driving operation, or a moving body that can move automatically (autonomously) without human driving operation. In this embodiment, a case where the moving body 2 is a vehicle will be described as an example. Examples of vehicles include a two-wheeled vehicle, a three-wheeled vehicle, and a four-wheeled vehicle. In this embodiment, a case where the vehicle is an autonomously moving four-wheeled vehicle will be described as an example.

[0014] Note that the image processing device 10, the photographing unit 12, the detection unit 14, and the display unit 16 are not limited to being mounted on the moving object 2. The image processing device 10 may be mounted on a stationary object. A stationary object is an object fixed to the ground. A stationary object is an object that cannot be moved or an object that is stationary relative to the ground. Examples of stationary objects include traffic lights, parked vehicles, and road signs. The image processing device 10 may also be mounted on a cloud server that executes processing on the cloud.

[0015] The photographing unit 12 photographs the periphery of the moving object 2 and acquires photographed image data. In the following description, the photographed image data will be simply referred to as a photographed image. The photographing unit 12 is, for example, a digital camera capable of shooting video. Photographing refers to converting an image of a subject formed by an optical system such as a lens into an electrical signal. The photographing unit 12 outputs the photographed image to the image processing device 10. In addition, in this embodiment, the photographing unit 12 will be described assuming that it is a monocular fisheye camera (for example, with a viewing angle of 195 degrees).

[0016] In this embodiment, an example will be described in which four photographing units 12 (photographing units 12A to 12D) are mounted on a moving object 2. The multiple photographing units 12 (photographing units 12A to 12D) photograph subjects in their respective photographing areas E (photographing areas E1 to E4) and acquire photographed images. Note that the photographing directions of these multiple photographing units 12 are different from each other. Furthermore, the photographing directions of these multiple photographing units 12 are adjusted in advance so that at least a portion of the photographing area E between adjacent photographing units 12 overlaps.

[0017] Furthermore, the four image capturing units 12A to 12D are merely an example, and there is no limitation on the number of image capturing units 12. For example, if the moving object 2 has a vertically long shape like a bus or truck, it is possible to arrange one image capturing unit 12 at the front, rear, front of the right side, rear of the right side, front of the left side, and rear of the left side of the moving object 2, for a total of six image capturing units 12. In other words, the number and placement positions of the image capturing units 12 can be set arbitrarily depending on the size and shape of the moving object 2. Note that the boundary angle determination process described below can be realized by providing at least two image capturing units 12.

[0018] The detection unit 14 detects position information of each of a plurality of detection points around the moving object 2. In other words, the detection unit 14 detects position information of each of the detection points in the detection area F. A detection point refers to each of the points in real space that are individually observed by the detection unit 14. The detection points correspond to, for example, three-dimensional objects around the moving object 2. For example, the detection unit 14 irradiates light around the detection unit 14 and receives the reflected light reflected at a reflection point. This reflection point corresponds to the detection point.

[0019] The position information of the detection point is information indicating the position of the detection point in real space (three-dimensional space). For example, the position information of the detection point is information indicating the distance from the detection unit 14 (i.e., the position of the moving object 2) to the detection point and the direction of the detection point relative to the detection unit 14. These distances and directions can be expressed, for example, by position coordinates indicating the relative position of the detection point relative to the detection unit 14, position coordinates indicating the absolute position of the detection point, or a vector.

[0020] The detection unit 14 may be, for example, a 3D (Three-Dimensional) scanner, a 2D (Two-Dimensional) scanner, a distance sensor (millimeter-wave radar, laser sensor), a sonar sensor that detects objects using sound waves, an ultrasonic sensor, or the like. The laser sensor may be, for example, a three-dimensional LiDAR (Laser Imaging Detection and Ranging) sensor. The detection unit 14 may also be a device that uses SfM (Structure from Motion) technology to measure distance from an image captured by a monocular camera. A plurality of image capture units 12 may also be used as the detection unit 14. One of the plurality of image capture units 12 may also be used as the detection unit 14.

[0021] The display unit 16 displays various types of information and is, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display.

[0022] In this embodiment, the image processing device 10 is communicably connected to an electronic control unit (ECU) 3 mounted on the moving object 2. The ECU 3 is a unit that performs electronic control of the moving object 2. In this embodiment, the image processing device 10 is capable of receiving CAN (Controller Area Network) data such as the speed and moving direction of the moving object 2 from the ECU 3.

[0023] Next, the hardware configuration of the image processing device 10 will be described.

[0024] FIG. 2 is a diagram illustrating an example of the hardware configuration of the image processing device 10. As shown in FIG.

[0025] The image processing device 10 is, for example, a computer, and includes a CPU (Central Processing Unit) 10A, a ROM (Read Only Memory) 10B, a RAM (Random Access Memory) 10C, and an I / F (Interface) 10D. The CPU 10A, ROM 10B, RAM 10C, and I / F 10D are interconnected by a bus 10E, and have a hardware configuration similar to that of a typical computer.

[0026] The CPU 10A is a calculation device that controls the image processing device 10. The CPU 10A corresponds to an example of a hardware processor. The ROM 10B stores programs and the like that realize various processes by the CPU 10A. The RAM 10C stores data necessary for various processes by the CPU 10A. The I / F 10D is an interface that is connected to the imaging unit 12, the detection unit 14, the display unit 16, the ECU 3, etc., and is used to send and receive data.

[0027] A program for executing image processing executed by the image processing device 10 of this embodiment is provided by being pre-installed in a ROM 10B or the like. The program executed by the image processing device 10 of this embodiment may be provided by being recorded on a recording medium in a format that can be installed on the image processing device 10 or in a format that can be executed. The recording medium is a computer-readable medium. Examples of the recording medium include a CD (Compact Disc)-ROM, a flexible disk (FD), a CD-R (Recordable), a DVD (Digital Versatile Disk), a USB (Universal Serial Bus) memory, and an SD (Secure Digital) card.

[0028] Next, a functional configuration of the image processing device 10 according to this embodiment will be described. The image processing device 10 uses Visual SLAM (Simultaneous Localization and Mapping) to simultaneously estimate position information of the detection point and self-position information of the moving object 2 from images captured by the image capturing unit 12. The image processing device 10 stitches together a plurality of spatially adjacent captured images to generate and display a composite image overlooking the periphery of the moving object 2. In this embodiment, the image capturing unit 12 is used as the detection unit 14.

[0029] Fig. 3 is a diagram showing an example of the functional configuration of the image processing device 10. In Fig. 3, in order to clarify the data input / output relationship, the image capturing unit 12 and the display unit 16 are also shown in addition to the image processing device 10.

[0030] The image processing device 10 includes an acquisition unit 20, a selection unit 23, a matching unit 25, a self-position estimation unit 27, a detection point registration unit 29, a memory unit 26, a correction unit 28, a determination unit 30, a deformation unit 32, a virtual viewpoint line of sight determination unit 34, a projection transformation unit 36, and an image synthesis unit 38.

[0031] Some or all of the above-mentioned multiple units may be realized by, for example, causing a processing device such as CPU 10A to execute a program, that is, by software. Also, some or all of the above-mentioned multiple units may be realized by hardware such as an IC (Integrated Circuit), or may be realized by a combination of software and hardware.

[0032] The acquiring section 20 acquires the photographed images from the photographing section 12. The acquiring section 20 acquires the photographed images from each of the photographing sections 12 (photographing sections 12A to 12D).

[0033] The acquisition unit 20 outputs the acquired captured image to the projection transformation unit 36 ​​and the selection unit 23 every time the acquisition unit 20 acquires the captured image.

[0034] The selection unit 23 selects a detection area of ​​the detection point. In this embodiment, the selection unit 23 selects at least one of the multiple imaging units 12 (imaging units 12A to 12D), thereby selecting the detection area.

[0035] In this embodiment, the selection unit 23 selects at least one of the photographing units 12 using vehicle state information, detection direction information contained in the CAN data received from the ECU 3, or instruction information input by an operation instruction from the user.

[0036] The vehicle state information is information indicating, for example, the traveling direction of the moving body 2, the state of the direction indication of the moving body 2, the state of the gear of the moving body 2, etc. The vehicle state information can be derived from CAN data. The detected direction information is information indicating the direction in which noteworthy information is detected, and can be derived using POI (Point of Interest) technology. The instruction information is information indicating the direction in which noteworthy information is detected, and is input by an operational instruction from the user.

[0037] For example, the selection unit 23 selects the direction of the detection area using the vehicle state information. Specifically, the selection unit 23 uses the vehicle state information to identify parking information such as rear parking information indicating that the moving object 2 is parked backwards and parallel parking information indicating that the moving object 2 is parked parallel. The selection unit 23 associates the parking information with identification information of any one of the image capturing units 12 and stores them in advance. For example, the selection unit 23 stores in advance identification information of the image capturing unit 12D (see FIG. 1) that captures images behind the moving object 2 in association with the rear parking information. The selection unit 23 also stores in advance identification information of the image capturing units 12B and 12C (see FIG. 1) that capture images to the left and right of the moving object 2 in association with the parallel parking information.

[0038] Then, the selection unit 23 selects the direction of the detection area by selecting the image capturing unit 12 corresponding to the parking information derived from the received vehicle state information.

[0039] The selection unit 23 may also select the photographing unit 12 whose photographing area E is the direction indicated by the detected direction information. The selection unit 23 may also select the photographing unit 12 whose photographing area E is the direction indicated by the detected direction information derived using POI technology.

[0040] The selection unit 23 outputs the photographed image captured by the selected photographing unit 12 from among the photographed images acquired by the acquisition unit 20 to the matching unit 25.

[0041] The matching unit 25 performs a feature extraction process and a matching process between each of a plurality of captured images (a plurality of captured images in different frames) captured at different times. In detail, the matching unit 25 performs a feature extraction process from these plurality of captured images. The matching unit 25 performs a matching process for identifying corresponding points between the plurality of captured images captured at different times, using the feature amounts between the plurality of captured images. The matching unit 25 outputs the matching process result to the self-position estimation unit 27. The matching unit 25 also registers information on the identified corresponding points between the plurality of captured images in the environmental map information 26A.

[0042] The storage unit 26 stores various types of data. The storage unit 26 is, for example, a semiconductor memory element such as a RAM or a flash memory, a hard disk, an optical disk, or the like. The storage unit 26 may be a storage device provided outside the image processing device 10. The storage unit 26 may also be a storage medium. Specifically, the storage medium may store or temporarily store programs and various types of information downloaded via a LAN (Local Area Network), the Internet, or the like.

[0043] The environment map information 26A is map information that indicates the surrounding conditions of the moving object 2. The environment map information 26A is information in which position information of each detection point and self-position information of the moving object 2 are registered in a three-dimensional coordinate space with a predetermined position in real space as the origin. The predetermined position in real space may be determined based on, for example, predetermined conditions.

[0044] For example, the predetermined position is the position of the moving object 2 when the image processing device 10 executes the image processing of this embodiment. For example, assume that image processing is executed at a predetermined timing, such as a parking scene of the moving object 2. In this case, the image processing device 10 may set the position of the moving object 2 when it determines that the predetermined timing has been reached as the predetermined position. For example, the image processing device 10 may determine that the predetermined timing has been reached when it determines that the behavior of the moving object 2 has become behavior indicative of a parking scene. Examples of behavior indicative of a parking scene include when the speed of the moving object 2 drops below a predetermined speed, when the gear of the moving object 2 is shifted into reverse gear, or when a signal indicating the start of parking is received through a user's operation instruction. Note that the predetermined timing is not limited to a parking scene.

[0045] Fig. 4 is a schematic diagram of an example of the environment map information 26A. As shown in Fig. 4, the environment map information 26A is information in which the position information of each of the detection points P and the self-position information of the self-position S of the moving object 2 are registered at corresponding coordinate positions in the three-dimensional coordinate space.

[0046] The self-position estimation unit 27 estimates the self-position information of the moving object 2 by triangulation using information on points identified as corresponding that are included in the matching processing result acquired from the matching unit 25. Here, the self-position information is information indicating the attitude of the moving object 2. The attitude of the moving object 2 indicates the position and inclination of the moving object 2. The self-position information includes, for example, information such as the position of the moving object 2 and the orientation of the imaging unit 12 corresponding to each of the different imaging timings.

[0047] Then, the self-position estimation unit 27 registers the calculated self-position information in the environmental map information 26A.

[0048] The self-position estimation unit 27 may estimate the self-position information using odometry. In this case, the self-position estimation unit 27 may estimate new self-position information by odometry-based integral calculation using the previously calculated self-position information and the movement amount of the moving object 2. The self-position estimation unit 27 can acquire the movement amount by reading the movement amount of the moving object 2 included in the CAN data acquired from the ECU 3.

[0049] The detection point registration unit 29 calculates the amount of movement (translation amount and rotation amount) of the moving object 2 using the self-position information of the moving object 2 corresponding to each of the different image capturing timings estimated by the self-position estimation unit 27. Then, from the amount of movement, the matching unit 25 calculates the relative coordinates (detection points) of the corresponding points between the multiple captured images specified by the matching unit 25 with respect to the self-position of the moving object 2.

[0050] Furthermore, the detection point registration unit 29 registers the coordinates in the environment map information 26A as the coordinates of the detection point P. Note that the coordinates of the detection point P registered in the environment map information 26A may be converted into coordinates with a predetermined position as the origin.

[0051] Therefore, as the moving object 2 moves, the position information of new detection points P and the self-position information of the self-position S are sequentially added and registered in the environmental map information 26A. As an example, the self-positions S of the self-positions S1 to S3 are shown in Fig. 4. The larger the value of the number following S, the closer the self-position S is to the current timing.

[0052] The correction unit 28 corrects the position information of each of the multiple detection points P registered in the environment map information 26A and the self-position information of the moving object 2. The correction unit 28 corrects the position information and self-position information already registered in the environment map information 26A, using the position information of each of the corresponding detection points P acquired at an acquisition timing later than the acquisition timing of the detection point P.

[0053] That is, the correction unit 28 corrects the position information of the detection points P registered in the environment map information 26A using the position information of the detection points P newly registered by the detection point registration unit 29. At this time, the correction unit 28 may correct the position information and the self-position information by further using various parameters used in calculating the position information of each of the detection points P registered in the environment map information 26A. Through this correction process, the correction unit 28 corrects errors in the position information of the detection points P registered in the environment map information 26A. The correction unit 28 may correct the position information of the detection points P registered in the environment map information 26A using the least squares method or the like. Through the correction process by the correction unit 28, the accumulated error in the position information of the detection points P is corrected.

[0054] The timing of the correction process by the correction unit 28 is not limited. For example, the correction unit 28 may execute the correction process at a predetermined timing. The predetermined timing may be determined based on a predetermined condition, for example. In this embodiment, the image processing device 10 is described as being configured to include the correction unit 28 as an example. However, the image processing device 10 may not be configured to include the correction unit 28.

[0055] The determination unit 30 determines the projection shape of the projection surface using the position information of the detection points P stored in the environment map information 26A, and generates projection shape information. The determination unit 30 outputs the generated projection shape information to the deformation unit 32.

[0056] Here, the projection surface is a three-dimensional surface onto which an image of the periphery of the moving object 2 is projected. The image of the periphery of the moving object 2 is a captured image of the periphery of the moving object 2, which is captured by each of the image capturing units 12A to 12D. The projection shape of the projection surface is a three-dimensional (3D) shape that is virtually formed in a virtual space corresponding to the real space. In this embodiment, the determination of the projection shape of the projection surface executed by the determination unit 30 is referred to as a "projection shape determination process."

[0057] The determination unit 30 determines a boundary area using position information including a plurality of detection points P around the moving object 2, which is stored in the environmental map information 26A, and the self-position information of the moving object 2, and generates boundary area information. The determination unit 30 outputs the generated boundary area information to the image synthesis unit 38.

[0058] Here, the boundary area refers to an area where two peripheral images acquired by spatially adjacent imaging units 12 among imaging units 12A to 12D are superimposed when generating a composite image by spatially connecting multiple peripheral images of moving object 2. This boundary area is determined, for example, by a boundary angle corresponding to the center position of the boundary area and a boundary width. In this embodiment, the determination of the boundary area executed by determination unit 30 is referred to as a "boundary area determination process."

[0059] In addition, the determination unit 30 calculates an asymptotic curve using position information including multiple detection points P around the moving body 2 stored in the environmental map information 26A and the self-position information of the moving body 2, and generates asymptotic curve information.

[0060] FIG. 5 is an explanatory diagram of an asymptotic curve Q generated by the determination unit 30. Here, the asymptotic curve is an asymptotic curve of a plurality of detection points P in the environment map information 26A. FIG. 5 shows an example in which the asymptotic curve Q is displayed in a projected image obtained by projecting a captured image onto a projection surface when the moving object 2 is viewed from above. For example, it is assumed that the determination unit 30 has identified three detection points P in descending order of proximity to the self-position S of the moving object 2. In this case, the determination unit 30 generates the asymptotic curve Q of these three detection points P.

[0061] The determination unit 30 outputs the self-position and the asymptotic curve information to the virtual viewpoint line-of-sight determination unit 34.

[0062] The configuration of the determination unit 30 will be explained in detail later.

[0063] The deformation unit 32 deforms the projection surface based on the projection shape information received from the determination unit 30.

[0064] FIG. 6 is a schematic diagram showing an example of a reference projection plane 40. FIG. 7 is a schematic diagram showing an example of a projection shape 41 determined by the determination unit 30. That is, the deformation unit 32 deforms the reference projection plane shown in FIG. 6, which is stored in advance, based on the projection shape information, and determines a deformed projection plane 42 as the projection shape 41 shown in FIG. 7. The determination unit 30 generates deformed projection plane information based on the projection shape 41. The deformation of this reference projection plane is performed using the detection point P closest to the moving object 2 as a reference. The deformation unit 32 outputs the deformed projection plane information to the projection conversion unit 36.

[0065] Furthermore, for example, the deformation unit 32 deforms the reference projection plane into a shape that follows the asymptotic curve of a predetermined number of detection points P in order of proximity to the moving object 2, based on the projection shape information.

[0066] It is preferable that the deformation unit 32 deforms the reference projection plane using the position information of the detection point P acquired before the first timing and the self-position information of the self-position S.

[0067] Here, the first timing refers to the latest timing at which the matching unit 25 detects the position information of the detection point P, or any timing prior to the latest timing. For example, the detection point P acquired before the first timing includes the position information of a specific object in the vicinity of the moving body 2, while the detection point P acquired at the first timing does not include the position information of the specific object in the vicinity. The determination unit 30 may determine the projection shape in the same manner as described above using the position information of the detection point P acquired before the first timing, which is included in the environmental map information 26A. Then, the deformation unit 32 may generate a deformed projection surface in the same manner as described above using the projection shape information of the projection shape.

[0068] For example, there are cases where the position information of the detection point P detected at the first timing by the matching unit 25 does not include the position information of the detection point P detected before that timing. Even in this case, the deformation unit 32 can generate a deformed projection plane according to the detection point P detected before.

[0069] The virtual viewpoint line of sight determining unit 34 determines virtual viewpoint line of sight information based on the self-position and the asymptotic curve information.

[0070] Determination of virtual viewpoint line-of-sight information will be described with reference to FIGS. 5 and 7. The virtual viewpoint line-of-sight determination unit 34 determines, for example, a direction passing through a detection point P closest to the self-position S of the moving object 2 and perpendicular to the deformed projection plane as the line-of-sight direction. Furthermore, the virtual viewpoint line-of-sight determination unit 34 fixes, for example, the direction of the line-of-sight direction L, and determines the coordinates of the virtual viewpoint O as an arbitrary Z coordinate and arbitrary XY coordinates in a direction away from the asymptotic curve Q toward the self-position S. In this case, the XY coordinates may be coordinates of a position farther away from the asymptotic curve Q than the self-position S. The virtual viewpoint line-of-sight determination unit 34 then outputs virtual viewpoint line-of-sight information indicating the virtual viewpoint O and the line-of-sight direction L to the projection transformation unit 36. Note that, as shown in FIG. 7, the line-of-sight direction L may be a direction from the virtual viewpoint O toward the position of the vertex W of the asymptotic curve Q.

[0071] The projection conversion unit 36 ​​generates a projection image by projecting the captured image acquired from the image capturing unit 12 onto the deformed projection surface based on the deformed projection surface information and the virtual viewpoint line of sight information. The projection conversion unit 36 ​​converts the generated projection image into a virtual viewpoint image and outputs it to the image synthesis unit 38. Here, the virtual viewpoint image is an image obtained by viewing the projection image in an arbitrary direction from a virtual viewpoint.

[0072] With reference to FIG. 7, the projection image generation process by the projection transformation unit 36 ​​will be described in detail. The projection transformation unit 36 ​​projects the captured image onto the modified projection surface 42. Then, the projection transformation unit 36 ​​generates a virtual viewpoint image (not shown), which is an image obtained by viewing the captured image projected onto the modified projection surface 42 from an arbitrary virtual viewpoint O in a line of sight direction L. The position of the virtual viewpoint O may be set to, for example, the latest self-position S of the moving object 2. In this case, the X and Y coordinate values ​​of the virtual viewpoint O may be set to the X and Y coordinate values ​​of the latest self-position S of the moving object 2. Furthermore, the Z coordinate value (vertical position) of the virtual viewpoint O may be set to the Z coordinate value of the detection point P closest to the self-position S of the moving object 2. The line of sight direction L may be determined based on, for example, a predetermined criterion.

[0073] The line of sight direction L may be, for example, a direction from the virtual viewpoint O toward the detection point P that is closest to the self-position S of the moving object 2. The line of sight direction L may also be a direction that passes through the detection point P and is perpendicular to the deformed projection plane 42. Virtual viewpoint line of sight information indicating the virtual viewpoint O and the line of sight direction L is created by the virtual viewpoint line of sight determination unit 34.

[0074] For example, the virtual viewpoint line-of-sight determination unit 34 may determine, as the line-of-sight direction L, a direction that passes through the detection point P closest to the self-position S of the moving object 2 and is perpendicular to the deformed projection plane 42. Alternatively, the virtual viewpoint line-of-sight determination unit 34 may fix the direction of the line-of-sight direction L and determine the coordinates of the virtual viewpoint O as an arbitrary Z coordinate and arbitrary X and Y coordinates in a direction away from the asymptotic curve Q toward the self-position S. In this case, the X and Y coordinates may be coordinates of a position farther away from the asymptotic curve Q than the self-position S. The virtual viewpoint line-of-sight determination unit 34 then outputs virtual viewpoint line-of-sight information indicating the virtual viewpoint O and the line-of-sight direction L to the projection transformation unit 36. Note that, as shown in FIG. 6 , the line-of-sight direction L may be a direction from the virtual viewpoint O toward the position of the vertex W of the asymptotic curve Q.

[0075] The projection transformation unit 36 ​​receives virtual viewpoint line-of-sight information from the virtual viewpoint line-of-sight determination unit 34. By receiving the virtual viewpoint line-of-sight information, the projection transformation unit 36 ​​identifies a virtual viewpoint O and a line-of-sight direction L. Then, the projection transformation unit 36 ​​generates a virtual viewpoint image, which is an image viewed from the virtual viewpoint O in the line-of-sight direction L, from the captured image projected onto the modified projection surface 42. The projection transformation unit 36 ​​outputs the virtual viewpoint image to the image synthesis unit 38.

[0076] The image synthesis unit 38 generates a synthesized image by extracting part or all of the virtual viewpoint images. For example, the image synthesis unit 38 performs a stitching process of a plurality of virtual viewpoint images (here, four virtual viewpoint images corresponding to the imaging units 12A to 12D) in the boundary area, and a blending process of each image in the stitching process. Here, the blending process is a process of blending spatially adjacent images in the boundary area at a predetermined ratio (blend ratio). The blend ratio can be set in stages so that it changes with increasing distance from the center position of the boundary area, for example. Note that the value of the blend ratio and the staged setting of the blend ratio can also be adjusted arbitrarily. Furthermore, the blending process may be omitted.

[0077] The image synthesis unit 38 outputs the generated synthetic image to the display unit 16. The synthetic image may be a bird's-eye view image with a virtual viewpoint O above the moving object 2, or may be an image in which the moving object 2 is displayed semi-transparently with a virtual viewpoint O inside the moving object 2.

[0078] The projection transformation unit 36 ​​and the image synthesis unit 38 constitute an image generation unit 37. The image generation unit 37 is an example of an image generation unit that uses a boundary region to generate a synthetic image using spatially adjacent peripheral images.

[0079] (Configuration example of the determination unit 30) Next, an example of the detailed configuration of the determination unit 30 will be described.

[0080] Fig. 8 is a schematic diagram showing an example of the configuration of the determination unit 30. As shown in Fig. 8, the determination unit 30 includes an absolute distance conversion unit 30A, an extraction unit 30B, a nearest neighbor identification unit 30C, a reference projection surface shape selection unit 30D, a scale determination unit 30E, an asymptotic curve calculation unit 30F, a shape determination unit 30G, and a boundary region determination unit 30H.

[0081] The absolute distance conversion unit 30A reads the environment map information 26A from the storage unit 26. The absolute distance conversion unit 30A converts the position information of each of the plurality of detection points P included in the read environment map information 26A into distance information of the absolute distance from the current position, which is the latest self-position S of the moving object 2, to each of the plurality of detection points P. Note that if the detection unit 14 acquires the distance information of the detection points P, the absolute distance conversion unit 30A may be omitted.

[0082] More specifically, the absolute distance conversion unit 30A calculates the current position of the moving object 2 using the speed data of the moving object 2 included in the CAN data received from the ECU 3 of the moving object 2.

[0083] Specifically, for example, the absolute distance conversion unit 30A calculates the distance between the self-positions S registered in the environment map information 26A using the speed data of the moving object 2 included in the CAN data. For example, assume the environment map information 26A shown in FIG. 4. In this case, the absolute distance conversion unit 30A calculates the distance between the self-positions S1 and S2 and the distance between the self-positions S2 and S3 using the speed data included in the CAN data. Then, the absolute distance conversion unit 30A calculates the current position of the moving object 2 using these distances.

[0084] Then, the absolute distance conversion unit 30A calculates distance information, which is the distance from the current position of the moving object 2 to each of the plurality of detection points P included in the environment map information 26A. In detail, the absolute distance conversion unit 30A converts the position information of each of the plurality of detection points P included in the environment map information 26A into distance information from the current position of the moving object 2. Through this processing, the absolute distance conversion unit 30A calculates distance information, which is the absolute distance of each of the detection points P.

[0085] Then, the absolute distance conversion unit 30A outputs the calculated distance information of each of the multiple detection points P to the extraction unit 30B. In addition, the absolute distance conversion unit 30A outputs the calculated current position of the moving object 2 to the virtual viewpoint line of sight determination unit 34 as self-position information of the moving object 2.

[0086] The extraction unit 30B extracts detection points P that are present within a specific range from the multiple detection points P for which distance information has been received from the absolute distance conversion unit 30A. The specific range is, for example, a range from the road surface on which the moving object 2 is placed to a height equivalent to the vehicle height of the moving object 2. Note that the range is not limited to this range.

[0087] By extracting the detection points P within the range, the extraction unit 30B can extract, for example, the detection points P of objects that are obstacles to the progress of the moving body 2.

[0088] Then, the extraction unit 30B outputs the distance information of each of the extracted detection points P to the nearest neighbor identification unit 30C.

[0089] The nearest neighbor identification unit 30C divides the area around the self-position S of the moving body 2 into specific angle ranges, and for each angle range, identifies the detection point P closest to the moving body 2, or multiple detection points P in order of proximity to the moving body 2. The nearest neighbor identification unit 30C identifies the detection point P using the distance information received from the extraction unit 30B. In this embodiment, a form in which the nearest neighbor identification unit 30C identifies multiple detection points P in order of proximity to the moving body 2 for each angle range will be described as an example.

[0090] The nearest neighbor specifying unit 30C outputs distance information of the detection point P specified for each angle range to the reference projection surface shape selecting unit 30D, the scale determining unit 30E, the asymptotic curve calculating unit 30F, and the boundary region determining unit 30H.

[0091] The reference projection surface shape selection unit 30D selects the shape of the reference projection surface.

[0092] Here, the reference projection surface will be described in detail with reference to Fig. 6. The reference projection surface 40 is, for example, a projection surface having a shape that serves as a reference when changing the shape of the projection surface. The shape of the reference projection surface 40 is, for example, bowl-shaped, cylindrical, etc. Note that Fig. 6 shows an example of a bowl-shaped reference projection surface 40.

[0093] The bowl-shaped container has a bottom surface 40A and a side wall surface 40B, one end of which is continuous with the bottom surface 40A and the other end of which is open. The width of the horizontal cross section of the side wall surface 40B increases from the bottom surface 40A toward the open end of the other end. The bottom surface 40A is, for example, circular. Here, a circular shape includes a perfect circle and other circular shapes such as an ellipse. The horizontal cross section is an orthogonal plane perpendicular to the vertical direction (arrow Z direction). The orthogonal plane is a two-dimensional plane along the arrow X direction, which is perpendicular to the arrow Z direction, and the arrow Y direction, which is perpendicular to the arrow Z direction and the arrow X direction. Hereinafter, the horizontal cross section and the orthogonal plane may be referred to as the XY plane. The bottom surface 40A may have a shape other than a circle, such as an egg shape.

[0094] The cylindrical shape is a shape consisting of a circular bottom surface 40A and a side wall surface 40B that is continuous with the bottom surface 40A. The side wall surface 40B that constitutes the cylindrical reference projection surface 40 has a cylindrical shape with an opening at one end that is continuous with the bottom surface 40A and an open other end. However, the side wall surface 40B that constitutes the cylindrical reference projection surface 40 has a shape in which the diameter in the XY plane is approximately constant from the bottom surface 40A side toward the opening at the other end. The bottom surface 40A may have a shape other than a circle, such as an egg shape.

[0095] In this embodiment, the case where the shape of the reference projection plane 40 is bowl-shaped as shown in Fig. 6 will be described as an example. The reference projection plane 40 is a three-dimensional model virtually formed in a virtual space, with the bottom surface 40A being a surface that substantially coincides with the road surface below the moving object 2 and the center of the bottom surface 40A being the self-position S of the moving object 2.

[0096] The reference projection surface shape selection unit 30D selects the shape of the reference projection surface 40 by reading one specific shape from the storage unit 26, which stores a plurality of types of shapes of the reference projection surface 40. For example, the reference projection surface shape selection unit 30D selects the shape of the reference projection surface 40 based on distance information and the positional relationship between the self-position and surrounding three-dimensional objects. The shape of the reference projection surface 40 may also be selected based on an operational instruction from the user. The reference projection surface shape selection unit 30D outputs shape information of the determined reference projection surface 40 to the shape determination unit 30G. In this embodiment, as described above, an example will be described in which the reference projection surface shape selection unit 30D selects a bowl-shaped reference projection surface 40.

[0097] The scale determination unit 30E determines the scale of the reference projection plane 40 of the shape selected by the reference projection plane shape selection unit 30D. For example, the scale determination unit 30E makes a decision to reduce the scale when there are multiple detection points P within a predetermined distance range from the self-position S. The scale determination unit 30E outputs scale information of the determined scale to the shape determination unit 30G.

[0098] The asymptotic curve calculation unit 30F uses the distance information of the detection point P closest to the self-position S for each angle range from the self-position S received from the nearest neighbor identification unit 30C, and outputs asymptotic curve information of the calculated asymptotic curve Q to the shape determination unit 30G and the virtual viewpoint line of sight determination unit 34. Note that the asymptotic curve calculation unit 30F may calculate the asymptotic curve Q of the detection point P accumulated for each of multiple portions of the reference projection plane 40. Then, the asymptotic curve calculation unit 30F may output the asymptotic curve information of the calculated asymptotic curve Q to the shape determination unit 30G and the virtual viewpoint line of sight determination unit 34.

[0099] The shape determination unit 30G enlarges or reduces the reference projection plane 40, which has a shape indicated by the shape information received from the reference projection plane shape selection unit 30D, to the scale of the scale information received from the scale determination unit 30E. Then, the shape determination unit 30G determines, as the projection shape, a shape obtained by deforming the reference projection plane 40 after enlarging or reducing it so that it becomes a shape that follows the asymptotic curve information of the asymptotic curve Q received from the asymptotic curve calculation unit 30F.

[0100] Here, the determination of the projection shape will be described in detail with reference to Fig. 7. As shown in Fig. 7, the shape determination unit 30G determines, as the projection shape 41, a shape obtained by deforming the reference projection plane 40 into a shape that passes through a detection point P that is closest to the self-position S of the moving object 2, which is the center of the bottom surface 40A of the reference projection plane 40. The shape that passes through the detection point P means that the side wall surface 40B after deformation passes through the detection point P. The self-position S is the latest self-position S calculated by the self-position estimation unit 27. Furthermore, the shape that passes through the detection point P means that the side wall surface 40B after deformation passes through the detection point P.

[0101] That is, the shape determination unit 30G identifies the detection point P that is closest to the self-position S among the multiple detection points P registered in the environmental map information 26A. In detail, the XY coordinates of the center position (self-position S) of the moving object 2 are set to (X, Y) = (0, 0). Then, the shape determination unit 30G determines the X 2 +Y 2 The detected point P where the value of is the smallest is identified as the detected point P closest to the self-position S. Then, the shape determination unit 30G determines, as the projected shape 41, a shape obtained by deforming the side wall surface 40B of the reference projection plane 40 so that it passes through the detected point P.

[0102] More specifically, the shape determination unit 30G determines the deformed shape of the bottom surface 40A and a portion of the side wall surface 40B as the projected shape 41 so that when the reference projection plane 40 is deformed, a portion of the side wall surface 40B becomes a wall surface that passes through the detection point P closest to the moving object 2. The deformed projected shape 41 is, for example, a shape that is raised from a rising line 44 on the bottom surface 40A in a direction approaching the center of the bottom surface 40A. "Raised" means, for example, bending or folding the side wall surface 40B and a portion of the bottom surface 40A in a direction approaching the center of the bottom surface 40A so that the angle formed between the side wall surface 40B of the reference projection plane 40 and the bottom surface 40A becomes smaller.

[0103] The shape determination unit 30G determines to deform the specific region on the reference projection plane 40 so that it protrudes to a position that passes through the detection point P when viewed from the viewpoint (planar view) of the XY plane. The shape and range of the specific region may be determined based on predetermined criteria. Then, the shape determination unit 30G determines to deform the reference projection plane 40 so that the distance from the self-position S continuously increases from the protruding specific region toward regions other than the specific region on the side wall surface 40B.

[0104] For example, it is preferable to determine the projected shape 41 so that the outer periphery of the cross section along the XY plane has a curved shape, as shown in Fig. 7. Note that the outer periphery of the cross section of the projected shape 41 is, for example, a circle, but may have a shape other than a circle.

[0105] The shape determination unit 30G may determine, as the projected shape 41, a shape obtained by deforming the reference projection plane 40 so that the shape follows an asymptotic curve. The shape determination unit 30G generates an asymptotic curve of a predetermined number of detection points P in a direction away from the detection point P closest to the self-position S of the moving object 2. The number of detection points P may be more than one. For example, the number of detection points P is preferably three or more. In this case, the shape determination unit 30G preferably generates an asymptotic curve of a plurality of detection points P located at positions that are at least a predetermined angle away from the self-position S. For example, the shape determination unit 30G may determine, as the projected shape 41, a shape obtained by deforming the reference projection plane 40 so that the shape follows the generated asymptotic curve Q for the asymptotic curve Q shown in FIG. 6.

[0106] The shape determination unit 30G may divide the area around the self-position S of the moving body 2 into specific angle ranges, and for each angle range, identify the detection point P closest to the moving body 2, or multiple detection points P in order of proximity to the moving body 2. Then, the shape determination unit 30G may determine, as the projection shape 41, a shape obtained by deforming the reference projection plane 40 so that the shape passes through the identified detection point P or the shape follows the asymptotic curve Q of the identified multiple detection points P for each angle range.

[0107] Furthermore, there are no limitations on the timing at which shape determination unit 30G determines projection shape 41. For example, in this embodiment, shape determination unit 30G may determine projection shape 41 using a captured image obtained at a time when the size of the subject included in the detection area selected by selection unit 23 becomes larger due to movement of moving object 2.

[0108] Then, the shape determination unit 30G outputs the projection shape information of the determined projection shape 41 to the deformation unit 32.

[0109] The boundary area determination unit 30H executes a boundary area determination process to determine a boundary area of ​​a plurality of peripheral images of the moving object 2, using position information including a plurality of detection points P around the moving object 2, which is stored in the environmental map information 26A, and the self-position information of the moving object 2. That is, the boundary area determination unit 30H determines the boundary area based on the position of the nearest detection point P identified by the nearest neighbor identification unit 30C.

[0110] FIG. 9 is a diagram for explaining the boundary angle and boundary width. FIG. 9 shows the line-of-sight direction AA of the image capture unit 12A, the line-of-sight direction AC of the image capture unit 12C, the line-of-sight direction AD of the image capture unit 12D, and the angular ranges R1 and R2 for which the boundary angle is set when the moving object 2 is viewed from directly above. Pillar C is the nearest three-dimensional object from the moving object 2. C0 indicates the center position of pillar C as viewed from the moving object 2, and CW indicates the width of pillar C as viewed from the moving object 2. In the situation shown in FIG. 9, assume that the moving object 2 is parked backward into a parking space line PL. In this case, the line-of-sight direction AD of the image capture unit 12D corresponds to a boundary angle of 0 degrees, and the line-of-sight direction AC of the image capture unit 12C corresponds to a boundary angle of 90 degrees.

[0111] The boundary area determination unit 30H identifies the center position C0 and width CW of the pillar C, which is the three-dimensional object closest to the moving object 2, based on the position of the nearest detection point P identified by the nearest neighbor identification unit 30C. The boundary area determination unit 30H determines a boundary angle (e.g., 45 degrees) corresponding to the identified center position C0 of the pillar C. The boundary area determination unit 30H determines an angle range corresponding to a predetermined boundary width based on the boundary angle.

[0112] It is preferable that the boundary width does not exceed the width CW of the pillar C, which is the three-dimensional object closest to the moving object 2. The boundary area can be set to any size by adjusting the boundary width. For example, the size of the boundary area may be set based on the width CW of the detected pillar C. In addition, in the example of FIG. 9, for ease of explanation, a case where the boundary angle is set in an angle range R1 of 0 to 90 degrees is illustrated. However, if the imaging unit 12 has an imaging range of about 195 degrees, for example, it is also possible to set the boundary angle to an angle of 90 degrees or more.

[0113] 9 illustrates an example in which only a pillar C exists around the moving object 2. In contrast, when multiple three-dimensional objects exist around the moving object 2, the boundary area determination unit 30H determines the boundary area based on the nearest three-dimensional object. The boundary area determination unit 30H also determines the boundary area based on three-dimensional objects whose size satisfies a predetermined standard (for example, a size that satisfies the size detectable by Visual SLAM).

[0114] 9, for example, pillar C moves diagonally forward to the right of moving body 2. In this case, boundary area determination unit 30H similarly sets a boundary angle and determines a boundary area as moving body 2 moves within angle range R2 between line of sight AC of image capture unit 12C and line of sight AA of image capture unit 12A. This is because image processing device 10 continuously generates environment map information 26A, and therefore can determine the positional relationship between moving body 2 and pillar C in real time.

[0115] 9 illustrates an example in which the three-dimensional object closest to the moving body 2 is a pillar C. In contrast to this, if the three-dimensional object closest to the moving body 2 is a wall rather than a pillar C, the boundary region determination unit 30H determines the boundary region based on the wall portion closest to the moving body 2. Furthermore, for example, if the entire right side of the moving body 2 is a wall, the boundary region determination unit 30H determines the wall portion closest to the moving body 2, for example, using a top view image (Top View) of the moving body 2, and determines the boundary region based on that portion.

[0116] The boundary region determination unit 30H outputs the determined boundary angle and a predetermined angle range based on the boundary angle to the image synthesis unit 38 as boundary region information.

[0117] Next, we will explain in detail the boundary area adaptation control realized by the image processing device 10 according to this embodiment. Here, the boundary area adaptation control is a control that sets a boundary area based on the three-dimensional object closest to the moving object 2, thereby resolving problems that occur when a composite image made by stitching together multiple peripheral images is projected and displayed on a projection surface.

[0118] In the following, for the sake of concrete explanation, an example will be taken in which a composite image is generated using a peripheral image captured by the image capturing unit 12D located behind the moving body 2 and a peripheral image captured by the image capturing unit 12C located on the right side, similar to the situation shown in Figure 9.

[0119] In the following, as an example, a case where boundary area adaptation control and projection shape adaptation control are used together will be described. Here, by setting the projection shape based on the three-dimensional object closest to the moving body 2, problems that occur when an image is projected and displayed on the projection surface are eliminated. However, using boundary area adaptation control and projection shape adaptation control together is merely an example. In other words, boundary area adaptation control can also be used independently of projection shape adaptation control.

[0120] 10 shows a situation in which the moving object 2 is parked backward within a parking space (parking space) within a parking space line PL near a pillar C. FIG. 10 shows a peripheral image V obtained by perspective projection transformation of an image captured by the image capturing unit 12C. C1 and a peripheral image V obtained by perspective projection transformation of the image captured by the imaging unit 12D. D1 FIG. 11 is a conceptual diagram showing the surrounding image V C1 and surrounding image V D1 The composite image V is generated by combining T1 FIG. 1 is a diagram showing an example of the above.

[0121] In addition, in Figure 10, column CC1 , Parking space line PL C1 is the surrounding image V C1 The pillar C and parking space line PL are visualized in D1 , Parking space line PL D1 is the surrounding image V D1 The bottom of Fig. 10 shows the surrounding image V D1 and surrounding image V C1 When the two are overlapped, column C C1 , Parking space line PL C1 , pillar C D1 , Parking space line PL D1 , the boundary area BR is projected onto the reference projection plane 40.

[0122] In FIG. 10, the peripheral image V projected onto the reference projection plane 40 D1 Pillar C D1 and surrounding image V C1 Pillar C C1 There are some non-overlapping portions on the side surfaces of the reference projection plane 40 in the boundary region BR. D1 and surrounding image V C1 Therefore, as shown in the area H in FIG. 11, the composite image V T1 In the above example, part of pillar C may not be visualized and may disappear.

[0123] 12 is a conceptual diagram showing a case where projection shape adaptation control is executed on the reference projection plane 40 shown in FIG. 10, and the reference projection plane 40 is transformed into a projection shape 41. In FIG. 12, a peripheral image V obtained by perspective projection transformation of an image captured by the image capturing unit 12C is transformed into the projection shape 41. C2 and a peripheral image V obtained by perspective projection transformation of the image captured by the imaging unit 12D. D2 13 shows the composite image V shown in FIG. T2 12 is a diagram showing an example of the column C. C2 , Parking space line PL C2 is the surrounding image V C2 The pillar C and parking space line PL are visualized inD2 , Parking space line PL D2 is the surrounding image V D2 The pillar C and parking space line PL are shown in the image.

[0124] As shown in Fig. 12, the projection shape 41 is generated by the projection shape adaptation control based on the position of the pillar C, which is the nearest three-dimensional object. C2 and Pillar C D2 As a result, as shown in the area H in FIG. 13, the composite image V T2 Above, pillar C is visualized, and composite image V T2 The disappearance of part of column C above is resolved.

[0125] 14 is a conceptual diagram of a case where the moving object 2 further moves toward the parking space line PL from the state shown in FIG. 12 and FIG. 13. In FIG. 14, a peripheral image obtained by perspective projection conversion of an image photographed by the photographing unit 12C and a peripheral image obtained by perspective projection conversion of an image photographed by the photographing unit 12D are superimposed. Also, FIG. 15 shows a composite image V shown in FIG. 14. T2 14 and 15 are diagrams showing an example of a situation where the moving object 2 moves further toward the parking space line PL, and therefore the position of the pillar C on the right side of the moving object 2 moves from the rear to the front of the moving object 2.

[0126] If boundary area adaptation control is not executed, the boundary area BR will remain at the position shown in Fig. 12, for example. Therefore, as a result of the moving object 2 moving further toward the parking space line, the boundary area BR will move from the area overlapping with the pillar C as shown in Fig. 12 to an area not overlapping with the pillar C as shown in Fig. 14. The moved boundary area BR is projected in the area where the projection shape 41 rises, but the parking space line PL as a horizontal plane image included in the boundary area BR will not move. C2 , P.L. D2 As a result, as shown in the area H in FIG. 15, the composite image V T2In the above, the parking space line PL is in the boundary area BR. C2 , P.L. D2 It may be visualized twice as:

[0127] Fig. 16 is a conceptual diagram showing an example in which the projection process is adapted by boundary area adaptation control in the case shown in Fig. 14. In Fig. 16, a peripheral image obtained by perspective projection conversion of an image photographed by the photographing unit 12C and a peripheral image obtained by perspective projection conversion of an image photographed by the photographing unit 12D are superimposed. Fig. 17 shows the composite image V shown in Fig. 16. T2 FIG. 1 is a diagram showing an example of the above.

[0128] As shown in Fig. 16, the image processing device 10 according to this embodiment sets a boundary angle at the center position of the pillar C, which is the nearest three-dimensional object (position indicated by the arrow I in Fig. 17), by boundary area adaptation control, and sets a boundary area BR so that it fits within the width of the pillar C. Therefore, the parking space line PL is positioned in the peripheral image V of the right side with respect to the projected shape 41. C2 Parking space line PL corresponding to C2 , rear surrounding image V D2 Parking space line PL corresponding to D2 In the boundary area BR, the nearest solid object is a pillar C. C2 , C D2 are projected so as to overlap on the horizontal and side surfaces of the projected shape 41. As a result, as shown in the region H in FIG. T2 Above, the double image of parking space lines PL has been eliminated.

[0129] Next, an example of the flow of image processing including boundary area adaptation control, which is executed by the image processing device 10 according to the first embodiment, will be described.

[0130] FIG. 18 is a flowchart showing an example of the flow of image processing executed by the image processing device 10.

[0131] The acquisition unit 20 acquires the captured image from the image capture unit 12 (step S10). The acquisition unit 20 also directly acquires the specified content (for example, the gear of the moving object 2 is in reverse gear, etc.) and the vehicle state (for example, the stopped state, etc.).

[0132] The selection unit 23 selects at least two of the imaging units 12A to 12D (step S12).

[0133] The matching unit 25 extracts features and performs matching processing using a plurality of captured images selected in step S12 and captured by the imaging unit 12 at different timings among the captured images acquired in step S10 (step S14). The matching unit 25 also registers information on corresponding points between the plurality of captured images at different timings, identified by the matching processing, in the environment map information 26A.

[0134] The self-position estimation unit 27 reads the environmental map information 26A (map and self-position information) (step S16). Using information on the points identified as corresponding that are registered in the environmental map information 26A, the self-position estimation unit 27 calculates (estimates) the self-position information of the moving object 2 corresponding to each of the different image capturing timings by triangulation (step S18).

[0135] Then, the self-position estimation unit 27 adds (registers) the calculated self-position information to the environment map information 26A (step S20).

[0136] The detection point registration unit 29 reads the environment map information 26A (map and self-position information) (step S22) and determines the amount of movement (translation amount and rotation amount) of the moving object 2 using the self-position information of the moving object 2 corresponding to each of the different image capture times. From the amount of movement, the detection point registration unit 29 determines the three-dimensional coordinates relative to the self-position of the moving object 2 of the corresponding points between the multiple captured images captured at different image capture times, which were identified by the matching process in step S14, and performs three-dimensional reconstruction. The detection point registration unit 29 registers the three-dimensional coordinates as the coordinates of the detection point P in the environment map information 26A (step S24). Note that the coordinates of the detection point P registered in the environment map information 26A may be converted into coordinates with a predetermined position as the origin.

[0137] The correction unit 28 takes in the environment map information 26A (map and self-position information). The correction unit 28 corrects the position information of the detection point P registered in the environment map information 26A by, for example, error minimization processing, further using the position information of the detection point P newly registered by the detection point registration unit 29 (step S26). As a result, the map and self-position information in the environment map information 26A are updated.

[0138] The absolute distance conversion unit 30A takes in the speed data (host vehicle speed) of the moving object 2 included in the CAN data received from the ECU 3 of the moving object 2. Using the speed data of the moving object 2, the absolute distance conversion unit 30A converts the position information of each of the multiple detection points P included in the environmental map information 26A into distance information of the absolute distance from the current position, which is the latest host position S of the moving object 2, to each of the multiple detection points P (step S28). The absolute distance conversion unit 30A outputs the calculated distance information of each of the multiple detection points P to the extraction unit 30B. In addition, the absolute distance conversion unit 30A outputs the calculated current position of the moving object 2 to the virtual viewpoint line of sight determination unit 34 as host position information of the moving object 2.

[0139] The extraction unit 30B extracts detection points P that exist within a specific range from among the plurality of detection points P for which distance information has been received (step S30).

[0140] The nearest neighbor identification unit 30C divides the area around the self-position S of the moving body 2 into specific angle ranges, identifies the detection point P closest to the moving body 2 for each angle range, or multiple detection points P in order of closest to the moving body 2, and extracts the distance to the nearest object (step S32). The nearest neighbor identification unit 30C outputs the distance information (distance to the nearest object) of the detection point P identified for each angle range to the reference projection surface shape selection unit 30D, the scale determination unit 30E, the asymptotic curve calculation unit 30F, and the boundary region determination unit 30H.

[0141] The asymptotic curve calculation unit 30F calculates an asymptotic curve (step S34), and outputs it to the shape determination unit 30G and the virtual viewpoint line of sight determination unit 34 as asymptotic curve information.

[0142] The reference projection plane shape selection unit 30D selects the shape of the reference projection plane 40 (step S36), and outputs shape information of the selected reference projection plane 40 to the shape determination unit 30G.

[0143] The scale determination unit 30E determines the scale of the reference projection plane 40 of the shape selected by the reference projection plane shape selection unit 30D (step S38), and outputs scale information of the determined scale to the shape determination unit 30G.

[0144] The shape determination unit 30G determines a projection shape, which is how to deform the shape of the reference projection plane, based on the scale information and the asymptotic curve information (step S40). The shape determination unit 30G outputs projection shape information of the determined projection shape 41 to the deformation unit 32.

[0145] The projection transformation unit 35 deforms the reference projection plane based on the projection shape information (step S42). The projection transformation unit 35 determines virtual line-of-sight viewpoint information including a virtual viewpoint and a virtual line of sight for drawing the minimum value of the asymptote at the center of the screen (step S44).

[0146] Furthermore, the projection transformation unit 35 executes perspective projection using the deformed projection shape and the determined virtual viewpoint and virtual line of sight, and generates a virtual viewpoint image relating to the peripheral images in the four directions (step S46).

[0147] The boundary area determination unit 30H determines a boundary area based on the distance to the nearest object identified for each angle range. That is, the boundary area determination unit 30H determines a boundary area as an overlap area of ​​spatially adjacent peripheral images based on the position of the object nearest to the moving object 2 (step S48). The boundary area determination unit 30H outputs the determined boundary area to the image synthesis unit 38.

[0148] The image synthesis unit 38 generates a synthesized image by stitching spatially adjacent perspective projection images together using the boundary area (step S50). That is, the image synthesis unit 38 generates a synthesized image by stitching together perspective projection images from four directions according to the boundary area set at the angle of the nearest object direction. In the boundary area, the spatially adjacent perspective projection images are blended at a predetermined ratio.

[0149] The display unit 16 displays the composite image (step S52).

[0150] The image processing device 10 determines whether or not to end the image processing (step S54). For example, the image processing device 10 makes the determination in step S54 by determining whether or not a signal indicating that the position of the moving object 2 has stopped moving has been received from the ECU 3. Alternatively, for example, the image processing device 10 may make the determination in step S54 by determining whether or not an instruction to end the image processing has been received through an operation instruction by a user or the like.

[0151] If a negative determination is made in step S54 (step S54: No), the processes from step S10 to step S54 are repeatedly executed. Therefore, the boundary area for joining spatially adjacent peripheral images is set in accordance with the angle of the nearest object direction.

[0152] On the other hand, if the determination in step S54 is affirmative (step S54: Yes), this routine ends.

[0153] When returning from step S54 to step S10 after performing the correction process of step S26, the correction process of the subsequent step S26 may be omitted. Also, when returning from step S54 to step S10 without performing the correction process of step S26, the correction process of the subsequent step S26 may be performed.

[0154] As described above, the image processing device according to the embodiment includes a boundary region determination unit 30H as a determination unit, and an image generation unit 37 as a generation unit. The boundary region determination unit 30H determines a boundary region in an overlapping region of spatially adjacent peripheral images among a plurality of peripheral images of the moving object 2, using position information including a plurality of detection points P around the moving object 2 and self-position information of the moving object 2. The image generation unit 37 uses the boundary region to generate a composite image using the spatially adjacent peripheral images.

[0155] Therefore, the boundary area for joining spatially adjacent peripheral images among the multiple peripheral images of the moving body 2 can be appropriately set according to the position information including multiple detection points P around the moving body 2 and the self-position information of the moving body 2.

[0156] Furthermore, the boundary area determination unit 30H determines the boundary area based on the position information of the detection point P corresponding to the three-dimensional object closest to the moving object 2. Therefore, in the boundary area, a plurality of spatially adjacent peripheral images are used to project the three-dimensional object closest to the moving object 2 so that they overlap on the projection surface. Furthermore, in areas other than the boundary area, a projection process is performed using one of the spatially adjacent peripheral images. As a result, it is possible to prevent problems such as double imaging of parking space lines on the composite image.

[0157] Furthermore, the boundary area determination unit 30H determines the boundary area based on the width of the three-dimensional object closest to the moving object 2. Therefore, the boundary area can be set so that it fits within the width of the three-dimensional object closest to the moving object 2. As a result, the boundary area does not extend beyond the three-dimensional object closest to the moving object 2, and the occurrence of double imaging of parking space lines in horizontal areas, etc. can be suppressed.

[0158] The boundary area determination unit 30H determines multiple boundary areas that track the three-dimensional object using the sequentially generated position information and self-position information. The image generation unit 37 sequentially generates a composite image using spatially adjacent peripheral images using the multiple boundary areas that track the three-dimensional object. Therefore, the boundary area for connecting spatially adjacent peripheral images is set to track the angle of the nearest object direction. As a result, the user can use a highly accurate composite image that suppresses the occurrence of double images of parking space lines in horizontal areas, etc.

[0159] The image processing device 10 according to this embodiment further includes a deformation unit 32 that deforms the reference projection plane using the position information and the self-position information. The image generation unit 37 generates a composite image using the deformed reference projection plane and spatially adjacent peripheral images. Therefore, the boundary area for connecting spatially adjacent peripheral images is set to follow the reference projection plane that is deformed according to the position of the nearest object. As a result, the user can use a highly accurate composite image in which the occurrence of double imaging of three-dimensional objects surrounding the moving object 2 is suppressed.

[0160] The effects of the image processing device 10 according to this embodiment will be described in more detail with reference to Figs. 19 to 24. Figs. 19, 21, and 23 are diagrams for explaining a comparative example in which boundary area adaptive control is not executed. Figs. 20, 22, and 24 are diagrams showing examples of composite images obtained by image processing including boundary area adaptive control. Note that Figs. 19 to 24 all show examples in which the moving object 2 is parked backward.

[0161] In the comparative example shown in Fig. 19, the boundary angle is set at a position (approximately 45 degrees) away from the three-dimensional object J closest to the moving object 2. Therefore, in the area H shown in Fig. 19, a phenomenon occurs in which the parking space lines are visualized as double.

[0162] Fig. 20 is an example of a composite image obtained by executing boundary area adaptation control according to this embodiment for the comparative example shown in Fig. 19. By boundary area adaptation control, the boundary angle is set to a position I (approximately 75 degrees) that overlaps with the three-dimensional object J closest to the moving object 2, and the boundary width is set to fit within the width of the three-dimensional object J closest to the moving object 2. Therefore, in Fig. 20, the double imaging of the parking space lines is eliminated.

[0163] 21 and 22 show examples of composite images when the moving object 2 moves from the position shown in FIGS. 19 and 20 and parks in a different location. FIG. 21 shows a comparative example in which the boundary angle set in FIG. 20 is maintained even when the moving object 2 moves. In this case, the boundary angle is set at a position (approximately 75 degrees) that is away from the three-dimensional object K closest to the moving object 2. Therefore, in the area H shown in FIG. 21, a phenomenon occurs in which the parking space lines are displayed double. Note that FIGS. 21 and 22 may also show images in which the moving object 2 has moved forward from the position shown in FIGS. 19 and 20, for example, by switching to a rear-parking position.

[0164] Fig. 22 is an example of a composite image obtained by executing boundary area adaptation control according to this embodiment for the comparative example shown in Fig. 21. By boundary area adaptation control, the boundary angle is set to position I (approximately 25 degrees) where it overlaps with the three-dimensional object K closest to the moving object 2, and the boundary area is set so as to fit within the width of the three-dimensional object K closest to the moving object 2. Therefore, in Fig. 22, the double imaging of the parking space lines is eliminated.

[0165] 23 and 24, the moving object 2 has moved further rearward than its position in FIGS. 21 and 22. FIG. 23 shows a comparative example in which the boundary angle set in FIG. 22 is maintained even as the moving object 2 moves. In this case, the boundary angle is set at a position (approximately 25 degrees) that is away from the three-dimensional object K closest to the moving object 2. Therefore, in area H shown in FIG. 23, a phenomenon occurs in which the parking space lines are visualized as double.

[0166] FIG. 24 is an example of a composite image obtained by executing boundary area adaptive control according to this embodiment for the comparative example shown in FIG. 23. By boundary area adaptive control, the boundary angle is set to position I (approximately 60 degrees) where it overlaps with the three-dimensional object K closest to the moving object 2, and the boundary area is set so as to fit within the width of the three-dimensional object K closest to the moving object 2. In this case, the boundary angle is continuously controlled as the moving object 2 moves rearward. Therefore, in FIG. 24, the double imaging of parking space lines is eliminated. In this way, by executing boundary area adaptive control in accordance with the movement of the moving object 2, it is possible to continuously suppress the occurrence of double imaging of three-dimensional objects around the moving object 2.

[0167] (Second embodiment) In the first embodiment described above, an example of a form in which position information of a detection point P is acquired from an image captured by the imaging unit 12, i.e., a form in which Visual SLAM is used, has been described. In contrast, in the second embodiment, an example of a form in which position information of a detection point P is detected by the detection unit 14 will be described. That is, the image processing device 10 according to the second embodiment is an example in which three-dimensional LiDAR SLAM or the like is used.

[0168] 25 is a diagram showing an example of the functional configuration of the image processing device 10 of the second embodiment. Similar to the image processing device 10 of the first embodiment, the image processing device 10 is connected to the imaging unit 12, the detection unit 14, and the display unit 16 so as to be able to exchange data or signals with them.

[0169] The image processing device 10 includes an acquisition unit 20, a self-position estimation unit 27, a detection point registration unit 29, a memory unit 26, a correction unit 28, a determination unit 30, a deformation unit 32, a virtual viewpoint line of sight determination unit 34, a projection transformation unit 36, and an image synthesis unit 38.

[0170] Some or all of the above-mentioned multiple units may be realized by software, for example, by causing a processing device such as CPU 10A shown in Fig. 2 to execute a program. Also, some or all of the above-mentioned multiple units may be realized by hardware such as an IC, or may be realized by a combination of software and hardware.

[0171] 25, the storage unit 26, correction unit 28, determination unit 30, deformation unit 32, virtual viewpoint line of sight determination unit 34, projection conversion unit 36, and image synthesis unit 38 are the same as those in the first embodiment. The storage unit 26 stores environment map information 26A. The environment map information 26A is the same as that in the first embodiment.

[0172] The acquisition unit 20 acquires the captured images from the photographing unit 12. The acquisition unit 20 also acquires position information of the detection points from the detection unit 14. The acquisition unit 20 acquires the captured images from each of the photographing units 12 (photographing units 12A to 12D). The detection unit 14 detects position information of each of the multiple detection points. Therefore, the acquisition unit 20 acquires the position information of each of the multiple detection points and the captured images by each of the multiple photographing units 12.

[0173] Every time the acquisition unit 20 acquires position information of each of the plurality of detection points, it outputs the acquired position information to the detection point registration unit 29 and the self-position estimation unit 27. In addition, every time the acquisition unit 20 acquires a captured image, it outputs the acquired captured image to the projection transformation unit 36.

[0174] Every time the detection point registration unit 29 acquires position information of each of the plurality of detection points, the detection point registration unit 29 registers the acquired position information in the environment map information 26A. The environment map information 26A is stored in the storage unit 26.

[0175] Each time the detection point registration unit 29 acquires position information of a detection point P from the detection unit 14 via the acquisition unit 20, it identifies the same detection point P that is already registered in the environment map information 26A by scan matching. The same detection point P means that the timing at which the detection unit 14 acquired the position information is different, but the detection point P is the same in real space. The detection point registration unit 29 adds and registers, to the environment map information 26A, the position information of the detection point P that is not registered in the environment map information 26A, among the position information of the detection point P acquired from the detection unit 14. At this time, the detection point registration unit 29 converts the acquired position information of the detection point P into coordinates with the above-mentioned predetermined position as the origin, and then registers it in the environment map information 26A.

[0176] As a result of scan matching by the detection point registration unit 29, there may be cases where the newly acquired detection point P and the detection point P already registered in the environment map information 26A do not match by a predetermined percentage or more. In this case, the detection point registration unit 29 may discard the position information of the newly acquired detection point P and omit registration of the newly acquired detection point P in the environment map information 26A. The predetermined percentage may be determined in advance. This process can increase the reliability of the environment map information 26A.

[0177] It is preferable that the detection point registration unit 29 registers various parameters used in calculating the position information of the detection point P together with the position information in the environment map information 26A.

[0178] The self-position estimation unit 27 estimates self-position information indicating the self-position S of the moving object 2 based on the position information of each of the plurality of detection points P registered in the environmental map information 26A. Then, the self-position estimation unit 27 registers the self-position information of the estimated self-position S in the environmental map information 26A.

[0179] The self-position estimation unit 27 uses the position information of each of the detection points P registered in the environmental map information 26A and the position information of each of the latest detection points P acquired from the detection unit 14 via the acquisition unit 20 to identify corresponding detection points P included in this position information. Corresponding detection points P mean that the detection unit 14 acquired the position information at different times but that the detection points P are the same detection points P in real space. The self-position estimation unit 27 calculates self-position information of the self-position S of the moving object 2 by triangulation using the position information of each of the identified corresponding detection points P. Then, the self-position estimation unit 27 registers the calculated self-position information in the environmental map information 26A. Note that the self-position estimation unit 27 may estimate the self-position information using odometry.

[0180] As described above, in this embodiment, the image processing device 10 simultaneously registers the position information of the detection point P in the environmental map information 26A and estimates the self-position information of the moving object 2 by SLAM.

[0181] Next, an example of the flow of image processing including boundary area adaptation control, which is executed by the image processing device 10 according to the second embodiment, will be described.

[0182] FIG. 26 is a flowchart showing an example of the flow of image processing executed by the image processing device 10.

[0183] The acquisition unit 20 acquires a captured image from the imaging unit 12 (step S100). The acquisition unit 20 also acquires position information of each of the plurality of detection points P from the detection unit 14 (step S102).

[0184] The detection point registration unit 29 identifies, by scan matching, the same detection points P that have already been registered in the environment map information 26A among the detection points P in the position information acquired in step S102 (step S104). Then, the detection point registration unit 29 registers, in the environment map information 26A, the position information of the detection points P that have not been registered in the environment map information 26A among the position information of the detection points P acquired in step S102 (step S106).

[0185] The self-position estimation unit 27 estimates self-position information indicating the self-position S of the moving object 2 based on the position information of each of the plurality of detection points P registered in the environmental map information 26A and the position information acquired in step S102 (step S108). Then, the self-position estimation unit 27 registers the estimated self-position information in the environmental map information 26A (step S110).

[0186] The correction unit 28 corrects the position information of the detection point P registered in the environment map information 26A using the position information of the detection point P acquired in step S102 (step S112). As described above, the correction process of step S112 may not be executed.

[0187] The absolute distance conversion unit 30A of the determination unit 30 converts the position information of each of the multiple detection points P included in the environmental map information 26A into distance information of the absolute distance from the current position of the moving object 2 to each of the multiple detection points P (step S114).

[0188] Extraction unit 30B extracts detection points P that exist within a specific range from among the detection points P for which distance information of absolute distances has been calculated by absolute distance conversion unit 30A (step S116).

[0189] The nearest neighbor identification unit 30C uses the distance information of each of the detection points P extracted in step S116 to identify a plurality of detection points P in order of proximity to the moving object 2 for each angle range around the moving object 2 (step S118).

[0190] The asymptotic curve calculation unit 30F calculates the asymptotic curve Q using the distance information of each of the plurality of detection points P for each angle range identified in step S118 (step S120).

[0191] The reference projection surface shape selecting unit 30D selects the shape of the reference projection surface 40 (step S122). As described above, the reference projection surface shape selecting unit 30D selects the bowl-shaped reference projection surface 40 as an example.

[0192] The scale determination unit 30E determines the scale of the reference projection plane 40 of the shape selected in step S122 (step S124).

[0193] The shape determination unit 30G enlarges or reduces the reference projection plane 40 of the shape selected in step S122 to the scale determined in step S124. Then, the shape determination unit 30G deforms the reference projection plane 40 after the enlargement or reduction so that the shape follows the asymptotic curve Q calculated in step S120. The shape determination unit 30G determines this deformed shape as the projection shape 41 (step S126).

[0194] The deformation unit 32 deforms the reference projection plane 40 to the projection shape 41 determined by the determination unit 30 (step S128). Through this deformation process, the deformation unit 32 generates a deformed projection plane 42, which is the deformed reference projection plane 40.

[0195] The virtual viewpoint line-of-sight determination unit 34 determines virtual viewpoint line-of-sight information (step S130). For example, the virtual viewpoint line-of-sight determination unit 34 determines the self-position S of the moving object 2 as the virtual viewpoint O, and determines the direction from the virtual viewpoint O toward the position of the vertex W of the asymptotic curve Q as the line-of-sight direction L. In detail, the virtual viewpoint line-of-sight determination unit 34 may determine, as the line-of-sight direction L, the direction toward the vertex W of the asymptotic curve Q of a specific angle range among the asymptotic curves Q calculated for each angle range in step S120.

[0196] The projection transformation unit 36 ​​projects the captured image acquired in step S100 onto the deformed projection surface 42 generated in step S128. Then, the projection transformation unit 36 ​​converts the projected image into a virtual viewpoint image, which is an image obtained by viewing the captured image projected onto the deformed projection surface 42 from the virtual viewpoint O determined in step S130 in the line of sight L (step S132).

[0197] The boundary area determination unit 30H determines a boundary area based on the distance to the nearest object identified for each angle range. That is, the boundary area determination unit 30H determines a boundary area as an overlap area of ​​spatially adjacent peripheral images based on the position of the object nearest to the moving object 2 (step S134). The boundary area determination unit 30H outputs the determined boundary area to the image synthesis unit 38.

[0198] The image synthesis unit 38 generates a synthesized image by stitching spatially adjacent perspective projection images together using the boundary area (step S136). That is, the image synthesis unit 38 generates a synthesized image by stitching together perspective projection images in four directions according to the boundary area set to the angle of the nearest object direction. In the boundary area, the spatially adjacent perspective projection images are blended at a predetermined ratio.

[0199] The display unit 16 executes display control to display the generated composite image 54 (step S138).

[0200] Next, the image processing device 10 determines whether or not to end the image processing (step S140). For example, the image processing device 10 makes the determination in step S140 by determining whether or not a signal indicating that the position of the moving object 2 has stopped moving has been received from the ECU 3. Alternatively, for example, the image processing device 10 may make the determination in step S140 by determining whether or not an instruction to end the image processing has been received through an operation instruction by a user or the like.

[0201] If the determination in step S140 is negative (step S140: No), the processes from step S100 to step S140 are repeatedly executed. Therefore, the boundary area for joining spatially adjacent peripheral images is set according to the angle of the nearest object direction. On the other hand, if the determination in step S140 is positive (step S140: Yes), this routine ends.

[0202] When returning from step S140 to step S100 after executing the correction process of step S112, the correction process of the subsequent step S112 may be omitted. Also, when returning from step S140 to step S100 without executing the correction process of step S112, the correction process of the subsequent step S112 may be executed.

[0203] As described above, the image processing device 10 according to the second embodiment uses SLAM to generate position information including a plurality of detection points P around the moving object 2 and self-position information of the moving object 2, using the captured image captured by the image capturing unit 12 and the position information detected by the detection unit 14. The boundary area determination unit 30H uses the position information and the self-position information to determine a boundary area in an overlapping area between spatially adjacent peripheral images among the plurality of peripheral images of the moving object 2. The image generation unit 37 generates a composite image using the boundary area and the spatially adjacent peripheral images. Therefore, the image processing device 10 according to the second embodiment can achieve the same effects as the image processing device 10 according to the first embodiment.

[0204] (Variation) In the above embodiments, the image processing device 10 uses SLAM. However, it is also possible to create environmental map information using immediate values ​​from a sensor array constructed with multiple distance sensors without using SLAM, and to use this to perform image processing including boundary area adaptation control.

[0205] Although the embodiments and modifications have been described above, the image processing device, image processing method, and image processing program disclosed herein are not limited to the above-described embodiments, and the components can be modified and embodied in each implementation stage without departing from the spirit of the invention. Furthermore, various inventions can be created by appropriately combining multiple components disclosed in the above-described embodiments. For example, some components may be deleted from all of the components shown in the embodiments.

[0206] The image processing device 10 of the first and second embodiments can be applied to various devices. For example, the image processing device 10 of the first and second embodiments can be applied to a surveillance camera system that processes images obtained from a surveillance camera, or an in-vehicle system that processes images of the surrounding environment outside the vehicle. [Explanation of symbols]

[0207] 10 Image processing device 12, 12A~12D Photography Department 14 Detector 20 Acquisition Department 23 Selection section 25 Matching Section 26 Memory section 26A Environmental Map Information 27 Self-position estimation part 28 Correction section 29 Detection point registration unit 30 Decision Section 30A Absolute distance conversion unit 30B Extraction part 30C Nearest neighbor identification part 30D Reference projection surface shape selection section 30E Scale determination section 30F Asymptotic curve calculation section 30G shape determining section 30H Boundary area determination part 32 Deformed part 34 Virtual viewpoint line of sight determination unit 36 Projection transformation unit 37 Image generation unit 38 Image synthesis unit

Claims

1. a determination unit that uses position information including a plurality of detection points around the moving object and self-position information of the moving object to set a boundary angle of an image during image synthesis so as to overlap with a nearest three-dimensional object, and determines a boundary area in an overlapping area of ​​spatially adjacent peripheral images of the moving object using the boundary angle; an image generating unit that generates a composite image using the spatially adjacent peripheral images by using the boundary region; An image processing device comprising:

2. the determination unit sets the boundary angle of the images at the time of the image synthesis so as to fall within a width of the nearest three-dimensional object. The image processing device according to claim 1 .

3. The determination unit determines the boundary region based on a width of the three-dimensional object. The image processing device according to claim 2 .

4. The determination unit determines a plurality of the boundary regions that follow the three-dimensional object by using the position information and the self-position information that are sequentially generated; The image generation unit sequentially generates the composite image using the spatially adjacent peripheral images by using a plurality of the boundary regions that follow the three-dimensional object.

4. The image processing device according to claim 2 or 3.

5. a deformation unit that deforms a reference projection plane by using the position information and the self-position information, The image generating unit generates the composite image by using the transformed reference projection plane and the peripheral images spatially adjacent to the reference projection plane.

5. The image processing device according to claim 2.

6. The detection point is detected using the peripheral image.

6. An image processing device according to claim 1.

7. The detection point is detected by a detection unit mounted on the moving object.

6. An image processing device according to claim 1.

8. 1. A computer-implemented image processing method, comprising the steps of: a step of setting a boundary angle of an image to overlap with a nearest three-dimensional object when synthesizing the images using position information including a plurality of detection points around the moving body and self-position information of the moving body, and determining a boundary area in an overlapping area of ​​spatially adjacent peripheral images of the moving body using the boundary angle; and generating a composite image using the spatially adjacent surrounding images using the boundary region. Image processing methods.

9. On the computer, a step of setting a boundary angle of an image to overlap with a nearest three-dimensional object when synthesizing the images using position information including a plurality of detection points around the moving body and self-position information of the moving body, and determining a boundary area in an overlapping area of ​​spatially adjacent peripheral images of the moving body using the boundary angle; generating a composite image using the spatially adjacent surrounding images using the boundary region; An image processing program for executing the above.

Images