Image processing apparatus, image processing method, and recording medium

Abstract

The image processing apparatus of the present application, in one embodiment, has a deformation section (32). The deformation section (32) deforms a projection plane, i.e., a reference projection plane, of a surrounding image of a mobile body using position information of a plurality of detection points of the surrounding of the mobile body and self position information of the mobile body.

Description

Technical Field

[0001] The present invention relates to an image processing device, an image processing method, and a recording medium. Background Art

[0002] A technology has been disclosed for generating a composite image from an arbitrary viewpoint using a projection image obtained by projecting a captured image of the surroundings of a moving object onto a virtual projection surface. Furthermore, a technology has been disclosed for detecting the distance from the moving object to an object surrounding the moving object using a sensor and deforming the shape of the projection surface based on the detected distance (e.g., see Patent Documents 1-3).

[0003] Patent Document 1: Japanese Patent Application Laid-Open No. 2013-207637;

[0004] Patent Document 2: Japanese Patent Application No. 2014-531078;

[0005] Patent Document 3: Japanese Patent No. 5369465.

[0006] Non-patent document 1: "Mobile radio environment awareness-terrain construction and own position estimation" システム / Control / Information (Journal of the システムInformation Society), Vol. 60 No. 12, pp509-514, 2016.

[0007] However, if the relative position of the object to the moving object is outside the detection area of ​​the sensor, it is difficult to obtain the distance to the object. Therefore, in the conventional technology, it is sometimes difficult to deform the projection surface according to the object. Summary of the Invention

[0008] In one aspect, an object of the present invention is to provide an image processing device, an image processing method, and a recording medium capable of deforming a projection surface according to an object.

[0009] In one embodiment, the image processing device disclosed in the present application includes a deformation unit that deforms a reference projection plane, which is a projection plane of an image of the surroundings of the moving object, using position information of a plurality of detection points storing the surroundings of the moving object and the position information of the moving object itself.

[0010] According to one embodiment of the image processing device disclosed in the present application, it is possible to deform a projection surface according to an object. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 It is a diagram showing the overall configuration of the image processing system according to the first embodiment.

[0012] Figure 2 This is a diagram showing the hardware configuration of the image processing device according to the first embodiment.

[0013] Figure 3 This is a diagram showing the functional configuration of the image processing device according to the first embodiment.

[0014] Figure 4 Schematic diagram of the environment map information according to the first embodiment.

[0015] Figure 5A It is a schematic diagram showing the reference projection plane of the first embodiment.

[0016] Figure 5B It is a schematic diagram showing the projection shape of the first embodiment.

[0017] Figure 6 This is an explanatory diagram of the asymptotic curve of the first embodiment.

[0018] Figure 7 It is a schematic diagram showing the functional configuration of the determination unit according to the first embodiment.

[0019] Figure 8 This is a flowchart showing the flow of image processing in the first embodiment.

[0020] Figure 9A It is a schematic diagram showing a captured image according to the first embodiment.

[0021] Figure 9B Schematic diagram showing a composite image according to the first embodiment.

[0022] Figure 9C FIG. 1 is a schematic diagram showing a timing chart of image processing according to the first embodiment.

[0023] Figure 10A This is an explanatory diagram of conventional projection surface deformation processing.

[0024] Figure 10B This is an explanatory diagram of conventional projection surface deformation processing.

[0025] Figure 11A This is an explanatory diagram of conventional projection surface deformation processing.

[0026] Figure 11B This is an explanatory diagram of conventional projection surface deformation processing.

[0027] Figure 12A It is an explanatory diagram of the deformation of the reference projection plane in the first embodiment.

[0028] Figure 12B It is an explanatory diagram of the deformation of the reference projection plane in the first embodiment.

[0029] Figure 13A It is an explanatory diagram of the deformation of the reference projection plane in the first embodiment.

[0030] Figure 13B It is an explanatory diagram of the deformation of the reference projection plane in the first embodiment.

[0031] Figure 14A This is an explanatory diagram of the determination of the viewing direction L according to the first embodiment.

[0032] Figure 14B This is an explanatory diagram of the determination of the viewing direction L according to the first embodiment.

[0033] Figure 14C This is an explanatory diagram of the determination of the viewing direction L according to the first embodiment.

[0034] Figure 15A It is a schematic diagram showing a captured image according to the first embodiment.

[0035] Figure 15B Schematic diagram showing a composite image according to the first embodiment.

[0036] Figure 16 This is a diagram showing the functional configuration of an image processing device according to the second embodiment.

[0037] Figure 17 This is a flowchart showing the flow of image processing in the second embodiment.

[0038] Figure 18 FIG. 1 is a schematic diagram showing a timing chart of image processing according to the second embodiment.

[0039] Figure 19 This is an image showing a composite image according to the second embodiment. DETAILED DESCRIPTION

[0040] The following describes in detail embodiments of the image processing device, image processing method, and recording medium disclosed in this application with reference to the accompanying drawings. The following embodiments do not limit the disclosed technology. Furthermore, the various embodiments can be appropriately combined to the extent that the processing contents do not conflict.

[0041] (First embodiment)

[0042] Figure 1 1 is a diagram showing an example of the overall configuration of an image processing system 1 according to this embodiment. The image processing system 1 includes an image processing device 10, an imaging unit 12, a detection unit 14, and a display unit 16. The image processing device 10, the imaging unit 12, the detection unit 14, and the display unit 16 are connected so as to be able to transmit and receive data or signals.

[0043] In this embodiment, a mode in which the image processing device 10 , the imaging unit 12 , the detection unit 14 , and the display unit 16 are mounted on the mobile object 2 will be described as an example.

[0044] The so-called mobile body 2 is a movable object. The mobile body 2 is, for example, a vehicle, a flyable object (manned aircraft, unmanned aircraft (for example, UAV (Unmanned Aerial Vehicle), pilotless aircraft)), a robot, or the like. In addition, the mobile body 2 is, for example, a mobile body that travels by a driving operation of a person, a mobile body that is capable of automatically traveling (autonomous traveling) without a driving operation of a person. In the present embodiment, a case where the mobile body 2 is a vehicle will be described as one example. The vehicle is, for example, a two-wheeled car, a three-wheeled car, a four-wheeled car, or the like. In the present embodiment, a case where the vehicle is a four-wheeled car capable of autonomous traveling will be described as one example.

[0045] Further, it is not limited to a case where the image processing apparatus 10, the photographing section 12, the detection section 14, and the display section 16 are all mounted on the mobile body 2. The image processing apparatus 10 can also be mounted on a stationary object. The stationary object is an object fixed to the ground. The stationary object is an object that is not movable, an object in a state of being stationary with respect to the ground. The stationary object is, for example, a signal light, a parked vehicle, a road sign, or the like. In addition, the image processing apparatus 10 can also be mounted on a cloud server that performs processing on a cloud.

[0046] The photographing section 12 photographs the periphery of the mobile body 2, and acquires photographing image data. Hereinafter, the photographing image data will be simply referred to as a photographing image. The photographing section 12 is, for example, a digital camera. Further, the so-called photographing refers to conversion of an image of a photographed object, which is imaged by an optical system such as a lens, into an electric signal. The photographing section 12 outputs the photographed photographing image to the image processing apparatus 10.

[0047] In the present embodiment, a case where four photographing sections 12 (photographing section 12A to photographing section 12D) are mounted on the mobile body 2 will be described as one example. The plurality of photographing sections 12 (photographing section 12A to photographing section 12D) photograph the photographed object in each photographing region E (photographing region E1 to photographing region E4), and acquire a photographing image. Further, the photographing directions of these plurality of photographing sections 12 are different from each other. Specifically, the photographing directions of these plurality of photographing sections 12 are adjusted in advance so that at least a part of the photographing regions E do not overlap with each other.

[0048] The detection section 14 detects each position information of a plurality of detection points in the periphery of the mobile body 2. In other words, the detection section 14 detects each position information of the detection points of the detection region F. The so-called detection point indicates each point in a real space that is respectively observed by the detection section 14. For example, the detection section 14 irradiates light to the surroundings of the detection section 14, and receives reflected light reflected by a reflection point. The reflection point corresponds to the detection point.

[0049] The position information of a detection point is information indicating the position of the detection point in real space (three-dimensional space). For example, the position information of a detection point is information indicating the distance from the detection unit 14 (i.e., the position of the mobile object 2) to the detection point and the direction of the detection point relative to the detection unit 14. These distances and directions can be represented, for example, by position coordinates indicating the relative position of the detection point relative to the detection unit 14, position coordinates indicating the absolute position of the detection point, or vectors.

[0050] Detection unit 14 may be, for example, a 3D (three-dimensional) scanner, a 2D (two-dimensional) scanner, a distance sensor (millimeter-wave radar, laser sensor), a sonar sensor that detects objects using sound waves, or an ultrasonic sensor. An example of a laser sensor is a three-dimensional LIDAR (Laser Imaging Detection and Ranging) sensor. Alternatively, detection unit 14 may be a device that uses SfM (Structure from Motion) technology to measure distance based on images captured by a single-lens reflex camera.

[0051] The display unit 16 displays various information and is, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display.

[0052] In this embodiment, the image processing device 10 is communicatively connected to an electronic control unit (ECU) 3 mounted on a mobile object 2. The ECU 3 is a unit that performs electronic control of the mobile object 2. In this embodiment, the image processing device 10 can receive CAN (Controller Area Network) data, such as the speed and direction of movement of the mobile object 2, from the ECU 3.

[0053] Next, the hardware configuration of the image processing device 10 will be described. Figure 2 1 is a diagram showing an example of the hardware configuration of the image processing device 10 .

[0054] Image processing device 10 includes a CPU (Central Processing Unit) 10A, a ROM (Read Only Memory) 10B, a RAM (Random Access Memory) 10C, and an interface (Interface) 10D, and is, for example, a computer. CPU 10A, ROM 10B, RAM 10C, and I / F 10D are interconnected via a bus 10E, forming a hardware configuration utilizing a conventional computer.

[0055] The CPU 10A is a computing device that controls the image processing device 10. The CPU 10A corresponds to an example of a hardware processor. The ROM 10B stores programs and other information that implement the various processes performed by the CPU 10A. The RAM 10C stores data required for the various processes performed by the CPU 10A. The I / F 10D is an interface for transmitting and receiving data to and from the imaging unit 12, the detection unit 14, the display unit 16, and the ECU 3.

[0056] The program for executing the image processing executed in the image processing device 10 of the present embodiment is pre-loaded into the ROM 10B or the like and provided. In addition, the program executed in the image processing device 10 of the present embodiment can also be configured to be provided by recording it in a recording medium in a form that can be installed in the image processing device 10 or in an executable form. The recording medium is a medium that can be read by a computer. The recording medium is a CD (Compact Disc)-ROM, a floppy disk (FD), a CD-R (Recordable), a DVD (Digital Versatile Disk), a USB (Universal Serial Bus) memory, an SD (Secure Digital) card, etc.

[0057] Next, the functional configuration of the image processing device 10 will be described. Figure 3 1 is a diagram showing an example of the functional configuration of the image processing device 10. Figure 3 In order to clarify the input and output relationship of data, in addition to the image processing device 10, the imaging unit 12, the detection unit 14, and the display unit 16 are also shown.

[0058] The image processing device 10 includes an acquisition unit 20 , a detection point registration unit 22 , a self-position estimation unit 24 , a storage unit 26 , a correction unit 28 , a determination unit 30 , a deformation unit 32 , a virtual viewpoint line of sight determination unit 34 , a projection conversion unit 36 ​​, and an image synthesis unit 38 .

[0059] Some or all of the above-mentioned components may be implemented by, for example, a processing device such as the CPU 10A executing a program, i.e., software. In addition, some or all of the above-mentioned components may be implemented by hardware such as an IC (Integrated Circuit), or by a combination of software and hardware.

[0060] The acquisition unit 20 acquires captured images from the imaging unit 12. Furthermore, the acquisition unit 20 acquires positional information of detection points from the detection unit 14. The acquisition unit 20 acquires captured images from each imaging unit 12 (imaging units 12A to 12D). The detection unit 14 detects positional information of each of the multiple detection points. Therefore, the acquisition unit 20 acquires positional information of each of the multiple detection points and images captured by each of the multiple imaging units 12.

[0061] Each time the acquisition unit 20 acquires position information of each of the plurality of detection points, it outputs the acquired position information to the detection point registration unit 22 and the self-position estimation unit 24. Additionally, each time the acquisition unit 20 acquires a captured image, it outputs the acquired captured image to the projection conversion unit 36.

[0062] Each time the detection point registration unit 22 acquires position information of each of a plurality of detection points, it registers the acquired position information in the environment map information 26A. The environment map information 26A is stored in the storage unit 26.

[0063] The storage unit 26 stores various data. Examples of the storage unit 26 include semiconductor memory devices such as RAM and flash memory, a hard disk, and an optical disk. Furthermore, the storage unit 26 may be a storage device external to the image processing apparatus 10. Furthermore, the storage unit 26 may be a storage medium. Specifically, the storage medium may be a medium that stores or temporarily stores programs and various information downloaded via a LAN (Local Area Network) or the Internet.

[0064] Environmental map information 26A is map information representing the surrounding conditions of mobile object 2. Environmental map information 26A registers position information of each detection point and the position information of mobile object 2 in a three-dimensional coordinate space with a predetermined position in real space as the origin. The predetermined position in real space may be determined based on, for example, pre-set conditions.

[0065] For example, the prescribed position is the position of the mobile body 2 when the image processing device 10 performs the image processing of the present embodiment. For example, assume that image processing is performed at a prescribed moment in a parking scene of the mobile body 2. In this case, the image processing device 10 only needs to set the position of the mobile body 2 when it is determined that the prescribed moment has arrived as the prescribed position. For example, when the image processing device 10 determines that the operating status of the mobile body 2 is an operating status representing a parking scene, it can be determined that the prescribed moment has arrived. The operating status representing a parking scene is, for example, a case where the speed of the mobile body 2 becomes lower than a prescribed speed, a case where the gear of the mobile body 2 enters the rear reverse wheel, a case where a signal indicating the start of parking is received according to a user's operating instruction, etc. In addition, the prescribed moment is not limited to a parking scene.

[0066] Figure 4 2 is a diagram showing an example of the environment map information 26A. The environment map information 26A is information in which position information of each detection point P and the own position information of the own position S of the mobile object 2 are registered at corresponding coordinate positions in the three-dimensional coordinate space.

[0067] Each time the detection point registration unit 22 acquires the position information of a detection point P from the detection unit 14 via the acquisition unit 20, it identifies the same detection point P registered in the environment map information 26A through scan matching. The term "same detection point P" refers to the same detection point P in real space, even though the acquisition time of the position information by the detection unit 14 is different. The detection point registration unit 22 additionally registers the position information of any detection point P acquired from the detection unit 14 that is not registered in the environment map information 26A in the environment map information 26A. To do this, the detection point registration unit 22 converts the coordinates of the acquired position information of the detection point P into coordinates with the predetermined position as the origin, and then registers the coordinates in the environment map information 26A.

[0068] Furthermore, as a result of scan matching performed by the detection point registration unit 22, there may be cases where the detection point P already registered in the environment map information 26A and the newly acquired detection point P do not match by a predetermined ratio or more. In this case, the detection point registration unit 22 may discard the position information of the newly acquired detection point P and omit registration of the newly acquired detection point P in the environment map information 26A. The predetermined ratio may be predetermined. This process can improve the reliability of the environment map information 26A.

[0069] Furthermore, it is preferable that the detection point registration unit 22 registers various parameters used for calculating the position information of the detection point P in the environment map information 26A together with the position information.

[0070] Return to Figure 3Continuing the explanation. The self-position estimation section 24 estimates self-position information indicating the self-position S of the mobile body 2 based on the respective position information of the plurality of detection points P registered in the environmental map information 26A. Also, the self-position estimation section 24 registers the estimated self-position information of the self-position S in the environmental map information 26A.

[0071] The so-called self-position information is information indicating the posture of the mobile body 2. The posture of the mobile body 2 indicates the position and the inclination of the mobile body 2.

[0072] The self-position estimation section 24 determines the corresponding detection points P included in the respective position information of the detection points P registered in the environmental map information 26A and the respective position information of the latest detection points P acquired from the detection section 14 via the acquisition section 20. The so-called corresponding detection points P refer to the same detection points P in the real space, but the acquisition time of the position information of the detection section 14 is different. The self-position estimation section 24 calculates the self-position information of the self-position S of the mobile body 2 by triangulation using the respective position information of the determined corresponding detection points P. Then, the self-position estimation section 24 registers the calculated self-position information in the environmental map information 26A.

[0073] Therefore, in the environmental map information 26A, the position information of the new detection points P and the self-position information of the self-position S are additionally registered in association with the movement of the mobile body 2. In Figure 4 In the drawing, as one example, the self-positions S of the self-position S1 to the self-position S3 are shown. The larger the value of the number following S is, the closer it means to the self-position S of the current time.

[0074] Further, the self-position estimation section 24 can estimate the self-position information using the odometry method. In this case, the self-position estimation section 24 can estimate new self-position information by integration calculation based on the odometry method using the last calculated self-position information and the movement amount of the mobile body 2. Further, the self-position estimation section 24 can acquire the movement amount by reading the movement amount of the mobile body 2 included in the CAN data acquired from the ECU 3.

[0075] Thus, in the present embodiment, the image processing apparatus 10 simultaneously performs the registration of the position information of the detection points P in the environmental map information 26A and the estimation of the self-position information of the mobile body 2 by SLAM (Simultaneous Localization and Mapping).

[0076] The correction unit 28 corrects the position information of each of the plurality of detection points P registered in the environment map information 26A, as well as the own position information of the mobile object 2. The correction unit 28 corrects the position information registered in the environment map information 26A and the own position information using the position information of each corresponding detection point P acquired at a later acquisition time than the acquisition time of the detection point P.

[0077] Specifically, correction unit 28 uses the position information of the corresponding detection point P re-detected by detection unit 14 to correct the position information of detection point P registered in environment map information 26A. In this case, correction unit 28 may also use various parameters used in calculating the position information of each detection point P registered in environment map information 26A to correct the position information as well as its own position information. Through this correction process, correction unit 28 corrects the error in the position information of detection point P registered in environment map information 26A. Correction unit 28 may use a least squares method, etc., to correct the position information of detection point P registered in environment map information 26A. The correction process performed by correction unit 28 corrects the accumulated error in the position information of detection point P.

[0078] The timing of the correction process performed by correction unit 28 is not limited. For example, correction unit 28 may perform the correction process at predetermined times. The predetermined times may also be determined based on pre-set conditions. Furthermore, in this embodiment, the image processing device 10 is described as including the correction unit 28. However, the image processing device 10 may not include the correction unit 28.

[0079] Next, the determination unit 30 will be described. The determination unit 30 uses the environment map information 26A to determine the projection shape of the projection surface. Specifically, the determination unit 30 uses the position information of the detection point P stored in the environment map information 26A to determine the projection shape of the projection surface.

[0080] The projection plane is a three-dimensional plane for projecting the surrounding image of the moving object 2. The projection shape of the projection plane is a three-dimensional (3D) shape virtually formed in a virtual space corresponding to the real space.

[0081] The so-called peripheral image of the moving object 2 is a captured image of the periphery of the moving object 2. In the present embodiment, the peripheral image of the moving object 2 is a captured image captured by each of the imaging units 12A to 12D.

[0082] In the present embodiment, the decision unit 30 decides, as the projection shape, a shape obtained by deforming the reference projection plane based on the position information of the detection point P registered in the environment map information 26A.

[0083] Figure 5A: is a schematic diagram showing an example of the reference projection plane 40. The reference projection plane 40 is, for example, a projection plane having a shape that serves as a reference when changing the shape of the projection plane. The shape of the reference projection plane 40 is, for example, a bowl shape, a cylinder shape, or the like.

[0084] The so-called bowl-shaped shape has a bottom surface 40A and a side wall surface 40B, one end of the side wall surface 40B is continuous with the bottom surface 40A, and the other end is open. The width of the horizontal section of the side wall surface 40B increases from the bottom surface 40A side toward the open side of the other end. The bottom surface 40A is, for example, circular. Here, the so-called circle is a circular shape other than a perfect circle such as a perfect circle and an elliptical shape. The so-called horizontal section is an orthogonal plane perpendicular to the vertical direction (arrow Z direction). The orthogonal plane is a two-dimensional plane along the arrow X direction perpendicular to the arrow Z direction, and the arrow Y direction perpendicular to the arrow Z direction and the arrow X direction. In the following, the horizontal section and the orthogonal plane are sometimes referred to as the XY plane for explanation. In addition, the bottom surface 40A may also be a shape other than a circle such as an egg shape.

[0085] The so-called cylindrical shape is a shape consisting of a circular bottom surface 40A and a sidewall surface 40B continuous with the bottom surface 40A. Furthermore, the sidewall surface 40B constituting the cylindrical reference projection surface 40 is cylindrical, with an opening at one end continuous with the bottom surface 40A and an opening at the other end. The sidewall surface 40B constituting the cylindrical reference projection surface 40 has a shape with a substantially constant diameter in the XY plane from the bottom surface 40A toward the opening at the other end. Furthermore, the bottom surface 40A may have a shape other than circular, such as an egg shape.

[0086] In this embodiment, a bowl-shaped reference projection plane 40 is used as an example for description. The reference projection plane 40 is a three-dimensional model virtually formed in a virtual space, with a bottom surface 40A substantially aligned with the road surface below the moving object 2 and the center of the bottom surface 40A being the position S of the moving object 2.

[0087] The determination unit 30 determines the projected shape as the shape obtained by deforming the reference projection surface 40 so as to pass through the detection point P closest to the moving object 2. The shape passing through the detection point P means that the deformed side wall surface 40B passes through the detection point P.

[0088] Figure 5B is a schematic diagram showing an example of projection shape 41. The determination unit 30 determines, as projection shape 41, a shape obtained by deforming the reference projection plane 40 into a shape passing through the detection point P closest to the self-position S of the mobile object 2, which is the center of the bottom surface 40A of the reference projection plane 40. This self-position S is the latest self-position S calculated by the self-position estimating unit 24.

[0089] The decision unit 30 specifies the detection point P closest to the self-position S among the plurality of detection points P registered in the environment map information 26A. Specifically, the XY coordinates of the center position (self-position S) of the mobile object 2 are set to (X, Y) = (0, 0). 2 +Y 2 The detection point P having the minimum value of is determined as the detection point P closest to the self-position S. The determination unit 30 determines the shape of the side wall surface 40B of the reference projection surface 40 deformed to pass through the detection point P as the projection shape 41 .

[0090] Specifically, the determining unit 30 determines the deformed shape of a portion of the bottom surface 40A and the side wall surface 40B as a projected shape 41, such that when the reference projection surface 40 is deformed, the portion of the side wall surface 40B becomes a wall surface passing through the detection point P closest to the mobile object 2. The deformed projected shape 41 is, for example, a shape that rises from a rising line 44 on the bottom surface 40A toward the center of the bottom surface 40A. The rising refers to, for example, bending or folding the side wall surface 40B and the portion of the bottom surface 40A toward the center of the bottom surface 40A so that the angle between the side wall surface 40B and the bottom surface 40A of the reference projection surface 40 becomes smaller.

[0091] The determining unit 30 determines to deform the reference projection surface 40 so that the specific region protrudes toward a position passing through the detection point P in the XY plane (as viewed from above). The shape and range of the specific region can also be determined based on a predetermined reference. Furthermore, the determining unit 30 determines to deform the reference projection surface 40 so that the distance from the self-position S continuously increases from the protruding specific region toward the area of ​​the side wall surface 40B outside the specific region.

[0092] In detail, Figure 5B As shown, the projection shape 41 is preferably determined so that the outer periphery of the cross section along the XY plane is a curved shape. In addition, the outer periphery of the cross section of the projection shape 41 is, for example, circular, but may also be a shape other than a circular shape.

[0093] Alternatively, the determination unit 30 may determine, as the projected shape 41, a shape obtained by deforming the reference projection plane 40 so as to conform to an asymptotic curve. The asymptotic curve is an asymptotic curve for a plurality of detection points P. The determination unit 30 generates asymptotic curves for a predetermined number of detection points P, moving away from the detection point P closest to the own position S of the mobile object 2. The number of detection points P can be any number. For example, the number of detection points P is preferably three or more. In this case, the determination unit 30 preferably generates asymptotic curves for a plurality of detection points P located at positions separated by a predetermined angle or more, as viewed from the own position S.

[0094] Figure 6 It is an explanatory diagram of the asymptotic curve Q. Figure 6 This example shows an asymptotic curve Q in a projected image 51, which is a result of projecting a captured image onto a projection surface, when the moving object 2 is viewed from above. For example, assume that the determination unit 30 identifies three detection points P in the order in which they approach the self-position S of the moving object 2. In this case, the determination unit 30 generates an asymptotic curve Q for these three detection points P. The determination unit 30 then determines the shape of the reference projection surface 40 deformed so as to conform to the generated asymptotic curve Q as the projected shape 41.

[0095] Alternatively, the determination unit 30 may divide the area around the self-position S of the mobile object 2 into specific angular ranges, and for each of the angular ranges, determine the detection point P closest to the mobile object 2, or determine a plurality of detection points P in order of proximity to the mobile object 2. Furthermore, the determination unit 30 may determine, as the projection shape 41, a shape obtained by deforming the reference projection plane 40 so as to pass through the determined detection point P or along an asymptotic curve Q of the determined plurality of detection points P, for each of the angular ranges.

[0096] Next, an example of a detailed configuration of the determination unit 30 will be described.

[0097] Figure 7 : is a schematic diagram showing an example of the configuration of the determination unit 30. The determination unit 30 includes an absolute distance conversion unit 30A, an extraction unit 30B, a nearest neighbor identification unit 30C, a reference projection surface shape selection unit 30D, a scale determination unit 30E, an asymptotic curve calculation unit 30F, and a shape determination unit 30G.

[0098] The absolute distance conversion unit 30A reads the environment map information 26A from the storage unit 26. The absolute distance conversion unit 30A converts the position information of each of the plurality of detection points P included in the read environment map information 26A into distance information representing the absolute distance from the current position, i.e., the latest self-position S, of the mobile object 2 to each of the plurality of detection points P. If the detection unit 14 acquires the distance information of the detection points P, the absolute distance conversion unit 30A may be omitted.

[0099] Specifically, the absolute distance conversion unit 30A calculates the current position of the moving object 2 using the speed data of the moving object 2 included in the CAN data received from the ECU 3 of the moving object 2 .

[0100] Specifically, for example, the absolute distance conversion unit 30A calculates the distance between the own position S registered in the environment map information 26A using the speed data of the mobile body 2 included in the CAN data. Figure 4The illustrated environmental map information 26A. In this case, the absolute distance conversion section 30A calculates the distance of the own position S1 from the own position S2 and the distance of the own position S2 from the own position S3 using the speed data included in the CAN data. Furthermore, the absolute distance conversion section 30A calculates the current position of the mobile body 2 using these distances.

[0101] Furthermore, the absolute distance conversion section 30A calculates distance information that is the distance to each of the plurality of detection points P included in the environmental map information 26A, from the current position of the mobile body 2. In detail, the absolute distance conversion section 30A converts each of the position information of the plurality of detection points P included in the environmental map information 26A into distance information from the current position of the mobile body 2. Through this processing, the absolute distance conversion section 30A calculates distance information that is each of the absolute distances of the detection points P.

[0102] Furthermore, the absolute distance conversion section 30A outputs each of the distance information of the plurality of detection points P calculated to the extraction section 30B. In addition, the absolute distance conversion section 30A outputs the current position of the mobile body 2 calculated as the own position information of the mobile body 2 to the virtual viewpoint sight line determination section 34.

[0103] The extraction section 30B extracts a detection point P that exists within a specific range from among the plurality of detection points P for which distance information is received from the absolute distance conversion section 30A. The specific range is, for example, a range from the road surface on which the mobile body 2 is disposed to a height that corresponds to the vehicle height of the mobile body 2. Furthermore, the range is not limited to this range.

[0104] By extracting the detection point P within the range by the extraction section 30B, it is possible to extract, for example, a detection point P of an object that becomes an obstacle to the travel of the mobile body 2.

[0105] Furthermore, the extraction section 30B outputs each of the distance information of the extracted detection points P to the closest determination section 30C.

[0106] The closest determination section 30C divides the surroundings of the own position S of the mobile body 2 by a specific each angle range, and determines, for each angle range, a detection point P that is closest to the mobile body 2 or a plurality of detection points P in order of proximity to the mobile body 2. The closest determination section 30C determines the detection points P using the distance information received from the extraction section 30B. In the present embodiment, a manner in which the closest determination section 30C determines a plurality of detection points P in order of proximity to the mobile body 2 for each angle range will be described as one example.

[0107] The closest determination section 30C outputs the distance information of the detection points P determined for each angle range to the reference projection face shape selection section 30D, the scale determination section 30E, and the asymptotic curve calculation section 30F.

[0108] The reference projection surface shape selection unit 30D selects the shape of the reference projection surface 40. The reference projection surface shape selection unit 30D reads a specific shape from the storage unit 26 that stores a plurality of shapes of the reference projection surface 40, thereby selecting the shape of the reference projection surface 40. For example, the reference projection surface shape selection unit 30D selects the shape of the reference projection surface 40 based on the positional relationship between its own position and the surrounding three-dimensional objects, distance information, etc. In addition, the shape of the reference projection surface 40 can also be selected by the user's operation instructions. The reference projection surface shape selection unit 30D outputs the shape information of the determined reference projection surface 40 to the shape determination unit 30G. In this embodiment, as described above, the method in which the reference projection surface shape selection unit 30D selects a bowl-shaped reference projection surface 40 is explained as an example.

[0109] The scale determination unit 30E determines the scale of the reference projection plane 40 having the shape selected by the reference projection plane shape selection unit 30D. For example, the scale determination unit 30E determines to reduce the scale when there are multiple detection points P within a predetermined distance from the self-position S. The scale determination unit 30E outputs scale information of the determined scale to the shape determination unit 30G.

[0110] The asymptotic curve calculation unit 30F outputs asymptotic curve information of the asymptotic curve Q calculated using each piece of information on the distance of the detection point P closest to the self-position S within each angular range from the self-position S, received from the nearest neighbor determination unit 30C, to the shape determination unit 30G and the virtual viewpoint line of sight determination unit 34. Alternatively, the asymptotic curve calculation unit 30F may calculate the asymptotic curve Q for each of the detection points P stored in a plurality of portions of the reference projection plane 40. Furthermore, the asymptotic curve calculation unit 30F may output the asymptotic curve information of the calculated asymptotic curve Q to the shape determination unit 30G and the virtual viewpoint line of sight determination unit 34.

[0111] The shape determination unit 30G enlarges or reduces the reference projection plane 40, whose shape is represented by the shape information received from the reference projection plane shape selection unit 30D, to the scale of the scale information received from the scale determination unit 30E. Furthermore, the shape determination unit 30G determines, as a projection shape 41, a shape obtained by deforming the enlarged or reduced reference projection plane 40 into a shape based on the asymptotic curve information of the asymptotic curve Q received from the asymptotic curve calculation unit 30F.

[0112] Then, the shape determination unit 30G outputs projection shape information of the determined projection shape 41 to the deformation unit 32 .

[0113] Return to Figure 3Next, the deformation unit 32 will be described. The deformation unit 32 deforms the reference projection surface 40 into a projection shape 41 represented by the projection shape information received from the determination unit 30. Through this deformation process, the deformation unit 32 generates the deformed reference projection surface 40, that is, the deformed projection surface 42 (see Figure 5B ).

[0114] That is, the deformation unit 32 deforms the reference projection plane 40 using the position information of the detection point P stored in the environment map information 26A and the own position information of the moving object 2 .

[0115] Specifically, for example, the deformation unit 32 deforms the reference projection plane 40 into a curved surface shape passing through the detection point P closest to the moving object 2 based on the projection shape information. Through this deformation process, the deformation unit 32 generates a deformed projection plane 42 .

[0116] Furthermore, for example, the deformation unit 32 deforms the reference projection plane 40 into a shape along an asymptotic curve Q of a plurality of detection points P whose number is predetermined in order of approaching the moving object 2 based on the projection shape information.

[0117] Furthermore, it is preferable that the deformation unit 32 deforms the reference projection plane 40 using the position information of the detection point P and the own position information of the own position S acquired before the first time.

[0118] Here, the term "first moment" refers to the most recent moment at which the detection unit 14 detected the position information of the detection point P, or any moment before that most recent moment. For example, the detection point P acquired before the first moment includes the position information of a specific object located in the vicinity of the mobile object 2, while the detection point P acquired at the first moment does not include the position information of the specific object located in the vicinity. The determination unit 30 can use the position information of the detection point P acquired before the first moment, included in the environment map information 26A, to determine the projection shape 41 in the same manner as described above. Furthermore, the deformation unit 32 can use the projection shape information of the projection shape 41 to generate the deformed projection surface 42 in the same manner as described above.

[0119] In this case, for example, even if the position information of the detection point P detected by the detection unit 14 at the first moment does not include the position information of the detection point P detected before that moment, the deformation unit 32 can generate a deformation projection surface 42 corresponding to the detection point P detected in the past.

[0120] Next, the projection conversion unit 36 ​​will be described. The projection conversion unit 36 ​​generates a projection image 51 by projecting the captured image acquired from the imaging unit 12 onto the deformed projection plane 42 , that is, the reference projection plane 40 deformed by the deforming unit 32 .

[0121] Specifically, the projection conversion unit 36 ​​receives deformed projection surface information of the deformed projection surface 42 from the deformation unit 32. The deformed projection surface information is information indicating the deformed projection surface 42. The projection conversion unit 36 ​​projects the captured image acquired from the imaging unit 12 via the acquisition unit 20 onto the deformed projection surface 42 indicated by the received deformed projection surface information. Through this projection processing, the projection conversion unit 36 ​​generates a projected image 51.

[0122] The projection conversion unit 36 ​​converts the projected image 51 into a virtual viewpoint image. The virtual viewpoint image is an image that allows the projected image 51 to be visually recognized from a virtual viewpoint in an arbitrary direction.

[0123] use Figure 5B To explain. The projection conversion unit 36 ​​projects the captured image 50 onto the deformed projection surface 42. Moreover, the projection conversion unit 36 ​​generates a virtual viewpoint image, which is an image (not shown) of the captured image 50 projected onto the deformed projection surface 42 visually recognized from an arbitrary virtual viewpoint O in the line of sight direction L. The position of the virtual viewpoint O can be set to, for example, the latest self-position S of the moving body 2. In this case, the value of the XY coordinate of the virtual viewpoint O can be set to the value of the XY coordinate of the latest self-position S of the moving body 2. In addition, the value of the Z coordinate (position in the vertical direction) of the virtual viewpoint O can be set to the value of the Z coordinate of the detection point P closest to the self-position S of the moving body 2. The line of sight direction L can also be determined based on a predetermined reference, for example.

[0124] The line of sight direction L can be, for example, a direction from the virtual viewpoint O toward the detection point P closest to the own position S of the mobile object 2. Alternatively, the line of sight direction L can be a direction passing through the detection point P and perpendicular to the deformed projection plane 42. Virtual viewpoint line of sight information indicating the virtual viewpoint O and the line of sight direction L is generated by the virtual viewpoint line of sight determination unit 34.

[0125] Return to Figure 3 The virtual viewpoint sight line determination unit 34 determines virtual viewpoint sight line information.

[0126] For example, the virtual viewpoint line of sight determination unit 34 may also determine the direction passing through the detection point P closest to the own position S of the moving body 2 and perpendicular to the deformation projection surface 42 as the line of sight direction L. In addition, the virtual viewpoint line of sight determination unit 34 may also fix the direction of the line of sight direction L and determine the coordinates of the virtual viewpoint O as arbitrary Z coordinates and arbitrary XY coordinates in the direction away from the asymptotic curve Q toward the own position S. In this case, the XY coordinates may also be the coordinates of a position farther away from the asymptotic curve Q than the own position S. Moreover, the virtual viewpoint line of sight determination unit 34 outputs the virtual viewpoint line of sight information representing the virtual viewpoint O and the line of sight direction L to the projection conversion unit 36. In addition, as Figure 6As shown, the sight line direction L may be a direction from the virtual viewpoint O toward the vertex W of the asymptotic curve Q.

[0127] Return to Figure 3 Continuing the description, the projection conversion unit 36 ​​receives virtual viewpoint line of sight information from the virtual viewpoint line of sight determination unit 34. Upon receiving this virtual viewpoint line of sight information, the projection conversion unit 36 ​​determines a virtual viewpoint O and a line of sight direction L. Furthermore, the projection conversion unit 36 ​​generates a virtual viewpoint image, which is an image visually recognized from the virtual viewpoint O in the line of sight direction L, based on the captured image 50 projected onto the deformed projection surface 42. The projection conversion unit 36 ​​outputs the virtual viewpoint image to the image synthesis unit 38.

[0128] The image synthesis unit 38 generates a composite image by extracting part or all of the virtual viewpoint image. For example, the image synthesis unit 38 determines the width of the overlapping portion of the multiple captured images 50 included in the virtual viewpoint image, performs a process of gluing the captured images 50, and determines the mixed image 50 to be displayed in the overlapping portion. This generates a composite image 54. The image synthesis unit 38 then outputs the composite image 54 to the display unit 16. Furthermore, the composite image 54 can be a bird's-eye view image with the top of the moving object 2 as the virtual viewpoint O, or a semi-transparent image of the moving object 2 with the interior of the moving object 2 as the virtual viewpoint O.

[0129] Next, an example of the flow of image processing executed by the image processing apparatus 10 will be described.

[0130] Figure 8 This is a flowchart showing an example of the flow of image processing executed by the image processing apparatus 10 .

[0131] The acquisition unit 20 acquires the captured image 50 from the imaging unit 12 (step S100 ). Furthermore, the acquisition unit 20 acquires position information of each of the plurality of detection points P from the detection unit 14 (step S102 ).

[0132] The detection point registration unit 22 uses scan matching to identify the same detection point P registered in the environment map information 26A among the detection points P whose position information was acquired in step S102 (step S104). Furthermore, the detection point registration unit 22 registers the position information of the detection point P acquired in step S102, which is not registered in the environment map information 26A, in the environment map information 26A (step S106).

[0133] The own position estimating unit 24 estimates own position information indicating the own position S of the mobile object 2 based on the position information of each of the plurality of detection points P registered in the environment map information 26A and the position information acquired in step S102 (step S108). The own position estimating unit 24 then registers the estimated own position information in the environment map information 26A (step S110).

[0134] The correction unit 28 corrects the position information of the detection point P registered in the environment map information 26A using the position information of the detection point P acquired in step S102 (step S112). As described above, the correction process of step S112 may not be performed.

[0135] The absolute distance conversion unit 30A of the determination unit 30 converts the position information of each of the plurality of detection points P included in the environment map information 26A into distance information of the absolute distance from the current position of the mobile body 2 to each of the plurality of detection points P (step S114 ).

[0136] The extraction unit 30B extracts the detection points P existing within a specific range from among the detection points P whose distance information has been calculated by the absolute distance conversion unit 30A (step S116 ).

[0137] The closest identifying unit 30C identifies a plurality of detection points P in order of proximity to the moving object 2 for each angular range around the moving object 2 using the distance information of each detection point P extracted in step S116 (step S118 ).

[0138] The asymptotic curve calculation unit 30F calculates the asymptotic curve Q using each piece of distance information of the plurality of detection points P for each angle range determined in step S118 (step S120 ).

[0139] The reference projection surface shape selection unit 30D selects the shape of the reference projection surface 40 (step S122). As described above, the method in which the reference projection surface shape selection unit 30D selects the bowl-shaped reference projection surface 40 is described as an example.

[0140] The scale determination unit 30E determines the scale of the reference projection plane 40 of the shape selected in step S122 (step S124 ).

[0141] The shape determination unit 30G enlarges or reduces the reference projection plane 40 of the shape selected in step S122 to the scale determined in step S124. Furthermore, the shape determination unit 30G deforms the enlarged or reduced reference projection plane 40 so that it becomes a shape that follows the asymptotic curve Q calculated in step S120. The shape determination unit 30G determines this deformed shape as the projected shape 41 (step S126).

[0142] The deformation section 32 deforms the reference projection surface 40 into the projection shape 41 decided in the decision section 30 (step S128). Through this deformation processing, the deformation section 32 generates the deformed reference projection surface 40, that is, the deformation projection surface 42 (refer to FIG. 6B). Figure 5B

[0143] The virtual viewpoint sight line decision section 34 decides the virtual viewpoint sight line information (step S130). For example, the virtual viewpoint sight line decision section 34 sets the own position S of the mobile body 2 as the virtual viewpoint O, and decides the direction from the virtual viewpoint O toward the position of the vertex W of the asymptote curve Q as the sight line direction L. In detail, the virtual viewpoint sight line decision section 34 can decide the direction toward the vertex W of the asymptote curve Q of the specific one of the asymptote curves Q calculated for each of the angle ranges in step S120 as the sight line direction L.

[0144] The projection conversion section 36 projects the captured image 50 acquired in step S100 onto the deformation projection surface 42 generated in step S128. Further, the projection conversion section 36 converts this projection image 51 into a virtual viewpoint image, which is an image of the captured image 50 projected onto the deformation projection surface 42 visually recognized in the sight line direction L from the virtual viewpoint O decided in step S130 (step S132).

[0145] The image synthesis section 38 generates a synthesized image 54 obtained by extracting a part or all of the virtual viewpoint image generated in step S132 (step S134). For example, the image synthesis section 38 performs decision of the width of the overlapping part of the plurality of captured images 50 included in the virtual viewpoint image, a bonding process of the captured images 50, and a mixing process of deciding the captured images 50 displayed in the overlapping part. Further, the image synthesis section 38 performs display control of outputting the generated synthesized image 54 to the display section 16 (step S136).

[0146] Next, the image processing apparatus 10 judges whether or not to end the image processing (step S138). For example, the image processing apparatus 10 performs the judgment of step S138 by discriminating whether or not a signal indicating the stop of the position movement of the mobile body 2 is received from the ECU 3. Alternatively, for example, the image processing apparatus 10 can perform the judgment of step S138 by discriminating whether or not an instruction of the end of the image processing is accepted based on the user's operation instruction or the like.

[0147] ​If the determination in step S138 is negative (step S138: No), the routine returns to step S100. If the determination in step S138 is positive (step S138: Yes), the routine ends. Furthermore, when returning from step S138 to step S100 after executing the correction process in step S112, the subsequent correction process in step S112 may be omitted. Furthermore, when returning from step S138 to step S100 without executing the correction process in step S112, the subsequent correction process in step S112 may be executed.

[0148] Next, a specific example of image processing executed by the image processing device 10 according to this embodiment will be described using a timing chart.

[0149] Figure 9A 1 and 2 are diagrams showing examples of captured images 50 captured at respective times of conditions T1 to T4. Figure 9A Each of the captured images 50 is, for example, an image captured by the imaging unit 12D of the mobile object 2. Situations T1 through T4 assume situations in which the mobile object 2 is parked in a parking space near the pillar C. Furthermore, a larger number after T indicates a later time. In other words, situations T1 through T4 represent situations at later times. Captured images 50T1 through 50T4 are examples of captured images 50 captured at respective times in situations T1 through T4.

[0150] At the time of situation T1, the pillar C existing in the real space is not detected by the detection unit 14. That is, at the time of situation T1, the pillar C is outside the detection area F of the detection unit 14. Furthermore, at the times of situation T2 and T3, the pillar C existing in the real space is within the detection area F of the detection unit 14. At the time of situation T4, the pillar C existing in the real space is outside the detection area F of the detection unit 14.

[0151] Figure 9B Schematic diagram showing an example of the composite image 54 outputted by the image processing device 10 for each time point of situations T1 to T4. The composite images 54T1 to 54T4 are examples of the composite images 54 corresponding to each time point of situations T1 to T4, respectively.

[0152] Figure 9C 1 is a schematic diagram showing an example of a time chart of image processing by the image processing device 10 .

[0153] exist Figure 9CIn the figure, "DR" indicates the reading process of the environment map information 26A. "SM" indicates the scan matching process by the detection point registration unit 22. "MW" indicates the registration process of the position information of the detection point registration unit 22 into the environment map information 26A. "PO" indicates the estimation process of the own position information of the mobile object 2 by the own position estimation unit 24. "PW" indicates the registration of the own position information of the mobile object 2 into the environment map information 26A by the own position estimation unit 24. "DW" indicates the updating process of the position information of the environment map information 26A by the correction unit 28.

[0154] In addition, Figure 9C Among them, the "detection point registration process" is a process for registering the position information of the detection point P in the environment map information 26A by the detection point registration unit 22. The "own position estimation process" is a process for estimating the own position information of the mobile object 2 by the own position estimation unit 24 and registering it in the environment map information 26A. The "correction process" is a process for correcting the position information by the correction unit 28. The "display control process" is a process that includes acquisition of the captured image 50 by the acquisition unit 20, determination of the projection shape 41 by the determination unit 30 (projection shape determination), deformation of the reference projection plane 40 by the deformation unit 32 (projection shape conversion), generation of a virtual viewpoint image by the projection conversion unit 36 ​​(projection conversion), generation of a composite image 54 by the image synthesis unit 38 (composite image generation), and display control of the composite image 54 by the image synthesis unit 38.

[0155] also, Figure 9C The illustrated situation T5 is a situation at a time point after the situation T4 , and assumes a situation in which the pillar C existing in the real space is outside the detection area of ​​the detection unit 14 .

[0156] like Figure 9C As shown, at each time point in situations T1 through T5, processing is performed using the captured images 50 (captured images 50T1 through 50T4) for each situation and the position information of the detection points P. Therefore, the position information of each of the plurality of detection points P detected at each time point is sequentially added and registered in the environment map information 26A. Furthermore, the estimated own position information is sequentially registered in the environment map information 26A.

[0157] Furthermore, the image processing device 10 executes the above-described display control process at each time point of the status T1 to the status T5 using the position information of the detection point P and the own position information of the own position S registered in the environment map information 26A.

[0158] That is, the environment map information 26A sequentially stores the position information of the detection points P detected once. The image processing device 10 then deforms the reference projection plane 40 using the environment map information 26A to execute the above-described display control process.

[0159] Therefore, if Figure 9A and Figure 9B As shown, in situations T2 and T3, where a pillar C in real space is within the detection area F of the detection unit 14, the image processing device 10 outputs a composite image 54T2 and a composite image 54T3, respectively. Composite images 54T2 and 54T3 include an image C' of the pillar C. Furthermore, even in situation T4, where a pillar C in real space is outside the detection area F of the detection unit 14, the image processing device 10 can output a composite image 54T4 in which the shape of the projection surface is deformed based on the position information of the pillar C. This is because the image processing device 10 of this embodiment generates the composite image 54 using the deformed projection surface 42, which is a projection surface deformed using the position information of the detection points P around the mobile object 2.

[0160] As described above, the deformation unit 32 of the image processing device 10 of this embodiment uses the position information of multiple detection points P around the moving body 2 and the own position information of the moving body 2 to deform the projection surface of the surrounding image of the moving body 2, that is, the reference projection surface 40 of a predetermined shape.

[0161] Sometimes, due to the movement of the mobile object 2 or the like, a portion of an object surrounding the mobile object 2 may be located outside the detection area F of the detection point P. However, the image processing device 10 of this embodiment deforms the reference projection plane 40 using the stored positional information of the detection points P surrounding the mobile object 2. Therefore, the image processing device 10 can deform the reference projection plane 40 in accordance with the objects surrounding the mobile object 2.

[0162] On the other hand, in the conventional technology, it is sometimes difficult to deform the projection plane (reference projection plane 40 ) according to the object.

[0163] Figure 10A 、 Figure 10B 、 Figure 11A as well as Figure 11B This is an explanatory diagram of an example of conventional deformation processing of a projection surface. Figure 10A and Figure 10B FIG. 1 is a schematic diagram showing an example of the positional relationship between the detection area F of the detection unit 14 and surrounding objects when the moving body 2 moves in the direction of arrow B. Figure 10A and Figure 10B In FIG, pillar C is shown as an example of a surrounding object.

[0164] For example, in Figure 10A In the case of the positional relationship shown in FIG, there is a column C in the detection area F of the detection unit 14. In this case, in the prior art, the reference projection plane 40 is deformed according to the detection result of the detection area F, so that, for example, Figure 11A The composite image 530A is shown. The composite image 530A is an example of a conventional composite image 530. The composite image 530A includes an image C' in which the column C is captured.

[0165] However, if Figure 10B As shown in FIG. 1 , as the moving body 2 moves in the direction of arrow B, the pillar C may sometimes be outside the detection area F of the detection unit 14. In this case, the pillar C does not exist in the detection area F, so in the conventional art, the reference projection surface 40 cannot be deformed according to the pillar C. Therefore, in the conventional art, for example, the output Figure 11B The composite image 530B is shown. The composite image 530B is an example of a conventional composite image 530. The composite image 530B does not include the image C' of the pillar C. In addition, the reference projection plane 40 is not deformed by the pillar C, so the area CA' corresponding to the pillar C in the composite image 530B is a deformed image.

[0166] As described above, in the conventional technology, it is sometimes difficult to deform the projection plane (reference projection plane 40 ) according to the distance from the actual object.

[0167] On the other hand, the image processing device 10 of the present embodiment deforms the reference projection plane 40 using the position information of the detection points P stored around the moving object 2 and the position information of the moving object 2 itself.

[0168] Figure 12A 、 Figure 12B 、 Figure 13A as well as Figure 13B This is an explanatory diagram of deformation of the reference projection plane 40 by the image processing device 10 according to the present embodiment. Figure 12A and Figure 12B Schematic diagram showing an example of the environment map information 26A. Figure 12A Yes Figure 10A This is a diagram showing an example of environment map information 26A when the positional relationship is . Figure 12B Yes Figure 10B This is a diagram showing an example of environment map information 26A when the positional relationship is .

[0169] For example, in Figure 10A In the case of the positional relationship shown in FIG, the column C exists in the detection area F of the detection unit 14. In addition, the position information of the newly detected detection point P is additionally registered in the environmental map information 26A. Figure 10AWhen the positional relationship is shown, a plurality of detection points P of the pillar C are registered in the environment map information 26A (see Figure 12A ).

[0170] Furthermore, the image processing apparatus 10 of this embodiment deforms the reference projection plane 40 using the environment map information 26A storing the position information of the detection point P. Therefore, the image processing apparatus 10 of this embodiment deforms the reference projection plane 40. Figure 10A When the positional relationship shown is Figure 13A The composite image 54A is shown. The composite image 54A is an example of the composite image 54 of the present embodiment. The composite image 54A includes an image C' in which the column C is captured.

[0171] Furthermore, it is assumed that the pillar C becomes outside the detection area F due to the moving body 2 moving in the direction of arrow B (see Figure 10B ). In the environment map information 26A, the position information of the newly detected detection point P is additionally registered in the state where the position information of the detected detection point P has already been registered. Figure 10B When the positional relationship is shown, the detection point P of the column C outside the detection area F is also registered in the environment map information 26A (see Figure 12B ).

[0172] Furthermore, the image processing device 10 of this embodiment deforms the reference projection plane 40 using the environment map information 26A that stores the position information of the detection point P. Specifically, even when pillar C is outside the detection area F, the image processing device 10 can calculate the positional relationship (distance information) between the mobile object 2 and the detection point P using the stored position information of the detection point P and the estimated self-position information of the self-position S. Furthermore, even when pillar C is outside the detection area F, the image processing device 10 can estimate the self-position information of the mobile object 2 using newly detected detection points P of other objects (e.g., detection points P on a wall D, etc.). Furthermore, by using the calculated distance information and self-position information, the image processing device 10 can deform the reference projection plane 40 to appropriately display pillar C outside the detection area F.

[0173] Therefore, the image processing device 10 of this embodiment Figure 10B When the positional relationship shown is Figure 13B The composite image 54B shown is an example of the composite image 54 of this embodiment. The composite image 54B includes an image C' of the column C. In addition, the reference projection surface 40 is deformed according to the column C, so it is different from the conventional technology (see Figure 11B ) compared to the composite image 54B, the distortion included in the region corresponding to the column C is suppressed.

[0174] Therefore, the image processing device 10 of this embodiment can deform the reference projection plane 40 , which is the projection plane, according to the object.

[0175] Furthermore, even when an object exists outside the detection area F, the image processing device 10 of this embodiment can deform the reference projection plane 40 in accordance with the object.

[0176] As described above, in the image processing device 10 of this embodiment, the detection point registration unit 22 additionally registers the position information of the detection points P acquired from the detection unit 14 that is not registered in the environment map information 26A into the environment map information 26A. Furthermore, the absolute distance conversion unit 30A of the determination unit 30 converts the position information of each of the plurality of detection points P included in the environment map information 26A into distance information representing the absolute distance from the current position of the mobile object 2. Through this processing, the absolute distance conversion unit 30A calculates distance information representing the absolute distance of each of the detection points P.

[0177] Therefore, the distance information from the past self-position S to the detection point P according to the movement of the mobile body 2 is suppressed from becoming inconsistent with the distance information from the current self-position S to the detection point P. Therefore, the image processing device 10 of this embodiment deforms the reference projection plane 40 using the distance information calculated by the absolute distance conversion unit 30A, thereby being able to deform the reference projection plane 40 according to the object.

[0178] Furthermore, as described above, the determination unit 30 can determine, as the projection shape 41, a shape obtained by deforming the reference projection plane 40 so as to form a curved surface shape passing through the detection point P closest to the mobile object 2. In other words, the determination unit 30 causes a specific region of the reference projection plane 40, including a point where the detection point P is projected in a direction along the XY plane, to protrude to a position passing through the detection point P. Furthermore, the determination unit 30 can determine the shape obtained by deforming the reference projection plane 40 so that the distance from the self-position S continuously increases from the protruding specific region toward the region of the side wall surface 40B other than the specific region.

[0179] When the projection shape 41 is set to a shape that only causes the specific area to protrude relative to the side wall surface 40B of the reference projection plane 40, a synthetic image 54 may be generated in which the boundary between the specific area and areas other than the specific area is unnaturally discontinuous. Therefore, the determination unit 30 determines the shape that deforms the reference projection plane 40 so as to become a curved surface shape passing through the detection point P as the projection shape 41, thereby providing a natural-looking synthetic image 54.

[0180] As described above, the virtual viewpoint line of sight determining unit 34 of the image processing device 10 of this embodiment can determine the self-position S of the mobile object 2 as the virtual viewpoint O, and determine the direction perpendicular to the deformation projection plane 42, which passes through the detection point P closest to the self-position S of the mobile object 2, as the line of sight direction L. Furthermore, the projection conversion unit 36 ​​converts the projected image 51 into a virtual viewpoint image that is visually recognized from the virtual viewpoint O in the line of sight direction L.

[0181] Here, it is assumed that the downstream end of the line of sight L is fixed at the detection point P closest to the position S of the moving object 2, and the angle of the line of sight L relative to the deformed projection plane 42 is variable. In this case, the rising line 44 (see Figure 5B ) area is deformed.

[0182] Figures 14A to 14C This is an explanatory diagram of an example of determining the line of sight direction L.

[0183] Figure 14A : This is a schematic diagram showing an example of a composite image 54E in which the line of sight L, which is the direction from the virtual viewpoint O toward the detection point P closest to the moving object 2, is perpendicular to the deformed projection surface 42. The composite image 54E is an example of the composite image 54. The composite image 54E includes a line 45 that is a picture of a straight line existing in the real space. In this case, Figure 14A As shown, in the composite image 54E, the rising line 44 (also see Figure 5B ) does not include distortion. Therefore, the line 45 in the synthesized image 54E, which is a picture of the straight line existing in the real space, becomes a straight line similar to the straight line existing in the real space.

[0184] Figure 14B This is a schematic diagram showing an example of the relationship between the line of sight direction L and the deformed projection plane 42 when the line of sight direction L and the deformed projection plane 42 become an angle smaller than 90° (for example, 45° or less) due to the movement of the movable body 2 or the like. Figure 14C It means setting the virtual viewpoint O and the sight direction L to Figure 14B . Synthetic image 54F is an example of a synthetic image 54. In this case, the line 45 in the synthetic image 54F, which is a picture of a straight line existing in the real space, becomes a line that is deformed with the region corresponding to the rising line 44 of the deformed projection surface 42 as the boundary.

[0185] Therefore, it is preferable that the virtual viewpoint sight line determination unit 34 of this embodiment determines the own position S of the mobile object 2 as the virtual viewpoint O, and determines the direction that passes through the detection point P closest to the own position S of the mobile object 2 and is perpendicular to the deformation projection plane 42 as the sight line direction L. Furthermore, it is preferable that the projection conversion unit 36 ​​converts the projected image 51 into an image that is visually recognized from the virtual viewpoint O in the sight line direction L, that is, a virtual viewpoint image.

[0186] In this case, even when the moving body 2 moves, the sight line direction L is maintained in a direction perpendicular to the deformation projection plane 42 (see FIG. Figure 5B ). Therefore, if Figure 14A As shown, the synthesized image 54E does not include deformation in the region corresponding to the rising line 44 of the deformed projection surface 42. Therefore, the line 45 in the synthesized image 54E, which is a reflection of the straight line existing in the real space, is a straight line similar to the straight line existing in the real space.

[0187] Therefore, the image processing device 10 of this embodiment can achieve improved accuracy of the synthesized image 54 in addition to the above-mentioned effects.

[0188] Here, objects such as pillars C may be included in different directions around the mobile body 2. In such a situation, if the direction from the virtual viewpoint O toward the nearest detection point P is defined as the line of sight direction L, the result may be a synthetic image 54 that does not include at least one of these objects existing in the real space.

[0189] Figure 15A is a diagram showing an example of a captured image 50H. The captured image 50H is an example of the captured image 50. Assume that the captured image 50H includes a plurality of objects C1 and C2.

[0190] In this case, if the direction toward the detection point P located closest to the moving object 2 is defined as the line of sight direction L, the output may be Figure 15B Composite image 54H is shown. Figure 15B Schematic diagram showing an example of a composite image 54H. The composite image 54H is an example of the composite image 54.

[0191] like Figure 15B As shown, the synthesized image 54 includes the image C2 ′ of the object C2 but does not include the image C1 ′ of the object C1 .

[0192] Therefore, the determination unit 30 can determine, as the projected shape 41, a shape that is deformed into a shape along the asymptotic curve Q of a plurality of detection points P, a number of which is predetermined in the order of approaching the own position S of the mobile object 2. Furthermore, the virtual viewpoint line of sight determination unit 34 can set the own position S of the mobile object 2 as the virtual viewpoint O, and determine the direction from the virtual viewpoint O toward the position of the vertex W of the asymptotic curve Q as the line of sight direction L.

[0193] By setting the sight line direction L to the direction of the vertex W, as Figure 6 As shown, the projected image 51 and the synthesized image 54 are images C1' and C2' of a plurality of objects. Therefore, the image processing device 10 of this embodiment can improve the accuracy of the synthesized image 54 in addition to the above-mentioned effects.

[0194] Furthermore, as described above, in this embodiment, based on the scan matching results of the detection point registration unit 22, if the position information of a newly acquired detection point P and a detection point already registered in the environment map information 26A do not match by a predetermined ratio or more, the position information of the newly acquired detection point P can be discarded. Therefore, in addition to the aforementioned effects, the image processing device 10 of this embodiment can also improve the reliability of the environment map information 26A. Furthermore, the image processing device 10 can suppress temporal fluctuations in the deformed projection plane 42, thereby stabilizing the synthesized image 54.

[0195] (Second embodiment)

[0196] In the first embodiment, the detection unit 14 detects the position information of the detection point P. In this embodiment, the position information of the detection point P is acquired from the captured image 50 by the imaging unit 12.

[0197] Figure 16 This diagram illustrates an example of the functional configuration of an image processing device 11 according to this embodiment. Similar to the image processing device 10 according to the first embodiment, the image processing device 11 is connected to an imaging unit 12 and a display unit 16 so that data or signals can be transmitted and received. The imaging unit 12 and the display unit 16 are similar to those in the first embodiment. Furthermore, in this embodiment, the imaging unit 12 is described as a single-lens reflex camera.

[0198] exist Figure 16 In order to clarify the input and output relationship of data, in addition to the image processing device 11, the imaging unit 12 and the display unit 16 are also shown.

[0199] The image processing device 11 includes: an acquisition unit 21, a selection unit 23, a matching unit 25, a self-position estimation unit 27, a detection point registration unit 29, a storage unit 26, a correction unit 28, a determination unit 30, a deformation unit 32, a virtual viewpoint line of sight determination unit 34, a projection conversion unit 36 ​​and an image synthesis unit 38.

[0200] Part or all of the above-mentioned parts may be realized by, for example, causing a processing device such as CPU 10A to execute a program, i.e., software. In addition, part or all of the above-mentioned parts may be realized by hardware such as IC, or by using both software and hardware.

[0201] The storage unit 26, correction unit 28, determination unit 30, deformation unit 32, virtual viewpoint line of sight determination unit 34, projection conversion unit 36, and image synthesis unit 38 are the same as those in the first embodiment. The storage unit 26 stores environment map information 26A. The environment map information 26A is the same as that in the first embodiment.

[0202] The acquisition unit 21 acquires the captured image 50 from the imaging unit 12. The acquisition unit 21 acquires the captured image 50 from each imaging unit 12 (the imaging unit 12A to the imaging unit 12D).

[0203] Every time the acquisition unit 21 acquires the captured image 50 , it outputs the acquired captured image 50 to the projection conversion unit 36 ​​and the selection unit 23 .

[0204] The selection unit 23 selects a detection area of ​​the detection point P. In the present embodiment, the selection unit 23 selects a detection area by selecting any one of the plurality of imaging units 12 (imaging units 12A to 12D).

[0205] In the present embodiment, the selection unit 23 selects any one of the imaging units 12 using vehicle state information or detection direction information included in CAN data received from the ECU 3 or instruction information input by a user's operation instruction.

[0206] Vehicle status information includes, for example, information indicating the direction of travel of vehicle 2, the status of the direction indicator of vehicle 2, and the status of the gears of vehicle 2. Vehicle status information can be derived from CAN data. Detection direction information indicates the direction in which information of interest is detected and can be derived using POI (Point of Interest) technology. Indication information indicates the direction of interest and is input via user operation.

[0207] For example, the selection unit 23 uses the vehicle state information to select the direction of the detection area. Specifically, the selection unit 23 uses the vehicle state information to determine parking information such as rear parking information indicating rear parking of the mobile body 2 and side parking information indicating side parking. The selection unit 23 associates the parking information with the identification information of any of the imaging units 12 and stores it in advance. For example, the selection unit 23 stores the imaging unit 12D (refer to the image capturing unit 12D) that captures the rear of the mobile body 2. Figure 1 ) is stored in advance in correspondence with the rear parking information. In addition, the selection unit 23 selects the imaging unit 12B and the imaging unit 12C (refer to Figure 1 ) is pre-stored in correspondence with the identification information of each shooting unit of the vehicle and the parallel parking information.

[0208] Then, the selection unit 23 selects the direction of the detection area by selecting the imaging unit 12 corresponding to the parking information derived from the received vehicle state information.

[0209] Alternatively, the selection unit 23 may select the imaging unit 12 that captures the direction indicated by the detection direction information as the imaging area E. Alternatively, the selection unit 23 may select the imaging unit 12 that captures the direction indicated by the detection direction information derived using the POI technology as the imaging area E.

[0210] The selection unit 23 outputs the captured image 50 captured by the selected imaging unit 12 among the captured images 50 acquired by the acquisition unit 21 to the matching unit 25 .

[0211] The matching unit 25 extracts features from multiple captured images 50 captured at different times and performs matching between the images. Specifically, the matching unit 25 extracts features from the multiple captured images 50. Furthermore, the matching unit 25 uses the features between the multiple captured images 50 captured at different times to identify corresponding points between the multiple captured images 50. The matching unit 25 then outputs the matching results to the self-position estimating unit 27. Furthermore, the matching unit 25 registers information on the identified corresponding points between the multiple captured images 50 in the environment map information 26A.

[0212] The self-position estimating unit 27 estimates the self-position information of the mobile object 2 by triangulation using the information of the corresponding points included in the matching processing results obtained from the matching unit 25. The self-position information includes, for example, information such as the position of the mobile object 2 corresponding to each of the different imaging times and the orientation of the imaging unit 12.

[0213] Then, the self-position estimating unit 27 registers the calculated self-position information in the environment map information 26A.

[0214] The detection point registration unit 29 uses the self-position information of the mobile object 2 corresponding to each of the different capture times, estimated by the self-position estimation unit 27, to determine the amount of movement (translation and rotation) of the mobile object 2. Furthermore, based on this amount of movement, the detection point registration unit 29 determines the relative coordinates of corresponding points between the multiple captured images 50 determined by the matching unit 25, relative to the self-position of the mobile object 2. The detection point registration unit 29 then registers these coordinates as the coordinates of the detection point P in the environment map information 26A. The coordinates of the detection point P registered in the environment map information 26A may also be converted to coordinates with a predetermined position as the origin.

[0215] As described above, in this embodiment, the image processing device 11 estimates the position information of the detection point P and the own position information of the moving object 2 simultaneously based on the captured image 50 captured by the imaging unit 12 by Visual SLAM (Simultaneous Localization and Mapping).

[0216] Furthermore, the timing of determining the projection shape 41 by the determination unit 30 is not limited. For example, in the present embodiment, the determination unit 30 may determine the projection shape 41 using the captured image 50 obtained at a time when the size of the subject included in the detection area selected by the selection unit 23 further increases due to the movement of the mobile object 2.

[0217] Next, an example of the flow of image processing executed by the image processing device 11 will be described.

[0218] Figure 17 This is a flowchart showing an example of the flow of image processing executed by the image processing device 11 .

[0219] The acquisition unit 21 acquires the captured image 50 from the imaging unit 12 (step S200 ).

[0220] The selection unit 23 selects any one of the plurality of imaging units 12 (imaging units 12A to 12D) (step S202 ).

[0221] The matching unit 25 uses the multiple captured images 50 acquired in step S200 and captured at different times by the capturing unit 12 selected in step S202 to extract features and perform matching processing (step S204). Furthermore, the matching unit 25 registers information on corresponding points between the multiple captured images 50 at different times, as determined by the matching processing, in the environment map information 26A.

[0222] The own position estimation section 27 reads the environmental map information 26A (step S206), and using the information of the points determined to be corresponding points registered in the environmental map information 26A, estimates the own position information of the mobile body 2 corresponding to each of the different photographing times by triangulation (step S208).

[0223] Then, the own position estimation section 27 registers the calculated own position information in the environmental map information 26A (step S210).

[0224] The detected point registration section 29 reads the environmental map information 26A (step S212), and using the own position information of the mobile body 2 corresponding to each of the different photographing times, calculates the movement amount (translation amount and rotation amount) of the mobile body 2. Further, from the movement amount, the relative coordinates of the corresponding points between the plurality of photographing images 50 of different photographing times determined by the matching process of step S204 with respect to the own position of the mobile body 2 are calculated. Then, the detected point registration section 29 registers the coordinates as the coordinates of the detected point P in the environmental map information 26A (step S214). Further, the coordinates of the detected point P registered in the environmental map information 26A can be coordinates after being converted to coordinates with a prescribed position as the origin.

[0225] Further, the image processing apparatus 11 executes the processes of steps S216 to S242 similarly to steps S112 to S138 of the first embodiment, and ends this routine. Further, in the case of returning from step S242 to step S200 after the correction process of step S216 is executed, the correction process of step S216 after that can be omitted at times. In addition, in the case of returning from step S242 to step S200 without executing the correction process of step S216, the correction process of step S216 after that can be executed at times.

[0226] Figure 18 is a schematic view showing one example of a time chart based on the image processing of the image processing apparatus 11.

[0227] In Figure 18In the figure, "CI" means acquisition of a captured image. "RI" means acquisition of instruction information input by a user's operation instruction. "RC" means acquisition of CAN data by the acquisition unit 21. "SC" means selection processing by the imaging unit 12 by the selection unit 23. "FM" means feature extraction processing and matching processing by the matching unit 25. "RD" means reading processing of the environment map information 26A. "PO" means estimation processing of the own position information by the own position estimation unit 27. "WP" means registration of the own position information by the own position estimation unit 27 into the environment map information 26A. "CM" means creation processing of the environment map information 26A. "WM" means writing processing into the environment map information 26A. "WD" means writing processing of the position information of the detection point P and the own position information into the environment map information 26A.

[0228] In addition, Figure 18 In the figure, "RS" refers to the process of reading the speed data of the mobile object 2 included in the CAN data by the absolute distance conversion unit 30A. "AD" refers to the process of calculating the absolute distance, or distance information, of each detection point P by the absolute distance conversion unit 30A. "2D" refers to the process of extracting detection points P within a specific range by the extraction unit 30B. "DD" refers to the process of identifying the detection point P closest to the mobile object 2, or a plurality of detection points P in order of proximity to the mobile object 2, by the nearest proximity identification unit 30C. Alternatively, "DD" may refer to the process of identifying the nearest detection point P for each angular range as viewed from the own position S of the mobile object 2, or the process of identifying a plurality of detection points P in order of proximity to the mobile object 2. "CB" refers to the process of selecting the shape of the reference projection plane 40 by the reference projection plane shape selection unit 30D. "CS" refers to the process of determining the scale of the reference projection plane 40 by the scale determination unit 30E. "CC" refers to the process of calculating the asymptotic curve Q by the asymptotic curve calculation unit 30F. “DS” means a process of determining the projection shape 41 by the shape determining unit 30G. “VP” means a process of determining virtual viewpoint-line-of-sight information by the virtual viewpoint-line-of-sight determining unit 34 .

[0229] in addition, Figure 18The “detection point registration process” is a process of registering the position information of the detection point P based on the detection point registration unit 29 to the environmental map information 26A. The “self-position estimation process” is a process of estimating the self-position information of the mobile body 2 based on the self-position estimation unit 27 and registering it to the environmental map information 26A. The “correction process” is a process of correcting the position information of the detection point P or the self-position information of the mobile body 2 based on the correction unit 28. The “display control process” is a process that includes the acquisition of the captured image 50 based on the acquisition unit 20, the determination of the projection shape 41 based on the determination unit 30 (projection shape determination), the deformation of the reference projection plane 40 based on the deformation unit 32 (projection shape conversion), the generation of a virtual viewpoint image based on the projection conversion unit 36 ​​(projection conversion), the generation of a synthetic image 54 based on the image synthesis unit 38 (synthetic image generation), and the display control of the synthetic image 54 based on the image synthesis unit 38. In addition, Figure 18 The illustrated conditions T1 to T4 are the same as those in the first embodiment.

[0230] like Figure 18 As shown, processing using the captured image 50 for each of the states T1 through T4 is performed. Therefore, the position information for each of the plurality of detection points P detected at each time is sequentially added to the environment map information 26A. Furthermore, the estimated own position information is sequentially registered in the environment map information 26A.

[0231] Furthermore, the image processing device 10 executes the above-described display control process at each time point of the status T1 to the status T4 using the position information of the detection point P and the own position information of the own position S registered in the environment map information 26A.

[0232] That is, the image processing device 11 of this embodiment simultaneously estimates the position information of the detection point P and the own position information of the moving object 2 based on the captured image 50 captured by the imaging unit 12 using Visual SLAM, and registers them in the environment map information 26A.

[0233] Here, in the conventional technology, it is sometimes difficult to deform the projection plane (reference projection plane 40 ) according to the distance from the actual object.

[0234] Specifically, if you use Figure 10B and Figure 11B As described above, in the conventional technique, the reference projection plane 40 is not deformed by the pillars C outside the detection area F, and therefore the area CA′ corresponding to the pillars C in the conventional synthetic image 530B is a deformed image.

[0235] On the other hand, in the image processing device 11 of the second embodiment, the position information of the detection point P of the object in the past captured by the imaging unit 12 is also stored in the environment map information 26A. Therefore, even if the object is not reflected in the current image captured by the imaging unit 12, the image processing device 11 can provide a composite image 54 in which the shape of the projection surface is deformed based on the position information of the detection point P of the object in the past.

[0236] Figure 19 is an image showing an example of a composite image 54H output by the image processing device 11. The composite image 54H is an example of the composite image 54. Figure 19 As shown, the image processing device 11 can deform the projection plane (reference projection plane 40 ) according to the distance from the actual object and output a synthesized image 54H.

[0237] Therefore, the image processing device 11 of this embodiment can provide the composite image 54 by using the imaging unit 12 without using the detection unit 14. Furthermore, the image processing device 11 can provide the composite image 54 by using the detection unit 14 in addition to the imaging unit 12.

[0238] Furthermore, the image processing device 11 of the present embodiment can be particularly suitably used when the mobile body 2 is traveling at a low speed of a predetermined speed or less, such as when the mobile body 2 is parked.

[0239] While the embodiments have been described above, the image processing device, image processing method, and image processing program disclosed in this application are not limited to the aforementioned embodiments per se. During implementation, the components may be modified and embodied within the scope of the present invention. Furthermore, various inventions may be formed by appropriately combining the multiple components disclosed in the aforementioned embodiments. For example, some components may be deleted from all the components shown in the embodiments.

[0240] Furthermore, the image processing devices 10 and 11 of the first and second embodiments can be applied to various devices. For example, the image processing devices 10 and 11 of the first and second embodiments can be applied to a surveillance camera system that processes images obtained from a surveillance camera, or an in-vehicle system that processes images of the surrounding environment outside a vehicle.

Claims

1. An image processing device, wherein: A deformation unit is provided for deforming a reference projection plane, which is a projection plane of an image of the surroundings of the moving object, using position information of a plurality of detection points stored around the moving object and the position information of the moving object itself. The position information includes position information of the detection points at different times.

2. The image processing apparatus according to claim 1, wherein: The image processing device comprises: an acquiring unit, configured to acquire position information of each of the plurality of detection points from the detecting unit; a detection point registration unit, registering each position information acquired by the acquisition unit into the environmental map information; as well as a self-position estimating unit that estimates the self-position information based on the position information registered in the environmental map information and registers the self-position information in the environmental map information; The deformation unit deforms the reference projection plane based on the environment map information.

3. The image processing apparatus according to claim 2, wherein: A correction unit is provided for correcting the position information of each of the plurality of detection points registered in the environment map information and the own position information using the position information of each corresponding detection point acquired at an acquisition time after the acquisition time of the detection point.

4. The image processing device according to any one of claims 1 to 3, wherein: The deformation unit deforms the shape of the reference projection plane based on the detection point closest to the moving object.

5. The image processing apparatus according to claim 4, wherein: The deformation unit deforms the reference projection surface into a curved surface shape passing through the detection point closest to the moving object.

6. The image processing device according to any one of claims 1 to 3, wherein: The deformation section deforms the reference projection surface into a shape of an asymptotic curve along the detection point in each of a plurality of portions of the reference projection surface.

7. The image processing device according to any one of claims 1 to 3, wherein: A projection conversion unit is provided for converting a projection image obtained by projecting the peripheral image onto the deformed projection surface, which is the reference projection surface, into a virtual viewpoint image in which the direction of the deformed projection surface is visually recognized from a virtual viewpoint.

8. The image processing device according to any one of claims 1 to 3, wherein: The detection point is a detection point detected by a detection unit mounted on the moving object.

9. The image processing device according to any one of claims 1 to 3, wherein: The detection points are detection points detected based on the surrounding image.

10. An image processing method is an image processing method executed by a computer, wherein: The image processing method comprises the following steps: The projection surface of the surrounding image of the moving body, i.e., the reference projection surface, is deformed using the position information of multiple detection points stored around the moving body and the position information of the moving body itself, wherein the position information includes the position information of the detection points stored at different times.

11. A recording medium storing an image processing program, wherein the image processing program is configured to cause a computer to execute the following steps: The projection surface of the surrounding image of the moving body, i.e., the reference projection surface, is deformed using the position information of multiple detection points stored around the moving body and the position information of the moving body itself, wherein the position information includes the position information of the detection points stored at different times.

Images