Image processing device, image processing method, and image processing program

Abstract

The image processing device (10) comprises a determination unit (30) and a deformation unit (32) in one aspect. The determination unit (30), on the basis of a magnitude relationship between a first threshold value and a measured distance between a moving body (2) and a three-dimensional object in the periphery of the moving body (2), executes a first distance stabilization process for converting the measured distance into a first distance or a second distance smaller than the first distance as a stabilized distance. The deformation unit (32) deforms a projection plane of a peripheral image of the moving body (2) on the basis of the stabilized distance.

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, an image processing method, and an image processing program.

Background Art

[0002] A technique for generating a composite image from an arbitrary viewpoint using a projected image obtained by projecting a captured image of the surroundings of a moving body onto a virtual projection plane is disclosed. An improved technique for deforming the projection plane according to the distance from the moving body to an obstacle has also been proposed.

Prior Art Literature

Patent Literature

[0003]

Patent Literature 1

Patent Literature 2

Patent Literature 3

Patent Literature 4

Non-Patent Literature

[0004]

Non-Patent Literature 1

Summary of the Invention

Problem to be Solved by the Invention

[0005] However, when deforming the projection plane according to the distance from the moving body to an obstacle, the displayed image may become unnatural.

[0006] In one aspect, the present invention aims to provide an image processing apparatus, an image processing method, and an image processing program that eliminate the problem of unnatural-looking images when the projection surface is deformed according to the distance from a moving object to an obstacle. [Means for solving the problem]

[0007] In one embodiment, the image processing apparatus disclosed in this application stabilizes the measurement distance based on the relationship between the measurement distance between the moving object and a three-dimensional object surrounding the moving object and a first threshold value. of The system includes a conversion unit that converts to a first distance or a second distance smaller than the first distance, and a deformation unit that deforms the projection plane of the surrounding image of the moving body based on the stabilization distance. [Effects of the Invention]

[0008] According to one embodiment of the image processing apparatus disclosed in this application, when the projection surface is deformed according to the distance from the moving object to the obstacle, the problem of the displayed image becoming unnatural can be eliminated. [Brief explanation of the drawing]

[0009] [Figure 1] Figure 1 shows an example of the overall configuration of the image processing system according to the first embodiment. [Figure 2] Figure 2 shows an example of the hardware configuration of the image processing device according to the first embodiment. [Figure 3] Figure 3 shows an example of the functional configuration of the image processing apparatus according to the first embodiment. [Figure 4] Figure 4 is a schematic diagram of an example of environmental map information according to the first embodiment. [Figure 5] Figure 5 is an explanatory diagram of an example of an asymptotic curve according to the first embodiment. [Figure 6] Figure 6 is a schematic diagram showing an example of a reference projection plane according to the first embodiment. [Figure 7] Figure 7 is a schematic diagram showing an example of a projected shape determined by the shape determination unit according to the first embodiment. [Figure 8] FIG. 8 is a schematic diagram illustrating an example of a functional configuration of a determination unit according to the first embodiment. [Figure 9] FIG. 9 is a plan view illustrating an example of a situation where a moving body is backed into a parking space having a pillar. [Figure 10] FIG. 10 is a plan view illustrating an example of a situation where the moving body is closer to the pillar compared with FIG. 9 when the moving body is backed into the parking space having the pillar. [Figure 11] FIG. 11 is a diagram illustrating an example of temporal change in a measured distance of a detection point of a nearest-neighbor pillar from a moving body. [Figure 12] FIG. 12 is a graph illustrating an example of a relationship between an input and an output of a distance stabilization processing unit. [Figure 13] FIG. 13 is a diagram illustrating an example of temporal change in a stabilized distance obtained by a first distance stabilization process that receives the measured distance of the pillar detection point illustrated in FIG. 11 as an input. [Figure 14] FIG. 14 is a diagram illustrating an example of temporal change in a stabilized distance D obtained by first and second distance stabilization processes that receive the measured distance of the pillar detection point illustrated in FIG. 11 as an input. [Figure 15] FIG. 15 is a flowchart illustrating an example of a flow of image processing executed by the image processing apparatus according to the first embodiment. [Figure 16] FIG. 16 is a diagram illustrating an example of a functional configuration of an image processing apparatus according to a second embodiment. [Figure 17] FIG. 17 is a flowchart illustrating an example of a flow of image processing executed by the image processing apparatus according to the second embodiment. DESCRIPTION OF EMBODIMENTS

[0010] Hereinafter, embodiments of an image processing apparatus, an image processing method, and an image processing program disclosed in the present application will be described in detail with reference to the accompanying drawings. The following embodiments do not limit the disclosed technology. Furthermore, each embodiment can be appropriately combined within a range that does not make processing contents inconsistent with each other.

[0011] (First Embodiment) Figure 1 shows an example of the overall configuration of the image processing system 1 of this embodiment. The image processing system 1 comprises an image processing device 10, an imaging unit 12, a detection unit 14, and a display unit 16. The image processing device 10, the imaging unit 12, the detection unit 14, and the display unit 16 are connected to each other so as to be able to send and receive data or signals.

[0012] In this embodiment, the image processing device 10, the imaging unit 12, the detection unit 14, and the display unit 16 will be described as an example of a configuration mounted on a mobile body 2. The image processing device 10 according to the first embodiment is an example that utilizes Visual SLAM (Simultaneous Localization and Mapping) processing.

[0013] Mobile object 2 is a movable object. Examples of mobile object 2 include vehicles, flying objects (manned aircraft, unmanned aircraft (e.g., UAVs (Unmanned Aerial Vehicles), drones)), robots, etc.). Mobile object 2 can also be a mobile object that moves through human operation or a mobile object that can move automatically (autonomously) without human operation. In this embodiment, the case where mobile object 2 is a vehicle will be described as an example. Examples of vehicles include two-wheeled vehicles, three-wheeled vehicles, four-wheeled vehicles, etc. In this embodiment, the case where the vehicle is an autonomously moving four-wheeled vehicle will be described as an example.

[0014] Furthermore, the image processing device 10, the imaging unit 12, the detection unit 14, and the display unit 16 are not limited to being mounted on the mobile body 2. The image processing device 10 may be mounted on a stationary object. A stationary object is an object fixed to the ground. A stationary object is an object that cannot be moved or is stationary relative to the ground. Examples of stationary objects include traffic lights, parked vehicles, and road signs. In addition, the image processing device 10 may be mounted on a cloud server that performs processing on the cloud.

[0015] The imaging unit 12 photographs the area around the moving object 2 and acquires captured image data. Hereafter, the captured image data will simply be referred to as the captured image. The imaging unit 12 is, for example, a digital camera capable of shooting video. Note that "shooting" refers to converting the image of a subject formed by an optical system such as a lens into an electrical signal. The imaging unit 12 outputs the captured image to the image processing device 10. In this embodiment, the imaging unit 12 is assumed to be a monocular fisheye camera (for example, with a field of view of 195 degrees).

[0016] In this embodiment, a configuration in which four imaging units 12 (imaging units 12A to 12D) are mounted on a mobile body 2 will be described as an example. Each of the multiple imaging units 12 (imaging units 12A to 12D) photographs a subject in its respective imaging area E (imaging area E1 to imaging area E4) and acquires an image. These multiple imaging units 12 are assumed to have different imaging directions. Furthermore, the imaging directions of these multiple imaging units 12 are assumed to be pre-adjusted so that at least a portion of the imaging area E overlaps between adjacent imaging units 12.

[0017] Furthermore, the four imaging units 12A to 12D are just examples, and there is no limit to the number of imaging units 12. For example, if the mobile body 2 has a vertically elongated shape like a bus or truck, one imaging unit 12 can be placed at the front, rear, front right side, rear right side, front left side, and rear left side of the mobile body 2, for a total of six imaging units 12. In other words, the number and placement of the imaging units 12 can be arbitrarily set depending on the size and shape of the mobile body 2. Note that the boundary angle determination process, which will be described later, can be achieved by providing at least two imaging units 12.

[0018] The detection unit 14 detects the positional information of each of the multiple detection points around the moving object 2. In other words, the detection unit 14 detects the positional information of each of the detection points in the detection region F. A detection point refers to each of the points in real space that are individually observed by the detection unit 14. A detection point corresponds to, for example, a three-dimensional object around the moving object 2.

[0019] The position information of a detection point refers to information indicating the position of the detection point in real space (three-dimensional space). For example, the position information of a detection point includes the distance from the detection unit 14 (i.e., the position of the moving body 2) to the detection point, and the direction of the detection point relative to the detection unit 14. These distances and directions can be represented, for example, by position coordinates indicating the relative position of the detection point relative to the detection unit 14, position coordinates indicating the absolute position of the detection point, or by vectors.

[0020] The detection unit 14 may be, for example, a 3D (Three-Dimensional) scanner, a 2D (Two-Dimensional) scanner, a distance sensor (millimeter-wave radar, laser sensor), a sonar sensor that detects objects using sound waves, or an ultrasonic sensor. The laser sensor may be, for example, a three-dimensional LiDAR (Laser imaging Detection and Ranging) sensor. The detection unit 14 may also be a device using SfM (Structure from Motion) technology that measures distance from images captured by a monocular camera. Alternatively, multiple imaging units 12 may be used as the detection unit 14. Alternatively, one of the multiple imaging units 12 may be used as the detection unit 14.

[0021] The display unit 16 displays various types of information. The display unit 16 is, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display.

[0022] In this embodiment, the image processing device 10 is communicatively connected to an electronic control unit (ECU) 3 mounted on the mobile body 2. The ECU 3 is a unit that electronically controls the mobile body 2. In this embodiment, the image processing device 10 is capable of receiving CAN (Controller Area Network) data such as the speed and direction of movement of the mobile body 2 from the ECU 3.

[0023] Next, the hardware configuration of the image processing device 10 will be described.

[0024] Figure 2 shows an example of the hardware configuration of the image processing device 10.

[0025] The image processing device 10 includes a CPU (Central Processing Unit) 10A, a ROM (Read Only Memory) 10B, a RAM (Random Access Memory) 10C, and an I / F (Interface) 10D, and is, for example, a computer. The CPU 10A, ROM 10B, RAM 10C, and I / F 10D are interconnected by a bus 10E, resulting in a hardware configuration that utilizes a typical computer.

[0026] The CPU 10A is an arithmetic unit that controls the image processing device 10. The CPU 10A corresponds to an example of a hardware processor. The ROM 10B stores programs and other data that implement various processes performed by the CPU 10A. The RAM 10C stores data necessary for various processes performed by the CPU 10A. The I / F 10D is an interface for sending and receiving data to and from the imaging unit 12, detection unit 14, display unit 16, and ECU 3, etc.

[0027] The program for performing image processing to be executed by the image processing device 10 of this embodiment is provided pre-installed in a ROM 10B or the like. Alternatively, the program executed by the image processing device 10 of this embodiment may be provided as a file in a format installable or executable by the image processing device 10, recorded on a recording medium. The recording medium is a medium readable by a computer. Examples of recording media include CD (Compact Disc)-ROM, flexible disk (FD), CD-R (Recordable), DVD (Digital Versatile Disk), USB (Universal Serial Bus) memory, and SD (Secure Digital) card.

[0028] Next, the functional configuration of the image processing device 10 according to this embodiment will be described. The image processing device 10 simultaneously estimates the position information of the detection point and the self-position information of the moving object 2 from the captured image captured by the imaging unit 12 using Visual SLAM processing. The image processing device 10 stitches together a plurality of spatially adjacent captured images to generate and display a composite image that provides an overview of the area around the moving object 2. In this embodiment, the imaging unit 12 is used as the detection unit 14.

[0029] Figure 3 shows an example of the functional configuration of the image processing device 10. In addition to the image processing device 10, the imaging unit 12 and the display unit 16 are also shown in Figure 3 to clarify the data input / output relationship.

[0030] The image processing device 10 includes an acquisition unit 20, a selection unit 23, a Visual-SLAM processing unit 24 (hereinafter referred to as "VSLAM processing unit 24"), a determination unit 30, a deformation unit 32, a virtual viewpoint line-of-sight determination unit 34, a projection transformation unit 36, and an image synthesis unit 38.

[0031] Some or all of the above-mentioned parts may be implemented by having a processing unit such as a CPU 10A execute a program, that is, by software. Alternatively, some or all of the above-mentioned parts may be implemented by hardware such as an IC (Integrated Circuit), or by using a combination of software and hardware.

[0032] The acquisition unit 20 acquires captured images from the shooting unit 12. The acquisition unit 20 acquires captured images from each of the shooting units 12 (shooting units 12A to 12D).

[0033] Each time the acquisition unit 20 acquires a captured image, it outputs the acquired image to the projection conversion unit 36 ​​and the selection unit 23.

[0034] The selection unit 23 selects the detection area of ​​the detection point. In this embodiment, the selection unit 23 selects the detection area by selecting at least one imaging unit 12 from among the multiple imaging units 12 (imaging units 12A to imaging units 12D).

[0035] In this embodiment, the selection unit 23 selects at least one imaging unit 12 using vehicle status information, detection direction information, or instruction information input by user operation instructions included in the CAN data received from the ECU 3.

[0036] Vehicle status information includes, for example, the direction of travel of the mobile unit 2, the status of the mobile unit 2's direction indicator, and the gear status of the mobile unit 2. Vehicle status information can be derived from CAN data. Detected direction information indicates the direction in which the information of interest was detected and can be derived using POI (Point of Interest) technology. Instruction information indicates the direction of interest and is input by user operation instructions.

[0037] For example, the selection unit 23 uses vehicle status information to select the direction of the detection area. Specifically, the selection unit 23 uses vehicle status information to identify parking information such as rear parking information indicating that the moving body 2 is parked behind the vehicle, and parallel parking information indicating parallel parking. The selection unit 23 stores in advance the identification information of one of the imaging units 12 in association with the parking information. For example, the selection unit 23 stores in advance the identification information of imaging unit 12D (see Figure 1) which photographs the rear of the moving body 2 in association with the rear parking information. The selection unit 23 also stores in advance the identification information of imaging units 12B and 12C (see Figure 1) which photograph the left and right directions of the moving body 2 in association with the parallel parking information.

[0038] Then, the selection unit 23 selects the direction of the detection area by selecting the shooting unit 12 that corresponds to the parking information derived from the received vehicle status information.

[0039] Furthermore, the selection unit 23 may select an imaging unit 12 whose imaging area E is the direction indicated by the detection direction information. Alternatively, the selection unit 23 may select an imaging unit 12 whose imaging area E is the direction indicated by the detection direction information derived by POI technology.

[0040] The selection unit 23 outputs the image captured by the selected imaging unit 12 from the images acquired by the acquisition unit 20 to the VSLAM processing unit 24.

[0041] The VSLAM processing unit 24 uses the captured image received from the selection unit 23 to perform Visual SLAM processing to generate environmental map information, and outputs the generated environmental map information to the determination unit 30.

[0042] Specifically, the VSLAM processing unit 24 includes a matching unit 25, a storage unit 26, a self-position estimation unit 27, a correction unit 28, and a three-dimensional reconstruction unit 29.

[0043] The matching unit 25 performs feature extraction and matching processes for multiple captured images (multiple captured images with different frames) taken at different timings. Specifically, the matching unit 25 performs feature extraction from these multiple captured images. The matching unit 25 uses the features to identify corresponding points between the multiple captured images taken at different timings. The matching unit 25 outputs the matching results to the self-localization unit 27.

[0044] The self-position estimation unit 27 uses multiple matching points obtained from the matching unit 25 to estimate its own position relative to the captured image through projection transformation and other means. Here, the self-position includes information on the position (three-dimensional coordinates) and tilt (rotation) of the imaging unit 12, and the self-position estimation unit 27 stores this as self-position information in the environmental map information 26A.

[0045] The three-dimensional reconstruction unit 29 performs perspective projection transformation processing using the amount of self-position movement (translation and rotation) estimated by the self-position estimation unit 27 to determine the three-dimensional coordinates (relative coordinates to its own position) of the matching point. The three-dimensional reconstruction unit 29 stores the determined three-dimensional coordinates as surrounding position information in the environmental map information 26A.

[0046] As a result, new surrounding location information and self-location information are sequentially added to the environmental map information 26A as the mobile body 2, on which the camera unit 12 is mounted, moves.

[0047] The storage unit 26 stores various types of data. The storage unit 26 may be, for example, a semiconductor memory element such as RAM or flash memory, a hard disk, or an optical disc. The storage unit 26 may also be a storage device located outside the image processing device 10. Furthermore, the storage unit 26 may be a storage medium. Specifically, the storage medium may be a medium on which programs and various types of information have been downloaded or temporarily stored via a LAN (Local Area Network) or the Internet.

[0048] Environmental map information 26A is information that registers surrounding position information calculated by the three-dimensional reconstruction unit 29 and self-position information calculated by the self-position estimation unit 27 in a three-dimensional coordinate space with a predetermined position in real space as the origin. The predetermined position in real space may be determined, for example, based on pre-set conditions.

[0049] For example, the predetermined position is the position of the moving body 2 when the image processing device 10 performs the image processing of this embodiment. For example, consider a case where image processing is performed at a predetermined timing, such as when the moving body 2 is parking. In this case, the image processing device 10 can set the position of the moving body 2 when it determines that the predetermined timing has been reached as the predetermined position. For example, the image processing device 10 can determine that the predetermined timing has been reached when it determines that the behavior of the moving body 2 is indicative of a parking scene. Behavior indicating a parking scene includes, for example, when the speed of the moving body 2 falls below a predetermined speed, when the gear of the moving body 2 is put into reverse, or when a signal indicating the start of parking is received by a user's operation instruction. Note that the predetermined timing is not limited to a parking scene.

[0050] Figure 4 is a schematic diagram of an example of environmental map information 26A. As shown in Figure 4, environmental map information 26A is information in which the positional information (surrounding positional information) of each detection point P and the self-position information of the moving object 2's self-position S are registered at the corresponding coordinate positions in the three-dimensional coordinate space. As an example, self-positions S1 to S3 are shown. The larger the numerical value following S, the closer the self-position S is to the current timing.

[0051] The correction unit 28 corrects the surrounding position information and self-position information registered in the environmental map information 26A for points that have been matched multiple times across multiple frames, using, for example, the least squares method, so that the sum of the difference in distance in three-dimensional space between the previously calculated three-dimensional coordinates and the newly calculated three-dimensional coordinates is minimized. The correction unit 28 may also correct the amount of self-position movement (translation and rotation) used in the process of calculating the self-position information and surrounding position information.

[0052] The timing of the correction process performed by the correction unit 28 is not limited. For example, the correction unit 28 may perform the correction process at predetermined intervals. The predetermined timing may be determined, for example, based on pre-set conditions. In this embodiment, the image processing device 10 is described as having a configuration that includes the correction unit 28 as an example. However, the image processing device 10 may also have a configuration that does not include the correction unit 28.

[0053] The determination unit 30 receives environmental map information from the VSLAM processing unit 24 and uses the surrounding position information and self-position information stored in the environmental map information 26A to calculate the measured distance between the moving object 2 and surrounding three-dimensional objects. Here, the measured distance refers to the distance between objects measured by processing using distance sensors or images (VSLAM processing in this embodiment). Since the measured distance is a distance obtained by processing using distance sensors or images, it can take any value depending on the situation. In that sense, the measured distance is a continuous value.

[0054] The determination unit 30 performs a distance stabilization process that converts the measured distance into a stabilized distance. Here, the stabilized distance refers to a discrete distance (non-continuous value) obtained based on the measured distance. This distance stabilization process will be explained in detail later. Note that the determination unit 30 is an example of a conversion unit.

[0055] Furthermore, the determination unit 30 determines the projection shape of the projection surface using the stabilized distance obtained by the distance stabilization process and generates projection shape information. The determination unit 30 outputs the generated projection shape information to the deformation unit 32.

[0056] Here, the projection surface is a three-dimensional surface onto which the surrounding image of the moving object 2 is projected. The surrounding image of the moving object 2 is a captured image of the area around the moving object 2, and is a captured image taken by each of the imaging units 12A to 12D. The projected shape of the projection surface is a three-dimensional (3D) shape virtually formed in a virtual space corresponding to real space. In this embodiment, the determination of the projected shape of the projection surface performed by the determination unit 30 is called the projection shape determination process.

[0057] Furthermore, the determination unit 30 uses the surrounding position information and self-position information of the moving object 2 stored in the environmental map information 26A to calculate an asymptotic curve of the surrounding position information with respect to the self-position.

[0058] Figure 5 is an explanatory diagram of the asymptotic curve Q generated by the determination unit 30. Here, the asymptotic curve is the asymptotic curve of multiple detection points P in the environmental map information 26A. Figure 5 is an example in which the asymptotic curve Q is shown on a projected image obtained by projecting the captured image onto a projection surface when the moving object 2 is viewed from above in a bird's-eye view. For example, suppose the determination unit 30 identifies three detection points P in order of proximity to the self-position S of the moving object 2. In this case, the determination unit 30 generates the asymptotic curve Q of these three detection points P.

[0059] The determination unit 30 outputs its own position and asymptotic curve information to the virtual viewpoint line-of-sight determination unit 34.

[0060] The configuration of the determination unit 30 will be explained in detail later.

[0061] The deformation unit 32 deforms the projection surface based on the projection shape information received from the determination unit 30.

[0062] Figure 6 is a schematic diagram showing an example of a reference projection plane 40. Figure 7 is a schematic diagram showing an example of a projection shape 41 determined by the determination unit 30. That is, the deformation unit 32 deforms the reference projection plane shown in Figure 6, which is stored in advance, based on the projection shape information, and determines the deformed projection plane 42 as the projection shape 41 shown in Figure 7. The determination unit 30 generates deformed projection plane information based on the projection shape 41. This deformation of the reference projection plane is performed, for example, with the detection point P closest to the moving object 2 as the reference. The deformation unit 32 outputs the deformed projection plane information to the projection conversion unit 36.

[0063] Furthermore, for example, the deformation unit 32 deforms the reference projection plane to a shape that follows the asymptotic curves of a predetermined number of detection points P, ordered from closest to the moving body 2, based on the projection shape information.

[0064] The virtual viewpoint line-of-sight determination unit 34 determines virtual viewpoint line-of-sight information based on its own position and asymptotic curve information.

[0065] The determination of virtual viewpoint line-of-sight information will be explained with reference to Figures 5 and 7. The virtual viewpoint line-of-sight determination unit 34 determines the line-of-sight direction as a direction that passes through the detection point P closest to the self-position S of the moving object 2 and is perpendicular to the deformation projection plane. The virtual viewpoint line-of-sight determination unit 34 also determines the coordinates of the virtual viewpoint O as an arbitrary Z coordinate and arbitrary XY coordinates in a direction away from the asymptotic curve Q toward the self-position S, for example, by fixing the direction of the line-of-sight direction L. In this case, the XY coordinates may be coordinates at a position further away from the asymptotic curve Q than the self-position S. The virtual viewpoint line-of-sight determination unit 34 then outputs virtual viewpoint line-of-sight information indicating the virtual viewpoint O and the line-of-sight direction L to the projection transformation unit 36. As shown in Figure 7, the line-of-sight direction L may be the direction toward the vertex W of the asymptotic curve Q from the virtual viewpoint O.

[0066] The projection conversion unit 36 ​​generates a projected image by projecting the captured image acquired from the imaging unit 12 onto the deformed projection surface based on the deformed projection surface information and the virtual viewpoint line-of-sight information. The projection conversion unit 36 ​​converts the generated projected image into a virtual viewpoint image and outputs it to the image synthesis unit 38. Here, the virtual viewpoint image is an image obtained by viewing the projected image from a virtual viewpoint in any direction.

[0067] The projection image generation process by the projection transformation unit 36 ​​will be explained in detail with reference to Figure 7. The projection transformation unit 36 ​​projects the captured image onto the deformable projection surface 42. The projection transformation unit 36 ​​then generates a virtual viewpoint image (not shown) which is an image viewed from an arbitrary virtual viewpoint O in the line of sight direction L, based on the captured image projected onto the deformable projection surface 42. The position of the virtual viewpoint O can be, for example, the latest self-position S of the moving object 2. In this case, the XY coordinate values ​​of the virtual viewpoint O can be set to the XY coordinate values ​​of the latest self-position S of the moving object 2. Also, the Z coordinate value (vertical position) of the virtual viewpoint O can be set to the Z coordinate value of the detection point P closest to the self-position S of the moving object 2. The line of sight direction L may be determined, for example, based on a predetermined criterion.

[0068] The line of sight direction L may be, for example, the direction from the virtual viewpoint O toward the detection point P closest to the self-position S of the moving object 2. Alternatively, the line of sight direction L may be a direction that passes through the detection point P and is perpendicular to the deformation projection plane 42. Virtual viewpoint line of sight information indicating the virtual viewpoint O and the line of sight direction L is created by the virtual viewpoint line of sight determination unit 34.

[0069] For example, the virtual viewpoint line of sight determination unit 34 may determine the line of sight direction L as a direction that passes through the detection point P closest to the self-position S of the moving object 2 and is perpendicular to the deformation projection plane 42. Alternatively, the virtual viewpoint line of sight determination unit 34 may fix the direction of the line of sight direction L and determine the coordinates of the virtual viewpoint O as an arbitrary Z coordinate and arbitrary XY coordinates in a direction away from the asymptotic curve Q toward the self-position S. In this case, the XY coordinates may be coordinates at a position further away from the asymptotic curve Q than the self-position S. The virtual viewpoint line of sight determination unit 34 then outputs virtual viewpoint line of sight information indicating the virtual viewpoint O and the line of sight direction L to the projection transformation unit 36. As shown in Figure 7, the line of sight direction L may be the direction toward the vertex W of the asymptotic curve Q from the virtual viewpoint O.

[0070] The projection transformation unit 36 ​​receives virtual viewpoint line-of-sight information from the virtual viewpoint line-of-sight determination unit 34. By receiving this virtual viewpoint line-of-sight information, the projection transformation unit 36 ​​identifies the virtual viewpoint O and the line-of-sight direction L. The projection transformation unit 36 ​​then generates a virtual viewpoint image from the captured image projected onto the deformed projection surface 42, which is the image viewed from the virtual viewpoint O in the line-of-sight direction L. The projection transformation unit 36 ​​outputs the virtual viewpoint image to the image synthesis unit 38.

[0071] The image synthesis unit 38 generates a composite image by extracting part or all of the virtual viewpoint images. For example, the image synthesis unit 38 performs processing such as stitching together multiple virtual viewpoint images (in this case, four virtual viewpoint images corresponding to the imaging units 12A to 12D) in the boundary region between the imaging units.

[0072] The image synthesis unit 38 outputs the generated composite image to the display unit 16. The composite image may be a bird's-eye view image with the virtual viewpoint O above the moving object 2, or it may be an image with the virtual viewpoint O inside the moving object 2, displaying the moving object 2 semi-transparently.

[0073] Furthermore, the projection conversion unit 36 ​​and the image synthesis unit 38 constitute the image generation unit 37.

[0074] [Example of configuration of the decision unit 30] Next, we will describe an example of the detailed configuration of the determination unit 30.

[0075] Figure 8 is a schematic diagram showing an example of the configuration of the determination unit 30. As shown in Figure 8, the determination unit 30 comprises an absolute distance conversion unit 30A, an extraction unit 30B, a nearest neighbor identification unit 30C, a distance stabilization processing unit 30I, a reference projection plane shape selection unit 30D, a scale determination unit 30E, an asymptotic curve calculation unit 30F, a shape determination unit 30G, and a boundary region determination unit 30H.

[0076] The absolute distance conversion unit 30A converts the relative positional relationship between the user's own position and surrounding three-dimensional objects, which can be obtained from the environmental map information 26A, into the absolute value of the distance from the user's own position to the surrounding three-dimensional objects.

[0077] Specifically, for example, the speed data of the mobile body 2, which is included in the CAN data received from the ECU 3 of the mobile body 2, is used. For example, in the case of the environmental map information 26A shown in Figure 4, the relative positional relationship between the self-position S and multiple detection points P can be known, but the absolute value of the distance is not calculated. Here, the distance between the self-position S3 and the self-position S2 can be determined from the inter-frame period used for self-position calculation and the speed data from the CAN data during that period. Since the relative positional relationship of the environmental map information 26A is similar to that of real space, knowing the distance between the self-position S3 and the self-position S2 allows the absolute value of the distance (measured distance) from the self-position S to all other detection points P to be determined. Note that when the detection unit 14 acquires distance information for detection points P, the absolute distance conversion unit 30A may be omitted.

[0078] The absolute distance conversion unit 30A then outputs the measured distance of each of the multiple detection points P calculated to the extraction unit 30B. The absolute distance conversion unit 30A also outputs the calculated current position of the moving object 2 as the self-position information of the moving object 2 to the virtual viewpoint line of sight determination unit 34.

[0079] The extraction unit 30B extracts detection points P that are within a specific range from among a plurality of detection points P that have received the measured distance from the absolute distance conversion unit 30A. The specific range is, for example, the range from the road surface on which the mobile body 2 is placed to a height equivalent to the height of the mobile body 2. However, this range is not limited to this range.

[0080] The extraction unit 30B extracts detection points P within the range, which can, for example, extract detection points P such as objects that obstruct the movement of the moving body 2 or objects located adjacent to the moving body 2.

[0081] The extraction unit 30B then outputs the measurement distance of each extracted detection point P to the nearest neighbor identification unit 30C.

[0082] The nearest neighbor identification unit 30C divides the area around the moving body 2's own position S into specific ranges (e.g., angular ranges), and for each range, identifies the detection point P closest to the moving body 2, or multiple detection points P in order of proximity to the moving body 2. The nearest neighbor identification unit 30C identifies the detection points P using the measurement distance received from the extraction unit 30B. In this embodiment, one example described is a configuration in which the nearest neighbor identification unit 30C identifies multiple detection points P in order of proximity to the moving body 2 for each range.

[0083] The nearest location identification unit 30C outputs the measured distance of the detection point P identified for each range to the distance stabilization processing unit 30I.

[0084] The distance stabilization processing unit 30I performs a distance stabilization process that converts the measured distance into a stabilized distance. The distance stabilization process performed by the distance stabilization processing unit 30I consists of a first distance stabilization process and a second distance stabilization process. The first and second distance stabilization processes will be described in detail below with reference to Figures 9 to 14.

[0085] (First distance stabilization process) The first distance stabilization process converts the measured distance to a first distance or a second distance smaller than the first distance, based on the relationship between the measured distance of a detection point P identified for each range and a threshold value.

[0086] Figure 9 is a plan view showing an example of a situation where the mobile body 2 is parked in the reverse direction along the parking space line PL where pillar C is located. Figure 10 is a plan view showing an example of a situation where the mobile body 2 is moving in the reverse direction, and is even closer to pillar C than in Figure 9. To make the explanation more concrete, the distance stabilization process when the mobile body 2 is parking in the reverse direction will be described below.

[0087] In the situations shown in Figures 9 and 10, column C is located within the imaging area E4 of the imaging unit 12D. Therefore, the measurement distances of multiple detection points P related to column C, identified for each range, are sequentially output from the nearest neighbor identification unit 30C to the distance stabilization processing unit 30I.

[0088] Figure 11 shows an example of the temporal change in the measurement distance P at the detection point on the nearest pillar C from the moving object 2. The circles in Figure 11 represent the acquisition of a single measurement distance. As shown in Figure 11, as the moving object 2 moves backward and approaches pillar C, the measurement distance d at the detection point P decreases. r The increase in the measured distance d indicates that the moving body 2 moved forward temporarily, for example, to change direction during reverse parking.

[0089] Here, the measured distance d is, for example, time t, as shown in Figure 11. r During this period, the measurement range fluctuates, including slight increases or decreases, in addition to the changes associated with the backward movement of the moving body 2. This fluctuation is due to measurement errors (noise, fluctuations in measured values ​​due to other factors) in the measurement distance obtained using the sensor (in this embodiment, the measurement distance obtained by Visual SLAM processing using the captured image).

[0090] Therefore, when the projection surface is deformed in accordance with the fluctuating measurement distance d as shown in Figure 11, the projection surface will frequently deform in the direction of approaching the moving object 2 or moving away from the moving object 2 in conjunction with the fluctuations in the measurement distance d (hereinafter, this phenomenon will also be called the "first fluctuation of the projection surface"). As a result, when an image (projected image) is displayed on a projection surface where temporal fluctuations have occurred, the projected image will fluctuate and the image will appear distorted. The first distance stabilization process resolves this problem of the projected image becoming unnatural due to the first fluctuation of the projection surface.

[0091] Figure 12 is a graph showing an example of the relationship between the input and output of the distance stabilization processing unit 30I. In the graph of Figure 12, the horizontal axis represents the measured distance d, which is the input to the distance stabilization processing unit 30I, and the vertical axis represents the stabilized distance D, which is the output of the distance stabilization processing unit 30I.

[0092] As the moving body 2 moves backward, it approaches the column C, and the measured distance d between the moving body 2 and the column C fluctuates, as shown in Figure 11, including a slight increase or decrease in addition to the change associated with the backward movement of the moving body 2. However, as shown in Figure 12, the distance stabilization processing unit 30I converts the measured distance d to a stabilized distance D1 and outputs it until the measured distance d, which gradually decreases from d1 as the moving body 2 moves backward, becomes smaller than the threshold d3. On the other hand, when the measured distance d becomes smaller than the threshold d3, the distance stabilization processing unit 30I converts the measured distance d to a stabilized distance D2 and outputs it. In this case, the threshold d3 is an example of a first threshold as a down determination threshold (a threshold for determining a decrease in the stabilized distance). Also, the stabilized distance D1 and stabilized distance D2 are examples of a first distance as the stabilized distance before down and a second distance as the stabilized distance after down, respectively.

[0093] Furthermore, as the moving body 2 moves backward, the measured distance d between the moving body 2 and the column C becomes even smaller. As shown in Figure 12, the distance stabilization processing unit 30I converts the measured distance d to a stabilized distance D2 and outputs it until the measured distance d becomes smaller than threshold d3 and then smaller than threshold d5, which is even smaller than threshold d3. On the other hand, if the input measured distance d becomes smaller than threshold d5, the distance stabilization processing unit 30I converts the measured distance d to a stabilized distance D3 and outputs it. In this case, threshold d5 is an example of a third threshold as a down determination threshold. Also, stabilized distance D2 and stabilized distance D3 are examples of a second distance as a stabilized distance before down and a third distance as a stabilized distance after down, respectively.

[0094] Similarly, the distance stabilization processing unit 30I converts the measured distance d, which is input sequentially, into a stabilized distance D3 and outputs it until the measured distance d becomes smaller than the threshold d7. On the other hand, when the measured distance d, which is input sequentially, becomes smaller than the threshold d7 which is the threshold for determining a down state, the distance stabilization processing unit 30I converts the measured distance d into a stabilized distance D4 which is the stabilized distance after the down state and outputs it. As a result, even if there are fluctuations in the measured distance d, the deformation unit 32 deforms the projection plane using the information of the stabilized distance D, so that the occurrence of fluctuations in the projection plane can be suppressed.

[0095] (Second distance stabilization process) Next, the second distance stabilization process will be described. The second distance stabilization process further resolves the problem of unnatural projection images caused by temporal fluctuations of the projection surface when the first distance stabilization process is performed. Specifically, after the measured distance becomes smaller than the first threshold, the second distance stabilization process converts the measured distance to either the first distance or the second distance based on the relationship between the acquired measured distance and a second threshold that is larger than the first threshold.

[0096] Figure 13 shows an example of the change over time of the stabilized distance D obtained by the first distance stabilization process, which takes the measured distance d of the detection point P of column C shown in Figure 11 as input. According to the first distance stabilization process, as shown in Figure 13, the measured distance d in Figure 11, which contains many fluctuations, can be converted into a stabilized distance D with fewer fluctuations. Therefore, a stable projected image can be displayed using a projection surface with suppressed fluctuations.

[0097] On the other hand, in Figure 13, the stabilization distance D fluctuates between D1 and D2 during the period from time t2 to time t3. Furthermore, the stabilization distance D fluctuates between D2 and D3 during the period from time t4 to time t5, between D3 and D4 during the period from time t6 to time t7, and between D4 and D5 during the period from time t8 to time t9.

[0098] The fluctuations in the stabilization distance D during each of the above periods are due to the fact that, for example, during the period from time t2 to time t3 shown in Figure 13, the measured distance d repeatedly becomes larger or smaller than the threshold d3 shown in Figure 12 due to factors such as noise, and the stabilization distance D also fluctuates between D1 and D2 in conjunction with these fluctuations. The shape of the projection surface is determined according to the value of the stabilization distance D. Therefore, during the period from time t2 to time t3, which corresponds to the period before and after the switching of the stabilization distance, deformation of the projection surface occurs frequently in conjunction with the fluctuations in the stabilization distance D (hereinafter, this phenomenon will also be called the "second fluctuation of the projection surface"). As a result, the projected image, which is displayed stably by the first distance stabilization process, becomes distorted and the projected image fluctuates during the period before and after the switching of the stabilization distance (i.e., the period before and after the measured distance d crosses the down judgment threshold). The second distance stabilization process resolves this problem of the projected image becoming unnatural due to such a second fluctuation of the projection surface.

[0099] For example, as shown in Figure 12, if the distance stabilization processing unit 30I has converted the stabilization distance D to D2 when the measured distance d is less than the threshold d3, it will not convert the stabilization distance D to D1 even if the measured distance d becomes greater than the threshold d3. In other words, when the measured distance d is less than the threshold d3, the distance stabilization processing unit 30I will not convert the stabilization distance D to D1 unless the measured distance d becomes greater than the threshold d2, which is greater than the threshold d3. Therefore, there is a dead zone between threshold d3 and threshold d2. In this case, threshold d3 is an example of a first threshold as a down judgment threshold. Threshold d2 is an example of a second threshold as an up judgment threshold (a threshold for determining an increase in the stabilization distance).

[0100] Furthermore, for example, as shown in Figure 12, if the distance stabilization processing unit 30I has converted the stabilization distance D to D3 when the measured distance d is smaller than the threshold d5, it will not convert the stabilization distance D to D2 even if the measured distance d becomes larger than the threshold d5. In other words, when the measured distance d is smaller than the threshold d5, the distance stabilization processing unit 30I will not convert the stabilization distance D to D2 unless the measured distance d becomes larger than the threshold d4 which is larger than the threshold d5. Therefore, there is a dead zone between the threshold d5 and the threshold d4. In this case, the threshold d5 is an example of a first threshold as a down judgment threshold, and the threshold d4 is an example of a second threshold as an up judgment threshold.

[0101] Similarly, as shown in Figure 12, if the measured distance d is smaller than the threshold d7 and the stabilized distance D is converted to D4, the distance stabilization processing unit 30I will not convert the stabilized distance D to D3 even if the measured distance d becomes larger than the threshold d7. In other words, when the measured distance d is smaller than the threshold d7, the distance stabilization processing unit 30I will not convert the stabilized distance D to D3 unless the measured distance d becomes larger than the threshold d6 which is larger than the threshold d7. Therefore, there is a dead zone between the threshold d7 and the threshold d6. In this case, the threshold d7 is an example of a first threshold as a down judgment threshold, and the threshold d6 is an example of a second threshold as an up judgment threshold.

[0102] Furthermore, the second distance stabilization process, which follows the input-output relationship shown in Figure 12, determines the value of the stabilization distance D not by the relationship between the measured distance d and the first threshold once the measured distance d has fallen below the first threshold, but by the relationship between the measured distance d and the second threshold, which is larger than the first threshold. In other words, it controls the value of the stabilization distance D based on the history of the measured distance d. In this sense, the second distance stabilization process can be called a hysteresis process for the stabilization distance D.

[0103] Figure 14 shows an example of the change over time of the stabilized distance D obtained by the first and second distance stabilization processes, which use the measured distance of detection point P of column C shown in Figure 11 as input. In Figure 14, the time when the stabilized distance D is converted from D1 to D2 is t'3, the time when it is converted from D2 to D3 is t'5, the time when it is converted from D3 to D4 is t'7, and the time when it is converted from D4 to D3 is t'8. As can be seen by comparing Figure 14 with Figure 13, fluctuations in the stabilized distance D during the periods before and after each down judgment threshold or up judgment threshold are eliminated.

[0104] Furthermore, in Figure 12, the dead zones defined by thresholds d3 and d2, d5 and d4, and d7 and d6 each have different lengths. This is because the accuracy of the distance sensor (in this case, the VSLAM processing unit 24) changes depending on the measured distance from the moving object 2 to the three-dimensional object. The width of each of these dead zones can be arbitrarily set by adjusting each threshold according to the measurement accuracy of the distance sensor. For example, if the measurement accuracy of the distance sensor is ±5% of the absolute distance, the width of the dead zone may be set to increase as the measured distance d increases, as shown in the example in Figure 12.

[0105] Furthermore, after the distance stabilization processing unit 30I has converted the stabilization distance D to D2 when the measured distance d becomes smaller than the threshold d3, if the measured distance d becomes smaller than the threshold d5 as the moving body 2 moves backward, it will convert the stabilization distance D to D3 according to the first distance stabilization process. In this case, thresholds d3 and d5 are examples of the first and third thresholds, respectively, and stabilization distances D2 and D3 are examples of the second and third distances, respectively.

[0106] The distance stabilization processing unit 30I outputs the stabilization distances of the detection points P, which have been identified for each range and obtained by the distance stabilization process, to the reference projection plane shape selection unit 30D, the scale determination unit 30E, the asymptotic curve calculation unit 30F, and the boundary region determination unit 30H.

[0107] The reference projection plane shape selection unit 30D selects the shape of the reference projection plane.

[0108] Here, the reference projection plane will be explained in detail with reference to Figure 6. The reference projection plane 40 is, for example, a projection plane of a shape that serves as a reference when changing the shape of the projection plane. The shape of the reference projection plane 40 can be, for example, a bowl shape, a cylinder shape, etc. Figure 6 shows an example of a bowl-shaped reference projection plane 40.

[0109] A bowl shape is a shape having a base surface 40A and side walls 40B, where one end of the side wall surface 40B is continuous with the base surface 40A and the other end is open. The width of the horizontal cross-section of the side wall surface 40B increases from the base surface 40A side toward the open end. The base surface 40A is, for example, circular. Here, circular shape includes shapes other than perfect circles, such as ellipses. A horizontal cross-section is an orthogonal plane perpendicular to the vertical direction (arrow Z direction). An orthogonal plane is a two-dimensional plane along the arrow X direction perpendicular to the arrow Z direction, and the arrow Y direction perpendicular to both the arrow Z direction and the arrow X direction. In the following, the horizontal cross-section and orthogonal plane may be referred to as the XY plane. The base surface 40A may also be a shape other than a circle, such as an egg shape.

[0110] A cylindrical shape consists of a circular base surface 40A and side wall surfaces 40B continuous with the base surface 40A. The side wall surfaces 40B that constitute the reference projection plane 40 of the cylindrical shape are cylindrical in shape, with one end opening continuous with the base surface 40A and the other end open. However, the side wall surfaces 40B that constitute the reference projection plane 40 of the cylindrical shape have a shape in which the diameter of the XY plane is approximately constant from the base surface 40A side toward the opening side of the other end. The base surface 40A may be a shape other than a circle, such as an egg shape.

[0111] In this embodiment, the case in which the shape of the reference projection plane 40 is a bowl shape as shown in Figure 6 will be described as an example. The reference projection plane 40 is a three-dimensional model virtually formed in a virtual space, with its bottom surface 40A being a surface that substantially coincides with the road surface below the moving body 2, and the center of the bottom surface 40A being the self-position S of the moving body 2.

[0112] The reference projection plane shape selection unit 30D selects the shape of the reference projection plane 40 by reading a specific shape from among several types of reference projection planes 40. For example, the reference projection plane shape selection unit 30D selects the shape of the reference projection plane 40 based on the positional relationship between its own position and surrounding three-dimensional objects, the stabilization distance, etc. Alternatively, the shape of the reference projection plane 40 may be selected by user operation instructions. The reference projection plane shape selection unit 30D outputs the shape information of the determined reference projection plane 40 to the shape determination unit 30G. In this embodiment, as described above, the reference projection plane shape selection unit 30D will be described as selecting a bowl-shaped reference projection plane 40 as an example.

[0113] The scale determination unit 30E determines the scale of the reference projection plane 40 of the shape selected by the reference projection plane shape selection unit 30D. The scale determination unit 30E makes decisions such as reducing the scale when there are multiple detection points P within a predetermined distance range from its own position S. The scale determination unit 30E outputs the scale information of the determined scale to the shape determination unit 30G.

[0114] The asymptotic curve calculation unit 30F uses the stabilization distances of the detection points P closest to its own position S for each range from its own position S, received from the nearest neighbor identification unit 30C, to calculate the asymptotic curve Q and outputs the asymptotic curve information of the calculated asymptotic curve Q to the shape determination unit 30G and the virtual viewpoint line of sight determination unit 34. The asymptotic curve calculation unit 30F may also calculate the asymptotic curve Q of the detection points P accumulated for each of the multiple parts of the reference projection plane 40. The asymptotic curve calculation unit 30F may then output the asymptotic curve information of the calculated asymptotic curve Q to the shape determination unit 30G and the virtual viewpoint line of sight determination unit 34.

[0115] The shape determination unit 30G enlarges or reduces the reference projection plane 40 of the shape indicated by the shape information received from the reference projection plane shape selection unit 30D to the scale information received from the scale determination unit 30E. Then, the shape determination unit 30G determines the shape of the enlarged or reduced reference projection plane 40 as the projected shape, which is deformed to conform to the asymptotic curve information of the asymptotic curve Q received from the asymptotic curve calculation unit 30F.

[0116] Here, the determination of the projected shape will be explained in detail with reference to Figure 7. As shown in Figure 7, the shape determination unit 30G determines the projected shape 41 by deforming the reference projection plane 40 into a shape that passes through the detection point P closest to the self-position S of the moving body 2, which is the center of the bottom surface 40A of the reference projection plane 40. The shape that passes through the detection point P means that the deformed side wall surface 40B has a shape that passes through the detection point P. The self-position S is the latest self-position S calculated by the self-position estimation unit 27.

[0117] In other words, the shape determination unit 30G identifies the detection point P closest to its own position S from among the multiple detection points P registered in the environmental map information 26A. Specifically, the XY coordinates of the center position (self position S) of the moving body 2 are set to (X,Y)=(0,0). Then, the shape determination unit 30G determines X 2 +Y 2The detection point P where the value of is the minimum is identified as the detection point P closest to its own position S. Then, the shape determination unit 30G determines the shape obtained by deforming the side wall surface 40B of the reference projection plane 40 so that it passes through the detection point P as the projected shape 41.

[0118] More specifically, the shape determination unit 30G determines the deformed shape of a portion of the bottom surface 40A and the side wall surface 40B as the projected shape 41, such that when the reference projection plane 40 is deformed, a portion of the side wall surface 40B becomes a wall surface passing through the detection point P closest to the moving body 2. The deformed projected shape 41 is, for example, a shape that rises from a rising line 44 on the bottom surface 40A in a direction approaching the center of the bottom surface 40A from the viewpoint of the XY plane (plan view). To "rise" means, for example, bending or folding a portion of the side wall surface 40B and the bottom surface 40A in a direction approaching the center of the bottom surface 40A, so that the angle between the side wall surface 40B and the bottom surface 40A of the reference projection plane 40 becomes a smaller angle. In the raised shape, the rising line 44 may be located between the bottom surface 40A and the side wall surface 40B, and the bottom surface 40A may remain undeformed.

[0119] The shape determination unit 30G determines that a specific region on the reference projection plane 40 should be deformed so that it protrudes to a position passing through the detection point P in the viewpoint (plan view) of the XY plane. The shape and range of the specific region may be determined based on predetermined criteria. The shape determination unit 30G then determines that the reference projection plane 40 should be deformed so that the distance from its own position S increases continuously from the protruding specific region toward the region on the side wall surface 40B other than the specific region.

[0120] For example, as shown in Figure 7, it is preferable to determine the projected shape 41 such that the outer periphery of the cross-section along the XY plane is curved. The outer periphery of the cross-section of the projected shape 41 is, for example, circular, but may be a shape other than a circle.

[0121] The shape determination unit 30G may also determine the projected shape 41 as a shape obtained by deforming the reference projection plane 40 so that it follows the asymptotic curve. The shape determination unit 30G generates a predetermined number of asymptotic curves for multiple detection points P in a direction away from the detection point P closest to the self-position S of the moving body 2. The number of these detection points P may be multiple. For example, it is preferable that the number of detection points P be three or more. In this case, it is also preferable that the shape determination unit 30G generates asymptotic curves for multiple detection points P located at positions that are at a predetermined angle or more away from the self-position S. For example, the shape determination unit 30G can determine the projected shape 41 as a shape obtained by deforming the reference projection plane 40 so that it follows the generated asymptotic curve Q in the asymptotic curve Q shown in Figure 5.

[0122] The shape determination unit 30G may divide the area around the self-position S of the moving body 2 into specific ranges, and for each range, it may identify the detection point P closest to the moving body 2, or multiple detection points P in order of proximity to the moving body 2. The shape determination unit 30G may then determine the projected shape 41 as the shape obtained by deforming the reference projection plane 40 so that it is a shape that passes through the detection points P identified for each range, or a shape that follows the asymptotic curve Q of the multiple detection points P identified.

[0123] The shape determination unit 30G then outputs the projection shape information of the determined projection shape 41 to the deformation unit 32.

[0124] Next, an example of the image processing flow, including distance stabilization processing, performed by the image processing apparatus 10 according to the first embodiment will be described.

[0125] Figure 15 is a flowchart showing an example of the image processing flow performed by the image processing device 10.

[0126] The acquisition unit 20 acquires the captured image from the shooting unit 12 (step S10). The acquisition unit 20 also captures directly specified information (for example, when the gear of the moving body 2 is in reverse gear) and the vehicle status (for example, when it is stopped).

[0127] The selection unit 23 selects at least two of the imaging units 12A to 12D (step S12).

[0128] The matching unit 25 uses multiple captured images taken by the camera unit 12 at different timings, selected in step S12 from the captured images acquired in step S10, to perform feature extraction and matching processing (step S14).

[0129] The self-position estimation unit 27 reads the environmental map information 26A (surrounding location information and self-position information) (step S16). The self-position estimation unit 27 uses multiple matching points obtained from the matching unit 25 to estimate the self-position relative to the captured image through projection transformation, etc. (step S18), and registers the calculated self-position information in the environmental map information 26A (step S20).

[0130] The three-dimensional reconstruction unit 29 reads the environmental map information 26A (surrounding location information and self-location information) (step S22). The three-dimensional reconstruction unit 29 performs perspective projection transformation processing using the amount of self-position movement (translation amount and rotation amount) estimated by the self-position estimation unit 27 to determine the three-dimensional coordinates (relative coordinates to the self-position) of the matching point and registers them in the environmental map information 26A as surrounding location information (step S24).

[0131] The correction unit 28 reads the environmental map information 26A (surrounding location information and self-location information). For points that have been matched multiple times across multiple frames, the correction unit 28 corrects the surrounding location information and self-location information registered in the environmental map information 26A (step S26) using, for example, the least squares method, so that the sum of the difference in distance in three-dimensional space between the previously calculated three-dimensional coordinates and the newly calculated three-dimensional coordinates is minimized, and updates the environmental map information 26A.

[0132] The absolute distance conversion unit 30A takes in the speed data (vehicle speed) of the mobile body 2, which is included in the CAN data received from the ECU 3 of the mobile body 2. Using the speed data of the mobile body 2, the absolute distance conversion unit 30A converts the surrounding position information included in the environmental map information 26A into distance information from the current position, which is the latest self-position S of the mobile body 2, to each of the multiple detection points P (step S28). The absolute distance conversion unit 30A outputs the calculated distance information for each of the multiple detection points P to the extraction unit 30B. The absolute distance conversion unit 30A also outputs the calculated current position of the mobile body 2 as the self-position information of the mobile body 2 to the virtual viewpoint line of sight determination unit 34.

[0133] The extraction unit 30B extracts detection points P that are within a specific range from among a plurality of detection points P that have received distance information (step S30).

[0134] The nearest neighbor identification unit 30C divides the area around the moving body 2's own position S into specific ranges, and for each range, identifies the detection point P closest to the moving body 2, or multiple detection points P in order of proximity to the moving body 2, and extracts the distance to the nearest neighbor object (step S32). The nearest neighbor identification unit 30C outputs the measured distance d of the detected points P identified for each range (measured distance between the moving body 2 and the nearest neighbor object) to the distance stabilization processing unit 30I.

[0135] The distance stabilization processing unit 30I takes the measured distance d of the detection point P specified for each range as input, performs a first distance stabilization process and a second distance stabilization process, and outputs the stabilized distance D to the reference projection plane shape selection unit 30D, the scale determination unit 30E, the asymptotic curve calculation unit 30F, and the boundary region determination unit 30H (step S33).

[0136] The asymptotic curve calculation unit 30F calculates an asymptotic curve (step S34) and outputs the asymptotic curve information to the shape determination unit 30G and the virtual viewpoint line-of-sight determination unit 34.

[0137] The reference projection plane shape selection unit 30D selects the shape of the reference projection plane 40 (step S36) and outputs the shape information of the selected reference projection plane 40 to the shape determination unit 30G.

[0138] The scale determination unit 30E determines the scale of the reference projection plane 40 of the shape selected by the reference projection plane shape selection unit 30D (step S38), and outputs the scale information of the determined scale to the shape determination unit 30G.

[0139] The shape determination unit 30G determines the projection shape, which is how to deform the shape of the reference projection plane, based on the scale information and asymptotic curve information (step S40). The shape determination unit 30G outputs the projection shape information of the determined projection shape 41 to the deformation unit 32.

[0140] The deformation unit 32 deforms the shape of the reference projection plane based on the projection shape information (step S42). The deformation unit 32 outputs the deformed projection plane information to the projection conversion unit 36.

[0141] The virtual viewpoint line of sight determination unit 34 determines virtual viewpoint line of sight information based on its own position and asymptotic curve information (step S44). The virtual viewpoint line of sight determination unit 34 outputs virtual viewpoint line of sight information indicating the virtual viewpoint O and line of sight direction L to the projection transformation unit 36.

[0142] The projection conversion unit 36 ​​generates a projected image by projecting the captured image acquired from the imaging unit 12 onto the deformed projection surface based on the deformed projection surface information and the virtual viewpoint line-of-sight information. The projection conversion unit 36 ​​converts the generated projected image into a virtual viewpoint image (step S46) and outputs it to the image synthesis unit 38.

[0143] The boundary area determination unit 30H determines the boundary area based on the distance to the nearest neighbor object identified for each range. That is, the boundary area determination unit 30H determines the boundary area as the superposition area of ​​spatially adjacent surrounding images based on the position of the nearest neighbor object of the moving object 2 (step S48). The boundary area determination unit 30H outputs the determined boundary area to the image synthesis unit 38.

[0144] The image synthesis unit 38 generates a composite image by joining spatially adjacent perspective projection images using a boundary region (step S50). That is, the image synthesis unit 38 joins perspective projection images in four directions according to a boundary region set at an angle to the direction of the nearest object to generate a composite image. In the boundary region, spatially adjacent perspective projection images are blended at a predetermined ratio.

[0145] The display unit 16 displays the composite image (step S52).

[0146] The image processing device 10 determines whether or not to terminate image processing (step S54). For example, the image processing device 10 makes the determination in step S54 by determining whether or not it has received a signal from the ECU 3 indicating that the movement of the moving object 2 has stopped. Alternatively, for example, the image processing device 10 may make the determination in step S54 by determining whether or not it has received an instruction to terminate image processing based on an operation instruction from the user.

[0147] If a negative result is obtained in step S54 (step S54: No), the processes from step S10 to step S54 described above are repeated.

[0148] On the other hand, if the decision in step S54 is affirmative (step S54: Yes), this routine terminates.

[0149] Furthermore, if the process returns from step S54 to step S10 after executing the correction process in step S26, the subsequent correction process in step S26 may be omitted. Also, if the process returns from step S54 to step S10 without executing the correction process in step S26, the subsequent correction process in step S26 may be executed.

[0150] As described above, the image processing apparatus 10 according to the embodiment includes a determination unit 30 as a conversion unit and a deformation unit 32. The determination unit 30 determines the measurement distance as a stabilization distance based on the relationship between the measured distance between the moving body 2 and the surrounding three-dimensional objects and the moving body 2 and a first threshold. ofThe distance is converted to a first distance or a second distance smaller than the first distance. The deformation unit 32 deforms the projection plane of the peripheral image of the moving body 2 based on the stabilization distance.

[0151] Therefore, even if the measured distance fluctuates, including slight increases or decreases in addition to the change due to the backward movement of the moving body 2, the measured distance can be converted into a stable distance with less fluctuation. The deformation unit 32 deforms the projection plane of the peripheral image of the moving body 2 based on the stable distance with less fluctuation. As a result, temporal fluctuations of the projection plane can be suppressed, and the problem of the projected image becoming unnatural can be eliminated.

[0152] Furthermore, the determination unit 30 determines the measurement distance as the stabilization distance based on the relationship between the measurement distance and the first threshold. of Convert to a second distance, and based on the relationship between the measured distance and the second threshold which is greater than the first threshold, the measured distance is used as the stabilization distance. of Convert to the first distance.

[0153] Therefore, the phenomenon of the projection surface fluctuating in conjunction with fluctuations in the stabilization distance is suppressed during the period before and after the switching of the stabilization distance. As a result, the problem of unnatural projection images can be further eliminated.

[0154] Furthermore, if the measurement distance is smaller than the first threshold, the determination unit 30 determines the measurement distance as the stabilization distance based on the relationship between the measurement distance and a third threshold that is smaller than the first threshold. of The measurement distance is converted to a second distance or a third distance smaller than the second distance. The deformation unit 32 deforms the projection plane of the peripheral image of the moving body 2 based on the second distance or third distance to which the measurement distance has been converted.

[0155] Therefore, even after the measurement distance becomes smaller than the first threshold, it is possible to stably suppress temporal fluctuations of the projection surface and eliminate the problem of unnatural projection images.

[0156] (Variation 1) In the above embodiment, image processing including distance stabilization was described using the case where the moving body 2 approaches the object by moving backward (or parking backward) as an example. However, the same image processing including distance stabilization can also be applied when the moving body 2 moves away from the object by moving forward.

[0157] In such cases, the distance stabilization processing unit 30I converts the measured distance d, which gradually increases from d8 as the moving body 2 moves forward, to a stabilized distance D4 and outputs it, as shown in Figure 12, for example. On the other hand, if the measured distance d exceeds the threshold d6, the distance stabilization processing unit 30I converts the measured distance d to a stabilized distance D3 and outputs it.

[0158] Furthermore, as shown in Figure 12, after the distance stabilization processing unit 30I converts the stabilization distance D to D3 when the measured distance d becomes greater than the threshold d6, it does not convert the stabilization distance D to D4 unless the measured distance d becomes less than the threshold d7 which is less than the threshold d6.

[0159] (Modification 2) In the above embodiment, at least one of the following processes may be performed on the measured distance: spatial outlier removal, temporal outlier removal, spatial smoothing, or temporal smoothing, before the distance stabilization processing unit 30I (for example, immediately before the determination unit 30). The distance stabilization processing unit 30I performs the distance stabilization process using the measured distance output from the preprocessing unit. This configuration makes it possible to achieve further improvements in accuracy.

[0160] (Variation 3) In the above embodiment, the stabilization distance D may be gradually changed. For example, when the measurement distance d becomes smaller than the threshold d3, the stabilization distance D may be gradually changed from D1 to D2. Also, when the measurement distance d becomes larger than the threshold d2, the stabilization distance D may be gradually changed from D2 to D1. Such processing may also be applied to other stabilization distances.

[0161] (Second embodiment) In the first embodiment described above, an example was described in which the position information (surrounding position information) of the detection point P is acquired from the image captured by the imaging unit 12, that is, an example of an embodiment using Visual SLAM. In contrast, in the second embodiment, an example is described in which the position information (surrounding position information) of the detection point P is detected by the detection unit 14. That is, the image processing device 10 according to the second embodiment is an example that uses three-dimensional LiDAR SLAM or the like.

[0162] Figure 16 shows an example of the functional configuration of the image processing device 10 of the second embodiment. Similar to the image processing device 10 of the first embodiment, the image processing device 10 is connected to the imaging unit 12, the detection unit 14, and the display unit 16 so as to be able to exchange data or signals.

[0163] The image processing device 10 includes an acquisition unit 20, a self-position estimation unit 27, a detection point registration unit 29B, a storage unit 26, a correction unit 28, a determination unit 30, a deformation unit 32, a virtual viewpoint line-of-sight determination unit 34, a projection transformation unit 36, and an image synthesis unit 38.

[0164] Some or all of the above-mentioned components may be implemented by having a processing unit, such as the CPU 10A shown in Figure 2, execute a program, i.e., by software. Alternatively, some or all of the above-mentioned components may be implemented by hardware such as an IC, or by a combination of software and hardware.

[0165] In Figure 16, the memory unit 26, correction unit 28, determination unit 30, deformation unit 32, virtual viewpoint line-of-sight determination unit 34, projection transformation unit 36, and image synthesis unit 38 are the same as in the first embodiment. The memory unit 26 stores environmental map information 26A. The environmental map information 26A is the same as in the first embodiment.

[0166] The acquisition unit 20 acquires captured images from the shooting unit 12. The acquisition unit 20 also acquires surrounding location information from the detection unit 14. The acquisition unit 20 acquires captured images from each of the shooting units 12 (shooting units 12A to 12D). The detection unit 14 detects the surrounding location information. Therefore, the acquisition unit 20 acquires the surrounding location information and the captured images from each of the multiple shooting units 12.

[0167] Each time the acquisition unit 20 acquires surrounding location information, it outputs the acquired surrounding location information to the detection point registration unit 29B. Furthermore, each time the acquisition unit 20 acquires a captured image, it outputs the acquired captured image to the projection conversion unit 36.

[0168] Each time the detection point registration unit 29B acquires new surrounding location information from the detection unit 14, it performs scan matching with the surrounding location information already registered in the environmental map information 26A, determines the relative positional relationship for adding the new surrounding location information to the registered surrounding location information, and then adds the new surrounding location information to the environmental map information 26A.

[0169] The correction unit 28 corrects the surrounding location information registered in the environmental map information 26A for the detected points that have been matched multiple times by scan matching, using, for example, the least squares method, so that the sum of the difference in distance in three-dimensional space between the three-dimensional coordinates calculated in the past and the newly calculated three-dimensional coordinates is minimized.

[0170] The self-position estimation unit 27 can calculate the translation and rotation amounts of its own position from the positional relationship between the surrounding position information already registered in the environmental map information 26A and the newly added surrounding position information, and estimate it as self-position information.

[0171] Thus, in this embodiment, the image processing device 10 simultaneously performs the updating of surrounding position information and the estimation of the self-position information of the moving object 2 using SLAM.

[0172] Next, an example of the image processing flow, including distance stabilization processing, performed by the image processing apparatus 10 according to the second embodiment will be described.

[0173] Figure 17 is a flowchart showing an example of the image processing flow performed by the image processing device 10.

[0174] The acquisition unit 20 acquires the captured image from the shooting unit 12 (step S100). The acquisition unit 20 also acquires surrounding location information from the detection unit 14 (step S102).

[0175] Each time the detection point registration unit 29B acquires new surrounding location information from the detection unit 14, it performs scan matching with the surrounding location information already registered in the environmental map information 26A (step S104). Then, the detection point registration unit 29B determines the relative positional relationship for adding the new surrounding location information to the surrounding location information already registered in the environmental map information 26A, and adds the new surrounding location information to the environmental map information 26A (step S106).

[0176] The self-position estimation unit 27 can calculate the translation and rotation amounts of its own position from the positional relationship between the surrounding position information already registered in the environmental map information 26A and the newly added surrounding position information, and estimate it as self-position information (step S108). Then, the self-position estimation unit 27 adds the self-position information to the environmental map information 26A (step S110).

[0177] The correction unit 28 corrects the surrounding location information registered in the environmental map information 26A for the detection points that have been matched multiple times by scan matching, using, for example, the least squares method, so that the sum of the difference in distance in three-dimensional space between the three-dimensional coordinates calculated in the past and the newly calculated three-dimensional coordinates is minimized (step S112), and updates the environmental map information 26A.

[0178] The absolute distance conversion unit 30A of the determination unit 30 determines distance information from the current position of the moving object 2 to the multiple detection points P in the surrounding area based on the environmental map information 26A (step S114).

[0179] The extraction unit 30B extracts detection points P that are within a specific range from among the detection points P whose absolute distance information has been calculated by the absolute distance conversion unit 30A (step S116).

[0180] The nearest-nearest-point identification unit 30C uses the distance information of each detection point P extracted in step S116 to identify multiple detection points P in order of proximity to the moving object 2 for each range around the moving object 2 (step S118).

[0181] The distance stabilization processing unit 30I takes the measured distance d of the detection point P specified for each range as input, performs a first distance stabilization process and a second distance stabilization process, and outputs the stabilized distance D to the reference projection plane shape selection unit 30D, the scale determination unit 30E, the asymptotic curve calculation unit 30F, and the boundary area determination unit 30H (step S119).

[0182] The asymptotic curve calculation unit 30F calculates an asymptotic curve Q using the distance information of each of the multiple detection points P for each range identified in step S118 (step S120).

[0183] The reference projection plane shape selection unit 30D selects the shape of the reference projection plane 40 (step S122). As described above, the reference projection plane shape selection unit 30D will be described as an example of selecting a bowl-shaped reference projection plane 40.

[0184] The scale determination unit 30E determines the scale of the reference projection plane 40 of the shape selected in step S122 (step S124).

[0185] The shape determination unit 30G enlarges or reduces the reference projection plane 40 of the shape selected in step S122 to the scale determined in step S124. Then, the shape determination unit 30G deforms the enlarged or reduced reference projection plane 40 so that it follows the asymptotic curve Q calculated in step S120. The shape determination unit 30G determines this deformed shape as the projected shape 41 (step S126).

[0186] The deformation unit 32 deforms the reference projection plane 40 into the projection shape 41 determined by the determination unit 30 (step S128). Through this deformation process, the deformation unit 32 generates a deformed projection plane 42, which is the deformed reference projection plane 40.

[0187] The virtual viewpoint line of sight determination unit 34 determines the virtual viewpoint line of sight information (step S130). For example, the virtual viewpoint line of sight determination unit 34 determines the self-position S of the moving object 2 as the virtual viewpoint O, and the direction toward the vertex W of the asymptotic curve Q from the virtual viewpoint O as the line of sight direction L. More specifically, the virtual viewpoint line of sight determination unit 34 only needs to determine the direction toward the vertex W of a specific range of asymptotic curve Q from among the asymptotic curve Q calculated for each range in step S120 as the line of sight direction L.

[0188] The projection conversion unit 36 ​​projects the captured image acquired in step S100 onto the deformed projection surface 42 generated in step S128. The projection conversion unit 36 ​​then converts the projected image into a virtual viewpoint image, which is the image viewed from the virtual viewpoint O determined in step S130 in the line of sight direction L, where the captured image projected onto the deformed projection surface 42 is viewed (step S132).

[0189] The boundary area determination unit 30H determines the boundary area based on the distance to the nearest neighbor object identified for each range. That is, the boundary area determination unit 30H determines the boundary area as the superposition area of ​​spatially adjacent surrounding images based on the position of the nearest neighbor object of the moving object 2 (step S134). The boundary area determination unit 30H outputs the determined boundary area to the image synthesis unit 38.

[0190] The image synthesis unit 38 generates a composite image by joining spatially adjacent perspective projection images using a boundary region (step S136). That is, the image synthesis unit 38 joins perspective projection images in four directions according to a boundary region set at an angle to the direction of the nearest object to generate a composite image. In the boundary region, spatially adjacent perspective projection images are blended at a predetermined ratio.

[0191] The display unit 16 performs display control to display the generated composite image 54 (step S138).

[0192] Next, the image processing device 10 determines whether or not to terminate the image processing (step S140). For example, the image processing device 10 makes the determination in step S140 by determining whether or not it has received a signal from the ECU 3 indicating that the movement of the moving object 2 has stopped. Alternatively, for example, the image processing device 10 may make the determination in step S140 by determining whether or not it has received an instruction to terminate the image processing, such as an operation instruction from the user.

[0193] If the decision in step S140 is negative (step S140: No), the process from step S100 to step S140 is repeated. On the other hand, if the decision in step S140 is positive (step S140: Yes), this routine terminates.

[0194] Furthermore, if the system returns from step S140 to step S100 after executing the correction process in step S112, the subsequent correction process in step S112 may be omitted. Also, if the system returns from step S140 to step S100 without executing the correction process in step S112, the subsequent correction process in step S112 may be executed.

[0195] As described above, the image processing apparatus 10 according to the second embodiment acquires the measured distance between the moving object 2 and surrounding three-dimensional objects by three-dimensional LiDAR SLAM processing. The determination unit 30 performs a first distance stabilization process that converts the measured distance to a first distance or a second distance smaller than the first distance, based on the relationship between the acquired measured distance and a first threshold. The deformation unit 32 deforms the projection plane of the surrounding image of the moving object 2 based on the first distance or second distance to which the measured distance has been converted. Therefore, the image processing apparatus 10 according to the second embodiment can achieve the same effects as the image processing apparatus 10 according to the first embodiment.

[0196] (Modification 4) In each of the embodiments described above, an image processing device 10 using SLAM was described. In contrast, it is also possible to create environmental map information using immediate values ​​from distance sensors (millimeter-wave radar, laser sensors), sonar sensors that detect objects using sound waves, ultrasonic sensors, sensor arrays constructed from multiple distance sensors, etc., without using SLAM, and then perform image processing including the distance stabilization process described above using this information.

[0197] Although various embodiments and modifications have been described above, the image processing apparatus, image processing method, and image processing program disclosed in this application are not limited to the above embodiments, etc., and the components can be modified and implemented in each implementation stage, etc., without departing from the gist of the invention. Furthermore, various inventions can be formed by appropriately combining the multiple components disclosed in the above embodiments, etc. For example, some components may be deleted from all the components shown in the embodiments.

[0198] Furthermore, the image processing apparatus 10 of the first and second embodiments described above can be applied to various devices. For example, the image processing apparatus 10 of the first and second embodiments can be applied to a surveillance camera system that processes video obtained from a surveillance camera, or an in-vehicle system that processes images of the surrounding environment outside the vehicle. [Explanation of Symbols]

[0199] 2 Mobile Units 10 Image Processing Device 12, 12A~12D Photography Department 14 Detection unit 20 Acquisition Department 23 Selection Section 24 VSLAM Processing Unit 25 Matching Department 26 Memory section 26A Environmental Map Information 27 Self-position estimation part 28 Correction section 29 Three-Dimensional Restoration Section 29B Detection point registration unit 30 Decision Section 30A Absolute Distance Conversion Unit 30B Extraction part 30C Nearest neighbor identification part 30D Reference Projection Plane Shape Selection Section 30E Scale Determination Unit 30F Asymptotic curve calculation section 30G shape determining section 30H Boundary area determination part 30I Distance Stabilization Processing Unit 32 Deformed part 34 Virtual viewpoint line of sight determination unit 36 Projection Transformation Unit 37 Image generation unit 38 Image Synthesis Unit

Claims

1. A conversion unit generates a stabilization distance by converting the measured distance to a first distance or a second distance smaller than the first distance, based on the relationship between the measured distance between the moving object and a three-dimensional object surrounding the moving object and the moving object, and a first threshold value. Based on the stabilization distance, a deformation unit deforms the projection plane of the surrounding image of the moving object so that the distance between the self-position corresponding to the moving object and the projection plane changes. An image processing device equipped with the following features.

2. The conversion unit, if the measurement distance is smaller than the first threshold, converts the measurement distance to the second distance to generate the stabilization distance, and after converting the measurement distance to the second distance, if the measurement distance is greater than the second threshold which is greater than the first threshold, converts the measurement distance back to the first distance to generate the stabilization distance. The image processing apparatus according to claim 1.

3. If the measurement distance is smaller than the first threshold, the conversion unit converts the measurement distance to the second distance or the third distance smaller than the second distance based on the relationship between the measurement distance and the third threshold which is smaller than the first threshold, in order to generate the stabilized distance. The image processing apparatus according to claim 2.

4. The system further includes a preprocessor that performs at least one of the following processes on the measurement distance: spatial outlier removal, temporal outlier removal, spatial smoothing, and temporal smoothing. The conversion unit uses the measurement distance output from the preprocessing unit to convert the measurement distance into the stabilization distance. The image processing apparatus according to any one of claims 1 to 3.

5. A SLAM processing unit generates map information of the area around the moving body using detection point position information, which is obtained by accumulating detection points corresponding to the three-dimensional objects around the moving body, and the self-position information of the moving body. A distance conversion unit calculates the distance information between the moving object and surrounding three-dimensional objects based on the map information as the measured distance. An image processing apparatus according to any one of claims 1 to 4.

6. The deformed portion has an image generation unit that projects the surrounding image onto the deformed projection surface, The SLAM processing unit performs Visual SLAM (Simultaneous Localization and Mapping) processing using the surrounding image to calculate the measurement distance. The image processing apparatus according to claim 5.

7. A computer-based image processing method, A step of obtaining the measured distance between the moving object and a three-dimensional object in the vicinity of the moving object, A step of generating a stabilized distance by converting the measured distance to a first distance or a second distance smaller than the first distance, based on the relationship between the measured distance and a first threshold; The step of deforming the projection plane of the surrounding image of the moving object based on the stabilization distance such that the distance between the self-position corresponding to the moving object and the projection plane changes, Image processing methods.

8. On the computer, A step of obtaining the measured distance between the moving object and a three-dimensional object in the vicinity of the moving object, A step of generating a stabilized distance by converting the measured distance to a first distance or a second distance smaller than the first distance, based on the relationship between the measured distance and a first threshold; Based on the stabilization distance, the step of deforming the projection plane of the surrounding image of the moving object so that the distance between the self-position corresponding to the moving object and the projection plane changes, An image processing program to execute this.

Images