Image processing apparatus, image processing method, and recording medium

Abstract

The image processing apparatus includes a decision section (30) and an image generation section (37) in one embodiment. The decision section (30) decides a boundary region in a repeated region of spatially adjacent peripheral images of a plurality of peripheral images of a mobile body using position information of a plurality of detection points including the periphery of the mobile body and self position information of the mobile body. The image generation section uses the boundary region to generate a composite image using the spatially adjacent peripheral images.

Description

Technical Field

[0001] This invention relates to an image processing apparatus, an image processing method, and a recording medium. Background Technology

[0002] The following technology is disclosed: using a projected image obtained by projecting images of the periphery of a moving object onto a virtual projection surface to generate a composite image from an arbitrary viewpoint.

[0003] Patent Document 1: Japanese Patent Application Publication No. 2013-207637

[0004] Patent Document 2: Japanese Patent Publication No. 2014-531078

[0005] Patent Document 3: Japanese Patent Application Publication No. 2009-232310

[0006] Patent Document 4: Japanese Patent Application Publication No. 2019-140518

[0007] Patent Document 5: Japanese Patent Application Publication No. 2019-153138

[0008] Non-patent literature 1: "Environmental Recognition - Map Building and Self-Position Inference of Mobile Robots" System / Control / Information (Journal of the Chinese Society for Systems Information Science), Vol. 60 No. 12, pp. 509-514, 2016

[0009] However, when a composite image composed of multiple superimposed images is projected and displayed on a projection surface, there are undesirable situations in the projected composite image, such as double parking lines on the horizontal plane. Summary of the Invention

[0010] In one aspect, the object of the present invention is to provide an image processing apparatus, an image processing method, and a recording medium that eliminates undesirable situations when a composite image formed by superimposing multiple images is projected and displayed on a projection surface.

[0011] The image processing apparatus disclosed in this application includes, in one embodiment, a determination unit that uses position information of multiple detection points including the periphery of a moving body and position information of the moving body itself to determine a boundary region among overlapping regions of spatially adjacent peripheral images in multiple peripheral images of the moving body; and an image generation unit that uses the boundary region to generate a composite image using the spatially adjacent peripheral images.

[0012] According to one aspect of the image processing apparatus disclosed in this application, it is possible to eliminate undesirable situations when a composite image formed by superimposing multiple images is projected and displayed on a projection surface. Attached Figure Description

[0013] Figure 1 This is a diagram showing the overall structure of the image processing system according to the first embodiment.

[0014] Figure 2 This is a diagram showing the hardware structure of the image processing apparatus according to the first embodiment.

[0015] Figure 3 This is a diagram showing the functional structure of the image processing apparatus according to the first embodiment.

[0016] Figure 4 This is a schematic diagram of the environmental map information in the first embodiment.

[0017] Figure 5 This is an explanatory diagram of the asymptotic curves of the first embodiment.

[0018] Figure 6 This is a schematic diagram showing an example of a reference projection plane of the first embodiment.

[0019] Figure 7 This is a schematic diagram illustrating an example of a projected shape determined by the shape determination unit of the first embodiment.

[0020] Figure 8 This is a schematic diagram illustrating an example of the functional structure of the decision-making unit in the first embodiment.

[0021] Figure 9 It is a diagram used to illustrate boundary angles and boundary widths.

[0022] Figure 10 This is a conceptual diagram representing the process of projecting a synthesized image onto a reference projection plane.

[0023] Figure 11 It is a schematic representation Figure 10 An example of a composite image is shown.

[0024] Figure 12 It means targeting Figure 10 The diagram shown illustrates the processing of a composite image under the condition that the projection of the reference projection plane has undergone projection shape determination processing.

[0025] Figure 13 It is a schematic representation Figure 12 An example of a composite image is shown.

[0026] Figure 14 It means from Figure 12 , Figure 13 The diagram shown illustrates the processing of a composite image based on the projected shape, assuming the moving body has advanced further toward the parking line.

[0027] Figure 15 It is a schematic representation Figure 14 An example of a composite image is shown.

[0028] Figure 16 This indicates that correction is achieved through adaptive control of the boundary region. Figure 14 A conceptual diagram illustrating an example of projection processing.

[0029] Figure 17 It is a schematic representation Figure 16 An example of a composite image is shown.

[0030] Figure 18 This is a flowchart illustrating an example of an image processing procedure performed by the image processing apparatus of the first embodiment.

[0031] Figure 19 This diagram is used to illustrate a comparative example where boundary region adaptive control is not performed.

[0032] Figure 20 This is a diagram representing an example of a synthetic image obtained through image processing that includes boundary region adaptation control.

[0033] Figure 21 This diagram is used to illustrate a comparative example where boundary region adaptive control is not performed.

[0034] Figure 22 This is a diagram representing an example of a synthetic image obtained through image processing that includes boundary region adaptation control.

[0035] Figure 23 This diagram is used to illustrate a comparative example where boundary region adaptive control is not performed.

[0036] Figure 24 This is a diagram representing an example of a synthetic image obtained through image processing that includes boundary region adaptation control.

[0037] Figure 25 This is a diagram illustrating an example of the functional structure of the image processing apparatus according to the second embodiment.

[0038] Figure 26 This is a flowchart illustrating an example of an image processing procedure performed by the image processing apparatus of the second embodiment. Detailed Implementation

[0039] Hereinafter, embodiments of the image processing apparatus, image processing method, and recording medium disclosed in this application will be described in detail with reference to the accompanying drawings. Furthermore, the following embodiments are not limited to the disclosed technology. Moreover, the embodiments can be appropriately combined without contradicting the processing content.

[0040] (First Implementation)

[0041] Figure 1 This diagram illustrates an example of the overall structure of the image processing system 1 according to this embodiment. The image processing system 1 includes an image processing device 10, an imaging unit 12, a detection unit 14, and a display unit 16. The image processing device 10, the imaging unit 12, the detection unit 14, and the display unit 16 are connected to transmit and receive data or signals.

[0042] In this embodiment, as an example, the image processing device 10, the imaging unit 12, the detection unit 14, and the display unit 16 are described as being mounted on the mobile body 2. Furthermore, the image processing device 10 of the first embodiment is an example using Visual SLAM.

[0043] The mobile body 2 refers to an object capable of movement. Examples of mobile body 2 include vehicles, flying objects (manned aircraft, unmanned aerial vehicles (e.g., UAVs), drones), robots, etc. Furthermore, mobile body 2 can be a mobile body that moves under human control or a mobile body that can move automatically (autonomous movement) without human control. In this embodiment, as an example, the case where mobile body 2 is a vehicle will be described. Vehicles include, for example, two-wheeled cars, three-wheeled cars, four-wheeled cars, etc. In this embodiment, as an example, the case where the vehicle is a four-wheeled car capable of autonomous movement will be described.

[0044] Furthermore, it is not limited to mounting the image processing unit 10, the imaging unit 12, the detection unit 14, and the display unit 16 all on the moving body 2. The image processing unit 10 can also be mounted on a stationary object. A stationary object is an object fixed to the ground. A stationary object is an object that cannot move and is stationary relative to the ground. Examples of stationary objects include traffic lights, parked vehicles, and road signs. In addition, the image processing unit 10 can also be mounted on a cloud server that performs processing in the cloud.

[0045] The imaging unit 12 captures images of the periphery of the moving object 2, acquiring captured image data. Hereinafter, the captured image data will be simply referred to as captured images. The imaging unit 12 is, for example, a digital camera capable of dynamic imaging. Furthermore, imaging refers to converting the image of the subject, which is imaged through an optical system such as a lens, into an electrical signal. The imaging unit 12 outputs the captured images to the image processing device 10. In this embodiment, it is assumed that the imaging unit 12 is a monocular fisheye camera (e.g., with a field of view of 195 degrees).

[0046] In this embodiment, as an example, a configuration in which four imaging units 12 (imaging units 12A to 12D) are mounted on the moving body 2 will be described. The multiple imaging units 12 (imaging units 12A to 12D) capture images of the subject within their respective imaging regions E (imaging regions E1 to E4), thereby acquiring images. Furthermore, the imaging directions of these multiple imaging units 12 are different from each other. Additionally, the imaging directions of these multiple imaging units 12 are pre-adjusted so that at least a portion of the imaging regions E overlaps with adjacent imaging units 12.

[0047] Furthermore, the four camera units 12A to 12D are just one example; the number of camera units 12 is not limited. For instance, if the moving body 2 has a longitudinal shape like a bus or truck, one camera unit 12 can be placed at the front, rear, front of the right side, rear of the right side, front of the left side, and rear of the left side of the moving body 2, for a total of six camera units 12. That is, the number and placement of the camera units 12 can be arbitrarily set according to the size and shape of the moving body 2. In addition, the boundary angle determination process described later can be implemented by providing at least two camera units 12.

[0048] The detection unit 14 detects the position information of multiple detection points around the moving body 2. In other words, the detection unit 14 detects the position information of the detection points in the detection area F. A detection point refers to an individual point in actual space that is independently observed by the detection unit 14. A detection point may correspond, for example, to a three-dimensional object surrounding the moving body 2. For example, the detection unit 14 illuminates light around itself and receives reflected light from a reflection point. This reflection point corresponds to a detection point.

[0049] The location information of the detection point refers to information indicating the position of the detection point in actual space (three-dimensional space). For example, the location information of the detection point indicates the distance from the detection unit 14 (i.e., the position of the moving body 2) to the detection point and the direction of the detection point with reference to the detection unit 14. These distances and directions can be represented, for example, by position coordinates indicating the relative position of the detection point with reference to the detection unit 14, position coordinates indicating the absolute position of the detection point, or vectors.

[0050] The detection unit 14 can be, for example, a 3D (Three-Dimensional) scanner, a 2D (Two-Dimensional) scanner, a distance sensor (millimeter-wave radar, laser sensor), a sonar sensor that detects objects using sound waves, an ultrasonic sensor, etc. The laser sensor can be, for example, a 3D LiDAR (Laser Imaging Detection and Ranging) sensor. Alternatively, the detection unit 14 can also be a device using SfM (Structure from Motion) technology, which measures distance based on images captured by a monocular camera. Alternatively, multiple imaging units 12 can be used as the detection unit 14. Alternatively, one of the multiple imaging units 12 can be used as the detection unit 14.

[0051] The display unit 16 displays various information. The display unit 16 is, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display.

[0052] In this embodiment, the image processing device 10 is connected to the electronic control unit (ECU) 3 mounted on the mobile body 2 and is able to communicate. The ECU 3 is a unit that performs electronic control of the mobile body 2. In this embodiment, the image processing device 10 can receive CAN (Controller Area Network) data such as the speed and direction of movement of the mobile body 2 from the ECU 3.

[0053] Next, the hardware structure of the image processing device 10 will be described.

[0054] Figure 2 This is a diagram illustrating an example of the hardware structure of the image processing device 10.

[0055] The image processing device 10 includes a CPU (Central Processing Unit) 10A, a ROM (Read Only Memory) 10B, a RAM (Random Access Memory) 10C, and an I / F (Interface) 10D, such as a computer. The CPU 10A, ROM 10B, RAM 10C, and I / F 10D are interconnected via a bus 10E, forming a hardware structure using a typical computer.

[0056] CPU 10A is a computing device that controls the image processing apparatus 10. CPU 10A corresponds to an example of a hardware processor. ROM 10B stores programs that implement various processes based on CPU 10A. RAM 10C stores data required for various processes based on CPU 10A. I / F 10D is an interface for transmitting and receiving data, connected to the imaging unit 12, detection unit 14, display unit 16, and ECU 3.

[0057] The program for performing image processing executed by the image processing apparatus 10 of this embodiment is provided pre-assembled in the ROM 10B or the like. Furthermore, the program executed by the image processing apparatus 10 of this embodiment may also be configured to be provided as a file recorded on a recording medium in a form that can be installed on or executed by the image processing apparatus 10. The recording medium is a computer-readable medium. Recording media include CD (Compact Disc)-ROM, floppy disk (FD), CD-R (Recordable), DVD (Digital Versatile Disk), USB (Universal Serial Bus) memory, SD (Secure Digital) card, etc.

[0058] Next, the functional structure of the image processing apparatus 10 in this embodiment will be described. The image processing apparatus 10 uses Visual SLAM (Simultaneous Localization and Mapping) to simultaneously infer the position information of the detection point and the position information of the moving body 2 based on the captured images obtained by the imaging unit 12. The image processing apparatus 10 connects multiple spatially adjacent captured images to generate and display a composite image overlooking the perimeter of the moving body 2. Furthermore, in this embodiment, the imaging unit 12 is used as the detection unit 14.

[0059] Figure 3 This is a diagram illustrating an example of the functional structure of the image processing apparatus 10. Furthermore, in Figure 3 In order to clarify the input and output relationship of data, the imaging unit 12 and the display unit 16 are illustrated together with the image processing device 10.

[0060] The image processing apparatus 10 includes an acquisition unit 20, a selection unit 23, a matching unit 25, a self-position inference unit 27, a detection point registration unit 29, a storage unit 26, a correction unit 28, a decision unit 30, a deformation unit 32, a virtual viewpoint line of sight determination unit 34, a projection transformation unit 36, and an image synthesis unit 38.

[0061] Some or all of the aforementioned components can also be implemented, for example, by having a processing device such as a CPU10A execute a program, i.e., by software. Alternatively, some or all of the aforementioned components can also be implemented by hardware such as an IC (Integrated Circuit), or by a combination of software and hardware.

[0062] The acquisition unit 20 acquires captured images from the imaging unit 12. The acquisition unit 20 acquires captured images from each imaging unit in the imaging unit 12 (imaging units 12A to imaging units 12D).

[0063] Each time an image is acquired, the acquisition unit 20 outputs the acquired image to the projection transformation unit 36 ​​and the selection unit 23.

[0064] The selection unit 23 selects the detection area of ​​the detection point. In this embodiment, the selection unit 23 selects the detection area by selecting at least one of the plurality of imaging units 12 (imaging units 12A to imaging units 12D).

[0065] In this embodiment, the selection unit 23 selects at least one imaging unit 12 using vehicle status information, detection direction information contained in the CAN data received from the ECU3, or instruction information input based on the user's operation instructions.

[0066] Vehicle status information includes, for example, information indicating the direction of travel of moving body 2, the status of the direction indicator of moving body 2, and the status of the gears of moving body 2. Vehicle status information can be derived from CAN data. Detection direction information indicates the direction in which information of interest has been detected and can be derived using POI (Point of Interest) technology. Indication information indicates the direction in which interest should be focused and is input based on user input.

[0067] For example, the selection unit 23 uses vehicle status information to select the direction of the detection area. Specifically, the selection unit 23 uses vehicle status information to determine parking information such as rear parking information indicating parking behind the moving body 2 and longitudinal parking information indicating parking in a tandem. The selection unit 23 pre-stores the parking information in a correspondence with the identification information of any one of the imaging units 12. For example, the selection unit 23 pre-stores the imaging unit 12D (refer to) that captures images of the moving body 2 from behind, in a correspondence with the rear parking information. Figure 1 The identification information of the camera 12B and camera 12C in the left and right directions of the camera moving body 2 is pre-stored in relation to the longitudinal parking information by the selection unit 23. Figure 1 Each of them has its own identification information.

[0068] Furthermore, the selection unit 23 selects the direction of the detection area by selecting the imaging unit 12 that corresponds to the parking information derived from the received vehicle status information.

[0069] Alternatively, the selection unit 23 can also select the imaging unit 12 that uses the direction indicated by the detection direction information as the imaging area E. Alternatively, the selection unit 23 can also select the imaging unit 12 that uses the direction indicated by the detection direction information derived from POI technology as the imaging area E.

[0070] The selection unit 23 outputs the captured image from the captured image acquired by the acquisition unit 20, which was captured by the selected capturing unit 12, to the matching unit 25.

[0071] The matching unit 25 performs feature extraction and matching processing on multiple captured images (multiple captured images with different frame rates) taken at different times. Specifically, the matching unit 25 performs feature extraction processing on these multiple captured images. For the multiple captured images taken at different times, the matching unit 25 uses the feature rates to determine corresponding points between the multiple captured images. The matching unit 25 outputs its matching processing results to its own position inference unit 27. Furthermore, the matching unit 25 registers the information of the corresponding points between the multiple captured images in the environmental map information 26A.

[0072] Storage unit 26 stores various types of data. Storage unit 26 can be, for example, a semiconductor memory element such as RAM or flash memory, a hard disk, or an optical disk. Alternatively, storage unit 26 can be a storage device located external to the image processing device 10. Furthermore, storage unit 26 can also be a storage medium. Specifically, the storage medium can download programs and various information via a LAN (Local Area Network), the Internet, etc., and store or temporarily store them.

[0073] Environmental map information 26A is map information representing the surrounding conditions of the moving body 2. Environmental map information 26A is information that registers the position information of each detection point and the position information of the moving body 2 in a three-dimensional coordinate space with a specified location in the actual space as the origin. The specified location in the actual space can, for example, be determined based on pre-set conditions.

[0074] For example, the predetermined position is the position of the moving body 2 when the image processing device 10 performs image processing according to this embodiment. For example, assume that image processing is performed at a predetermined time, such as in a parking scenario of the moving body 2. In this case, the image processing device 10 can simply set the position of the moving body 2 when it is determined that the predetermined time has been reached as the predetermined position. For example, when the image processing device 10 determines that the action of the moving body 2 is an action indicating a parking scenario, it can determine that the predetermined time has been reached. Actions indicating a parking scenario include, for example, the speed of the moving body 2 falling below a predetermined speed, the gear of the moving body 2 shifting into reverse, or receiving a signal indicating the start of parking based on user operation instructions. Furthermore, the predetermined time is not limited to parking scenarios.

[0075] Figure 4 This is a schematic diagram of an example of environmental map information 26A. For example... Figure 4 As shown, the environmental map information 26A is information that registers the position information of each detection point P and the position information of the moving body 2 itself in the corresponding coordinate positions in the three-dimensional coordinate space.

[0076] The self-position inference unit 27 uses information about points identified as corresponding points contained in the matching processing results obtained from the matching unit 25 to infer the self-position information of the moving body 2 through triangulation. Here, self-position information refers to information representing the posture of the moving body 2. The posture of the moving body 2 represents its position and tilt. The self-position information includes, for example, information such as the position of the moving body 2 corresponding to each shooting time in different shooting times, and the orientation of the shooting unit 12.

[0077] Furthermore, the self-position inference unit 27 registers the calculated self-position information in the environmental map information 26A.

[0078] Furthermore, the self-position inference unit 27 can also infer its own position information using the range measurement method. In this case, the self-position inference unit 27 can infer new self-position information by using the previously calculated self-position information and the movement amount of the moving body 2 through integral calculation based on the range measurement method. Additionally, the self-position inference unit 27 can obtain the movement amount by reading the movement amount of the moving body 2 contained in the CAN data acquired from the ECU 3.

[0079] The detection point registration unit 29 uses the self-position information of the moving body 2, which is inferred by the self-position inference unit 27, corresponding to each shooting time in different shooting times, to calculate the amount of movement (translation and rotation) of the moving body 2. Moreover, based on this amount of movement, the relative coordinates (detection points) of corresponding points among the multiple captured images determined by the matching unit 25 with respect to the self-position of the moving body 2 are calculated.

[0080] Additionally, the detection point registration unit 29 registers these coordinates as the coordinates of detection point P in the environmental map information 26A. Furthermore, the coordinates of detection point P registered in the environmental map information 26A can also be transformed into coordinates with a specified location as the origin.

[0081] Therefore, in environmental map information 26A, as the moving body 2 moves, the location information of the new detection point P and its own location information S are sequentially added and registered. Figure 4 In the example, S represents the position S of itself from position S1 to position S3. The larger the value of S, the closer it is to the current timing position S.

[0082] The calibration unit 28 calibrates the position information of each of the multiple detection points P registered in the environmental map information 26A, as well as the position information of the moving body 2 itself. The calibration unit 28 uses the position information of each of the corresponding detection points P acquired at an acquisition time after the acquisition time of the detection point P to calibrate the position information registered in the environmental map information 26A and its own position information.

[0083] That is, the correction unit 28 uses the location information of the newly registered detection point P by the detection point registration unit 29 to correct the location information of the detection point P registered in the environmental map information 26A. At this time, the correction unit 28 can also use various parameters used to calculate the location information of each detection point P registered in the environmental map information 26A to correct the location information and its own location information. Through this correction process, the correction unit 28 corrects the error in the location information of the detection point P registered in the environmental map information 26A. The correction unit 28 can correct the location information of the detection point P registered in the environmental map information 26A using methods such as least squares. Through the correction process based on the correction unit 28, the cumulative error in the location information of the detection point P is corrected.

[0084] The timing of the correction processing based on the correction unit 28 is not limited. For example, the correction unit 28 may perform the correction processing at predetermined time intervals. The predetermined time intervals may also be determined based on preset conditions. Furthermore, in this embodiment, as an example, the image processing apparatus 10 is described in the case where it has a structure including the correction unit 28. However, the image processing apparatus 10 may also have a structure without the correction unit 28.

[0085] The decision unit 30 uses the position information of the detection point P accumulated in the environmental map information 26A to determine the projection shape of the projection surface and generate projection shape information. The decision unit 30 outputs the generated projection shape information to the deformation unit 32.

[0086] Here, the projection surface refers to the three-dimensional surface used to project the peripheral image of the moving body 2. Furthermore, the peripheral image of the moving body 2 refers to the captured image of the periphery of the moving body 2, which is the captured image taken by each of the capturing units 12A to 12D. The projection shape of the projection surface is a three-dimensional (3D) shape virtually formed in a virtual space corresponding to the actual space. In this embodiment, the determination of the projection shape of the projection surface performed by the determination unit 30 is referred to as "projection shape determination processing."

[0087] The decision unit 30 uses the position information of multiple detection points P around the moving body 2, which are stored in the environmental map information 26A, and the position information of the moving body 2 itself, to determine the boundary region and generate boundary region information. The decision unit 30 outputs the generated boundary region information to the image synthesis unit 38.

[0088] Here, the boundary region refers to the area where two spatially adjacent peripheral images acquired by the imaging units 12 (12A-12D) are superimposed when multiple peripheral images of the moving body 2 are spatially connected to generate a composite image. This boundary region is determined, for example, by the boundary angle and boundary width corresponding to the center position of the boundary region. In this embodiment, the determination of the boundary region performed by the determination unit 30 is referred to as "boundary region determination processing".

[0089] In addition, the decision unit 30 uses the location information of multiple detection points P around the moving body 2 and the location information of the moving body 2 itself, which are stored in the environmental map information 26A, to calculate the asymptotic curve and generate asymptotic curve information.

[0090] Figure 5 This is an explanatory diagram of the asymptotic curve Q generated by the decision unit 30. Here, the asymptotic curve refers to the asymptotic curve of multiple detection points P in the environmental map information 26A. Figure 5 This example illustrates how an asymptotic curve Q is represented on a projected image obtained by projecting a captured image onto a projection surface when the moving body 2 is viewed from above. For instance, suppose the determination unit 30 determines three detection points P in order of proximity to the moving body 2's own position S. In this case, the determination unit 30 generates the asymptotic curve Q for these three detection points P.

[0091] The decision unit 30 outputs its own position and asymptotic curve information to the virtual viewpoint line-of-sight decision unit 34.

[0092] Furthermore, the structure of the decision section 30 will be explained in detail later.

[0093] The deformation unit 32 deforms the projection surface based on the projection shape information received from the determination unit 30.

[0094] Figure 6This is a schematic diagram representing an example of a reference projection plane 40. Figure 7 This is a schematic diagram illustrating an example of the projection shape 41 determined by the determination unit 30. That is, the deformation unit 32, based on the projection shape information, modulates the pre-stored... Figure 6 The reference projection plane shown is deformed to determine its role. Figure 7 The deformed projection surface 42 of the projected shape 41 is shown. The determination unit 30 generates deformed projection surface information based on the projected shape 41. The deformation of the reference projection surface is performed with the detection point P closest to the moving body 2 as a reference. The deformation unit 32 outputs the deformed projection surface information to the projection transformation unit 36.

[0095] Additionally, for example, the deformation unit 32 deforms the reference projection surface into a shape along an asymptotic curve of a plurality of detection points P in a predetermined number of steps in the order of approaching the moving body 2, based on the projection shape information.

[0096] Furthermore, the preferred deformation unit 32 uses the position information of the detection point P acquired before the first timing and its own position information of its own position S to deform the reference projection surface.

[0097] Here, the first timing refers to the latest timing at which the matching unit 25 detects the position information of the detection point P, or any timing earlier than the latest timing. For example, the detection point P acquired before the first timing includes the position information of a specific object located around the moving body 2, while the detection point P acquired at the first timing does not include the position information of that specific object located around the moving body 2. The determination unit 30 uses the position information of the detection point P acquired before the first timing, contained in the environment map information 26A, to determine the projection shape in the same way as described above. Furthermore, the deformation unit 32 uses the projection shape information of the projection shape to generate the deformed projection surface in the same way as described above.

[0098] For example, the position information of the detection point P detected by the matching unit 25 at the first timing may sometimes not include the position information of the detection point P detected in the past at that timing. In this case, the deformation unit 32 can also generate a deformable projection surface corresponding to the previously detected detection point P.

[0099] The virtual viewpoint line-of-sight determination unit 34 determines the virtual viewpoint line-of-sight information based on its own position and asymptotic curve information.

[0100] While referring to Figure 5 , Figure 7The determination of virtual viewpoint viewing information is explained. For example, the virtual viewpoint viewing determination unit 34 determines the viewing direction as the direction that passes through the detection point P closest to its own position S of the moving body 2 and is perpendicular to the deformable projection plane. Furthermore, the virtual viewpoint viewing determination unit 34, for example, fixes the direction of this viewing direction L and determines the coordinates of the virtual viewpoint O as an arbitrary Z coordinate and arbitrary XY coordinates in the direction away from its own position S from the asymptotic curve Q. In this case, the XY coordinates can also be the coordinates of a position further away from the asymptotic curve Q than its own position S. Moreover, the virtual viewpoint viewing determination unit 34 outputs virtual viewpoint viewing information representing the virtual viewpoint O and the viewing direction L to the projection transformation unit 36. Furthermore, as... Figure 7 As shown, the line of sight L can also be the direction from the virtual viewpoint O toward the vertex W of the asymptotic curve Q.

[0101] The projection transformation unit 36 ​​generates a projected image obtained by projecting the captured image acquired by the imaging unit 12 onto the distorted projection surface based on the deformed projection surface information and the virtual viewpoint viewing information. The projection transformation unit 36 ​​transforms the generated projected image into a virtual viewpoint image and outputs it to the image synthesis unit 38. Here, the virtual viewpoint image refers to the image obtained by visually confirming the projected image from a virtual viewpoint in any direction.

[0102] While referring to Figure 7 The projection image generation process based on the projection transformation unit 36 ​​will be described in detail below. The projection transformation unit 36 ​​projects the captured image onto the deformable projection surface 42. Furthermore, the projection transformation unit 36 ​​generates a virtual viewpoint image, which is an image (not shown) obtained by visually confirming the captured image projected onto the deformable projection surface 42 from any virtual viewpoint O in the viewing direction L. The position of the virtual viewpoint O can be, for example, the latest self-position S of the moving body 2. In this case, the XY coordinates of the virtual viewpoint O can be the XY coordinates of the latest self-position S of the moving body 2. In addition, the Z coordinate (vertical position) of the virtual viewpoint O can be the Z coordinate of the detection point P closest to the self-position S of the moving body 2. The viewing direction L can also be determined based on a predetermined reference, for example.

[0103] The viewing direction L is, for example, the direction from the virtual viewpoint O toward the detection point P, which is closest to the position S of the moving body 2. Alternatively, the viewing direction L can also be a direction that passes through the detection point P and is perpendicular to the deformable projection plane 42. The virtual viewpoint viewing information, representing the virtual viewpoint O and the viewing direction L, is generated by the virtual viewpoint viewing determination unit 34.

[0104] For example, the virtual viewpoint line-of-sight determination unit 34 can also determine the line-of-sight direction L as the direction that passes through the detection point P closest to its own position S of the moving body 2 and is perpendicular to the deformable projection plane 42. Alternatively, the virtual viewpoint line-of-sight determination unit 34 can fix the direction of this line-of-sight direction L and determine the coordinates of the virtual viewpoint O as an arbitrary Z coordinate and arbitrary XY coordinates in the direction away from its own position S from the asymptotic curve Q. In this case, the XY coordinates can also be the coordinates of a position further away from the asymptotic curve Q than its own position S. Furthermore, the virtual viewpoint line-of-sight determination unit 34 outputs the virtual viewpoint line-of-sight information representing the virtual viewpoint O and the line-of-sight direction L to the projection transformation unit 36. Furthermore, it can also be as follows... Figure 6 As shown, the line of sight L is the direction from the virtual viewpoint O toward the vertex W of the asymptotic curve Q.

[0105] The projection transformation unit 36 ​​receives virtual viewpoint viewing information from the virtual viewpoint viewing determination unit 34. By receiving this virtual viewpoint viewing information, the projection transformation unit 36 ​​determines the virtual viewpoint O and the viewing direction L. Furthermore, based on the captured image projected onto the distorted projection surface 42, the projection transformation unit 36 ​​generates an image visually confirmed from the virtual viewpoint O in the viewing direction L, i.e., a virtual viewpoint image. The projection transformation unit 36 ​​outputs the virtual viewpoint image to the image synthesis unit 38.

[0106] The image compositing unit 38 generates a composite image from which a portion or all of the virtual viewpoint images have been extracted. For example, the image compositing unit 38 performs a connection process for multiple virtual viewpoint images in the boundary region (here, the four virtual viewpoint images corresponding to the capturing units 12A to 12D), and a blending process for each image in this connection process. Here, the blending process refers to the process of blending spatially adjacent images in the boundary region at a predetermined ratio (blending ratio). The blending ratio can, for example, be set in stages to vary with the distance from the center of the boundary region. Furthermore, the value of the blending ratio and the staged setting of the blending ratio can be arbitrarily adjusted. Alternatively, the blending process can be omitted.

[0107] The image compositing unit 38 outputs the generated composite image to the display unit 16. Alternatively, the composite image may be a bird's-eye view image with the top of the moving body 2 as a virtual viewpoint O, or an image of the moving body 2 displayed semi-transparently with the interior of the moving body 2 as a virtual viewpoint O.

[0108] Furthermore, the projection transformation unit 36 ​​and the image synthesis unit 38 constitute the image generation unit 37. The image generation unit 37 is an example of an image generation unit that uses boundary regions to generate a synthesized image using spatially adjacent surrounding images.

[0109] (Structural example of the decision section 30)

[0110] Next, an example of the detailed structure of the decision section 30 will be explained.

[0111] Figure 8 This is a schematic diagram illustrating an example of the structure of the determination unit 30. (Example) Figure 8 As shown, the decision unit 30 includes an absolute distance conversion unit 30A, an extraction unit 30B, a nearest neighbor determination unit 30C, a reference projection plane shape selection unit 30D, a scale determination unit 30E, an asymptotic curve calculation unit 30F, a shape determination unit 30G, and a boundary region determination unit 30H.

[0112] The absolute distance conversion unit 30A reads the environment map information 26A from the storage unit 26. The absolute distance conversion unit 30A converts the position information of each of the multiple detection points P contained in the read environment map information 26A into distance information representing the absolute distance from the latest position S of the moving body 2 (i.e., its current position) to each of the multiple detection points P. Furthermore, if the detection unit 14 acquires the distance information of the detection points P, the absolute distance conversion unit 30A can be omitted.

[0113] In detail, the absolute distance conversion unit 30A uses the speed data of the mobile body 2 contained in the CAN data received from the ECU3 of the mobile body 2 to calculate the current position of the mobile body 2.

[0114] Specifically, for example, the absolute distance conversion unit 30A uses the speed data of the moving body 2 contained in the CAN data to calculate the distance between its own position S registered in the environmental map information 26A. For example, assuming Figure 4 The environmental map information 26A is shown. In this case, the absolute distance conversion unit 30A uses the speed data contained in the CAN data to calculate the distance between its own position S1 and its own position S2, and the distance between its own position S2 and its own position S3. Moreover, the absolute distance conversion unit 30A uses these distances to calculate the current position of the moving body 2.

[0115] Furthermore, the absolute distance conversion unit 30A calculates the distance, i.e., the distance information, from the current position of the moving body 2 to each of the multiple detection points P contained in the environmental map information 26A. Specifically, the absolute distance conversion unit 30A converts the position information of each of the multiple detection points P contained in the environmental map information 26A into distance information from the current position of the moving body 2. Through this processing, the absolute distance conversion unit 30A calculates the absolute distance, i.e., the distance information, of each detection point P.

[0116] Furthermore, the absolute distance conversion unit 30A outputs the calculated distance information of each of the multiple detection points P to the extraction unit 30B. Additionally, the absolute distance conversion unit 30A outputs the calculated current position of the moving body 2 as its own position information to the virtual viewpoint line-of-sight determination unit 34.

[0117] The extraction unit 30B extracts a detection point P that exists within a specific range from among multiple detection points P that have received distance information from the absolute distance conversion unit 30A. This specific range, for example, refers to the range from the road surface where the moving body 2 is positioned to a height equivalent to the vehicle height of the moving body 2. However, this range is not limited to the aforementioned range.

[0118] By extracting the detection point P within this range by the extraction unit 30B, for example, it is possible to extract the detection point P of an object that becomes an obstacle to the movement of the moving body 2.

[0119] Furthermore, the extraction unit 30B outputs the distance information of each extracted detection point P to the nearest neighbor determination unit 30C.

[0120] The nearest neighbor determination unit 30C divides the area around the moving body 2's own position S into defined angle ranges, and determines the detection point P closest to the moving body 2 in each angle range, or determines multiple detection points P in the order of proximity to the moving body 2. The nearest neighbor determination unit 30C uses distance information received from the extraction unit 30B to determine the detection point P. In this embodiment, as an example, the method by which the nearest neighbor determination unit 30C determines multiple detection points P in each angle range in the order of proximity to the moving body 2 will be described.

[0121] The nearest neighbor determination unit 30C outputs the distance information of the detection point P determined according to each angle range to the reference projection plane shape selection unit 30D, the scale determination unit 30E, the asymptotic curve calculation unit 30F, and the boundary region determination unit 30H.

[0122] The reference projection surface shape selection unit 30D selects the shape of the reference projection surface.

[0123] Here, on one side, refer to Figure 6 The reference projection plane is described in detail. The reference projection plane 40 is, for example, a projection plane that becomes a reference shape when the shape of the projection plane is changed. The shape of the reference projection plane 40 is, for example, bowl-shaped, cylindrical, etc. Furthermore, in Figure 6 The reference projection plane 40 of the bowl shape is shown in the example.

[0124] A bowl shape refers to a shape having a bottom surface 40A and a side wall surface 40B, with one end of the side wall surface 40B continuous with the bottom surface 40A and the other end open. For the side wall surface 40B, the width of the horizontal cross-section increases from the bottom surface 40A side towards the open side at the other end. The bottom surface 40A is, for example, a circle. Here, a circle refers to a shape other than a perfect circle, such as an ellipse. The horizontal cross-section is an orthogonal plane orthogonal to the vertical direction (arrow Z direction). The orthogonal plane is a two-dimensional plane along the arrow X direction orthogonal to arrow Z direction and the arrow Y direction orthogonal to both arrow Z and arrow X directions. Hereinafter, the horizontal cross-section and the orthogonal plane will sometimes be referred to as the XY plane. Furthermore, the bottom surface 40A can also be a shape other than a circle, such as an egg shape.

[0125] A cylinder refers to a shape consisting of a circular base surface 40A and a side wall surface 40B continuous with the base surface 40A. Furthermore, the side wall surface 40B of the reference projection plane 40 constituting the cylinder is cylindrical, with an opening at one end continuous with the base surface 40A and an opening at the other end. However, the side wall surface 40B of the reference projection plane 40 constituting the cylinder has a shape whose diameter in the XY plane is approximately constant from the base surface 40A side towards the opening at the other end. Additionally, the base surface 40A can also be a shape other than a circle, such as an egg shape.

[0126] In this embodiment, as an example, the shape of the reference projection plane 40 is as follows: Figure 6 The bowl-shaped case shown will be explained. The reference projection plane 40 is a three-dimensional model virtually formed in a virtual space where the bottom surface 40A is set to be roughly the same as the road surface below the moving body 2, and the center of the bottom surface 40A is set to the position S of the moving body 2 itself.

[0127] The reference projection surface shape selection unit 30D selects the shape of the reference projection surface 40 by reading a determined shape from the storage unit 26, which stores multiple types of reference projection surface 40 shapes. For example, the reference projection surface shape selection unit 30D selects the shape of the reference projection surface 40 based on its own position relative to surrounding three-dimensional objects, distance information, etc. Alternatively, the shape of the reference projection surface 40 can be selected based on user operation instructions. The reference projection surface shape selection unit 30D outputs the determined shape information of the reference projection surface 40 to the shape determination unit 30G. In this embodiment, as described above, the method by which the reference projection surface shape selection unit 30D selects a bowl-shaped reference projection surface 40 will be explained as an example.

[0128] The scale determination unit 30E determines the scale of the reference projection plane 40 of the shape selected by the reference projection plane shape selection unit 30D. For example, the scale determination unit 30E makes decisions such as reducing the scale when multiple detection points P exist within a range specified from its own position S. The scale determination unit 30E outputs the scale information of the determined scale to the shape determination unit 30G.

[0129] The asymptotic curve calculation unit 30F outputs asymptotic curve information of the asymptotic curve Q, calculated using distance information from each distance information received from the nearest neighbor determination unit 30C, representing the distance to the nearest detection point P within each angular range from its own position S, to the shape determination unit 30G and the virtual viewpoint line-of-sight determination unit 34. Furthermore, the asymptotic curve calculation unit 30F can also calculate the asymptotic curve Q of the detection points P accumulated according to multiple portions of the reference projection plane 40. Moreover, the asymptotic curve calculation unit 30F can also output the asymptotic curve information of the calculated asymptotic curve Q to the shape determination unit 30G and the virtual viewpoint line-of-sight determination unit 34.

[0130] The shape determination unit 30G enlarges or reduces the reference projection plane 40, which is the shape shown by the shape information received from the reference projection plane shape selection unit 30D, to the scale information received from the scale determination unit 30E. Furthermore, the shape determination unit 30G determines the shape obtained by deforming the reference projection plane 40, which is the shape along the asymptotic curve Q received from the asymptotic curve calculation unit 30F, as the projection shape for the enlarged or reduced reference projection plane 40.

[0131] Here, on one side, refer to Figure 7 The determination of the projection shape is explained in detail. For example... Figure 7 As shown, the shape determination unit 30G determines the shape of the reference projection surface 40 as the projection shape 41, obtained by deforming the reference projection surface 40 into a shape that passes through the detection point P, which is the center of the bottom surface 40A closest to the reference projection surface 40 and the self-position S of the moving body 2. The shape passing through the detection point P refers to the shape of the deformed side wall surface 40B passing through the detection point P. The self-position S is the latest self-position S calculated by the self-position inference unit 27. Furthermore, the shape passing through the detection point P refers to the shape of the deformed side wall surface 40B passing through the detection point P.

[0132] That is, the shape determination unit 30G determines the detection point P that is closest to its own position S among the multiple detection points P registered in the environmental map information 26A. Specifically, the XY coordinates of the center position (its own position S) of the moving body 2 are set to (X, Y) = (0, 0). Furthermore, the shape determination unit 30G sets the X... 2 +Y 2The value indicates that the detection point P with the minimum value is determined as the detection point P closest to its own position S. Moreover, the shape determination unit 30G determines the shape obtained by the side wall surface 40B of the reference projection surface 40 passing through the detection point P as the projection shape 41.

[0133] More specifically, the shape determination unit 30G determines the deformed shape of a portion of the bottom surface 40A and the side wall surface 40B as a projected shape 41, such that when the reference projection plane 40 is deformed, a portion of the side wall surface 40B becomes a wall that passes through the detection point P closest to the moving body 2. The deformed projected shape 41 is, for example, a shape that rises from the vertical line 44 on the bottom surface 40A toward a direction close to the center of the bottom surface 40A. "Rising" means, for example, bending or folding the side wall surface 40B and a portion of the bottom surface 40A toward a direction close to the center of the bottom surface 40A, so that the angle between the side wall surface 40B and the bottom surface 40A of the reference projection plane 40 becomes a smaller angle.

[0134] The shape determination unit 30G determines that a defined region in the reference projection plane 40 is deformed so that it protrudes from the viewpoint (top view) in the XY plane to a position passing through the detection point P. The shape and extent of the defined region can also be determined based on a predetermined reference. Furthermore, the shape determination unit 30G determines that the reference projection plane 40 is deformed into a shape in which the distance from its own position S continuously increases from the protruding defined region toward the region outside the defined region in the side wall surface 40B.

[0135] For example, such as Figure 7 As shown, the projection shape 41 is preferably determined such that the outer periphery of the cross-section along the XY plane is a curved shape. Furthermore, the outer periphery of this cross-section of the projection shape 41 is, for example, a circle, but it can also be a shape other than a circle.

[0136] Furthermore, the shape determination unit 30G can also determine the shape obtained by deforming the reference projection surface 40 into a shape along an asymptotic curve as the projected shape 41. The shape determination unit 30G generates an asymptotic curve for a predetermined number of detection points P in a direction away from the detection point P closest to the position S of the moving body 2. The number of detection points P can be multiple. For example, it is preferable that the number of detection points P is three or more. In this case, it is preferable that the shape determination unit 30G generates an asymptotic curve for a plurality of detection points P located at a position more than a predetermined angle away when viewed from its own position S. For example, the shape determination unit 30G can determine the shape obtained by deforming the reference projection surface 40 into a shape along an asymptotic curve. Figure 6 The shape obtained by deforming the reference projection plane 40 into the shape along the generated asymptotic curve Q is determined as the projection shape 41.

[0137] Furthermore, the shape determination unit 30G can also divide the area around the position S of the moving body 2 according to each defined angle range, and determine the detection point P closest to the moving body 2 according to each angle range, or determine multiple detection points P in the order of proximity to the moving body 2. Moreover, the shape determination unit 30G can also determine the shape obtained by deforming the reference projection plane 40 into a shape passing through the determined detection point P or along the asymptotic curve Q of the determined multiple detection points P according to each angle range as the projection shape 41.

[0138] Furthermore, the timing for determining the projection shape 41 by the shape determination unit 30G is not limited. For example, in this embodiment, the shape determination unit 30G may also determine the projection shape 41 based on the movement of the moving body 2, using an image obtained at a time when the size of the subject being photographed is larger in the detection area selected by the selection unit 23.

[0139] Furthermore, the shape determination unit 30G outputs the projection shape information of the determined projection shape 41 to the deformation unit 32.

[0140] The boundary region determination unit 30H performs boundary region determination processing, which uses the position information of multiple detection points P around the moving body 2 accumulated in the environment map information 26A and the position information of the moving body 2 itself, to determine the boundary regions of multiple surrounding images of the moving body 2. That is, the boundary region determination unit 30H determines the boundary region based on the position of the nearest detection point P specified by the nearest neighbor determination unit 30C.

[0141] Figure 9 This is a diagram used to illustrate boundary angles and boundary widths. In Figure 9 In the diagram, AA represents the line of sight of camera 12A, AC represents the line of sight of camera 12C, AD represents the line of sight of camera 12D, and R1 and R2 represent the angle ranges of the set boundary angles when viewing the moving body 2 from directly above. Column C is the three-dimensional object closest to the moving body 2. C0 represents the center position of column C as viewed from the moving body 2, and CW represents the width of column C as viewed from the moving body 2. Figure 9 In the scenario shown, it is assumed that the moving body 2 is parked behind the parking dividing line PL. In this case, the line of sight AD of the camera 12D corresponds to a boundary angle of 0 degrees, and the line of sight AC of the camera 12C corresponds to a boundary angle of 90 degrees.

[0142] The boundary region determination unit 30H determines the center position C0 and width CW of the three-dimensional object, i.e., the pillar C, closest to the moving body 2, based on the position of the nearest detection point P determined by the nearest neighbor determination unit 30C. The boundary region determination unit 30H determines the boundary angle (e.g., 45 degrees) corresponding to the determined center position C0 of the pillar C. The boundary region determination unit 30H determines the angle range corresponding to the prescribed boundary width based on the boundary angle.

[0143] Furthermore, it is preferable that the boundary width does not exceed the width CW of the three-dimensional object closest to the moving body 2, namely column C. Additionally, the boundary region can be made any size by adjusting the boundary width. For example, the size of the boundary region can be set based on the detected width CW of column C. Furthermore, in Figure 9 In the example, for ease of understanding, the case where the boundary angle is set within the angular range R1 from 0 degrees to 90 degrees is shown. In contrast, for example, if the shooting unit 12 has a shooting range of about 195 degrees, the boundary angle can also be set to an angle of 90 degrees or more.

[0144] In addition, Figure 9 The example illustrates a case where only pillar C exists around the moving body 2. In contrast, when multiple three-dimensional objects exist around the moving body 2, the boundary region determination unit 30H determines the boundary region based on the nearest three-dimensional object. Furthermore, the boundary region determination unit 30H selects three-dimensional objects that meet a predetermined reference size (e.g., a size that can be detected by Visual SLAM) as objects to determine the boundary region.

[0145] Additionally, when moving body 2 moves from Figure 9 When the situation shown progresses further backward, for example, when column C moves to the right-south front of moving body 2, the boundary region determination unit 30H sets the boundary angle within the angle range R2 between the viewing direction AC of imaging unit 12C and the viewing direction AA of imaging unit 12A, in tandem with the movement of moving body 2, thus determining the boundary region. This is because the image processing device 10 continuously generates environmental map information 26A, and therefore can determine the positional relationship between moving body 2 and column C in real time.

[0146] In addition, Figure 9 In the example shown, the nearest solid to the moving body 2 is a pillar C. Conversely, if the nearest solid to the moving body 2 is a wall instead of pillar C, the boundary region determination unit 30H determines the boundary region based on the nearest wall portion to the moving body 2. Furthermore, for example, if the entire right side of the moving body 2 is a wall, the boundary region determination unit 30H uses, for example, a top view of the moving body 2 to determine the nearest wall portion and determines the boundary region based on that portion.

[0147] The boundary region determination unit 30H outputs the determined boundary angle and the specified angle range based on the boundary angle as boundary region information to the image synthesis unit 38.

[0148] Next, the boundary region adaptation control implemented by the image processing apparatus 10 of this embodiment will be explained in detail. Here, boundary region adaptation control refers to eliminating undesirable situations when a composite image formed by connecting multiple peripheral images is projected and displayed on a projection surface by setting the boundary region based on the three-dimensional object closest to the moving body 2.

[0149] Furthermore, for the purpose of specific explanation below, and Figure 9 The situation shown is illustrated by the following example: a composite image is generated using the peripheral image captured by the camera unit 12D located at the rear of the moving body 2 and the peripheral image captured by the camera unit 12C located on the right side.

[0150] Furthermore, the following explanation, as an example, illustrates the use of projection shape adaptation control in conjunction with boundary region adaptation control. Here, by setting the projection shape based on the three-dimensional object closest to the moving body 2, undesirable situations arising from projecting and displaying an image on the projection surface are eliminated. However, the use of projection shape adaptation control in conjunction with boundary region adaptation control is merely an example. That is, boundary region adaptation control can also be used independently of projection shape adaptation control.

[0151] Figure 10 This describes the situation where the moving body 2 is parked behind the parking line PL (parking space) near the pillar C. Figure 10 This refers to the peripheral image V resulting from perspective projection transformation of the image captured by the imaging unit 12C. C1 And the peripheral image V after perspective projection transformation of the image captured by the 12D camera unit. D1 A concept diagram. Additionally... Figure 11 This is a schematic representation of the surrounding image V. C1 and surrounding image V D1 The synthesized image V generated by synthesis T1 A diagram of one example.

[0152] In addition, Figure 10 In the middle, column C C1 Parking lines PL C1 Indicates the surrounding image V C1 The image shows column C and parking lines PL. D1 Parking lines PL D1 Indicates the surrounding image V D1The image shows column C and parking lines PL. Additionally... Figure 10 The lower side indicates that the surrounding image V D1 With surrounding image V C1 In the case of superposition, column C C1 Parking lines PL C1 Column C D1 Parking lines PL D1 The case where the boundary region BR is projected onto the reference projection plane 40.

[0153] exist Figure 10 In the middle, the peripheral image V projected onto the reference projection plane 40 D1 Column C D1 and surrounding image V C1 Column C C1 There are non-overlapping portions on the side of the reference projection plane 40 located in the boundary region BR. Additionally, within the boundary region BR, the surrounding image V... D1 With surrounding image V C1 Mixed at a prescribed ratio. Therefore, sometimes, for example, Figure 11 As shown in region H, in the synthesized image V T1 Above, part of column C was not visualized and thus disappeared.

[0154] Figure 12 It means targeting Figure 10 The diagram shown illustrates the case where the reference projection surface 40 undergoes projection shape adaptation control, deforming into a projection shape 41. Figure 12 In the middle, regarding the projection shape 41, the peripheral image V after perspective projection transformation of the image captured by the imaging unit 12C is shown. C2 And the peripheral image V after perspective projection transformation of the image captured by the 12D camera unit. D2 .in addition, Figure 13 It is a schematic representation Figure 12 The composite image V shown T2 An example diagram. Furthermore, in Figure 12 In the middle, column C C2 Parking lines PL C2 Indicates the surrounding image V C2 The image shows column C and parking lines PL. D2 Parking lines PL D2 Indicates the surrounding image V D2 The image shows column C and parking lines PL.

[0155] like Figure 12 As shown, the projected shape 41 is generated based on the position of the nearest solid object, namely column C, through projection shape adaptation control. Therefore, column C in the boundary region BRC2 With column C D2 The deviation on the side of the projected shape 41 is eliminated. As a result, for example, as... Figure 13 As shown in region H, in the synthesized image V T2 The upper column C is imaged, and the composite image V is generated. T2 The disappearance of a portion of column C on the top was eliminated.

[0156] Figure 14 From Figure 12 , Figure 13 The diagram shown depicts a scenario where the moving body 2 has advanced further towards the parking dividing line PL. Figure 14 In this process, the peripheral image of the image captured by the imaging unit 12C after perspective projection transformation is superimposed with the peripheral image of the image captured by the imaging unit 12D after perspective projection transformation. Furthermore, Figure 15 It is a schematic representation Figure 14 The composite image V shown T2 An example diagram. In comparison Figure 14 , Figure 15 and Figure 12 , Figure 13 In this case, the moving body 2 moves further toward the parking dividing line PL, so the position of the post C moves from the rear to the front on the right side of the moving body 2.

[0157] Without implementing boundary region adaptation control, the boundary region BR, for example, remains... Figure 12 The position shown remains unchanged. Therefore, the result of the moving body 2 further advancing towards the parking dividing line is that the boundary region BR changes from the image... Figure 12 The region overlapping with column C as shown is imaged. Figure 14 The area that does not overlap with column C is moved as shown. The moved boundary area BR is projected into the area where projection shape 41 is erected, but the parking dividing line PL, which is a horizontal plane image, is included within the boundary area BR. C2 PL D2 The regions where the projected shapes 41 stand do not overlap. The result is, as... Figure 15 As shown in region H, it exists in the synthesized image V. T2 Above, the parking line PL serves as the parking line PL within the boundary area BR. C2 PL D2 And the situation of being double-imaged.

[0158] Figure 16 It means targeting Figure 14 The diagram illustrates a conceptual example of adapting projection processing through boundary region adaptation control. Figure 16In this process, the peripheral image of the image captured by the imaging unit 12C after perspective projection transformation is superimposed with the peripheral image of the image captured by the imaging unit 12D after perspective projection transformation. Furthermore, Figure 17 It is a schematic representation Figure 16 The composite image V shown T2 A diagram of one example.

[0159] like Figure 16 As shown, the image processing apparatus 10 of this embodiment uses boundary region adaptation control to locate the center position of the nearest three-dimensional object, i.e., the cylinder C. Figure 17 The position indicated by arrow I sets the boundary angle, and the boundary region BR is set to converge to the width of cylinder C. Therefore, the parking dividing line PL is defined by the surrounding image V on the right side. C2 Corresponding parking markings PL C2 , and the surrounding image V D2 Corresponding parking markings PL D2 Any one of the three objects is projected onto the projected shape 41. Additionally, in the boundary region BR, the nearest solid, namely cylinder C... C2 C D2 The horizontal and side planes of the projected shape 41 are projected to overlap. The result is, as... Figure 17 As shown in region H, in the synthesized image V T2 Above, the dual imaging of parking line PL is eliminated.

[0160] Next, an example of an image processing flow including boundary region adaptation control executed by the image processing apparatus 10 of the first embodiment will be described.

[0161] Figure 18 This is a flowchart illustrating an example of an image processing procedure performed by the image processing device 10.

[0162] The acquisition unit 20 acquires a captured image from the shooting unit 12 (step S10). In addition, the acquisition unit 20 performs the acquisition of directly specified content (e.g., the gear of the moving body 2 is in reverse gear, etc.) and the acquisition of vehicle status (e.g., stopped state, etc.).

[0163] Selection unit 23 selects at least any two of the shooting units 12A to 12D (step S12).

[0164] The matching unit 25 uses multiple captured images with different shooting times from the captured images acquired in step S10 and selected and captured by the shooting unit 12 in step S12 to perform feature extraction and matching processing (step S14). In addition, the matching unit 25 registers the information of corresponding points between the multiple captured images with different shooting times determined by the matching processing in the environmental map information 26A.

[0165] The self-position inference unit 27 reads the environmental map information 26A (map and self-position information) (step S16). The self-position inference unit 27 uses the information of the corresponding points registered in the environmental map information 26A to calculate (infer) the self-position information of the moving body 2 corresponding to each shooting time in different shooting times by triangulation (step S18).

[0166] Then, the self-position inference unit 27 adds (registers) the calculated self-position information to the environmental map information 26A (step S20).

[0167] The detection point registration unit 29 reads the environmental map information 26A (map and its own position information) (step S22), and uses the own position information of the moving body 2 corresponding to each shooting time in different shooting times to calculate the movement amount (translation and rotation amount) of the moving body 2. Based on this movement amount, the detection point registration unit 29 calculates the relative three-dimensional coordinates of the corresponding points between multiple shooting images with different shooting times, determined by the matching process in step S14, with respect to the own position of the moving body 2, and performs three-dimensional reconstruction. The detection point registration unit 29 registers these three-dimensional coordinates as the coordinates of the detection point P in the environmental map information 26A (step S24). In addition, the coordinates of the detection point P registered in the environmental map information 26A can also be transformed into coordinates with a specified position as the origin.

[0168] The calibration unit 28 acquires environmental map information 26A (map and its own position information). The calibration unit 28 also uses the position information of the newly registered detection point P by the detection point registration unit 29 to correct the position information of the detection point P registered in the environmental map information 26A, for example, by error minimization processing (step S26). As a result, the map and its own position information in the environmental map information 26A are updated.

[0169] The absolute distance conversion unit 30A acquires the speed data (vehicle speed) of the mobile body 2 contained in the CAN data received from the ECU 3 of the mobile body 2. Using the speed data of the mobile body 2, the absolute distance conversion unit 30A converts the position information of each of the multiple detection points P contained in the environmental map information 26A into distance information from the latest self-position S of the mobile body 2 (i.e., the current position) to each of the multiple detection points P (step S28). The absolute distance conversion unit 30A outputs the calculated distance information of each of the multiple detection points P to the extraction unit 30B. Furthermore, the absolute distance conversion unit 30A outputs the calculated current position of the mobile body 2 as the self-position information of the mobile body 2 to the virtual viewpoint line-of-sight determination unit 34.

[0170] The extraction unit 30B extracts the detection points P that exist within a defined range from the multiple detection points P that have received distance information (step S30).

[0171] The nearest neighbor determination unit 30C divides the area around the moving body 2's own position S according to each defined angle range, determines the detection point P closest to the moving body 2 according to each angle range, or determines multiple detection points P in the order of approaching the moving body 2, and extracts the distance between the nearest object and the nearest object (step S32). The nearest neighbor determination unit 30C outputs the distance information (distance between the nearest object) of the detection point P determined according to each angle range to the reference projection plane shape selection unit 30D, the scale determination unit 30E, the asymptotic curve calculation unit 30F, and the boundary region determination unit 30H.

[0172] The asymptotic curve calculation unit 30F calculates the asymptotic curve (step S34) and outputs it as asymptotic curve information to the shape determination unit 30G and the virtual viewpoint line determination unit 34.

[0173] The reference projection surface shape selection unit 30D selects the shape of the reference projection surface 40 (step S36) and outputs the shape information of the selected reference projection surface 40 to the shape determination unit 30G.

[0174] The scale determination unit 30E determines the scale of the reference projection plane 40 of the shape selected by the reference projection plane shape selection unit 30D (step S38), and outputs the scale information of the determined scale to the shape determination unit 30G.

[0175] The shape determination unit 30G determines the projection shape that deforms the shape of the reference projection plane based on the scale information and asymptotic curve information (step S40). The shape determination unit 30G outputs the projection shape information of the determined projection shape 41 to the deformation unit 32.

[0176] The projection transformation unit 35 deforms the reference projection surface based on the projection shape information (step S42). The projection transformation unit 35 determines the virtual viewing point information, which includes a virtual viewing point and a virtual viewing line used to draw the minimum value of the asymptote in the center of the screen (step S44).

[0177] In addition, the projection transformation unit 35 performs perspective projection using the deformed projection shape, the determined virtual viewpoint, and the virtual line of sight to generate a virtual viewpoint image related to the surrounding images in four directions (step S46).

[0178] The boundary region determination unit 30H determines the boundary region based on the distance to the nearest object determined according to each angle range. That is, the boundary region determination unit 30H determines the boundary region as the overlapping region of spatially adjacent surrounding images based on the position of the nearest object of the moving body 2 (step S48). The boundary region determination unit 30H outputs the determined boundary region to the image compositing unit 38.

[0179] The image compositing unit 38 uses boundary regions to connect spatially adjacent perspective projection images to generate a composite image (step S50). That is, the image compositing unit 38 connects perspective projection images in four directions based on boundary regions with angles set to the direction of the nearest object to generate a composite image. Furthermore, in the boundary regions, spatially adjacent perspective projection images are blended at a predetermined ratio.

[0180] Display unit 16 displays the composite image (step S52).

[0181] The image processing device 10 determines whether to end image processing (step S54). For example, the image processing device 10 performs the determination in step S54 by determining whether it has received a signal from the ECU3 indicating that the position movement of the moving body 2 has stopped. Alternatively, the image processing device 10 may also perform the determination in step S54 by determining whether it has received an instruction to end image processing based on user operation instructions, etc.

[0182] If a negative judgment is made in step S54 (step S54: No), the processing from step S10 to step S54 is repeated. Therefore, the boundary region used to connect spatially adjacent surrounding images is set at an angle following the direction of the nearest object.

[0183] On the other hand, if a positive judgment is made in step S54 (step S54: yes), then the procedure ends.

[0184] Furthermore, if the process returns from step S54 to step S10 after performing the correction process in step S26, the subsequent correction process in step S26 may sometimes be omitted. Alternatively, if the process returns from step S54 to step S10 without performing the correction process in step S26, the subsequent correction process in step S26 may sometimes be performed.

[0185] As described above, the image processing apparatus of this embodiment includes a boundary region determination unit 30H as a determination unit and an image generation unit 37 as a generation unit. The boundary region determination unit 30H uses position information of multiple detection points P including the periphery of the moving body 2 and the position information of the moving body 2 itself to determine the boundary region in the overlapping region of spatially adjacent peripheral images in multiple peripheral images of the moving body 2. The image generation unit 37 uses the boundary region to generate a composite image using spatially adjacent peripheral images.

[0186] Therefore, based on the position information of multiple detection points P surrounding the moving body 2 and the position information of the moving body 2 itself, a boundary region for connecting spatially adjacent surrounding images in multiple surrounding images of the moving body 2 can be appropriately set.

[0187] Furthermore, the boundary region determination unit 30H determines the boundary region based on the position information of the detection point P corresponding to the nearest moving body 2. Therefore, in the boundary region, multiple spatially adjacent peripheral images are used, and the nearest moving body 2 is projected onto the projection plane as overlapping images. In addition, in the region outside the boundary region, projection processing is performed using any one of the spatially adjacent peripheral images. As a result, in the synthesized image, undesirable conditions such as double-imaged parking lines can be suppressed.

[0188] Furthermore, the boundary region determination unit 30H determines the boundary region based on the width of the nearest moving body 2. Therefore, the boundary region can be set to converge to the width of the nearest moving body 2. As a result, the boundary region does not protrude from the nearest moving body 2, and the generation of double-imaged parking lines in horizontal areas, etc., can be suppressed.

[0189] Furthermore, the boundary region determination unit 30H uses sequentially generated position information and its own position information to determine multiple boundary regions following the three-dimensional object. The image generation unit 37 uses the multiple boundary regions following the three-dimensional object and sequentially generates a composite image using spatially adjacent peripheral images. Therefore, the boundary regions that connect spatially adjacent peripheral images are set at an angle following the direction of the nearest object. As a result, the user can use a high-precision composite image that suppresses the generation of double-imaged parking lines in horizontal areas, etc.

[0190] Furthermore, the image processing apparatus 10 of this embodiment also includes a deformation unit 32, which deforms a reference projection plane using position information and its own position information. The image generation unit 37 generates a composite image using the deformed reference projection plane and spatially adjacent peripheral images. Therefore, a boundary region for connecting spatially adjacent peripheral images is set along the reference projection plane, which is deformed according to the position of the nearest object. As a result, the user can use a high-precision composite image generated by suppressing double imaging of the surrounding three-dimensional objects of the moving body 2.

[0191] While referring to Figures 19-24 The effects of the image processing apparatus 10 of this embodiment will be explained in further detail. Figure 19 , Figure 21 , Figure 23 This is a diagram used to illustrate a comparative example where boundary region adaptive control is not performed. Figure 20 , Figure 22 , Figure 24 This is a diagram illustrating an example of a synthetic image obtained through image processing that includes boundary region adaptation control. Furthermore, Figures 19-24 These are all examples of situations where the moving object 2 is parked behind it.

[0192] exist Figure 19 In the comparative example shown, the boundary angle is set at a position offset from the nearest moving body 2 (the solid J) by approximately 45 degrees. Therefore, in Figure 19 In the area H shown, the parking lines are double-imaged.

[0193] Figure 20 It means targeting Figure 19 The comparative example shown is an example of a synthesized image obtained by performing the boundary region adaptation control of this embodiment. Through boundary region adaptation control, the boundary angle is set at position I (approximately 75 degrees) where it overlaps with the nearest moving body 2, and the boundary width is set to converge to the width of the nearest moving body 2, the solid object J. Therefore, in Figure 20 In this process, the dual imaging of parking lines is eliminated.

[0194] Figure 21 and Figure 22 Indicates that the moving body 2 is from Figure 19 and Figure 20 Examples of composite images showing situations where the location is moved and the vehicle is rear-ended to other locations. Figure 21 This means that even if the moving body 2 moves, it will remain Figure 20 A comparative example of the boundary angle set in the example. In this case, the boundary angle is set at a position offset from the nearest moving object 2 (approximately 75 degrees) from the solid object K. Therefore, in Figure 21In area H shown, the parking lines are double-imaged. Furthermore, Figure 21 and Figure 22 Alternatively, the moving body 2 can perform actions such as returning the wheels during rear parking, and thus... Figure 19 and Figure 20 The position shown is moved forward.

[0195] Figure 22 It means targeting Figure 21 The comparative example shown is an example of a synthesized image obtained by performing the boundary region adaptation control of this embodiment. Through boundary region adaptation control, the boundary angle is set at position I (approximately 25 degrees) overlapping with the nearest neighboring moving body 2, and the boundary region is set to converge to the width of the nearest neighboring moving body 2, the solid object K. Therefore, in Figure 22 In this process, the dual imaging of parking lines is eliminated.

[0196] exist Figure 23 and Figure 24 Medium moving body 2-way ratio Figure 21 and Figure 22 The position of the moving object 2 in the middle moves to the rear. Figure 23 This means that even if the moving body 2 moves, it will remain Figure 22 A comparative example of the boundary angle set in the example. In this case, the boundary angle is set at a position offset from the nearest moving object 2 (approximately 25 degrees) from the solid object K. Therefore, in Figure 23 In the area H shown, the parking lines are double-imaged.

[0197] Figure 24 It means targeting Figure 23 The comparative example shown is an example of a synthesized image obtained by performing the boundary region adaptation control of this embodiment. Through boundary region adaptation control, the boundary angle is set at position I (approximately 60 degrees) overlapping with the nearest neighboring moving body 2, and the boundary region is set to converge to the width of the nearest neighboring moving body 2, the solid object K. In this case, the boundary angle is continuously controlled as the moving body 2 moves backward. Therefore, in Figure 24 In this way, the double image of parking lines is eliminated. By performing boundary area adaptation control following the movement of the moving body 2, the generation of double image of surrounding three-dimensional objects of the moving body 2 can be continuously suppressed.

[0198] (Second Implementation)

[0199] In the first embodiment described above, an example of obtaining the position information of the detection point P from the image captured by the imaging unit 12, i.e., using Visual SLAM, was explained. In contrast, in the second embodiment, an example of detecting the position information of the detection point P using the detection unit 14 will be explained. That is, the image processing apparatus 10 of the second embodiment is an example using three-dimensional LiDAR SLAM or the like.

[0200] Figure 25 This is a diagram illustrating an example of the functional structure of the image processing apparatus 10 according to the second embodiment. The image processing apparatus 10, like the image processing apparatus 10 of the first embodiment, is connected to the imaging unit 12, the detection unit 14, and the display unit 16 to transmit and receive data or signals.

[0201] The image processing apparatus 10 includes an acquisition unit 20, a self-position inference unit 27, a detection point registration unit 29, a storage unit 26, a correction unit 28, a decision unit 30, a deformation unit 32, a virtual viewpoint line of sight determination unit 34, a projection transformation unit 36, and an image synthesis unit 38.

[0202] Some or all of the aforementioned components can also be made, for example, by making Figure 2 The CPU10A and other processing devices shown execute programs, that is, they are implemented through software. In addition, some or all of the above-mentioned components can also be implemented by hardware such as ICs, or they can be implemented by a combination of software and hardware.

[0203] exist Figure 25 In this embodiment, the storage unit 26, correction unit 28, determination unit 30, deformation unit 32, virtual viewpoint line-of-sight determination unit 34, projection transformation unit 36, and image synthesis unit 38 are the same as in the first embodiment. The storage unit 26 stores environmental map information 26A. The environmental map information 26A is the same as in the first embodiment.

[0204] The acquisition unit 20 acquires captured images from the imaging unit 12. Additionally, the acquisition unit 20 acquires position information of detection points from the detection unit 14. The acquisition unit 20 acquires captured images from each of the imaging units 12 (imaging units 12A to 12D). The detection unit 14 detects the position information of each of the multiple detection points. Therefore, the acquisition unit 20 acquires the position information of each of the multiple detection points and the captured images based on each of the multiple imaging units 12.

[0205] Each time the acquisition unit 20 acquires the position information of multiple detection points, it outputs the acquired position information to the detection point registration unit 29 and the self-position inference unit 27. Additionally, each time the acquisition unit 20 acquires a captured image, it outputs the acquired image to the projection transformation unit 36.

[0206] Each time the detection point registration unit 29 acquires the location information of multiple detection points, it registers each location information in the environmental map information 26A. The environmental map information 26A is then stored in the storage unit 26.

[0207] Each time the detection point registration unit 29 acquires the location information of detection point P from the detection unit 14 via the acquisition unit 20, it determines the same detection point P already registered in the environmental map information 26A through scanning and matching. The same detection point P refers to the same detection point P in actual space, even though the acquisition timing of the location information from the detection unit 14 is different. The detection point registration unit 29 additionally registers the location information of detection points P that are not registered in the environmental map information 26A from the location information of detection points P acquired from the detection unit 14 into the environmental map information 26A. At this time, the detection point registration unit 29 registers the acquired location information coordinates of the detection point P into coordinates with the aforementioned predetermined position as the origin in the environmental map information 26A.

[0208] Furthermore, the scanning and matching results based on the detection point registration unit 29 sometimes show a discrepancy between the detection point P already registered in the environmental map information 26A and the newly acquired detection point P at a predetermined ratio. In this case, the detection point registration unit 29 may also discard the location information of the newly acquired detection point P and omit the registration with the environmental map information 26A. The predetermined ratio can be determined in advance. This process can improve the reliability of the environmental map information 26A.

[0209] In addition, the preferred detection point registration unit 29 registers various parameters used for calculating the location information of detection point P together with the location information in the environmental map information 26A.

[0210] The self-position inference unit 27 infers the self-position information of the moving body 2, representing its own position S, based on the position information of each of the multiple detection points P registered in the environmental map information 26A. Furthermore, the self-position inference unit 27 registers the inferred self-position information S in the environmental map information 26A.

[0211] The self-position inference unit 27 uses the position information of each detection point P registered in the environmental map information 26A and the latest position information of each detection point P acquired from the detection unit 14 via the acquisition unit 20 to determine the corresponding detection point P contained in these position information. A corresponding detection point P refers to a detection point P that is the same in actual space, even though the acquisition timing based on the position information from the detection unit 14 is different. The self-position inference unit 27 calculates the self-position information of the moving body 2's own position S by using triangulation of the determined position information of each corresponding detection point P. Furthermore, the self-position inference unit 27 registers the calculated self-position information in the environmental map information 26A. Alternatively, the self-position inference unit 27 can also use a range method to infer its own position information.

[0212] Thus, in this embodiment, the image processing device 10 simultaneously performs SLAM to register the location information of the detection point P to the environmental map information 26A and to infer the location information of the moving body 2 itself.

[0213] Next, an example of an image processing flow including boundary region adaptation control executed by the image processing apparatus 10 of the second embodiment will be described.

[0214] Figure 26 This is a flowchart illustrating an example of an image processing procedure performed by the image processing device 10.

[0215] The acquisition unit 20 acquires the captured image from the imaging unit 12 (step S100). Additionally, the acquisition unit 20 acquires the position information of each of the multiple detection points P from the detection unit 14 (step S102).

[0216] The detection point registration unit 29 determines, through scanning and matching, the same detection point P among the detection points P of the location information obtained in step S102 that has been registered in the environmental map information 26A (step S104). Then, the detection point registration unit 29 registers the location information of the detection points P that have not been registered in the environmental map information 26A in the location information of the detection points P obtained in step S102 into the environmental map information 26A (step S106).

[0217] The self-position inference unit 27 infers the self-position information representing the self-position S of the moving body 2 based on the position information of each of the multiple detection points P registered in the environmental map information 26A and the position information obtained in step S102 (step S108). Then, the self-position inference unit 27 registers the inferred self-position information in the environmental map information 26A (step S110).

[0218] The calibration unit 28 uses the location information of the detection point P obtained in step S102 to correct the location information of the detection point P registered in the environmental map information 26A (step S112). Alternatively, as described above, the calibration process in step S112 can also be omitted.

[0219] The absolute distance conversion unit 30A of the decision unit 30 converts the position information of each of the multiple detection points P contained in the environmental map information 26A into distance information of the absolute distance from the current position of the moving body 2 to each of the multiple detection points P (step S114).

[0220] Extraction unit 30B extracts detection points P that exist within a defined range from the detection points P whose absolute distance information is calculated by absolute distance conversion unit 30A (step S116).

[0221] The nearest neighbor determination unit 30C uses the distance information of each detection point P extracted in step S116 to determine multiple detection points P in order of proximity to the moving body 2, according to each angle range around the moving body 2 (step S118).

[0222] The asymptotic curve calculation unit 30F uses each of the distance information of the multiple detection points P in each angle range determined in step S118 to calculate the asymptotic curve Q (step S120).

[0223] The reference projection surface shape selection unit 30D selects the shape of the reference projection surface 40 (step S122). As described above, the method by which the reference projection surface shape selection unit 30D selects a bowl-shaped reference projection surface 40 will be explained as an example.

[0224] The scale determination unit 30E determines the scale of the reference projection plane 40 of the shape selected in step S122 (step S124).

[0225] The shape determination unit 30G enlarges or reduces the reference projection plane 40 of the shape selected in step S122 to the scale determined in step S124. Then, the shape determination unit 30G deforms the enlarged or reduced reference projection plane 40 into a shape along the asymptotic curve Q calculated in step S120. The shape determination unit 30G determines the deformed shape as the projection shape 41 (step S126).

[0226] The deformation unit 32 deforms the reference projection surface 40 into a projection shape 41 determined by the determination unit 30 (step S128). Through this deformation process, the deformation unit 32 generates the deformed reference projection surface 40, i.e., the deformed projection surface 42.

[0227] The virtual viewpoint line-of-sight determination unit 34 determines the virtual viewpoint line-of-sight information (step S130). For example, the virtual viewpoint line-of-sight determination unit 34 determines the position S of the moving body 2 as the virtual viewpoint O, and determines the direction from the virtual viewpoint O toward the vertex W of the asymptotic curve Q as the line-of-sight direction L. In detail, the virtual viewpoint line-of-sight determination unit 34 determines the direction toward the vertex W of the asymptotic curve Q within a defined angle range, which is calculated in step S120 according to each angle range, as the line-of-sight direction L.

[0228] The projection transformation unit 36 ​​projects the captured image obtained in step S100 onto the deformed projection surface 42 generated in step S128. Then, the projection transformation unit 36 ​​transforms the projected image into an image obtained by visually confirming the captured image projected onto the deformed projection surface 42 from the virtual viewpoint O determined in step S130 in the viewing direction L, i.e., a virtual viewpoint image (step S132).

[0229] The boundary region determination unit 30H determines the boundary region based on the distance to the nearest object determined according to each angle range. That is, the boundary region determination unit 30H determines the boundary region as the overlapping region of spatially adjacent surrounding images based on the position of the nearest moving body 2 (step S134). The boundary region determination unit 30H outputs the determined boundary region to the image compositing unit 38.

[0230] The image compositing unit 38 generates a composite image by connecting spatially adjacent perspective projection images through boundary regions (step S136). That is, the image compositing unit 38 connects perspective projection images in four directions according to boundary regions set as the angle of the nearest object direction to generate a composite image. Furthermore, in the boundary regions, spatially adjacent perspective projection images are blended at a predetermined ratio.

[0231] Display unit 16 performs display control of the generated composite image 54 (step S138).

[0232] Next, the image processing device 10 determines whether to end image processing (step S140). For example, the image processing device 10 performs the determination in step S140 by determining whether it has received a signal from the ECU 3 indicating that the position movement of the moving body 2 has stopped. Alternatively, the image processing device 10 may also perform the determination in step S140 by determining whether it has received an instruction to end image processing based on user operation instructions, etc.

[0233] If a negative judgment is made in step S140 (step S140: No), the processing from step S100 to step S140 is repeated. Therefore, the boundary region used to connect spatially adjacent surrounding images is set at an angle following the direction of the nearest object. On the other hand, if a positive judgment is made in step S140 (step S140: Yes), the procedure ends.

[0234] Furthermore, if the process returns from step S140 to step S100 after performing the correction process in step S112, the subsequent correction process in step S112 may sometimes be omitted. Alternatively, if the process returns from step S140 to step S100 without performing the correction process in step S112, the subsequent correction process in step S112 may sometimes be performed.

[0235] As described above, the image processing apparatus 10 of the second embodiment uses SLAM to generate position information of multiple detection points P around the moving body 2 and position information of the moving body 2 itself, using the captured image taken by the imaging unit 12 and the position information detected by the detection unit 14. The boundary region determination unit 30H uses the aforementioned position information and the aforementioned position information to determine the boundary region in the overlapping region of spatially adjacent peripheral images in the multiple peripheral images of the moving body 2. The image generation unit 37 uses the boundary region and the spatially adjacent peripheral images to generate a composite image. Therefore, the image processing apparatus 10 according to the second embodiment can achieve the same operating effect as the image processing apparatus 10 of the first embodiment.

[0236] (Modified Example)

[0237] In the embodiments described above, the image processing apparatus 10 using SLAM has been described. In contrast, it is also possible to use environmental map information based on the immediate values ​​of a sensor array constructed from multiple distance sensors instead of SLAM, and to use this environmental map information to perform image processing including boundary region adaptation control.

[0238] The various embodiments and modifications have been described above. However, the image processing apparatus, image processing method, and recording medium disclosed in this application are not directly limited to the embodiments described above. In each implementation stage, the structural elements can be modified and customized without departing from the main idea. Furthermore, various inventions can be formed by appropriately combining the multiple structural elements disclosed in the above embodiments. For example, several structural elements may be deleted from all the structural elements shown in the embodiments.

[0239] Furthermore, the image processing apparatus 10 described in the first and second embodiments can be applied to various devices. For example, the image processing apparatus 10 described in the first and second embodiments can be applied to a surveillance camera system that processes images obtained from a surveillance camera, or to an in-vehicle system that processes images of the surrounding environment outside the vehicle.

[0240] Explanation of reference numerals: 10…Image processing device; 12, 12A-12D…Illumination unit; 14…Detection unit; 20…Acquisition unit; 23…Selection unit; 25…Matching unit; 26…Storage unit; 26A…Environmental map information; 27…Self-position inference unit; 28…Correction unit; 29…Detection point registration unit; 30…Decision unit; 30A…Absolute distance conversion unit; 30B…Extraction unit; 30C…Nearest neighbor determination unit; 30D…Reference projection plane shape selection unit; 30E…Scale determination unit; 30F…Asymptotic curve calculation unit; 30G…Shape determination unit; 30H…Boundary region determination unit; 32…Deformation unit; 34…Virtual viewpoint line of sight determination unit; 36…Projection transformation unit; 37…Image generation unit; 38…Image synthesis unit.

Claims

1. An image processing apparatus comprising: The decision unit uses position information of multiple detection points surrounding the moving object and the moving object's own position information to set the boundary angle of the image during image synthesis to overlap with the nearest 3D object, and uses the boundary angle to determine the boundary region in the overlapping region of spatially adjacent surrounding images of the moving object; and The image generation unit uses the boundary region to generate a composite image using the spatially adjacent surrounding images.

2. The image processing apparatus according to claim 1, wherein, The decision unit sets the boundary angle of the image during image synthesis to converge to the width of the nearest solid object.

3. The image processing apparatus according to claim 2, wherein, The determining unit determines the boundary region based on the width of the three-dimensional object.

4. The image processing apparatus according to claim 2 or 3, wherein, The decision-making unit uses the sequentially generated position information and its own position information to determine the multiple boundary regions that follow the three-dimensional object. The image generation unit uses multiple boundary regions that follow the stereoscopic object to sequentially generate the composite image using spatially adjacent surrounding images.

5. The image processing apparatus according to claim 2 or 3, wherein, The image processing device further includes a deformation unit that uses the position information and its own position information to deform the reference projection surface. The image generation unit uses the deformed reference projection plane and the spatially adjacent surrounding images to generate the composite image.

6. The image processing apparatus according to any one of claims 1 to 3, wherein, The detection point is detected using the surrounding image.

7. The image processing apparatus according to any one of claims 1 to 3, wherein, The detection point is detected by a detection unit mounted on the moving body.

8. An image processing method, which is an image processing method executed by a computer, wherein, The image processing method includes the following steps: Using the position information of multiple detection points surrounding the moving object and the moving object's own position information, the boundary angle of the image during image synthesis is set to overlap with the nearest 3D object, and the boundary angle is used to determine the boundary region in the overlapping region of spatially adjacent surrounding images of the moving object; and The boundary region is used to generate a composite image using spatially adjacent surrounding images.

9. A recording medium having an image processing program recorded thereon, the image processing program being used to cause a computer to perform the following steps: Using the position information of multiple detection points surrounding the moving object and the moving object's own position information, the boundary angle of the image during image synthesis is set to overlap with the nearest 3D object, and the boundary angle is used to determine the boundary region in the overlapping region of spatially adjacent surrounding images of the moving object; and The boundary region is used to generate a composite image using spatially adjacent surrounding images.

Images