IMAGE PROCESSING APPARATUS, IMAGE PROCESSING METHOD, AND IMAGE PROCESSING PROGRAM

Abstract

To reduce missing of an object in an overlapping part of a taken image or double appearing of the object in a projected image.SOLUTION: An image processor according to an embodiment includes an acquisition unit 20, a determination unit 22, and an adjusting unit 26. The acquisition unit 20 acquires a plurality of taken images overlapping partially. The determination unit 22 determines whether there is an object in an overlapping part of taken images adjacent to each other included in a projection image formed by projecting a plurality of taken images on a reference projection surface as an image projection surface virtually arranged in a virtual space corresponding to an actual space. The adjusting unit 26 performs adjusting processing of the overlapping region including the object in the reference projection surface which corresponds to the overlapping part including the object.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to an image processing device, an image processing method, and an image processing program. [Background technology]

[0002] A technology has been disclosed that generates a projected image by projecting multiple captured images of the periphery of a moving object onto a virtual projection surface. Also disclosed is a technology that detects the distance to an object around the moving object using a sensor and changes the shape of the projection surface according to the detected distance (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2013-207637 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, there are cases where an object exists in the overlapping portion of the captured image projected onto the projection surface. In this case, the object may disappear or the image of the object may appear twice in the projected area of ​​the projected image. The above Patent Document 1 does not mention the case where an object exists in the overlapping portion of the captured image, nor does it mention the phenomenon of the object disappearing or the image of the object appearing twice.

[0005] In one aspect, the present invention aims to provide an image processing device, an image processing method, and an image processing program that can suppress the disappearance of an object contained in an overlapping portion of a captured image or the double image of an object in a projected image. [Means for solving the problem]

[0006] In one aspect, the image processing device disclosed in the present application includes an acquisition unit that acquires a plurality of captured images in which a portion of a captured area overlaps, a determination unit that determines whether an object is included in an overlapping portion of adjacent captured images included in a projected image obtained by projecting the plurality of captured images onto a reference projection plane that is an image projection plane that is virtually disposed in a virtual space corresponding to a real space, and an adjustment process for an overlapping area including the object on the reference projection plane that corresponds to the overlapping area including the object, the adjustment process being performed within the overlapping area including the object on the reference projection plane as a portion of the overlapping area, Based on From the semi-final The central area of ​​the bottom surface of the reference projection plane is the moving body and an adjustment unit that executes an image synthesis adjustment process that adjusts the position and range of an image synthesis area, which is an area set by two dividing lines extending radially away from a reference position, by adjusting the position of the dividing lines and the angle between the two dividing lines so that the image synthesis area does not overlap with the object. The central area is a rectangular area surrounding the outline of the moving body, and the border lines constituting the outline of the overlapping area include a first border line extending, on the reference projection plane, from one of the four vertices of the rectangle in the central area toward the outside of the central area along a first side that forms the one vertex of the four sides that constitute the rectangle, and a second border line extending from the one vertex toward the outside of the central area along a second side different from the first side that forms the one vertex of the four sides, and the reference point is the intersection of the first border line and the second border line of the overlapping area. Effect of the Invention

[0007] According to one aspect of the image processing device disclosed in the present application, it is possible to prevent an object included in an overlapping portion of a captured image from disappearing or an object from appearing in a double image in a projected image. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing the overall configuration of an image processing system according to an embodiment. [Diagram 2] FIG. 2 is a diagram showing a hardware configuration of the image processing apparatus according to the embodiment. [Diagram 3] FIG. 3 is a diagram illustrating a functional configuration of the image processing device according to the embodiment. [Figure 4] FIG. 4 is a schematic diagram showing an example of a reference projection plane according to the embodiment. [Diagram 5] FIG. 5 is a schematic diagram showing a comparative projection image according to the prior art. [Figure 6A]FIG. 6A is an explanatory diagram of the projection plane deformation process according to the embodiment. [Figure 6B] FIG. 6B is an explanatory diagram of the projection plane deformation process according to the embodiment. [Figure 6C] FIG. 6C is an explanatory diagram of the projection plane deformation process according to the embodiment. [Figure 6D] FIG. 6D is an explanatory diagram of the projection plane deformation process according to the embodiment. [Figure 6E] FIG. 6E is an explanatory diagram of the projection plane deformation process according to the embodiment. [Figure 7A] FIG. 7A is an explanatory diagram of a projection plane deformation process of a component plane according to the embodiment. [Figure 7B] FIG. 7B is an explanatory diagram of a projection plane deformation process of a component surface according to the embodiment. [Figure 7C] FIG. 7C is an explanatory diagram of a projection plane deformation process of a component surface according to an embodiment. [Figure 8A] FIG. 8A is a schematic diagram of an adjusted projection plane according to an embodiment. [Figure 8B] FIG. 8B is a schematic diagram of an adjusted projection plane according to the embodiment. [Figure 9A] FIG. 9A is a schematic diagram of a cross-sectional shape of an adjusted projection plane according to an embodiment. [Figure 9B] FIG. 9B is a schematic diagram of a cross-sectional shape of the adjusted projection plane according to the embodiment. [Figure 9C] FIG. 9C is a schematic diagram of a cross-sectional shape of the adjusted projection plane according to the embodiment. [Figure 9D] FIG. 9D is a schematic diagram of a cross-sectional shape of the adjusted projection plane according to the embodiment. [Figure 10A] FIG. 10A is an explanatory diagram of the overlap region adjustment process according to the embodiment. [Figure 10B] FIG. 10B is an explanatory diagram of the overlap region adjustment process according to the embodiment. [Figure 11A] FIG. 11A is an explanatory diagram of adjustment of a synthesis coefficient according to the embodiment. [Figure 11B] FIG. 11B is an explanatory diagram of adjustment of the synthesis coefficient according to the embodiment. [Figure 12]FIG. 12 is a schematic diagram of a projected image according to the embodiment. [Figure 13A] FIG. 13A is a schematic diagram of a comparative projection image of the prior art. [Figure 13B] FIG. 13B is a schematic diagram of a comparative projection image of the prior art. [Figure 14] FIG. 14 is a schematic diagram of a projected image according to the embodiment. [Figure 15A] FIG. 15A is a schematic diagram showing a comparative projection image of the prior art. [Figure 15B] FIG. 15B is a schematic diagram of a projected image according to the embodiment. [Figure 16] FIG. 16 is a flowchart showing the flow of image processing according to the embodiment. [Figure 17] FIG. 17 is a flowchart showing the flow of the projection surface deformation process according to the embodiment. [Figure 18] FIG. 18 is a flowchart showing the flow of the overlap region adjustment process according to this embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, with reference to the attached drawings, embodiments of the image processing device, image processing method, and image processing program disclosed in the present application will be described in detail. Note that the following embodiments do not limit the disclosed technology. Moreover, each embodiment can be appropriately combined within a range that does not cause contradiction in the processing contents.

[0010] 1 is a diagram showing an example of the overall configuration of an image processing system 1 according to this embodiment. The image processing system 1 includes an image processing device 10, an image capturing unit 12, a detection unit 14, and a display unit 16. The image processing device 10, the image capturing unit 12, the detection unit 14, and the display unit 16 are connected to each other so as to be able to exchange data or signals.

[0011] In this embodiment, the image processing device 10, the photographing unit 12, the detection unit 14, and the display unit 16 are mounted on a moving object 2, for example.

[0012] The moving body 2 is an object that can move. The moving body 2 is, for example, a vehicle, a flyable object (a manned airplane, an unmanned airplane (e.g., a UAV (Unmanned Aerial Vehicle), a drone)), a robot, etc. The moving body 2 is, for example, a moving body that proceeds via a driving operation by a person, or a moving body that can proceed automatically (autonomously) without being driven by a person. In this embodiment, a case where the moving body 2 is a vehicle is described as an example. The vehicle is, for example, a two-wheeled vehicle, a three-wheeled vehicle, a four-wheeled vehicle, etc. In this embodiment, a case where the vehicle is an autonomously-traveling four-wheeled vehicle is described as an example.

[0013] Note that the image processing device 10, the photographing unit 12, the detection unit 14, and the display unit 16 are not limited to being mounted on the moving object 2. The image processing device 10 may be mounted on a stationary object. A stationary object is an object fixed to the ground. A stationary object is an object that cannot be moved or an object that is stationary relative to the ground. Examples of stationary objects include traffic lights, parked vehicles, and road signs. The image processing device 10 may also be mounted on a cloud server that executes processing on the cloud.

[0014] The photographing unit 12 photographs the periphery of the moving object 2 and acquires photographed image data. In the following description, the photographed image data will be simply referred to as a photographed image. The photographing unit 12 is, for example, a known digital camera. Note that photographing refers to converting an image of a subject formed by an optical system such as a lens into an electrical signal. The photographing unit 12 outputs the photographed image to the image processing device 10.

[0015] In this embodiment, a configuration in which four photographing units 12 (photographing units 12A to 12D) are mounted on a moving object 2 will be described as an example. The multiple photographing units 12 (photographing units 12A to 12D) photograph subjects in their respective photographing areas E (photographing areas E1 to E4) and obtain photographed images. Note that the number of photographing units 12 is not limited to four as long as it is more than one. In this embodiment, a configuration in which an image processing system 1 includes four photographing units 12 will be described as an example.

[0016] These multiple imaging units 12 have different imaging directions. Specifically, the imaging directions of these multiple imaging units 12 are adjusted in advance so that the imaging areas E partially overlap and partially do not overlap. For this reason, for example, the imaging areas E captured by the different imaging units 12 have overlapping areas D where adjacent imaging areas E overlap. In other words, the captured images of each of these imaging areas E have overlapping parts that correspond to the overlapping areas D and overlap between adjacent captured images.

[0017] In FIG. 1, overlapping area DA to overlapping area DD are shown as examples of overlapping area D. Overlapping area DA is the overlapping area D between shooting area E4 and shooting area E1. Overlapping area DB is the overlapping area D between shooting area E1 and shooting area E2. Overlapping area DC is the overlapping area D between shooting area E2 and shooting area E3. Overlapping area DD is the overlapping area D between shooting area E3 and shooting area E4.

[0018] The detection unit 14 detects position information of each of a plurality of detection points around the moving object 2. A detection point refers to each point in real space that is individually observed by the detection unit 14. For example, the detection unit 14 irradiates light, radio waves, or sound waves around the detection unit 14, and receives reflected light or waves reflected at reflection points. These reflection points correspond to the detection points.

[0019] The position information of the detection point is information indicating the position of the detection point in real space. The real space is a three-dimensional space. For example, the position information of the detection point is information including at least one of the distance from the detection unit 14 to the detection point and the direction of the detection point with respect to the detection unit 14. These distances and directions can be represented, for example, by a vector indicating the relative position of the detection point with respect to the detection unit 14.

[0020] The detection unit 14 is, for example, a 3D (Three-Dimensional) scanner, a 2D (Two Dimensions) scanner, a distance sensor (millimeter wave radar, laser sensor), a sonar sensor that detects an object by sound waves, an ultrasonic sensor, etc. The laser sensor is, for example, a three-dimensional LIDAR (Laser Imaging Detection and Ranging) sensor. The detection unit 14 may also be a device that uses a distance measurement technique (such as SfM (Structure from Motion)) that measures distance from multiple images captured by a monocular camera, or a device that uses a distance measurement technique (such as a stereo camera) that measures distance based on images captured by multiple cameras.

[0021] In this embodiment, a configuration in which four detection units 14 (detection units 14A to 14D) are mounted on the moving body 2 will be described as an example. At least a part of the detection areas of the multiple detection units 14 (detection units 14A to 14D) are different from each other. In this embodiment, the detection areas of the multiple detection units 14 (detection units 14A to 14D) include at least a part of the imaging area E of each of the multiple imaging units 12 (imaging units 12A to 12D). In this embodiment, a configuration in which the detection units 14A to 14D are provided at both ends of the front part of the moving body 2 and both ends of the rear part of the moving body 2 will be described as an example. Note that the detection units 14 only need to detect detection points in the detection areas including at least the imaging area E, and the number and installation positions of the detection units 14 are not limited.

[0022] The display unit 16 displays various types of information and is, for example, a known LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display.

[0023] Next, a description will be given of a hardware configuration of the image processing device 10. FIG 2 is a diagram showing an example of a hardware configuration of the image processing device 10.

[0024] The image processing device 10 includes a processor 10A, a memory 10B, a transmission / reception I / F (interface) 10C, a transmission / reception I / F 10D, and a display I / F 10E. The processor 10A, the memory 10B, the transmission / reception I / F 10C, the transmission / reception I / F 10D, and the display I / F 10E are interconnected by a bus 10F, and are a hardware configuration utilizing a normal computer.

[0025] The processor 10A is, for example, a micro-processing unit (MPU) and is a calculation device that controls the image processing device 10. The processor 10A is an example of a hardware processor. The memory 10B is, for example, a semiconductor memory element such as a random access memory (RAM), a read only memory (ROM), and a flash memory. The ROM and the flash memory store, for example, programs that realize various processes by the processor 10A. The RAM stores, for example, data required for various processes by the processor 10A. The transmission / reception I / F 10C is an interface that connects to the imaging unit 12 and transmits and receives data. The transmission / reception I / F 10D is an interface that connects to the detection unit 14 and transmits and receives data. The display I / F 10E is an interface that connects to the display unit 16 and transmits and receives data.

[0026] The program for executing image processing executed by the image processing device 10 of this embodiment is provided by being pre-installed in, for example, a ROM. The program executed by the image processing device 10 of this embodiment may be provided by being recorded on a recording medium in a format that can be installed on the image processing device 10 or in a format that can be executed. The recording medium is a medium that can be read by a computer. The recording medium may be a CD (Compact Disc)-ROM, a flexible disk (FD), a CD-R (Recordable), a DVD (Digital Versatile Disk), a USB (Universal Serial Bus) memory, an SD (Secure Digital) card, or the like.

[0027] Next, the functional configuration of the image processing device 10 will be described.

[0028] Fig. 3 is a diagram showing an example of a functional configuration of the image processing device 10. In order to clarify the data input / output relationship, Fig. 3 also shows an imaging unit 12, a detection unit 14, and a display unit 16 in addition to the image processing device 10.

[0029] The image processing device 10 of this embodiment generates a projected image by projecting a captured image obtained by the imaging unit 12 onto an adjusted projection plane obtained by adjusting the reference projection plane.

[0030] The reference projection plane is a projection plane having a predetermined reference shape, and is an image projection plane that is virtually disposed in a virtual space that corresponds to a real space.

[0031] Fig. 4 is a schematic diagram showing an example of the reference projection surface 40. The shape of the reference projection surface 40 is, for example, bowl-shaped, cylindrical, etc. Fig. 4 shows a bowl-shaped reference projection surface 40 as an example.

[0032] The bowl-like shape is a shape consisting of a bottom surface 400A, which is a circular flat surface, and a side wall surface 400B continuing to the bottom surface 400A. The circular shape includes a perfect circle shape, a circular shape, and an elliptical shape. The side wall surface 400B has a shape in which an opening at one end is continuing to the bottom surface 400A and the other end is open. The side wall surface 400B has a shape in which the diameter of the horizontal cross section increases from the bottom surface 400A side toward the opening side of the other end. The horizontal cross section is a cross section along a horizontal direction perpendicular to the vertical direction of the bottom surface 400A.

[0033] The cylindrical shape is a shape consisting of a circular bottom surface 400A and a side wall surface 400B that is continuous with the bottom surface 400A. The side wall surface 400B that constitutes the cylindrical reference projection surface 40 is cylindrical with an opening at one end that is continuous with the bottom surface 400A and an opening at the other end. However, the side wall surface 400B that constitutes the cylindrical reference projection surface 40 has a shape in which the diameter of the horizontal cross section is approximately constant from the bottom surface 400A side to the opening side of the other end.

[0034] In this embodiment, a case where the shape of the reference projection plane 40 is bowl-shaped will be described as an example. The reference projection plane 40 is a three-dimensional model virtually formed in a virtual space with a bottom surface 400A being a surface that substantially coincides with the ground below the moving object 2 and a central region of the bottom surface 400A being the reference position S of the moving object 2.

[0035] The range of the central region of the bottom surface 400A is, for example, the range occupied by an icon that diagrammatically shows the moving body 2. The icon is, for example, a rectangular symbol consisting of a straight line connecting both ends of the front part of the moving body 2, a straight line connecting both ends of the rear part, and a pair of straight lines along the overall length of the moving body 2. It is preferable that the center of the horizontal cross section of the icon coincides with the center of the bottom surface 400A. The reference position S is the position of the icon of the moving body 2 that is virtually arranged on the bottom surface 400A of the reference projection plane 40. The reference position S is defined by the position and range of the icon in the virtual space.

[0036] Continuing the description by returning to Fig. 3, the image processing device 10 generates a projection image by projecting a plurality of captured images obtained by a plurality of image capturing units 12 onto an adjusted projection plane obtained by performing an adjustment process on a reference projection plane 40.

[0037] Here, an object may be included in the overlapping portion of adjacent captured images projected onto the reference projection plane 40. The object is an object detected by the detection unit 14 as a group of one or more detection points, and is, for example, a solid object, a planar object, etc. The solid object is an object with a three-dimensional shape. The solid object is, for example, a fixed structure such as a pillar, a movable structure such as a car or a bicycle, a living thing such as a person or an animal, etc. The planar object is an object with a two-dimensional shape. The planar object is, for example, a line formed on the ground, etc. In this embodiment, a case where the object is a solid object will be described as an example.

[0038] When an object is included in the overlapping portion, the object may disappear or the image of the object may appear double in the area where the object is projected in the comparative projection image, which is a conventional projection image. The disappearance of an object means that at least a part of the area of ​​an object existing in real space does not exist in the area where the object in real space is projected in the projection image. The double image of an object means that the image of an object existing in real space is projected in a double or multiple overlapping state in the projection image.

[0039] Fig. 5 is a schematic diagram showing an example of a comparative projection image 600 of the prior art. It is assumed that a three-dimensional object B is included in an overlapping area DA between the photographing area E1 of the photographing unit 12A and the photographing area E2 of the photographing unit 12B. Fig. 5 shows objects BA and BB as object B. It is assumed that object BA is located closer to the moving object 2 than object BB.

[0040] In this case, when the image of object BA captured by the imaging unit 12B is projected onto the reference projection plane 40, the image BA1' of the object BA is projected with an inclination in the direction away from the side closer to the imaging unit 12B. On the other hand, when the image of object BA captured by the imaging unit 12A is projected onto the reference projection plane 40, the image BA2' of the object BA is projected with an inclination in the direction away from the side closer to the imaging unit 12A.

[0041] Similarly, when the image of the object BB captured by the imaging unit 12B is projected onto the reference projection plane 40, the image BB1' of the object BB is projected with an inclination in the direction away from the side closer to the imaging unit 12B. On the other hand, when the image of the object BB captured by the imaging unit 12A is projected onto the reference projection plane 40, the image BB2' of the object BB is projected with an inclination in the direction away from the side closer to the imaging unit 12A.

[0042] In this way, when images of one object B captured by different imaging units 12 are projected onto the reference projection surface 40, the images of the object B are projected onto the reference projection surface 40 inclined in different directions.

[0043] Here, as described later, when generating a projected image, for example, on the reference projection surface 40, for a portion of the overlap area DA that is continuous with the shooting area E1 (first area), only the captured image of the shooting area E1 is projected, for a portion of the overlap area DA that is continuous with the shooting area E2 (second area), only the captured image of the shooting area E2 is projected, and for an intermediate area (third area) between the first and second areas, the captured images of the two shooting areas E1 and E2 that make up the overlap area DA are synthesized and the synthesized image is projected.

[0044] For this reason, in the conventional technology, for an object BA existing in a position close to the moving object 2 in the overlapping region D, an image BA1' included in the captured image of the shooting region E2 is included in the first region on the reference projection plane 40, and an image BA2' included in the captured image of the shooting region E1 is included in the second region on the reference projection plane 40, so that the image BA1' of the object BA is not projected in the first region and the image BA2' of the object BA is not projected in the second region on the reference projection plane 40. As a result, the object BA existing in a position close to the moving object 2 in the overlapping region D may disappear.

[0045] Furthermore, for an object BB present at a position far from the moving object 2 in the overlapping area D, since both an image BA1' included in the captured image of the shooting area E2 and an image BA2' included in the captured image of the shooting area E1 are included in the third area on the reference projection plane 40, there were cases where both an image BB1' and an image BB2' of the object BB were projected in the third area on the reference projection plane 40. As a result, there were cases where an object BB present at a position far from the moving object 2 in the overlapping area D was projected as a double image.

[0046] Therefore, when the object B is included in the overlapping portion of the captured images, the image processing device 10 of this embodiment executes an adjustment process for the overlapping region D on the reference projection plane 40. The image processing device 10 will be described in detail below.

[0047] 3, the image processing device 10 includes an acquisition unit 20, a determination unit 22, a storage unit 24, an adjustment unit 26, a generation unit 28, and a display control unit 30.

[0048] Some or all of the above-mentioned multiple units may be realized by, for example, having a processing device such as the processor 10A execute a program, that is, by software. Also, some or all of the above-mentioned multiple units may be realized by hardware such as an IC (Integrated Circuit), or may be realized by a combination of software and hardware.

[0049] The acquisition unit 20 acquires the multiple captured images from the imaging unit 12. As described above, the multiple captured images partially overlap each other in the imaging area E. The acquisition unit 20 also acquires position information of the detection points from the detection unit 14. Each of the multiple detection units 14 (detection units 14A to 14D) detects position information of each of the multiple detection points at each timing. Therefore, the acquisition unit 20 acquires the position information of each of the multiple detection points and the captured images by each of the multiple imaging units 12 at each timing.

[0050] The acquisition unit 20 outputs the position information of each of the plurality of captured images and the plurality of detection points to the determination unit 22. The determination unit 22 receives the position information of each of the plurality of captured images and the plurality of detection points from the acquisition unit 20.

[0051] The determination unit 22 determines whether or not an object B is included in an overlapping portion of adjacent captured images included in a projected image obtained by projecting a plurality of captured images onto the reference projection plane 40.

[0052] For example, the determination unit 22 determines whether or not an object B is included in an overlapping portion of adjacent captured images by using position information of each of the multiple captured images and multiple detection points received from the acquisition unit 20, the reference projection plane information 24A, and the overlapping area information 24B. As described above, the position information of the detection point is information indicating at least one of the distance and the direction from the detection unit 14 to the object B.

[0053] The reference projection plane information 24A is information that represents the shape of the reference projection plane 40. The overlapping area information 24B is information that represents the position and range of the image synthesis area F that is set within the overlapping area D on the reference projection plane 40. For example, the shooting areas E of each of the multiple shooting units 12 are set in advance. For this reason, the relative position and range of the overlapping area D of each of the multiple shooting areas E with respect to the moving object 2 is determined in advance, and the relative position and range of the image synthesis area F with respect to the moving object 2 can be determined in advance.

[0054] For this reason, the reference projection plane information 24A and the overlapping area information 24B are stored in advance in the storage unit 24. The storage unit 24 stores various data. The storage unit 24 is, for example, a semiconductor memory element such as a RAM, a ROM, or a flash memory, a hard disk, an optical disk, or the like. The storage unit 24 may be a storage device provided outside the image processing device 10. The storage unit 24 may also be a storage medium that stores or temporarily stores programs and various information downloaded via a LAN (Local Area Network) or the Internet.

[0055] The determination unit 22 maps a plurality of detection points defined by the position information received from the acquisition unit 20 onto the reference projection plane 40 represented by the reference projection plane information 24A. Mapping means arranging the detection points in a virtual space defined by the reference projection plane 40. The determination unit 22 may specify, by a known method, positions in the virtual space corresponding to the directions and distances defined by the position information of the detection points, and arrange the detection points at the specified positions.

[0056] Furthermore, the determination unit 22 uses the overlap region information 24B to identify the position and range of the overlap region D on the reference projection plane 40. Then, the determination unit 22 determines whether or not a detection point constituting the object B is present in the overlap region D. Through this determination process, the determination unit 22 determines whether or not the object B is included in the overlap region D in the projection image.

[0057] The determination unit 22 may specify a group of detection points constituting the object B by a known method using position information of each of the multiple detection points mapped onto the reference projection plane 40. For example, the determination unit 22 specifies the group of detection points constituting the object B by performing a matching process between the mapped multiple detection points and a group of detection points constituting a predetermined outer shape of the object B.

[0058] Then, the judgment unit 22 determines whether or not the object B is included in the overlapping region D by determining whether or not at least a portion of the region of the object B formed by the group of identified detection points overlaps with the overlapping region D.

[0059] The determination unit 22 outputs the determination result as to whether or not the object B is included in the overlap region D to the adjustment unit 26. The adjustment unit 26 receives the determination result from the determination unit 22.

[0060] When the determination result indicates that the object B is included in the overlap region D, the adjustment unit 26 executes an adjustment process.

[0061] The adjustment process is a process of adjusting the overlapping region D including the object B on the reference projection plane 40. In other words, the adjustment process is a process of adjusting the overlapping region D including the object B among the multiple overlapping regions D included in the reference projection plane 40.

[0062] Adjusting the overlap area D means at least one of a projection surface deformation process that deforms the constituent surfaces of the overlap area D on the reference projection surface 40, and an image synthesis adjustment process that adjusts the position and range of the image synthesis area F that is set within the overlap area D on the reference projection surface 40.

[0063] First, a projection surface transformation process, which is an example of the adjustment process executed by the adjustment unit 26, will be described.

[0064] The projection plane deformation process is a process for deforming the constituent surfaces of overlapping area D including object B on the reference projection plane 40 into a shape that passes through the reference position S side of object B. The constituent surfaces of overlapping area D including object B on the reference projection plane 40 refer to the area of ​​the surfaces that constitute overlapping area D including object B out of the entire area of ​​reference projection plane 40.

[0065] A shape that passes on the reference position S side of object B means a shape that passes at least the detection point closest to the reference position S among the group of detection points that make up object B, or a shape that passes at least between that detection point and the reference position S.

[0066] 6A to 6E are diagrams illustrating an example of the projection surface transformation process.

[0067] 6A is a diagram of the reference projection plane 40 viewed from a direction inclined with respect to the vertical direction (see the direction of arrow W in FIG. 4). The adjustment unit 26 executes a deformation process to deform a constituent surface 40A of the overlapping area D on the reference projection plane 40. The constituent surface 40A is a constituent surface that is a part of the reference projection plane 40, and is a constituent surface that corresponds to the overlapping area D including the object B. By the adjustment unit 26 executing the deformation process of the constituent surface 40A, an adjusted projection plane 42 having an adjusted constituent surface 42A that passes on the reference position S side of the moving body 2 relative to the object B is generated, as shown in FIGS. 6B to 6D.

[0068] 6B to 6D show an example in which an object B exists in overlap region DB among multiple overlap regions D (overlap regions DA to DD) around the moving object 2. FIG. 6B is a view of adjusted projection surface 42 viewed from a direction inclined relative to the vertical direction (see the direction of arrow W in FIG. 4). FIG. 6C is a plan view of adjusted projection surface 42 viewed from the vertical direction. FIG. 6D is a schematic plan view of adjusted projection surface 42 viewed from the vertical direction.

[0069] 6D, the adjustment unit 26 deforms the constituent surface 40A of the overlapping region DB including the object B on the reference projection surface 40 into a shape that passes through the reference position S side of the moving body 2 from the object B, to obtain an adjusted constituent surface 42A. Therefore, the adjusted projection surface 42 becomes a projection surface in which the constituent surface 40A of the overlapping region DB including the object B is deformed into the adjusted constituent surface 42A (see FIGS. 6A to 6D).

[0070] Note that there may be a case where the object B is included in each of a plurality of overlapping regions D that are different from one another. In this case, the adjustment unit 26 may execute the projection surface transformation process for each of the plurality of overlapping regions D.

[0071] 6E shows an example in which an object B1 and an object B2 exist in each of overlapping areas DA and DB, respectively, among a plurality of overlapping areas D (overlapping areas DA to DD) around the moving object 2. Objects B1 and B2 are examples of object B. FIG 6E is a schematic diagram of a plan view of adjusted projection plane 42 viewed from the vertical direction.

[0072] In this case, the adjustment unit 26 deforms the constituent surface 40A1 of the overlapping region DB including the object B1 on the reference projection plane 40 into a shape that passes on the reference position S side of the moving body 2 from the object B1, and sets it as an adjusted constituent surface 42A1. The adjustment unit 26 also deforms the constituent surface 40A2 of the overlapping region DA including the object B2 on the reference projection plane 40 into a shape that passes on the reference position S side of the moving body 2 from the object B2, and sets it as an adjusted constituent surface 42A2. The constituent surfaces 40A1 and 40A2 are examples of the constituent surface 40A of the reference projection plane 40. The adjusted constituent surface 42A1 and 42A2 are examples of the adjusted constituent surface 42A.

[0073] In this manner, the adjustment unit 26 executes a projection plane deformation process for deforming the constituent plane 40A of the overlapping region D including the object B on the reference projection plane 40.

[0074] The deformation of the configuration plane 40A may be performed by changing the position coordinates of the configuration plane 40A in the virtual space. The adjustment unit 26 adjusts the configuration plane 40A of the reference projection plane 40 to the adjusted configuration plane 42A, for example, by the following method.

[0075] Figures 7A to 7C are explanatory diagrams of an example of a deformation process of the component surface 40 A. Figures 7A to 7C are schematic plan views of the reference projection surface 40 viewed from the vertical direction.

[0076] As shown in FIGS. 7A to 7C, in the virtual space, a constituent surface 40A of the reference projection surface 40 is composed of a plurality of divided regions divided by a plurality of dividing lines extending radially from a reference point 50.

[0077] The reference point 50 is a point that is a reference for the position coordinates on the reference projection plane 40. In this embodiment, as shown in FIG. 1, the points of the multiple overlapping areas D that are closest to the moving object 2 (reference position S) are determined as the reference points 50 (reference points 50A to 50D). The reference point 50A is a reference point 50 that is a reference for the position coordinates in the overlapping area DA between the shooting area E1 and the shooting area E4. The reference point 50B is a reference point 50 that is a reference for the position coordinates in the overlapping area DB between the shooting area E1 and the shooting area E2. The reference point 50C is a reference point 50 that is a reference for the position coordinates in the overlapping area DC between the shooting area E2 and the shooting area E3. The reference point 50D is a reference point 50 that is a reference for the position coordinates in the overlapping area DD between the shooting area E3 and the shooting area E4.

[0078] As shown in Fig. 7A, the adjustment unit 26 specifies an intersection Q between a straight line connecting multiple detection points P constituting the object B included in the overlapping area DA and a dividing line of each divided area constituting the constituent surface 40A2 of the reference projection surface 40. In Fig. 7A, detection points P1 and P2 are shown as an example of multiple detection points P constituting the object B. The detection points P1 and P2 are, for example, the top two detection points P that are closest to the reference position S among the multiple detection points P constituting the object B. In addition, when multiple intersection points Q are obtained, it is sufficient to select any one of the multiple intersection points Q, for example, it is sufficient to select the intersection point that is closest to the reference point 50.

[0079] As shown in FIG. 7B, next, the adjustment unit 26 searches for an index of the identified intersection point Q. An index is information for identifying a position on the reference projection plane 40. An index is previously assigned to each divided area obtained by dividing the reference projection plane 40 into a plurality of areas on the reference projection plane 40. In FIG. 7B, the numbers in parentheses are shown as an example of an index. For example, it is assumed that the adjustment unit 26 searches for the index "12" as the index of the intersection point Q.

[0080] As shown in FIG. 7C, next, adjustment unit 26 calculates distance L from reference point 50 to intersection point Q at the position specified by the index.

[0081] In the case of the form shown in FIG. 7C, the adjustment unit 26 calculates the distance L from the reference point 50A to the intersection Q. Here, the distance L is a horizontal distance parallel to the bottom surface 400A. Then, the adjustment unit 26 uses the distance L to change the position coordinates of the constituent surface 40A2 in the virtual space so that the constituent surface 40A2 of the overlapping area DA has a shape that at least passes through the intersection Q or between the intersection Q and the reference point 50A. In detail, the adjustment unit 26 changes the position coordinates of an area in the constituent surface 40A2 of the overlapping area DA whose horizontal distance from the reference point 50A is greater than the distance L so that the horizontal distance from the reference point 50A is equal to or less than the distance L. That is, the adjustment unit 26 deforms the constituent surface 40A2 so that the constituent surface 40A2 of the overlapping area DA becomes a surface that passes through the position of the object B or a position closer to the reference position S than the object B.

[0082] 6B to 6E, the constituent surface 40A of the overlapping area D including the object B on the reference projection surface 40 becomes an adjusted constituent surface 42A, which is deformed into a shape that passes through the reference position S side of the moving body 2 from the object B. Therefore, the reference projection surface 40 becomes an adjusted projection surface 42, in which the constituent surface 40A of the overlapping area D on the reference projection surface 40 is deformed into an adjusted constituent surface 42A.

[0083] It is preferable that the adjustment unit 26 adjusts the cross-sectional shape of the adjusted projection plane 42 so that it has an Nth-order curve passing on the reference position S side from the object B. N is an integer of 2 or more.

[0084] As shown in Figures 6B to 6E, the adjustment unit 26 deforms the shape of the constituent surface 40A of the overlap region D including the object B into a shape that passes on the reference position S side from the object B, but the shape of the constituent surfaces of the areas on the reference projection plane 40 other than the constituent surface 40A of the overlap region D does not necessarily need to be deformed.

[0085] In addition, the adjustment unit 26 may deform the shape of the constituent surface 40A of the overlap region D including the object B into a shape that passes through the reference position S side of the object B, and further deform the shape of the constituent surfaces of the area other than the constituent surface 40A of the overlap region D on the reference projection plane 40 in accordance with the shape of the adjusted constituent surface 42A, which is the deformed constituent surface 40A.

[0086] 8A and 8B are schematic diagrams of an example of adjusted projection surface 42. Fig. 8A is a schematic diagram of a plan view of adjusted projection surface 42 viewed from the vertical direction. Fig. 8B is a diagram of adjusted projection surface 42 viewed from a direction inclined relative to the vertical direction (see the direction of arrow W in Fig. 4).

[0087] For example, assume that the overlap region DA includes an object B2, and the overlap region DB includes an object B1. In this case, the adjustment unit 26 deforms the configuration surface 40A1 of the overlap region DB including the object B1 on the reference projection plane 40 into a shape that passes on the reference position S side of the moving body 2 from the object B1, and sets it as an adjusted configuration surface 42A1. Similarly, the adjustment unit 26 deforms the configuration surface 40A2 of the overlap region DA including the object B2 on the reference projection plane 40 into a shape that passes on the reference position S side of the moving body 2 from the object B2, and sets it as an adjusted configuration surface 42A2.

[0088] Then, the adjustment unit 26 changes the shape of the component surface 40B of the region other than the component surface 40A1 and the component surface 40A2 on the reference projection surface 40 into the adjusted component surface 42A1 and the adjusted component surface 42A2, thereby changing the shape of the region 40B into the adjusted component surface 42B. In detail, the adjustment unit 26 changes the shape of the region 40B into a curve of curvature whose rate of change with respect to the curvature of the adjusted component surface 42A1 and the adjusted component surface 42A2 changes within a range of a predetermined value or less. In this case, the adjusted projection surface 42 becomes a projection surface in which the overall shape of the reference projection surface 40 is adjusted according to the object B included in the overlap region D.

[0089] In this way, the adjustment unit 26 may generate the adjusted projection plane 42 by adjusting the overall shape of the reference projection plane 40 in accordance with the object B included in the overlapping region D.

[0090] The shape of the vertical cross section of the adjustment configuration surface 42A on the adjustment projection surface 42 is not limited. The vertical cross section is a cross section cut along the vertical direction of the bottom surface 400A. For example, the shape of the vertical cross section of the adjustment configuration surface 42A may be any one of a straight line that passes through the reference position S side of the object B included in the overlapping area D and is parallel to the vertical direction, a straight line that is inclined with respect to the vertical direction, a quadratic curve, and a square root curve.

[0091] 9A to 9D are schematic diagrams showing an example of a cross-sectional shape of adjusted projection surface 42 in the vertical direction.

[0092] 9A, the adjustment unit 26 adjusts the vertical cross-sectional shape of the constituent surface 40A of the overlapping region D including the object B on the adjusted projection surface 42 to a linear shape parallel to the vertical direction (arrow Z direction), for example. In this case, the cross-sectional shape along the vertical direction of the adjusted constituent surface 42A becomes a linear shape along the vertical direction.

[0093] 9B, the adjustment unit 26 adjusts the vertical cross-sectional shape of the constituent surface 40A of the overlapping region D including the object B on the adjusted projection surface 42 to a linear shape inclined with respect to the vertical direction (arrow Z direction), for example. In this case, the vertical cross-sectional shape of the adjusted constituent surface 42A becomes a linear shape inclined with respect to the vertical direction.

[0094] 9C, the adjustment unit 26 adjusts, for example, the shape of the vertical cross section of the constituent surface 40A of the overlapping region D including the object B on the adjusted projection surface 42 to a quadratic curve shape. In this case, the shape of the vertical cross section of the adjusted constituent surface 42A becomes a quadratic curve shape.

[0095] 9D, the adjustment unit 26 adjusts, for example, the shape of the vertical cross section of the constituent surface 40A of the overlapping region D including the object B on the adjusted projection surface 42 to a square root curved shape. In this case, the shape of the vertical cross section of the adjusted constituent surface 42A becomes a square root curved shape.

[0096] In this way, the adjustment unit 26 may perform a deformation process to adjust the constituent surface 40A of the overlap region D including object B to the shape of a straight line passing on the reference position S side of object B and parallel to the vertical direction, a straight line inclined with respect to the vertical direction, a quadratic curve, or a square root curve.

[0097] Next, an image synthesis adjustment process, which is an example of the adjustment process executed by the adjustment unit 26, will be described.

[0098] The image synthesis adjustment process is an adjustment process that adjusts the position and range of an image synthesis area F that is set within an overlapping area D including an object B on the reference projection plane 40 so that the image synthesis area F does not overlap with the object B. In other words, the image synthesis adjustment process is a process that changes the position and range within the overlapping area D where the captured images are synthesized and projected when a portion of each captured image of adjacent capturing areas E is synthesized and projected onto the projection plane.

[0099] 10A and 10B are explanatory diagrams of an example of the image synthesis adjustment process. Fig. 10A is a schematic diagram of a plan view of the reference projection plane 40 viewed from the vertical direction. Fig. 10A shows the position and range of the image synthesis area F in the overlap area D represented by the overlap area information 24B. As described above, the overlap area information 24B is information that represents the position and range of the image synthesis area F set in the overlap area D on the reference projection plane 40.

[0100] Fig. 10A shows an example in which an object B is included in image synthesis area FA, which is an image synthesis area F set within overlapping area D between shooting area E1 and shooting area E4. In this case, as shown in Fig. 10B, adjustment unit 26 adjusts this image synthesis area FA to image synthesis area FA', which is an area that does not include object B. Then, adjustment unit 26 generates information indicating the position and range of the adjusted image synthesis area FA'.

[0101] The information indicating the position and range of the adjusted image synthesis area FA' is, for example, information indicating a synthesis coefficient when each captured image of adjacent shooting areas E is synthesized and projected onto a projection surface. The synthesis coefficient is information indicating the blending ratio of two captured images. The synthesis coefficient is, for example, a transmission value. The transmission value may be referred to as an α value. In this embodiment, a case where the synthesis coefficient is an α value will be described.

[0102] Here, the image synthesis area F in the projected image is defined by adjusting the blending ratio of the captured images of the adjacent captured areas E. In detail, the image synthesis area F is realized by setting the synthesis coefficient of a part of the captured image in a range greater than 0 and less than 1.

[0103] When the synthesis coefficient is "0", the images captured in the two shooting areas E constituting the overlapping area D are not synthesized, and only the image captured in one of the two shooting areas E is projected. Similarly, when the synthesis coefficient is "1", the images captured in the two shooting areas E constituting the overlapping area D are not synthesized, and only the image captured in the other of the two shooting areas E is projected. On the other hand, an area where the synthesis coefficient is greater than 0 and less than 1 means that the images captured in the two shooting areas E constituting the overlapping area D are synthesized, and the synthesized image is projected.

[0104] 11A and 11B are diagrams illustrating an example of adjustment of the synthesis coefficient.

[0105] As shown in Fig. 11A, for example, an index is previously assigned to each divided region obtained by dividing the reference projection plane 40 into a plurality of regions on the reference projection plane 40. Fig. 11A shows an example of indexes according to angles from a reference line 51 passing through a reference point 50. The reference line 51 is, for example, a straight line connecting both ends of the rear part of the moving body 2, a straight line connecting both ends of the front part, and a straight line along the overall length direction of the moving body 2 and passing through the reference point 50. Fig. 11 shows, as an example, reference lines 51A and 51B passing through a reference point 50A.

[0106] Furthermore, a blending coefficient is preset for each index on the reference projection plane 40. For this reason, the overlapping area information 24B includes, as information indicating the position and range of the image blending area F, an index to which a blending coefficient in the range greater than 0 and less than 1 has been assigned, and the value of the blending coefficient set for each index.

[0107] For example, assume that the adjustment unit 26 adjusts the image synthesis area FA to the image synthesis area FA'. In this case, the adjustment unit 26 calculates the angle of the image synthesis area FA'. For example, the adjustment unit 26 calculates the angles θ1 and θ2 shown in FIG. 11A as the angles of the image synthesis area FA'. The angle θ2 is an angle that sets the range of the image synthesis area FA'. The angle of the image synthesis area FA' is information that represents the range occupied by the image synthesis area FA' as an angle centered on the reference point 50A. The angle θ1 is the angle between the reference line 51B and the overlapping area DA', and is an angle that sets the position of the image synthesis area FA'.

[0108] Then, the adjustment unit 26 searches for an index corresponding to the angle θ1 and the angle θ2 of the image synthesis area FA' on the reference projection plane 40. For example, it is assumed that the adjustment unit 26 has searched for indexes "7" to "13" as indexes corresponding to the angle θ1 and the angle θ2 of the image synthesis area FA'. The adjustment unit 26 sets a synthesis coefficient greater than 0 and less than 1 to the searched indexes "7" to "13". Then, the adjustment unit 26 assigns a synthesis coefficient for projecting only the captured image of the shooting area E adjacent to the area of ​​the index to the index of the overlapping area DA other than the image synthesis area FA'. For example, the adjustment unit 26 sets "0" to indexes "1" to "6", which is a synthesis coefficient indicating that only the captured image of the shooting area E4 is projected. Moreover, the adjustment unit 26 sets "1" to indexes "14" to "18", which is a synthesis coefficient indicating that only the captured image of the shooting area E1 is projected. Then, the adjuster 26 assigns synthesis coefficients that increase linearly from 0 to 1 to the indexes "7" to "13" that form the image synthesis area FA'.

[0109] 11B, a combination coefficient greater than 0 and less than 1 is assigned to the index of the angle θ2, which is the adjusted image combination area FA'. A combination coefficient of "0.0" is assigned to the index of the area in the overlap area DA other than the image combination area FA' that is continuous with the shooting area E4. A combination coefficient of "1.0" is assigned to the index of the area in the overlap area DA other than the image combination area FA' that is continuous with the shooting area E1.

[0110] Then, the adjuster 26 outputs, to the generator 28, synthesis coefficient information indicating the adjusted synthesis coefficients assigned to each of the indexes, as information indicating the position and range of the adjusted image synthesis area FA'.

[0111] In addition, as an adjustment process, the adjustment unit 26 only needs to perform at least one of a projection surface deformation process that deforms the constituent surface 40A of the reference projection surface 40, and an image synthesis adjustment process that adjusts the position and range of the image synthesis area F set within the overlap area D, and it is possible to perform a combination of the projection surface deformation process and the image synthesis adjustment process.

[0112] Returning to Fig. 3, the description will continue. The adjustment unit 26 outputs at least one of the adjusted projection surface information of the adjusted projection surface 42, which is the adjusted reference projection surface 40, and the blending coefficient information to the generation unit 28. In this embodiment, a form in which the adjustment unit 26 outputs both the adjusted projection surface information and the blending coefficient information to the generation unit 28 will be described as an example.

[0113] The generation unit 28 receives at least one of the adjusted projection surface information and the synthesis coefficient information from the adjustment unit 26, and generates a projection image by projecting the multiple captured images acquired by the acquisition unit 20 onto the adjusted projection surface 42 based on the received information.

[0114] Fig. 12 is a schematic diagram showing an example of the projected image 60. Fig. 12 shows a projected image 60A as an example of the projected image 60. The generation unit 28 projects each of the multiple captured images acquired by the acquisition unit 20 onto an adjusted projection surface 42 having a shape represented by the adjusted projection surface information received from the adjustment unit 26. The adjusted projection surface 42 is a projection surface on which a constituent surface 40A of the overlapping region D including the object B is deformed by the above-mentioned adjustment process.

[0115] Therefore, the projected image 60A is an image in which the double image and disappearance of the image of the object B included in the overlapping area D are suppressed.

[0116] 13A and 13B are schematic diagrams showing an example of a comparative projection image 600 of the prior art. Note that Fig. 13A and Fig. 13B are examples in which the captured image used to generate the projection image 60A shown in Fig. 12 is projected onto the reference projection surface 40.

[0117] As shown in Fig. 13A, in the conventional technology, there were cases where the image of object B existing in the overlapping area D was doubled in a comparative projection image 600A. Also, as shown in Fig. 13B, in the conventional technology, there were cases where the image of object B existing in the overlapping area D was lost in a comparative projection image 600B.

[0118] On the other hand, in this embodiment, the generation unit 28 generates the projection image 60 by projecting a plurality of captured images onto the adjusted projection surface 42. Therefore, as shown in Fig. 12, the generation unit 28 can generate the projection image 60 in which double imaging or disappearance of the image of the object B present in the overlap region D is suppressed.

[0119] In addition, when the synthesis coefficient information is received, the generation unit 28 changes the transmission value of each of the captured images of the multiple shooting areas E in accordance with the synthesis coefficient assigned to each of the indexes represented by the synthesis coefficient information, and then projects the captured images onto the adjusted projection surface 42.

[0120] FIG. 14 is a schematic diagram showing an example of the projected image 60. In FIG. 14, a projected image 60C is shown as an example of the projected image 60. The projected image 60C is an example of the projected image 60 in which a captured image in which the transmission value has been adjusted according to the total coefficient represented by the synthesis coefficient information is projected onto the adjusted projection surface 42 having a shape represented by the adjusted projection surface information. In other words, the projected image 60C is an example of the projected image 60 generated by executing a projection surface deformation process and an image synthesis adjustment process as adjustment processes, and is an example of the projected image 60 in which the shape of the projection surface 40 has been changed and the position and range of the image synthesis area F has been adjusted so as not to overlap with the object B.

[0121] Therefore, the projected image 60C is an image in which the image of the object B included in the overlapping region D is prevented from being doubled or lost.

[0122] Fig. 15A is a schematic diagram showing an example of a comparative projection image 600 of the prior art. Fig. 15A is an example in which the captured image used to generate the projection image 60C shown in Fig. 14 is projected onto the reference projection plane 40. Fig. 15B is a schematic diagram showing an example of a projection image 60B in which the captured image used to generate the projection image 60C shown in Fig. 14 is projected onto the adjusted projection plane 42. That is, the projection image 60B shown in Fig. 15B is an example of a projection image 60 generated by executing a transformation process as an adjustment process.

[0123] As shown in FIG. 15A, in the conventional technology, there are cases where the comparative projection image 600C is obtained in which the object B existing in the overlapping region D has disappeared. On the other hand, a case where a deformation process of the projection surface 40 is executed as an adjustment process is assumed. In this case, as shown in FIG. 15B, the projection image 60B is a projection image in which at least one of the double image and disappearance of the image of the object B included in the overlapping region D (disappearance of the object B in the case of FIG. 15B) is suppressed compared to the comparative projection image 600C. Then, a case where a deformation process and an overlapping region adjustment process are executed as an adjustment process is assumed. In this case, as shown in FIG. 14, the projection image 60C is a projection image in which the other of the double image and disappearance of the image of the object B included in the overlapping region D (double image of the object B in the case of FIG. 14) is further suppressed compared to the projection image 60B shown in FIG. 15B.

[0124] For this reason, it is preferable that the adjustment section 26 executes both the transformation process and the overlap area adjustment process as the adjustment process.

[0125] Returning to Fig. 3, the description will be continued. The display control unit 30 controls the display of the projection image 60 generated by the generation unit 28 on the display unit 16. Therefore, the display unit 16 displays the projection image 60 in which at least one of the double image and disappearance of the image of the object B existing in the real space is suppressed. Note that the display control unit 30 may perform various image processing such as changing the projection image 60 to an image viewed from a virtual viewpoint, and then perform display control to display the projection image on the display unit 16.

[0126] Next, an example of the flow of image processing executed by the image processing device 10 will be described.

[0127] FIG. 16 is a flowchart showing an example of the flow of image processing executed by the image processing device 10 described above.

[0128] The acquisition unit 20 acquires a plurality of captured images from the plurality of imaging units 12 (step S100). The acquisition unit 20 also acquires position information of each of the plurality of detection points P from the detection unit 14 (step S102).

[0129] The determination unit 22 reads the reference projection plane information 24A and the overlapping area information 24B from the storage unit 24 (step S104).

[0130] The determination unit 22 maps the multiple detection points P defined by the position information acquired in step S102 onto the reference projection plane 40 represented by the reference projection plane information 24A read in step S104 (step S106).

[0131] The determination unit 22 determines whether or not the object B is included in the overlap region D by using the overlap region information 24B read in step S104 and the mapping result in step S106 (step S108). If it is determined that the object B is included in the overlap region D (step S108: Yes), the process proceeds to step S110.

[0132] In step S110, the adjustment unit 26 executes an adjustment process of the constituent surface 40A of the reference projection plane 40 (step S110). The adjustment process of step S110 is a process of adjusting the overlapping area D including the object B on the reference projection plane 40. As described above, the adjustment unit 26 executes, as an adjustment process, at least one of a deformation process that deforms the constituent surface of the overlapping area D on the reference projection plane 40, and an image synthesis adjustment process that adjusts the position and range of the image synthesis area set within the overlapping area D on the reference projection plane 40. The flow of the adjustment process will be described in detail later.

[0133] By the processing of step S110, the adjusted projection plane 42 is generated, and adjusted projection plane information of the adjusted projection plane 42 obtained by transforming the constituent plane 40A of the reference projection plane 40, and synthesis coefficient information generated by the image synthesis adjustment processing are generated.

[0134] Next, the generation unit 28 synthesizes the multiple captured images acquired in step S100 using the synthesis coefficients indicated by the synthesis coefficient information generated in step S110, and projects the synthesized images onto the adjusted projection plane 42 generated in step S110 (step S112). By the process of step S112, the generation unit 28 generates a projection image 60. Then, the process proceeds to step S116, which will be described later.

[0135] On the other hand, if a negative decision is made in step S108 (step S108: No), the process proceeds to step S114. In step S114, the generation unit 28 projects the captured image acquired in step S100 onto the reference projection plane 40 (step S114). Then, the process proceeds to step S116.

[0136] In step S116, the display control unit 30 performs display control to display the projection image 60 generated in step S112 or the projection image generated in step S116 on the display unit 16 (step S116).

[0137] Next, the image processing device 10 judges whether to end the image processing (step S118). For example, the image processing device 10 executes the process of step S118 by judging whether or not an instruction input indicating the end of the image processing has been received. If a negative judgment is made in step S118 (step S118: No), the process returns to the above step S100. On the other hand, if a positive judgment is made in step S118 (step S118: Yes), the present routine ends. Note that when returning from step S118 to step S100, the process may be continued with the adjusted projection plane 42 generated in step S110 as a new reference projection plane 40.

[0138] Next, an example of the flow of the transformation process, which is the adjustment process in step S110, will be described.

[0139] FIG. 17 is a flowchart showing an example of the flow of the transformation process.

[0140] The adjustment unit 26 identifies the intersection Q between a straight line connecting the detection points P constituting the object B included in the overlapping area D, among the detection points whose position information was obtained in step S102 (see Figure 16), and the dividing line of each divided area constituting the constituent surface 40A of the reference projection plane 40 (step S200).

[0141] Next, the adjustment unit 26 searches for the index of the intersection Q identified in step S200 (step S202). Then, the adjustment unit 26 calculates the distance L from the reference point 50 to the intersection Q identified by the index (step S204). Here, the distance L is the distance in the horizontal direction parallel to the bottom surface 400A.

[0142] The adjustment unit 26 changes the position coordinates of the constituent surface 40A of the overlapping area D including the object B on the reference projection surface 40 so that the distance between the constituent surface 40A and the reference point 50 is equal to or less than the distance L (step S206). By the processing of step S206, the constituent surface 40A of the overlapping area D including the object B on the reference projection surface 40 is transformed into the adjusted constituent surface 42A, and the adjusted projection surface 42 is generated. Then, this routine ends.

[0143] Next, an example of the flow of the image synthesis adjustment process, which is the adjustment process in step S110, will be described.

[0144] FIG. 18 is a flowchart showing an example of the flow of the image synthesis adjustment process.

[0145] The adjustment unit 26 adjusts the image synthesis area F, which is set within the overlap area D including the object B on the reference projection plane 40, to an image synthesis area F' at an angle that does not include the object B, and calculates the angle (angle θ1, angle θ2) of the image synthesis area F' (step S300).

[0146] Next, the adjustment unit 26 searches for an index according to the angle of the image synthesis area F' calculated in step S300 (step S302).

[0147] Next, the adjustment unit 26 adjusts the blending coefficients set for each index of the overlap region D so that the region of the index searched for in step S302 becomes the image blending region F' (step S304). By the process of step S304, a blending coefficient greater than 0 and less than 1 is assigned to the index of the adjusted image blending region F' on the reference projection plane 40. Moreover, a blending coefficient of "0.0" or "1.0" is assigned to the region of the overlap region D other than the adjusted image blending region F'. Then, the adjustment unit 26 outputs blending coefficient information indicating the adjusted blending coefficients assigned to each index to the generation unit 28 as information indicating the position and range of the adjusted image blending region F'. Then, this routine ends.

[0148] As described above, the image processing device 10 of this embodiment includes the acquisition unit 20, the determination unit 22, and the adjustment unit 26. The acquisition unit 20 acquires a plurality of captured images in which a portion of the captured region E overlaps. The determination unit 22 determines whether or not an object B is included in an overlap region D of adjacent captured images on a reference projection plane 40 of a predetermined shape, which is an image projection plane virtually arranged in a virtual space corresponding to real space. The adjustment unit 26 executes an adjustment process for the overlap region D including the object B on the reference projection plane 40.

[0149] Here, in the conventional technology, no image processing was performed with a focus on the overlapping region D. For this reason, in the conventional technology, when object B exists in the overlapping region D of the photographed image projected onto the projection surface, there are cases in which object B disappears or the image of object B is doubled in the projected region of the projected image.

[0150] On the other hand, in the image processing device 10 of this embodiment, an adjustment process of the overlapping region D including the object B on the reference projection plane 40 is executed.

[0151] Therefore, the image processing device 10 of this embodiment can suppress the disappearance of object B contained in the overlapping area D of the captured image or the double image of object B by projecting the captured image onto the adjusted projection surface 42, which is the reference projection surface 40 after adjustment by the adjustment unit 26.

[0152] Although the embodiments have been described above, the image processing device, image processing method, and image processing program disclosed in the present application are not limited to the above-described embodiments as they are, and can be embodied by modifying the components in the implementation stage without departing from the gist of the invention. In addition, various inventions can be formed by appropriate combinations of the multiple components disclosed in the above-described embodiments. For example, some components may be deleted from all the components shown in the embodiments.

[0153] The image processing device 10 of the above embodiment can be applied to various devices and systems. For example, the image processing device 10 of the above embodiment can be applied to a surveillance camera system that processes images obtained from a surveillance camera, or an in-vehicle camera system that processes images of the surrounding environment outside the vehicle, such as an in-vehicle surround view monitor system, a parking assistance system, or an automatic driving system. [Explanation of symbols]

[0154] 10 Image processing device 20 Acquisition Department 22 Judgment Department 26 Adjustment section 28 Generation part

Claims

1. an acquisition unit that acquires a plurality of captured images in which the captured areas partially overlap; a determination unit that determines whether or not an object is included in an overlapping portion of adjacent captured images included in a projected image obtained by projecting a plurality of the captured images onto a reference projection plane that is an image projection plane virtually disposed in a virtual space corresponding to a real space; an adjustment unit that executes an image synthesis adjustment process for adjusting an overlapping area including the object on the reference projection plane corresponding to the overlapping portion including the object, the image synthesis adjustment process being a process for adjusting a position and a range of an image synthesis area that is set as a part of the overlapping area including the object on the reference projection plane within the overlapping area including the object, and that is set by two division lines extending radially from a reference point in the overlapping area toward a direction away from a reference position of a moving body that is a central area of ​​a bottom surface of the reference projection plane, by adjusting a position of the division lines and an angle between the two division lines so that the image synthesis area does not overlap with the object; Equipped with the central region is a rectangular region surrounding an outer shape of the moving object, The border that defines the outline of the overlapping area is: a first frame line extending from one of four vertices of the rectangle in the central area toward the outside of the central area along a first side that forms the one vertex of four sides constituting the rectangle, and a second frame line extending from the one vertex toward the outside of the central area along a second side different from the first side that forms the one vertex of the four sides, The reference point is an intersection point between the first border line and the second border line of the overlapping region; Image processing device.

2. a generation unit that generates the projected image by projecting the plurality of captured images onto an adjusted projection plane, which is the image projection plane on which the adjustment process has been performed; The image processing device according to claim 1 .

3. The object is a three-dimensional object.

3. The image processing device according to claim 1 or 2.

4. The reference projection plane has a predetermined shape. The image processing device according to any one of claims 1 to 3.

5. The reference projection plane is the image projection plane that is virtually disposed in the virtual space with a central region of a bottom surface of the reference projection plane as a reference position. The image processing device according to any one of claims 1 to 4.

6. The adjustment process includes: a projection plane deformation process for deforming the overlapping area including the object on the reference projection plane into a shape that passes through the reference position side of the object; The image processing device according to claim 5 .

7. The adjustment process includes: a projection plane deformation process for deforming a horizontal cross-sectional shape of the overlapping region including the object on the reference projection plane into a shape passing through the reference position side of the object; 7. The image processing device according to claim 5 or 6.

8. The adjustment process includes: a projection plane deformation process for deforming a shape of the overlapping region including the object on the reference projection plane into a shape that passes at least between a detection point that constitutes the object and the reference position and a detection point that is closest to the reference position; The image processing device according to any one of claims 5 to 7.

9. The adjustment process includes: deforming a shape of the overlapping area including the object on the reference projection plane into a shape that passes through the reference position side of the object; a projection plane deformation process that does not deform a shape of an area other than the overlap area including the object on the reference projection plane; The image processing device according to any one of claims 5 to 8.

10. The adjustment process includes: deforming a shape of the overlapping area including the object on the reference projection plane into a shape that passes through the reference position side of the object; a projection surface deformation process for deforming a shape of a region other than the overlap region including the object on the reference projection surface in accordance with a shape of the overlap region after deformation; The image processing device according to any one of claims 5 to 8.

11. The adjustment process includes: a projection plane deformation process for deforming a cross-sectional shape along the vertical direction of the overlapping region including the object on the reference projection plane into a shape that passes through the reference position side of the object and is any one of a straight line parallel to the vertical direction, a straight line inclined with respect to the vertical direction, a quadratic curve, and a square root curve; The image processing device according to any one of claims 5 to 10.

12. The image synthesis adjustment process includes: A process of adjusting a synthesis coefficient of the adjacent captured images in the overlapping region. The image processing device according to any one of claims 1 to 11.

13. The blending coefficient is a transmission value of the adjacent captured images. The image processing device according to claim 12.

14. The image synthesis adjustment process includes: The image synthesis area does not overlap with the object. setting the synthesis coefficient to a first value in a region other than the image synthesis region within the overlap region, where one of the adjacent captured images is projected; setting the synthesis coefficient to a second value different from the first value in a region other than the image synthesis region within the overlap region, the region being a region onto which the other of the adjacent photographed images is projected; A process of setting the synthesis coefficient to an intermediate value between the first value and the second value in the image synthesis area within the overlap area. The image processing device according to claim 12 or 13.

15. 1. A computer-implemented image processing method, comprising the steps of: acquiring a plurality of captured images in which a portion of a captured area overlaps; a step of determining whether or not an object is included in an overlapping portion of adjacent captured images included in a projected image obtained by projecting a plurality of the captured images onto a reference projection plane, which is an image projection plane virtually disposed in a virtual space corresponding to a real space; a step of executing an image synthesis adjustment process for adjusting an overlapping area including the object on the reference projection plane corresponding to the overlapping portion including the object, the image synthesis adjustment process being a process for adjusting a position and a range of an image synthesis area which is set as a part of the overlapping area including the object on the reference projection plane within the overlapping area including the object and which is set by two division lines extending radially from a reference point in the overlapping area in a direction away from a reference position of a moving body which is a central area of ​​a bottom surface of the reference projection plane, by adjusting a position of the division lines and an angle between the two division lines so that the image synthesis area does not overlap with the object; Including, the central region is a rectangular region surrounding an outer shape of the moving object, The border that defines the outline of the overlapping area is: a first frame line extending from one of four vertices of the rectangle in the central area toward the outside of the central area along a first side that forms the one vertex of four sides constituting the rectangle, and a second frame line extending from the one vertex toward the outside of the central area along a second side different from the first side that forms the one vertex of the four sides, The reference point is an intersection point between the first border line and the second border line of the overlapping region; Image processing methods.

16. An image processing program for causing a computer to execute the program, acquiring a plurality of captured images in which a portion of a captured area overlaps; a step of determining whether or not an object is included in an overlapping portion of adjacent captured images included in a projected image obtained by projecting a plurality of the captured images onto a reference projection plane, which is an image projection plane virtually disposed in a virtual space corresponding to a real space; a step of executing an image synthesis adjustment process for adjusting an overlapping area including the object on the reference projection plane corresponding to the overlapping portion including the object, the image synthesis adjustment process being a process for adjusting a position and a range of an image synthesis area which is set as a part of the overlapping area including the object on the reference projection plane within the overlapping area including the object and which is set by two division lines extending radially from a reference point in the overlapping area in a direction away from a reference position of a moving body which is a central area of ​​a bottom surface of the reference projection plane, by adjusting a position of the division lines and an angle between the two division lines so that the image synthesis area does not overlap with the object; Including, the central region is a rectangular region surrounding an outer shape of the moving object, The border that defines the outline of the overlapping area is: a first frame line extending from one of four vertices of the rectangle in the central area toward the outside of the central area along a first side that forms the one vertex of four sides constituting the rectangle, and a second frame line extending from the one vertex toward the outside of the central area along a second side different from the first side that forms the one vertex of the four sides, The reference point is an intersection point between the first border line and the second border line of the overlapping region; Image processing program.

Images