Image processing device, image processing method, and image processing program

Abstract

In one aspect, an information processing device (10) is provided with: a determination unit (30) that determines first information relating to transformation of a projection surface on which a peripheral image of a mobile object is projected; and a transformation unit (32) that transforms the projection surface on the basis of the first information. The determination unit (30) includes an information holding unit (308) that stores past second information used for determining the first information, and determines the first information on the basis of the past second information when the operation of the mobile object is started.

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an image processing method, and an image processing program. [Background technology]

[0002] There is a technology that generates an overhead image of the area around a moving object such as a car using images from multiple cameras mounted on the moving object. There is also a technology that changes the shape of the projection surface of the overhead image depending on the three-dimensional objects around the moving object. There is also a technology that acquires position information around a moving object using Visual SLAM (Simultaneous Localization and Mapping: VSLAM), which performs SLAM using images captured by cameras. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2021 / 065241 [Patent Document 2] Japanese Patent Publication No. 2020-083140 [Patent Document 3] International Publication No. 2019 / 039507 [Patent Document 4] Japanese Patent Publication No. 2020-034528 [Patent Document 5] Japanese Patent Application Publication No. 2019-191741 [Patent Document 6] Japanese Patent Application Laid-Open No. 2002-354467 Summary of the Invention [Problem to be solved by the invention]

[0004] However, when the projection surface of the overhead image is successively transformed in accordance with three-dimensional objects around the vehicle, for example, when the engine is started some time after parking, the overhead image may become unnatural.

[0005] In one aspect, the present invention aims to realize an image processing device, an image processing method, and an image processing program that provide a more natural overhead image than conventional methods when the projection surface of the overhead image is successively deformed in accordance with three-dimensional objects around a moving body. [Means for solving the problem]

[0006] In one aspect, the image processing device disclosed in the present application comprises a determination unit that determines first information regarding the deformation of a projection surface onto which a peripheral image of a moving body is projected, and a deformation unit that deforms the projection surface based on the first information, wherein the determination unit includes an information storage unit that accumulates past second information used to determine the first information, and determines the first information based on the past second information when the moving body starts operating. [Effects of the Invention]

[0007] According to one aspect of the image processing device disclosed in the present application, when the projection surface of the overhead image is successively transformed in accordance with three-dimensional objects around the moving body, it is possible to provide a more natural overhead image than conventionally possible. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of the overall configuration of an information processing system according to an embodiment. [Figure 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of the information processing device according to the embodiment. [Figure 3] FIG. 3 is a diagram illustrating an example of a functional configuration of the information processing device according to the embodiment. [Figure 4] FIG. 4 is a schematic diagram illustrating an example of environment map information according to the embodiment. [Figure 5] FIG. 5 is a schematic diagram illustrating an example of a functional configuration of the determination unit. [Figure 6] FIG. 6 is a schematic diagram showing an example of the reference projection plane. [Figure 7] FIG. 7 is an explanatory diagram of the asymptotic curve Q generated by the determination unit. [Figure 8] FIG. 8 is a schematic diagram showing an example of a projection shape determined by the determination unit. [Figure 9] FIG. 9 is a diagram showing an example of the situation around the moving object when the moving object has completed backward parking. [Figure 10] FIG. 10 is a diagram showing an example of the situation around the mobile object after a predetermined time has elapsed since the mobile object was backed into a parking space. [Figure 11] FIG. 11 is a diagram for explaining the information accumulation process executed in the projection surface transformation optimization process according to the embodiment. [Figure 12] FIG. 12 is a flowchart showing the flow of information selection processing executed in the projection surface transformation optimization processing according to the embodiment. [Figure 13] FIG. 13 is a flowchart showing an example of the flow of overhead image generation processing including projected shape optimization processing according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, with reference to the accompanying drawings, embodiments of the image processing device, image processing method, and image processing program disclosed herein will be described in detail. Note that the following embodiments do not limit the disclosed technology. Furthermore, each embodiment can be appropriately combined within a range that does not cause contradiction in the processing content.

[0010] FIG. 1 is a diagram showing an example of the overall configuration of an information processing system 1 of this embodiment. The information processing system 1 includes an information processing device 10, an imaging unit 12, a detection unit 14, and a display unit 16. The information processing device 10, the imaging unit 12, the detection unit 14, and the display unit 16 are connected so as to be able to exchange data or signals. The information processing device 10 is an example of an image processing device. Furthermore, the information processing method executed by the information processing device 10 is an example of an image processing method, and the information processing program used by the information processing device 10 to execute the information processing method is an example of an image processing program.

[0011] In this embodiment, the information processing device 10, the image capturing unit 12, the detection unit 14, and the display unit 16 are mounted on a moving object 2 as an example.

[0012] The moving body 2 is an object that can move. The moving body 2 is, for example, a vehicle, a flyable object (a manned airplane, an unmanned airplane (e.g., a UAV (Unmanned Aerial Vehicle), a drone)), a robot, etc. The moving body 2 is, for example, a moving body that moves through human driving operation, or a moving body that can move automatically (autonomously) without human driving operation. In this embodiment, a case where the moving body 2 is a vehicle will be described as an example. Examples of vehicles include a two-wheeled vehicle, a three-wheeled vehicle, and a four-wheeled vehicle. In this embodiment, a case where the vehicle is an autonomously moving four-wheeled vehicle will be described as an example.

[0013] Note that the information processing device 10, the photographing unit 12, the detection unit 14, and the display unit 16 are not limited to being all mounted on the moving object 2. The information processing device 10 may be mounted on, for example, a stationary object. A stationary object is an object fixed to the ground. A stationary object is an object that cannot be moved or an object that is stationary relative to the ground. Examples of stationary objects include traffic lights, parked vehicles, and road signs. Furthermore, the information processing device 10 may be mounted on a cloud server that executes processing on the cloud.

[0014] The photographing unit 12 photographs the periphery of the moving object 2 and acquires photographed image data. In the following description, the photographed image data will be simply referred to as a photographed image. In this embodiment, the photographing unit 12 will be described assuming, for example, a digital camera capable of photographing video, such as a monocular fisheye camera with a viewing angle of approximately 195 degrees. Note that photographing refers to converting an image of a subject formed by an optical system such as a lens into an electrical signal. The photographing unit 12 outputs the photographed image to the information processing device 10.

[0015] In this embodiment, an example will be described in which four imaging units 12, namely, a front imaging unit 12A, a left imaging unit 12B, a right imaging unit 12C, and a rear imaging unit 12D, are mounted on a moving object 2. The multiple imaging units 12 (front imaging unit 12A, left imaging unit 12B, right imaging unit 12C, and rear imaging unit 12D) each capture an image of a subject in a different imaging area E (front imaging area E1, left imaging area E2, right imaging area E3, and rear imaging area E4) and acquire a captured image. That is, the imaging directions of the multiple imaging units 12 are different from each other. Furthermore, the imaging directions of the multiple imaging units 12 are adjusted in advance so that at least a portion of the imaging area E of adjacent imaging units 12 overlaps. Furthermore, for convenience of explanation, the imaging area E in FIG. 1 is shown as being the same size as in FIG. 1, but in reality, it may include an area further away from the moving object 2.

[0016] Furthermore, the four photographing units 12A, 12B, 12C, and 12D are merely examples, and there is no limit to the number of photographing units 12. For example, if the moving body 2 has a vertically long shape like a bus or truck, it is possible to use a total of six photographing units 12, by arranging one photographing unit 12 at the front, one at the rear, one at the front of the right side, one at the rear of the right side, one at the front of the left side, and one at the rear of the left side of the moving body 2. In other words, the number and arrangement positions of the photographing units 12 can be set arbitrarily depending on the size and shape of the moving body 2.

[0017] The detection unit 14 detects position information of each of a plurality of detection points around the moving object 2. In other words, the detection unit 14 detects position information of each of the detection points in the detection area F. The detection points refer to each of the points in real space that are individually observed by the detection unit 14. The detection points correspond to, for example, three-dimensional objects around the moving object 2. The detection unit 14 is an example of an external sensor.

[0018] The detection unit 14 may be, for example, a 3D (Three-Dimensional) scanner, a 2D (Two-Dimensional) scanner, a distance sensor (millimeter-wave radar, laser sensor), a sonar sensor that detects objects using sound waves, an ultrasonic sensor, or the like. The laser sensor may be, for example, a three-dimensional LiDAR (Laser Imaging Detection and Ranging) sensor. The detection unit 14 may also be a device that uses a technology that measures distance from images captured by a stereo camera or a monocular camera, such as SfM (Structure from Motion) technology. A plurality of image capture units 12 may also be used as the detection unit 14. One of the plurality of image capture units 12 may also be used as the detection unit 14.

[0019] The display unit 16 displays various types of information and is, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display.

[0020] In this embodiment, the information processing device 10 is communicably connected to an electronic control unit (ECU) 3 mounted on the moving object 2. The ECU 3 is a unit that performs electronic control of the moving object 2. In this embodiment, the information processing device 10 is capable of receiving CAN (Controller Area Network) data such as the speed and moving direction of the moving object 2 from the ECU 3.

[0021] Next, the hardware configuration of the information processing device 10 will be described.

[0022] FIG. 2 is a diagram illustrating an example of a hardware configuration of the information processing device 10. As shown in FIG.

[0023] The information processing device 10 is, for example, a computer, and includes a CPU (Central Processing Unit) 10A, a ROM (Read Only Memory) 10B, a RAM (Random Access Memory) 10C, and an I / F (Interface) 10D. The CPU 10A, ROM 10B, RAM 10C, and I / F 10D are interconnected by a bus 10E, and have a hardware configuration similar to that of a typical computer.

[0024] The CPU 10A is a calculation device that controls the information processing device 10. The CPU 10A corresponds to an example of a hardware processor. The ROM 10B stores programs and the like that realize various processes by the CPU 10A. The RAM 10C stores data necessary for various processes by the CPU 10A. The I / F 10D is an interface that is connected to the imaging unit 12, the detection unit 14, the display unit 16, the ECU 3, etc., and is used to send and receive data.

[0025] A program for executing information processing executed by the information processing device 10 of this embodiment is provided by being pre-installed in a ROM 10B or the like. The program executed by the information processing device 10 of this embodiment may be provided by being recorded on a recording medium in a format that can be installed on the information processing device 10 or in a format that can be executed. The recording medium is a computer-readable medium. Examples of the recording medium include a CD (Compact Disc)-ROM, a flexible disk (FD), a CD-R (Recordable), a DVD (Digital Versatile Disk), a USB (Universal Serial Bus) memory, and an SD (Secure Digital) card.

[0026] Next, a functional configuration of the information processing device 10 according to this embodiment will be described. The information processing device 10 uses VSLAM processing to simultaneously estimate peripheral position information of the moving object 2 and self-position information of the moving object 2 from images captured by the image capturing unit 12. The information processing device 10 stitches together multiple spatially adjacent captured images to generate and display a composite image (overhead image) overlooking the periphery of the moving object 2. Note that in this embodiment, at least one of the image capturing units 12 is used as a detection unit 14, and the detection unit 14 processes images acquired from the image capturing unit 12.

[0027] Fig. 3 is a diagram showing an example of the functional configuration of the information processing device 10. In Fig. 3, in order to clarify the data input / output relationship, an image capturing unit 12 and a display unit 16 are also shown in addition to the information processing device 10.

[0028] The information processing device 10 executes a projection surface transformation optimization process. Here, the projection surface transformation optimization process accumulates past information used for the projection surface transformation, and makes the accumulated past information available for the projection surface transformation when starting the operation of the moving object 2. This projection surface transformation optimization process will be described in detail later.

[0029] The information processing device 10 includes an acquisition unit 20, a selection unit 21, an image comparison unit 22, a VSLAM processing unit 24, a distance conversion unit 27, a projection shape determination unit 29 having an information storage unit 308, and an image generation unit 37.

[0030] Some or all of the above-mentioned multiple units may be realized by, for example, causing a processing device such as CPU 10A to execute a program, that is, by software. Also, some or all of the above-mentioned multiple units may be realized by hardware such as an IC (Integrated Circuit), or may be realized by a combination of software and hardware.

[0031] The acquisition unit 20 acquires captured images from the imaging unit 12. That is, the acquisition unit 20 acquires captured images from each of the front imaging unit 12A, the left imaging unit 12B, the right imaging unit 12C, and the rear imaging unit 12D.

[0032] The acquisition unit 20 outputs the acquired captured image to the projection transformation unit 36 ​​and the selection unit 21 every time the acquisition unit 20 acquires the captured image.

[0033] The selection unit 21 selects a detection area of ​​the detection point. In this embodiment, the selection unit 21 selects at least one of the multiple imaging units 12 (imaging units 12A to 12D), thereby selecting the detection area.

[0034] The image comparison unit 22 sequentially inputs, via the acquisition unit 20, the front image, left image, right image, and rear image captured by the front photographing unit 12A, left photographing unit 12B, right photographing unit 12C, and rear photographing unit 12D.

[0035] In the projection surface transformation optimization process, the image comparison unit 22 calculates the similarity between a first image associated with a first trigger and a second image associated with a second trigger among a plurality of images captured around the moving object 2. More specifically, the image comparison unit 22 calculates the similarity between the first image associated with the first trigger and the second image associated with the second trigger for each of the forward, left, right, and backward directions.

[0036] Here, the first trigger is a signal (information) indicating that the moving object 2 is in a parking-completed state, such as the moving object 2 being powered off (operation end processing), or the moving object 2 being in a parking-completed state in automatic parking mode. The second trigger is a signal (information) that occurs after the first trigger, and indicates that the moving object 2 has completed parking and is about to start moving again, such as the moving object 2 being powered on (operation start processing), or the moving object 2 starting state. The first trigger and second trigger can be acquired, for example, from CAN data.

[0037] Furthermore, the image comparison unit 22 outputs an instruction based on the calculated similarity to the information storage unit 308 included in the projection shape determination unit 29. More specifically, if the calculated similarity for each direction exceeds a threshold, the image comparison unit 22 determines that the first image and the second image are similar, and outputs a first instruction to the information storage unit 308 to output the neighboring point information accumulated by the information storage unit 308. If the calculated similarity for each direction is below the threshold, the image comparison unit 22 determines that the first image and the second image are not similar, and outputs a second instruction to the information storage unit 308 to discard the neighboring point information accumulated by the information storage unit 308.

[0038] The VSLAM processing unit 24 generates first information including position information of three-dimensional objects around the moving body 2 and position information of the moving body 2 based on an image of the surroundings of the moving body 2. That is, the VSLAM processing unit 24 receives the captured image from the selection unit 21, performs VSLAM processing using the captured image to generate environmental map information, and outputs the generated environmental map information to the distance conversion unit 27.

[0039] More specifically, the VSLAM processing unit 24 includes a matching unit 240 , a storage unit 241 , a self-position estimation unit 242 , a three-dimensional reconstruction unit 243 , and a correction unit 244 .

[0040] The matching unit 240 performs a feature extraction process and a matching process between multiple captured images (multiple captured images in different frames) captured at different times. In detail, the matching unit 240 performs a feature extraction process from these multiple captured images. The matching unit 240 performs a matching process for identifying corresponding points between the multiple captured images captured at different times, using the feature amounts between the multiple captured images. The matching unit 240 outputs the matching process result to the storage unit 241.

[0041] The self-position estimation unit 242 estimates the self-position relative to the captured image by projective transformation or the like using the multiple matching points acquired by the matching unit 240. Here, the self-position includes information on the position (three-dimensional coordinates) and tilt (rotation) of the image capturing unit 12. The self-position estimation unit 242 stores the self-position information as point cloud information in the environment map information 241A.

[0042] The three-dimensional reconstruction unit 243 performs perspective projection transformation processing using the movement amount (translation amount and rotation amount) of the self-position estimated by the self-position estimation unit 242, and determines the three-dimensional coordinates (coordinates relative to the self-position) of the matching point. The three-dimensional reconstruction unit 243 stores the peripheral position information, which is the determined three-dimensional coordinates, in the environmental map information 241A as point cloud information.

[0043] As a result, new peripheral position information and new self-position information are sequentially added to the environment map information 241A as the moving object 2 on which the image capturing unit 12 is mounted moves.

[0044] The storage unit 241 stores various types of data. The storage unit 241 is, for example, a semiconductor memory element such as a RAM or a flash memory, a hard disk, an optical disk, or the like. The storage unit 241 may be a storage device provided outside the information processing device 10. The storage unit 241 may also be a storage medium. Specifically, the storage medium may store or temporarily store programs and various types of information downloaded via a LAN (Local Area Network), the Internet, or the like.

[0045] The environmental map information 241A is information in which point cloud information, which is peripheral position information calculated by the three-dimensional restoration unit 243, and point cloud information, which is self-position information calculated by the self-position estimation unit 242, are registered in a three-dimensional coordinate space with a predetermined position in the real space as the origin (reference position). The predetermined position in the real space may be determined based on, for example, a predetermined condition.

[0046] For example, the predetermined position used in the environmental map information 241A is the self-position of the moving object 2 when the information processing device 10 executes the information processing of this embodiment. For example, assume a case where information processing is executed at a predetermined timing, such as a parking scene of the moving object 2. In this case, the information processing device 10 may set the self-position of the moving object 2 when it determines that the predetermined timing has been reached as the predetermined position. For example, the information processing device 10 may determine that the predetermined timing has been reached when it determines that the behavior of the moving object 2 has become behavior indicative of a parking scene. Examples of behavior indicative of a parking scene due to backing up include when the speed of the moving object 2 drops below a predetermined speed, when the gear of the moving object 2 is shifted into reverse gear, or when a signal indicating the start of parking is received by a user's operation instruction, etc. Note that the predetermined timing is not limited to a parking scene.

[0047] Fig. 4 is a schematic diagram of an example in which information on a specific height is extracted from the environment map information 241A. As shown in Fig. 4, the environment map information 241A is information in which point cloud information, which is the position information (peripheral position information) of each detection point P, and point cloud information, which is the self-position information of the self-position S of the moving object 2, are registered at corresponding coordinate positions in the three-dimensional coordinate space. Note that Fig. 4 shows the self-position S of self-positions S1 to S3 as an example. The larger the value of the number following S, the closer the self-position S is to the current timing.

[0048] The correction unit 244 corrects the peripheral position information and self-position information registered in the environmental map information 241A using, for example, the least squares method, so that the sum of the differences in distance in three-dimensional space between previously calculated three-dimensional coordinates and newly calculated three-dimensional coordinates for points that have been matched multiple times between multiple frames is minimized. Note that the correction unit 244 may also correct the amount of movement (translation amount and rotation amount) of the self-position used in the process of calculating the self-position information and peripheral position information.

[0049] The timing of the correction process by the correction unit 244 is not limited. For example, the correction unit 244 may execute the correction process at a predetermined timing. The predetermined timing may be determined based on a predetermined condition, for example. Note that in this embodiment, a case where the information processing device 10 is configured to include the correction unit 244 will be described as an example. However, the information processing device 10 may not be configured to include the correction unit 244.

[0050] Distance conversion unit 27 converts the relative positional relationship between the self-position and the surrounding three-dimensional objects, which can be known from the environmental map information, into the absolute value of the distance from the self-position to the surrounding three-dimensional objects, generates detection point distance information of the surrounding three-dimensional objects, and outputs it to determination unit 30. Here, the detection point distance information is information obtained by offsetting the self-position to coordinates (0,0,0) and converting the measured distance (coordinates) to each of the calculated multiple detection points P into, for example, meters. In other words, information on the self-position of moving object 2 is included as the coordinates (0,0,0) of the origin in the detection point distance information.

[0051] The distance conversion unit 27 uses, for example, vehicle status information, such as the speed data of the moving object 2, included in the CAN data sent from the ECU 3. For example, in the case of the environment map information 241A shown in FIG. 4, the relative positional relationship between the self-position S and multiple detection points P can be known, but the absolute value of the distance has not been calculated. Here, the distance between the self-position S3 and the self-position S2 can be calculated based on the inter-frame period for calculating the self-position and the speed data during that period based on the vehicle status information. Because the relative positional relationship in the environment map information 241A is similar to that in real space, knowing the distance between the self-position S3 and the self-position S2 also makes it possible to calculate the absolute values ​​of the distances from the self-position S to all other detection points P. That is, the distance conversion unit 27 uses the actual speed data of the moving object 2 included in the CAN data to convert the relative positional relationship between the self-position and a surrounding three-dimensional object into the absolute value of the distance from the self-position to the surrounding three-dimensional object.

[0052] The vehicle state information included in the CAN data can be associated with the environmental map information output from the VSLAM processing unit 24 using time information. In addition, when the detection unit 14 acquires distance information of the detection point P, the distance conversion unit 27 may be omitted.

[0053] The projection shape determination unit 29 determines the shape of the projection surface for generating an overhead image by projecting an image acquired by the image capturing unit 12 mounted on the moving object 2.

[0054] Here, the projection surface is a three-dimensional surface onto which an image of the periphery of the moving object 2 is projected as a bird's-eye view image. The image of the periphery of the moving object 2 is a captured image of the periphery of the moving object 2, which is captured by each of the image capturing units 12A to 12D. The projection shape of the projection surface is a three-dimensional (3D) shape that is virtually formed in a virtual space corresponding to the real space. In this embodiment, the determination of the projection shape of the projection surface executed by the projection shape determination unit 29 is referred to as a projection shape determination process.

[0055] Specifically, the projection shape determination unit 29 includes a determination unit 30 having an information storage unit 308, a deformation unit 32, and a virtual viewpoint line of sight determination unit .

[0056] [Configuration example of the determination unit 30] An example of the detailed configuration of the determination unit 30 shown in FIG. 3 will be described below.

[0057] Fig. 5 is a schematic diagram showing an example of the functional configuration of the determination unit 30. As shown in Fig. 5, the determination unit 30 includes an extraction unit 305, a nearest neighbor identification unit 307, an information storage unit 308, a reference projection plane shape selection unit 309, a scale determination unit 311, an asymptotic curve calculation unit 313, a shape determination unit 315, and a boundary region determination unit 317.

[0058] The extraction unit 305 extracts detection points P that exist within a specific range from among the multiple detection points P for which the measurement distances have been received from the distance conversion unit 27, and generates a specific height extraction map. The specific range is, for example, a range from the road surface on which the moving object 2 is placed to a height equivalent to the vehicle height of the moving object 2. Note that the range is not limited to this range.

[0059] The extraction unit 305 extracts detection points P within the range and generates a specific height extraction map, thereby making it possible to extract detection points P of, for example, objects that obstruct the progress of the moving body 2 or objects located adjacent to the moving body 2.

[0060] Then, the extraction unit 305 outputs the generated specific height extraction map to the nearest neighbor identification unit 307.

[0061] The nearest neighbor identification unit 307 divides the surroundings of the self-position S of the moving body 2 into specific ranges (for example, angular ranges) using the specific height extraction map, and for each range, identifies the detection point P closest to the moving body 2, or multiple detection points P in order of proximity to the moving body 2, and generates neighbor point information. In this embodiment, an example will be described in which the nearest neighbor identification unit 307 identifies multiple detection points P for each range in order of proximity to the moving body 2 and generates neighbor point information.

[0062] The nearest neighbor specifying unit 307 outputs the measured distance of the detection point P specified for each range as neighbor point information to the information holding unit 308.

[0063] The information storage unit 308 accumulates past neighboring point information (second information) used to determine the projection surface shape information (first information) in the projection surface deformation optimization process, and outputs the accumulated past neighboring point information to the subsequent reference projection surface shape selection unit 309, scale determination unit 311, asymptotic curve calculation unit 313, and boundary area determination unit 317 when the operation of the moving body 2 is started.

[0064] Specifically, in response to a first trigger included in the CAN data from the ECU, the information holding unit 308 holds nearby point information corresponding to the end of operation of the moving object 2. In response to a first instruction from the image comparison unit 22, the information holding unit 308 outputs the held nearby point information to the reference projection surface shape selection unit 309, the scale determination unit 311, the asymptotic curve calculation unit 313, and the boundary area determination unit 317. In response to a second instruction from the image comparison unit 22, the information holding unit 308 discards the held nearby point information.

[0065] In this embodiment, for the sake of specificity, the neighboring point information is assumed to be acquired as the positions of neighboring points, for example, at 90-degree intervals in four directions, namely, forward, left, right, and backward, of the moving object 2. The information storage unit 308 also receives a first instruction and a second instruction from the image comparison unit 22 for each direction. Therefore, the information storage unit 308 outputs or discards the neighboring point information stored for each direction.

[0066] In addition, in response to a second instruction from the image comparison unit 22, the information storage unit 308 outputs new neighboring point information based on the environment map information newly generated by the VSLAM processing to the subsequent reference projection surface shape selection unit 309, scale determination unit 311, asymptotic curve calculation unit 313, and boundary area determination unit 317.

[0067] The information storage unit 308 may store, as the second information, point cloud information, which is the position information (peripheral position information) of each of the detected points P, in addition to the past neighboring point information used to determine the projection surface shape information.

[0068] The reference projection surface shape selection unit 309 selects the shape of the reference projection surface.

[0069] FIG. 6 is a schematic diagram showing an example of a reference projection surface 40. The reference projection surface will be described with reference to FIG. 6. The reference projection surface 40 is, for example, a projection surface having a shape that serves as a reference when changing the shape of the projection surface. The shape of the reference projection surface 40 is, for example, bowl-shaped, cylindrical, etc. Note that FIG. 6 shows an example of a bowl-shaped reference projection surface 40.

[0070] The bowl-shaped container has a bottom surface 40A and a side wall surface 40B, one end of which is continuous with the bottom surface 40A and the other end of which is open. The width of the horizontal cross section of the side wall surface 40B increases from the bottom surface 40A toward the open end of the other end. The bottom surface 40A is, for example, circular. Here, a circular shape includes a perfect circle and other circular shapes such as an ellipse. The horizontal cross section is an orthogonal plane perpendicular to the vertical direction (arrow Z direction). The orthogonal plane is a two-dimensional plane along the arrow X direction, which is perpendicular to the arrow Z direction, and the arrow Y direction, which is perpendicular to the arrow Z direction and the arrow X direction. Hereinafter, the horizontal cross section and the orthogonal plane may be referred to as the XY plane. The bottom surface 40A may have a shape other than a circle, such as an egg shape.

[0071] The cylindrical shape is a shape consisting of a circular bottom surface 40A and a side wall surface 40B that is continuous with the bottom surface 40A. The side wall surface 40B that constitutes the cylindrical reference projection surface 40 has a cylindrical shape with an opening at one end that is continuous with the bottom surface 40A and an open other end. However, the side wall surface 40B that constitutes the cylindrical reference projection surface 40 has a shape in which the diameter in the XY plane is approximately constant from the bottom surface 40A side toward the opening at the other end. The bottom surface 40A may have a shape other than a circle, such as an egg shape.

[0072] In this embodiment, the case where the shape of the reference projection plane 40 is bowl-shaped as shown in Fig. 6 will be described as an example. The reference projection plane 40 is a three-dimensional model virtually formed in a virtual space, with the bottom surface 40A being a surface that substantially coincides with the road surface below the moving object 2 and the center of the bottom surface 40A being the self-position S of the moving object 2.

[0073] The reference projection surface shape selection unit 309 selects the shape of the reference projection surface 40 by reading one specific shape from multiple types of reference projection surfaces 40. For example, the reference projection surface shape selection unit 309 selects the shape of the reference projection surface 40 based on the positional relationship and distance between the self-position and surrounding three-dimensional objects. The shape of the reference projection surface 40 may also be selected based on an operational instruction from the user. The reference projection surface shape selection unit 309 outputs shape information of the determined reference projection surface 40 to the shape determination unit 315. In this embodiment, as described above, an embodiment in which the reference projection surface shape selection unit 309 selects a bowl-shaped reference projection surface 40 will be described as an example.

[0074] The scale determination unit 311 determines the scale of the reference projection plane 40 of the shape selected by the reference projection plane shape selection unit 309. The scale determination unit 311 makes a decision to reduce the scale, for example, when the distance from the self-position S to a nearby point is shorter than a predetermined distance. The scale determination unit 311 outputs scale information of the determined scale to the shape determination unit 315.

[0075] The asymptotic curve calculation unit 313 calculates an asymptotic curve of the peripheral position information relative to the self-position, based on the peripheral position information and self-position information of the moving object 2 included in the environmental map information. The asymptotic curve calculation unit 313 uses the distances from the self-position S to the closest detection point P for each range from the self-position S received from the information storage unit 308, and outputs asymptotic curve information of the calculated asymptotic curve Q to the shape determination unit 315 and the virtual viewpoint line of sight determination unit 34.

[0076] FIG. 7 is an explanatory diagram of an asymptotic curve Q generated by the determination unit 30. Here, the asymptotic curve is an asymptotic curve of a plurality of detection points P in the environmental map information. FIG. 7 shows an example in which the asymptotic curve Q is displayed in a projected image obtained by projecting a captured image onto a projection surface when the moving object 2 is viewed from above. For example, it is assumed that the determination unit 30 has identified three detection points P in descending order of proximity to the self-position S of the moving object 2. In this case, the determination unit 30 generates the asymptotic curves Q of these three detection points P.

[0077] The asymptotic curve calculation unit 313 may obtain a representative point located at the center of gravity of a plurality of detection points P for each specific range (for example, an angular range) on the reference projection plane 40, and calculate an asymptotic curve Q for the representative point for each of the plurality of ranges. Then, the asymptotic curve calculation unit 313 outputs asymptotic curve information of the calculated asymptotic curve Q to the shape determination unit 315. The asymptotic curve calculation unit 313 may output the asymptotic curve information of the calculated asymptotic curve Q to the virtual viewpoint line of sight determination unit 34.

[0078] The shape determination unit 315 enlarges or reduces the reference projection plane 40, which has a shape indicated by the shape information received from the reference projection plane shape selection unit 309, to the scale of the scale information received from the scale determination unit 311. Then, the shape determination unit 315 determines, as the projection shape, a shape obtained by deforming the reference projection plane 40 after enlarging or reducing it so that the shape conforms to the asymptotic curve information of the asymptotic curve Q received from the asymptotic curve calculation unit 313.

[0079] Here, the determination of the projection shape will be described in detail. FIG. 8 is a schematic diagram showing an example of the projection shape 41 determined by the determination unit 30. As shown in FIG. 8, the shape determination unit 315 determines, as the projection shape 41, a shape obtained by deforming the reference projection plane 40 into a shape that passes through a detection point P that is closest to the self-position S of the moving object 2, which is the center of the bottom surface 40A of the reference projection plane 40. A shape that passes through the detection point P means that the deformed side wall surface 40B is a shape that passes through the detection point P. The self-position S is the self-position S calculated by the self-position estimation unit 242.

[0080] That is, the shape determination unit 315 identifies the detection point P that is closest to the self-position S among the multiple detection points P registered in the environmental map information. In detail, the XY coordinates of the center position (self-position S) of the moving object 2 are set to (X, Y) = (0, 0). Then, the shape determination unit 315 determines the X 2 +Y 2 The detected point P where the value of is the smallest is identified as the detected point P closest to the self-position S. Then, the shape determination unit 315 determines, as the projected shape 41, a shape obtained by deforming the side wall surface 40B of the reference projection plane 40 so that it passes through the detected point P.

[0081] More specifically, the shape determination unit 315 determines the deformed shape of the bottom surface 40A and a portion of the side wall surface 40B as the projected shape 41 so that, when the reference projection plane 40 is deformed, a portion of the side wall surface 40B becomes a wall surface passing through the detection point P closest to the moving object 2. The deformed projected shape 41 is, for example, a shape that is raised from a rising line 44 on the bottom surface 40A in a direction toward the center of the bottom surface 40A when viewed from the XY plane (planar view). "Raising" means, for example, bending or folding a portion of the side wall surface 40B and the bottom surface 40A in a direction toward the center of the bottom surface 40A so that the angle formed between the side wall surface 40B of the reference projection plane 40 and the bottom surface 40A becomes smaller. Note that in the raised shape, the rising line 44 may be located between the bottom surface 40A and the side wall surface 40B, and the bottom surface 40A may remain undeformed.

[0082] The shape determination unit 315 determines to deform the specific region on the reference projection plane 40 so as to project it to a position that passes through the detection point P when viewed from the viewpoint (planar view) of the XY plane. The shape and range of the specific region may be determined based on a predetermined criterion. Then, the shape determination unit 315 determines to deform the reference projection plane 40 so that the distance from the self-position S becomes continuously greater from the projected specific region toward a region other than the specific region on the side wall surface 40B. The shape determination unit 315 is an example of a projection shape determination unit.

[0083] For example, it is preferable to determine the projected shape 41 so that the outer periphery of the cross section along the XY plane has a curved shape, as shown in Fig. 8. Note that the outer periphery of the cross section of the projected shape 41 is, for example, a circle, but may have a shape other than a circle.

[0084] The shape determination unit 315 may determine, as the projected shape 41, a shape obtained by deforming the reference projection plane 40 so that the shape follows an asymptotic curve. The shape determination unit 315 generates an asymptotic curve of a predetermined number of detection points P in a direction away from the detection point P closest to the self-position S of the moving object 2. The number of detection points P may be more than one. For example, the number of detection points P is preferably three or more. In this case, the shape determination unit 315 preferably generates an asymptotic curve of a plurality of detection points P located at positions that are at least a predetermined angle away from the self-position S. For example, the shape determination unit 315 may determine, as the projected shape 41, a shape obtained by deforming the reference projection plane 40 so that the shape follows the generated asymptotic curve Q for the asymptotic curve Q shown in FIG. 7.

[0085] The shape determination unit 315 may divide the surroundings of the self-position S of the moving body 2 into specific ranges, and for each range, identify the detection point P closest to the moving body 2, or multiple detection points P in order of proximity to the moving body 2. Then, the shape determination unit 315 may determine, as the projection shape 41, a shape obtained by deforming the reference projection plane 40 so as to become a shape that passes through the detection points P identified for each range, or a shape that follows the asymptotic curve Q of the identified multiple detection points P.

[0086] Then, the shape determination unit 315 outputs the projection shape information of the determined projection shape 41 to the deformation unit 32.

[0087] 3, the deformation unit 32 deforms the projection plane based on the projection shape information received from the determination unit 30. This deformation of the reference projection plane is performed, for example, based on the detection point P closest to the moving object 2. The deformation unit 32 outputs the deformed projection plane information to the projection conversion unit 36.

[0088] Furthermore, for example, the deformation unit 32 deforms the reference projection plane into a shape that follows the asymptotic curve of a predetermined number of detection points P in order of proximity to the moving object 2, based on the projection shape information.

[0089] The virtual viewpoint line of sight determining unit 34 determines virtual viewpoint line of sight information based on the self-position and the asymptotic curve information, and outputs the information to the projection transformation unit 36 ​​.

[0090] Determination of virtual viewpoint line-of-sight information will be described with reference to FIGS. 7 and 8. The virtual viewpoint line-of-sight determination unit 34 determines, for example, a direction passing through a detection point P closest to the self-position S of the moving object 2 and perpendicular to the deformed projection plane as the line-of-sight direction. Furthermore, the virtual viewpoint line-of-sight determination unit 34 fixes, for example, the direction of the line-of-sight direction L, and determines the coordinates of the virtual viewpoint O as an arbitrary Z coordinate and arbitrary XY coordinates in a direction away from the asymptotic curve Q toward the self-position S. In this case, the XY coordinates may be coordinates of a position farther away from the asymptotic curve Q than the self-position S. The virtual viewpoint line-of-sight determination unit 34 then outputs virtual viewpoint line-of-sight information indicating the virtual viewpoint O and the line-of-sight direction L to the projection transformation unit 36. Note that, as shown in FIG. 8, the line-of-sight direction L may be a direction from the virtual viewpoint O toward the position of the vertex W of the asymptotic curve Q.

[0091] The image generation unit 37 uses the projection plane to generate an overhead image of the periphery of the moving object 2. Specifically, the image generation unit 37 includes a projection conversion unit and an image synthesis unit .

[0092] The projection conversion unit 36 ​​generates a projection image by projecting the captured image acquired from the image capturing unit 12 onto the deformed projection surface based on the deformed projection surface information and the virtual viewpoint line of sight information. The projection conversion unit 36 ​​converts the generated projection image into a virtual viewpoint image and outputs it to the image synthesis unit 38. Here, the virtual viewpoint image is an image obtained by viewing the projection image in an arbitrary direction from a virtual viewpoint.

[0093] With reference to FIG. 8, the projection image generation process by the projection transformation unit 36 ​​will be described in detail. The projection transformation unit 36 ​​projects the captured image onto the deformed projection surface 42. Then, the projection transformation unit 36 ​​generates a virtual viewpoint image (not shown), which is an image obtained by viewing the captured image projected onto the deformed projection surface 42 from an arbitrary virtual viewpoint O in a line of sight direction L. The position of the virtual viewpoint O may be set to, for example, the self-position S of the moving object 2 (used as the reference for the projection surface transformation process). In this case, the XY coordinate values ​​of the virtual viewpoint O may be set to the XY coordinate values ​​of the self-position of the moving object 2. Furthermore, the Z coordinate value (vertical position) of the virtual viewpoint O may be set to the Z coordinate value of the detection point P closest to the self-position of the moving object 2. The line of sight direction L may be determined based on, for example, a predetermined criterion.

[0094] The line of sight direction L may be, for example, a direction from the virtual viewpoint O toward the detection point P that is closest to the self-position S of the moving object 2. The line of sight direction L may also be a direction that passes through the detection point P and is perpendicular to the deformed projection plane 42. Virtual viewpoint line of sight information indicating the virtual viewpoint O and the line of sight direction L is created by the virtual viewpoint line of sight determination unit 34.

[0095] The image synthesis unit 38 generates a synthesized image by extracting a part or all of the virtual viewpoint images. For example, the image synthesis unit 38 performs a process of stitching together a plurality of virtual viewpoint images (here, four virtual viewpoint images corresponding to the imaging units 12A to 12D) in the boundary region between the imaging units.

[0096] The image synthesis unit 38 outputs the generated synthetic image to the display unit 16. The synthetic image may be a bird's-eye view image with a virtual viewpoint O above the moving object 2, or may be an image in which the moving object 2 is displayed semi-transparently with a virtual viewpoint O inside the moving object 2.

[0097] (Projection surface deformation optimization processing) Next, the projection surface transformation optimization process executed by the information processing device 10 according to this embodiment will be described in detail.

[0098] Fig. 10 is a diagram showing an example of the surrounding situation when the moving object 2 starts moving (when starting off). In the example shown in Fig. 10, Car 1 is located in parking area C1 on the right side as seen from the driver of the moving object 2, a wall WA is located behind the moving object 2, and parking area C2 is empty. Since Car 1 and wall WA are located near the moving object 2, it is necessary to perform projection surface deformation processing based on surrounding position information in order to obtain a natural bird's-eye view image.

[0099] However, in this embodiment, peripheral position information is calculated using VSLAM processing using an image (monocular image) acquired from one of the image capturing units 12 mounted on each side of the moving object 2. Therefore, immediately after the moving object 2 starts operating, peripheral position information cannot be calculated until the moving object 2 starts moving. This is because VSLAM processing using a monocular image uses the principle of motion stereo, which performs triangulation between images acquired at slightly different positions that are obtained sequentially in the time direction as the moving object 2 moves.

[0100] Here, in the information processing device 10 according to this embodiment, the nearby point information at the time of parking completion is stored in, for example, the information storage unit 308 included in the determination unit 30. Then, during the period from when the moving object 2 starts operating (when starting to move) until new peripheral position information is obtained due to the movement of the moving object 2, the information processing device 10 can perform the projection plane deformation process by using the nearby point information at the time of parking completion stored in the information storage unit 308.

[0101] Furthermore, as time passes after parking is completed, the surrounding conditions at the time of operation initiation (starting) may change from the surrounding conditions at the time parking is completed. Figure 9 is a diagram showing an example of the surrounding conditions of the mobile object 2 when reverse parking of the mobile object 2 is completed, a predetermined time before the lapse of Figure 10. In Figure 9, Car 2 is present in parking area C2.

[0102] In the information processing device 10 according to this embodiment, the image comparison unit 22 calculates similarities by comparing the images of the front, left, right, and rear at the time of parking completion with the images of the front, left, right, and rear at the time of operation startup. As a result, if the surrounding situation in a direction where the similarity falls below the threshold is determined to have changed since parking completion, the nearby point information at the time of parking completion is discarded, and the asymptotic curve is recalculated using the nearby point information in the remaining directions, and the projection surface shape is re-determined.

[0103] As described above, in this embodiment, the shape of the projection surface can be determined depending on whether or not there is a change in the surrounding situation when starting the operation of the moving object 2. This makes it possible to provide a more natural bird's-eye view image than before when starting the operation of the moving object 2.

[0104] 11 is a flowchart showing the flow of information accumulation processing executed in the projection plane transformation optimization processing according to the embodiment. Note that the information accumulation processing is executed, for example, when parking of the moving object 2 is completed or when the power is turned off (operation termination processing).

[0105] First, a photographed image is acquired by the photographing unit 12 (step S1).

[0106] The VSLAM processing unit 24 and the like calculate detection point distance information by VSLAM processing using the captured image (step S2).

[0107] The projection shape determination unit 29 deforms the projection surface using the acquired detection point distance information (step S3).

[0108] Image generation unit 37 generates an overhead image using the transformed projection surface (step S4). The generated overhead image is displayed on display unit 16.

[0109] The information processing device 10 determines whether to end the overhead image generating process based on whether the CAN data from the ECU includes the first trigger (step S5).

[0110] If the information processing device 10 determines not to end the overhead image generating process (No in step S5), the processes of steps S1 to S5 are repeatedly executed.

[0111] On the other hand, when the information processing device 10 determines to terminate the overhead image generation process (Yes in step S5), the image comparison unit 22 accumulates the captured image (first image) associated with the first trigger for each direction, and the projection shape determination unit 29 accumulates the nearby point information associated with the first trigger for each direction (step S6).

[0112] 12 is a flowchart showing the flow of information selection processing executed in the projection surface transformation optimization processing according to the embodiment. The information selection processing is executed, for example, at the timing when the operation of the moving object 2 is started. Note that the following steps S10 to S14 are executed for each of the forward, left, right, and backward directions of the moving object 2.

[0113] First, the photographing unit 12 acquires photographed images (second images) for each direction in response to a second trigger included in the CAN data from the ECU (step S10).

[0114] The image comparison unit 22 compares the captured image associated with the second trigger with the stored captured image associated with the first trigger (step S11). That is, the image comparison unit 22 calculates the similarity between the captured image associated with the second trigger and the stored captured image associated with the first trigger.

[0115] The image comparison unit 22 determines whether the similarity is equal to or less than a predetermined threshold value (step S12).

[0116] When image comparison unit 22 determines that the similarity is equal to or less than the predetermined threshold (Yes in step S12), it outputs a second instruction to information storage unit 308 to discard the accumulated neighboring point information. Information storage unit 308 discards the accumulated data, which is the neighboring point information accumulated based on the second instruction received from image comparison unit 22 (step S13). Thereafter, projection plane deformation and overhead image generation are executed using newly acquired neighboring point information.

[0117] On the other hand, if the image comparison unit 22 determines that the similarity is not equal to or less than the predetermined threshold (No in step S12), it outputs a first instruction to the information holding unit 308 to output accumulated data, which is the accumulated neighboring point information. The information holding unit 308 outputs the accumulated neighboring point information based on the first instruction received from the image comparison unit 22 (step S14).

[0118] The projection shape determination unit 29 deforms the projection surface using the neighboring point information for each direction acquired from the information storage unit 308 (step S15). Note that since the number of neighboring points in the direction for which the accumulated data has been discarded is zero, the projection surface shape is determined by determining the scale and calculating the asymptotic curve based on the neighboring point information for the remaining directions.

[0119] The image generating unit 37 generates an overhead image using the transformed projection plane (step S16). The drawn overhead image is displayed on the display unit 16.

[0120] The information processing device 10 determines whether the moving object 2 has moved or not based on the CAN data from the ECU (step S17).

[0121] If it is determined that the moving object 2 has moved (Yes in step S17), the acquisition unit 20 acquires images captured in each direction (step S18).

[0122] The VSLAM processing unit 24 and the like calculate new detection point distance information by VSLAM processing using the captured image (step S19).

[0123] Upon receiving the new detection point distance information, the information holding unit 308 discards the stored data, which is the stored neighboring point information (step S20).

[0124] The projection shape determination unit 29 deforms the projection surface using the new neighboring point information (step S21).

[0125] The image generating unit 37 generates an overhead image using the transformed projection surface (step S22). The generated overhead image is displayed on the display unit 16.

[0126] On the other hand, if it is determined that the moving object 2 is not moving (No in step S17), the processes of steps S10 to S17 are repeatedly executed. At this time, if it is determined that the similarity is equal to or less than a predetermined threshold due to the movement of an adjacent vehicle or the like, the accumulated neighboring point information may be discarded in accordance with an instruction from the image comparison unit 22.

[0127] FIG. 13 is a flowchart showing an example of the flow of overhead image generation processing including projected shape optimization processing according to the embodiment.

[0128] The acquisition unit 20 acquires the photographed image for each direction (step S30). The selection unit 21 selects the photographed image as the detection area (step S32).

[0129] The matching unit 240 extracts features and performs matching processing (step S34) using the multiple captured images selected in step S32 and captured by the imaging unit 12 at different capture times. The matching unit 240 also registers information on corresponding points between the multiple captured images at different capture times, which have been identified by the matching processing, in the storage unit 241.

[0130] The self-position estimation unit 242 reads the matching points and the environment map information 241A (peripheral position information and self-position information) from the storage unit 241 (step S36). The self-position estimation unit 242 estimates the self-position relative to the captured image by projective transformation or the like using the multiple matching points acquired from the matching unit 240 (step S38), and registers the calculated self-position information in the environment map information 241A (step S40).

[0131] The three-dimensional restoration unit 243 reads the environment map information 241A (peripheral position information and self-position information) (step S42). The three-dimensional restoration unit 243 performs perspective projection transformation processing using the movement amount (translation amount and rotation amount) of the self-position estimated by the self-position estimation unit 242, determines the three-dimensional coordinates of the matching point (coordinates relative to the self-position), and registers them as peripheral position information in the environment map information 241A (step S44).

[0132] The correction unit 244 reads the environment map information 241A (peripheral position information and self-position information). The correction unit 244 corrects (step S46) the peripheral position information and self-position information already registered in the environment map information 241A using, for example, the least squares method, so that the sum of the differences in distance in three-dimensional space between previously calculated three-dimensional coordinates and newly calculated three-dimensional coordinates for points that have been matched multiple times across multiple frames is minimized, and updates the environment map information 241A.

[0133] The distance conversion unit 27 acquires vehicle state information, including speed data (host vehicle speed) of the moving object 2, included in the CAN data received from the ECU 3 of the moving object 2 (step S48). The distance conversion unit 27 converts the coordinate distance between the point clouds included in the environment map information 241A into an absolute distance, for example, in meters, using the speed data of the moving object 2. The distance conversion unit 27 also offsets the origin of the environment map information to the host position S of the moving object 2, and generates detection point distance information indicating the distance from the moving object 2 to each of the plurality of detection points P (step S50). The distance conversion unit 27 outputs the detection point distance information to the extraction unit 305 and the virtual viewpoint line of sight determination unit 34.

[0134] The extraction unit 305 extracts the detection points P that exist within a specific range from the detection point distance information (step S52).

[0135] The nearest neighbor identification unit 307 divides the surroundings of the self-position S of the moving body 2 into specific ranges, and for each range, identifies the detection point P closest to the moving body 2, or multiple detection points P in order of proximity to the moving body 2, and extracts the distance to the nearest object (step S54). The nearest neighbor identification unit 307 outputs the measured distance d of the detection point P identified for each range (the measured distance between the moving body 2 and the nearest object) to the information storage unit 308 as nearby point information.

[0136] In response to the first trigger, the image comparison unit 22 saves the captured images in the forward, left, right, and rearward directions of the moving object 2 and the neighbor point information output by the nearest neighbor identification unit 307 (step S55).

[0137] In addition, the image comparison unit 22 calculates the similarity between the captured image associated with the second trigger and the accumulated captured image associated with the first trigger for each of the forward, left, right, and backward directions in response to the second trigger (step S56).

[0138] Furthermore, if the image comparison unit 22 determines that the similarity exceeds the predetermined threshold, it outputs a first instruction to the information holding unit 308 to output the accumulated neighboring point information. On the other hand, if the image comparison unit 22 determines that the similarity is equal to or less than the predetermined threshold, it discards the accumulated neighboring point information and outputs a second instruction to the information holding unit 308 to output the neighboring point information generated in step S54.

[0139] The information storage unit 308 selects neighboring point information based on the first instruction or the second instruction received from the image comparison unit 22 (step S57). The information storage unit 308 also outputs the neighboring point information to the reference projection surface shape selection unit 309, the scale determination unit 311, the asymptotic curve calculation unit 313, and the boundary area determination unit 317.

[0140] The reference projection plane shape selection unit 309 selects the shape of the reference projection plane 40 based on the neighboring point information input from the information storage unit 308 (step S60), and outputs shape information of the selected reference projection plane 40 to the shape determination unit 315.

[0141] The scale determination unit 311 determines the scale of the reference projection plane 40 of the shape selected by the reference projection plane shape selection unit 309 (step S62), and outputs scale information of the determined scale to the shape determination unit 315.

[0142] The asymptotic curve calculation unit 313 calculates an asymptotic curve based on the neighboring point information input from the information storage unit 308 (step S64), and outputs the asymptotic curve information to the shape determination unit 315 and the virtual viewpoint line of sight determination unit .

[0143] The shape determination unit 315 determines a projection shape, which is how to deform the shape of the reference projection plane, based on the scale information and the asymptotic curve information (step S66). The shape determination unit 315 outputs projection shape information of the determined projection shape 41 to the deformation unit 32.

[0144] The transformation unit 32 transforms the shape of the reference projection plane based on the projection shape information (step S68), and outputs the transformed transformed projection plane information to the projection conversion unit .

[0145] The virtual viewpoint line-of-sight determination unit 34 determines virtual viewpoint line-of-sight information based on the self-position and the asymptotic curve information (step S70). The virtual viewpoint line-of-sight determination unit 34 outputs the virtual viewpoint line-of-sight information indicating the virtual viewpoint O and the line-of-sight direction L to the projection transformation unit 36.

[0146] The projection conversion unit 36 ​​generates a projection image by projecting the captured image acquired from the image capturing unit 12 onto the deformed projection surface based on the deformed projection surface information and the virtual viewpoint line of sight information. The projection conversion unit 36 ​​converts (generates) the generated projection image into a virtual viewpoint image (step S72) and outputs it to the image synthesis unit 38.

[0147] The boundary area determination unit 317 determines a boundary area based on the distance to the nearest object identified for each range. That is, the boundary area determination unit 317 determines a boundary area as an overlap area of ​​spatially adjacent peripheral images based on the position of the object nearest to the moving object 2 (step S74). The boundary area determination unit 317 outputs the determined boundary area to the image synthesis unit 38.

[0148] The image synthesis unit 38 generates a synthesized image by joining spatially adjacent virtual viewpoint images together using a boundary region (step S76). In the boundary region, the spatially adjacent virtual viewpoint images are blended at a predetermined ratio.

[0149] The display unit 16 displays the composite image as an overhead image (step S78).

[0150] The information processing device 10 determines whether to end the information processing (step S80). For example, the information processing device 10 makes the determination in step S80 by determining whether or not a signal indicating that parking of the moving object 2 has been completed has been received from the ECU 3. Alternatively, for example, the information processing device 10 may make the determination in step S80 by determining whether or not an instruction to end the information processing has been received through an operation instruction by a user or the like.

[0151] If a negative determination is made in step S80 (step S80: No), the processes from step S30 to step S80 are repeatedly executed. On the other hand, if a positive determination is made in step S80 (step S80: Yes), the overhead image generation process including the projected shape optimization process according to the embodiment is terminated.

[0152] When returning from step S80 to step S30 after executing the correction process of step S46, the correction process of the subsequent step S46 may be omitted. Also, when returning from step S80 to step S30 without executing the correction process of step S46, the correction process of the subsequent step S46 may be executed.

[0153] The information processing device 10 according to the embodiment described above includes a determination unit 30, a deformation unit 32, and an information storage unit 308 included in the determination unit 30. The determination unit 30 determines projection shape information related to deformation of a projection plane onto which a peripheral image of the moving object 2 is projected. The deformation unit 32 deforms the projection plane based on the projection shape information. The information storage unit 308 accumulates neighboring point information as past second information used in the projection shape information, and outputs the accumulated past neighboring point information to a reference projection plane shape selection unit 309, a scale determination unit 311, an asymptotic curve calculation unit 313, and a boundary region determination unit 317 included in the determination unit 30 when starting operation of the moving object 2.

[0154] Therefore, even when the moving body 2 starts to operate and sufficient nearby point information has not yet been obtained through VSLAM processing, the information processing device 10 can appropriately deform the projection surface shape by using accumulated past nearby point information (for example, nearby point information corresponding to the end of the previous moving body operation).

[0155] The information processing device 10 according to the embodiment also includes an image comparison unit 22 as a determination unit. The image comparison unit 22 calculates the similarity between a first image associated with the moving object 2 at the time of its end of motion and a second image associated with the moving object 2 at the time of its start of motion, among the multiple peripheral images. If the similarity exceeds a threshold, the image comparison unit 22 outputs a first instruction to the information storage unit 308 to cause the determination unit 30 to output the accumulated past neighboring point information. If the similarity is equal to or less than the threshold, the image comparison unit 22 outputs a second instruction to the information storage unit 308 to cause the determination unit 30 to discard the accumulated past neighboring point information. In response to the first instruction, the information storage unit 308 outputs the accumulated past neighboring point information to the determination unit 30, and discards the accumulated past neighboring point information in response to the second instruction.

[0156] Therefore, the information processing device 10 can appropriately determine whether to use the past neighboring point information accumulated in the deformation of the projection surface shape, depending on the change in the surrounding situation between the time when the operation of the moving object 2 ends and the time when the operation of the moving object 2 starts. As a result, when the operation of the moving object 2 starts, it is possible to provide the user with a more natural bird's-eye view image than before.

[0157] (Variation 1) In the above embodiment, the similarity is calculated using captured images acquired in each of the directions forward, left, right, and rearward of the moving object 2, and the projection surface shape optimization process is performed for each direction. However, it is also possible to calculate the similarity using captured images acquired in at least one direction around the moving object 2, and perform the projection surface shape optimization process for at least one direction. In this case, the image comparison unit 22 calculates the similarity for at least one direction out of multiple directions around the moving object 2, and outputs a first instruction or a second instruction for at least one direction based on the calculated at least one similarity.

[0158] (Variation 2) In the above embodiment, when the first instruction is received, the projection surface shape can also be deformed in stages. In this case, the determination unit 30 determines projection shape information that gradually changes the projection surface shape using accumulated past neighboring point information and newly generated neighboring point information. The deformation unit 32 gradually deforms the projection surface using the projection shape information that gradually changes the projection surface shape. This configuration allows the projection surface to be gradually deformed more naturally.

[0159] (Variation 3) In the above embodiment, when the moving object 2 starts a new operation, a process can be executed to obtain useful peripheral position information as early as possible.

[0160] For example, the moving object 2 is moved slightly at a low speed around the stopping position of the moving object 2 to acquire multiple images at different positions. In this case, for example, the information processing device 10 is configured to further include an information generating unit that generates first control information for moving the moving object 2 to acquire multiple new images of the area around the moving object 2 when the similarity is equal to or less than a threshold value.

[0161] Furthermore, a mechanism provided in the photographing unit 12 may move the photographing unit 12 itself to acquire multiple images at different positions. In this case, for example, the information processing device 10 is configured to further include an information generating unit that generates second control information for moving the position of the photographing unit that photographs the periphery of the moving object and acquiring multiple new images of the periphery of the moving object when the similarity is equal to or less than a threshold value.

[0162] (Variation 4) In the above embodiment, the system can be used as a driver assistance system in any driving mode, including automatic parking mode, semi-automatic parking mode, manual driving mode, and automatic driving mode.

[0163] Although the embodiments and modifications have been described above, the image processing device, image processing method, and image processing program disclosed herein are not limited to the above-described embodiments, and the components can be modified and embodied in each implementation stage without departing from the spirit of the invention. Furthermore, various inventions can be created by appropriately combining multiple components disclosed in the above-described embodiments and modifications. For example, some components may be deleted from all of the components shown in the embodiments.

[0164] The information processing device 10 of the above embodiment and each modified example can be applied to various devices. For example, the information processing device 10 of the above embodiment and each modified example can be applied to a surveillance camera system that processes images obtained from a surveillance camera, or an in-vehicle system that processes images of the surrounding environment outside the vehicle. [Explanation of symbols]

[0165] 10. Information processing equipment 12, 12A~12D Photography Department 14 Detector 20 Acquisition Department 21 Selection section 22 Image Comparison Section 24 VSLAM processing unit 27 Distance conversion section 29 Projection shape determining section 30 Decision Section 32 Deformed part 34 Virtual viewpoint line of sight determination unit 36 Projection transformation unit 37 Image generation unit 38 Image synthesis unit 240 Matching Department 241 Storage section 241A Environmental Map Information 242 Self-position estimation part 243 3D Reconstruction Unit 244 Correction Unit 305 Extraction part 307 Nearest neighbor identification part 308 Information Holding Department 309 Reference projection surface shape selection section 311 Scale determination unit 313 Asymptotic curve calculation part 315 Shape determination unit

Claims

1. a determination unit that determines first information regarding deformation of a projection surface onto which a peripheral image of the moving object is projected; a deformation unit that deforms the projection surface based on the first information; a determination unit that outputs a first instruction to the determination unit when it is determined that a first image associated with the moving object at the time of the end of the movement of the moving object and a second image associated with the moving object at the time of the start of the movement of the moving object are similar to each other; an information generating unit that generates first control information for moving the moving object and acquiring a plurality of new images of the periphery of the moving object when the determining unit determines that the first image and the second image are not similar; Equipped with the determination unit includes an information storage unit that stores past second information used in determining the first information and corresponding to the time when the operation of the moving body ends, and determines the first information based on the past second information stored in the information storage unit in response to the first instruction when the operation of the moving body is started. Image processing device.

2. a determination unit that determines first information regarding deformation of a projection surface onto which a peripheral image of the moving object is projected; a deformation unit that deforms the projection surface based on the first information; a determination unit that outputs a first instruction to the determination unit when it is determined that a first image associated with the moving object at the time of the end of the movement of the moving object and a second image associated with the moving object at the time of the start of the movement of the moving object are similar to each other; an information generating unit that generates second control information for moving a position of an image capturing unit that captures images of the periphery of the moving object to acquire new images of the periphery of the moving object when the determining unit determines that the first image and the second image are not similar; Equipped with the determination unit includes an information storage unit that stores past second information used in determining the first information and corresponding to the time when the operation of the moving body ends, and determines the first information based on the past second information stored in the information storage unit in response to the first instruction when the operation of the moving body is started. Image processing device.

3. the determination unit determines the first information that changes gradually using the past second information and the newly generated second information after starting the operation of the moving object; the deformation unit gradually deforms the projection surface using the first information that gradually changes.

3. The image processing device according to claim 1.

4. 1. A computer-implemented image processing method comprising: determining first information regarding deformation of a projection surface onto which a peripheral image of the moving object is projected; deforming the projection surface based on the first information; outputting a first instruction when it is determined that a first image associated with the moving object at the time of completion of the movement of the moving object and a second image associated with the moving object at the time of starting the movement of the moving object are similar to each other; generating first control information for moving the moving object and acquiring a plurality of new images of the periphery of the moving object when it is determined that the first image and the second image are not similar; Including, the determining step includes a step of accumulating past second information corresponding to the time when the operation of the moving object used in determining the first information has ended, and the first information is determined based on the past second information accumulated in response to the first instruction at the time when the operation of the moving object is started. Image processing methods.

5. On the computer, determining first information regarding deformation of a projection surface onto which a peripheral image of the moving object is projected; deforming the projection surface based on the first information; outputting a first instruction when it is determined that a first image associated with the moving object at the time of completion of the movement of the moving object and a second image associated with the moving object at the time of starting the movement of the moving object are similar to each other; generating first control information for moving the moving object and acquiring a plurality of new images of the periphery of the moving object when it is determined that the first image and the second image are not similar; An image processing program for executing the determining step includes a step of accumulating past second information corresponding to the time when the operation of the moving object used in determining the first information has ended, and the first information is determined based on the past second information accumulated in response to the first instruction at the time when the operation of the moving object is started. Image processing program.

Images