Image processing device, image processing method, and image processing program

Abstract

An image processing apparatus (10) according to one aspect of the present invention comprises an action plan formulation unit (28) and a projection shape determination unit (29). The action plan formulation unit (28) generates, on the basis of action plan information of a mobile body (2), first information including scheduled self-position information of the mobile body (2) and position information of a peripheral three-dimensional object with the scheduled self-position information as a reference. The projection shape determination unit (29) determines, on the basis of the first information, the shape of a projection surface onto which a first image acquired by an imaging device mounted to the mobile body is projected to generate a bird's-eye view image.

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an image processing method, and an image processing program. [Background technology]

[0002] There is a technology that generates a bird's-eye view image of the area around a moving object, such as a car, using images from multiple cameras mounted on the object. There is also a technology that changes the shape of the projection surface of the bird's-eye view image depending on the three-dimensional objects around the moving object. There is also a technology that uses Visual SLAM (Simultaneous Localization and Mapping: VSLAM), which performs SLAM using images captured by cameras, to obtain positional information around the moving object and determine the moving path of the moving object. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2009-232310 [Patent Document 2] Japanese Patent Application Laid-Open No. 2013-207637 [Patent Document 3] Special Publication No. 2014-531078 [Patent Document 4] International Publication No. 2021 / 111531 [Patent Document 5] International Publication No. 2021 / 065241 [Patent Document 6] Japanese Patent Publication No. 2020-083140 Summary of the Invention [Problem to be solved by the invention]

[0004] However, when the projection surface of the overhead image is successively transformed in accordance with three-dimensional objects around the moving object, the transformation of the projection surface may lag behind the movement of the moving object, resulting in an unnatural overhead image.

[0005] In one aspect, the present invention aims to realize an image processing device, an image processing method, and an image processing program that provide a more natural overhead image than conventional methods when the projection surface of the overhead image is successively deformed in accordance with three-dimensional objects around a moving body. [Means for solving the problem]

[0006] In one aspect, the image processing device disclosed herein includes a behavior planning unit and a projection shape determination unit. The behavior planning unit generates first information based on behavior plan information of a moving object, the first information including planned self-location information indicating the planned self-location of the moving object and position information of a surrounding three-dimensional object based on the planned self-location information. The projection shape determination unit determines the shape of a projection surface onto which a first image acquired by an image capturing device mounted on the moving object is projected to generate an overhead image, based on the first information. [Effects of the Invention]

[0007] According to one aspect of the image processing device disclosed in the present application, when the projection surface of the overhead image is successively transformed in accordance with three-dimensional objects around the moving body, it is possible to provide a more natural overhead image than conventionally possible. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of the overall configuration of an information processing system according to an embodiment. [Figure 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of the information processing device according to the embodiment. [Figure 3] FIG. 3 is a diagram illustrating an example of a functional configuration of the information processing device according to the embodiment. [Figure 4] FIG. 4 is a schematic diagram illustrating an example of environment map information according to the embodiment. [Figure 5] FIG. 5 is a schematic diagram illustrating an example of a functional configuration of the action plan formulation unit of the information processing device according to the first embodiment. [Figure 6]FIG. 6 is a schematic diagram showing an example of a parking route plan generated by the planning processing unit. [Figure 7] FIG. 7 is a schematic diagram showing an example of the planned map information generated by the planned map information generating unit. [Figure 8] FIG. 8 is a schematic diagram illustrating an example of the functional configuration of the determination unit of the information processing device according to the first embodiment. [Figure 9] FIG. 9 is a schematic diagram showing an example of the reference projection plane. [Figure 10] FIG. 10 is an explanatory diagram of an asymptotic curve generated by the determination unit. [Figure 11] FIG. 11 is a schematic diagram showing an example of a projection shape determined by the determination unit. [Figure 12] FIG. 12 is a flowchart showing an example of the flow of the projection surface deformation process based on the action plan. [Figure 13] FIG. 13 is a flowchart showing an example of the flow of a process for generating an overhead image, including a projection surface deformation process based on an action plan, which is executed by an information processing device. [Figure 14] FIG. 14 is a diagram for explaining the projection surface deformation process executed by an information processing device according to a comparative example. [Figure 15] FIG. 15 is a diagram for explaining the projection surface deformation process executed by an information processing device according to a comparative example. [Figure 16] FIG. 16 is a schematic diagram illustrating an example of a functional configuration of an information processing device according to the second embodiment. [Figure 17] FIG. 17 is a schematic diagram illustrating an example of a functional configuration of an action plan formulation unit of an information processing device according to the second embodiment. [Figure 18] FIG. 18 is a schematic diagram illustrating an example of a functional configuration of an information processing device according to the third embodiment. [Figure 19] FIG. 19 is a schematic diagram illustrating an example of a functional configuration of an action plan formulation unit of an information processing device according to the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, with reference to the accompanying drawings, embodiments of the image processing device, image processing method, and image processing program disclosed herein will be described in detail. Note that the following embodiments do not limit the disclosed technology. Furthermore, each embodiment can be appropriately combined within a range that does not cause contradiction in the processing content.

[0010] (First embodiment) FIG. 1 is a diagram showing an example of the overall configuration of an information processing system 1 of this embodiment. The information processing system 1 includes an information processing device 10, an imaging unit 12, a detection unit 14, and a display unit 16. The information processing device 10, the imaging unit 12, the detection unit 14, and the display unit 16 are connected so as to be able to exchange data or signals. The information processing device 10 is an example of an image processing device. Furthermore, the information processing method executed by the information processing device 10 is an example of an image processing method, and the information processing program used by the information processing device 10 to execute the information processing method is an example of an image processing program.

[0011] In this embodiment, the information processing device 10, the image capturing unit 12, the detection unit 14, and the display unit 16 are mounted on a moving object 2 as an example.

[0012] The moving body 2 is an object that can move. The moving body 2 is, for example, a vehicle, a flyable object (a manned airplane, an unmanned airplane (e.g., a UAV (Unmanned Aerial Vehicle), a drone)), a robot, etc. The moving body 2 is, for example, a moving body that moves through human driving operation, or a moving body that can move automatically (autonomously) without human driving operation. In this embodiment, a case where the moving body 2 is a vehicle will be described as an example. Examples of vehicles include a two-wheeled vehicle, a three-wheeled vehicle, and a four-wheeled vehicle. In this embodiment, a case where the vehicle is an autonomously moving four-wheeled vehicle will be described as an example.

[0013] Note that the information processing device 10, the photographing unit 12, the detection unit 14, and the display unit 16 are not limited to being all mounted on the moving object 2. The information processing device 10 may be mounted on, for example, a stationary object. A stationary object is an object fixed to the ground. A stationary object is an object that cannot be moved or an object that is stationary relative to the ground. Examples of stationary objects include traffic lights, parked vehicles, and road signs. Furthermore, the information processing device 10 may be mounted on a cloud server that executes processing on the cloud.

[0014] The photographing unit 12 photographs the periphery of the moving object 2 and acquires photographed image data. In the following description, the photographed image data will be simply referred to as a photographed image. The photographing unit 12 is, for example, a digital camera capable of shooting video. Photographing refers to converting an image of a subject formed by an optical system such as a lens into an electrical signal. The photographing unit 12 outputs the photographed image to the information processing device 10. In addition, in this embodiment, the photographing unit 12 will be described assuming that it is a monocular fisheye camera (for example, with a viewing angle of 195 degrees).

[0015] In this embodiment, an example will be described in which four imaging units 12, namely, a front imaging unit 12A, a left imaging unit 12B, a right imaging unit 12C, and a rear imaging unit 12D, are mounted on a moving object 2. The multiple imaging units 12 (front imaging unit 12A, left imaging unit 12B, right imaging unit 12C, and rear imaging unit 12D) each capture an image of a subject in a different imaging area E (front imaging area E1, left imaging area E2, right imaging area E3, and rear imaging area E4) and acquire a captured image. That is, the imaging directions of the multiple imaging units 12 are different from each other. Furthermore, the imaging directions of the multiple imaging units 12 are adjusted in advance so that at least a portion of the imaging area E of adjacent imaging units 12 overlaps. Furthermore, for convenience of explanation, the imaging area E in FIG. 1 is shown as being the same size as in FIG. 1, but in reality, it may include an area further away from the moving object 2.

[0016] Furthermore, the four photographing units 12A, 12B, 12C, and 12D are merely examples, and there is no limit to the number of photographing units 12. For example, if the moving body 2 has a vertically long shape like a bus or truck, it is possible to use a total of six photographing units 12, by arranging one photographing unit 12 at the front, one at the rear, one at the front of the right side, one at the rear of the right side, one at the front of the left side, and one at the rear of the left side of the moving body 2. In other words, the number and arrangement positions of the photographing units 12 can be set arbitrarily depending on the size and shape of the moving body 2.

[0017] The detection unit 14 detects position information of each of a plurality of detection points around the moving object 2. In other words, the detection unit 14 detects position information of each of the detection points in the detection area F. The detection points refer to each of the points in real space that are individually observed by the detection unit 14. The detection points correspond to, for example, three-dimensional objects around the moving object 2. The detection unit 14 is an example of an external sensor.

[0018] The detection unit 14 may be, for example, a 3D (Three-Dimensional) scanner, a 2D (Two-Dimensional) scanner, a distance sensor (millimeter-wave radar, laser sensor), a sonar sensor that detects objects using sound waves, an ultrasonic sensor, or the like. The laser sensor may be, for example, a three-dimensional LiDAR (Laser Imaging Detection and Ranging) sensor. The detection unit 14 may also be a device that uses a technology that measures distance from images captured by a stereo camera or a monocular camera, such as SfM (Structure from Motion) technology. A plurality of image capture units 12 may also be used as the detection unit 14. One of the plurality of image capture units 12 may also be used as the detection unit 14.

[0019] The display unit 16 displays various types of information and is, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display.

[0020] In this embodiment, the information processing device 10 is communicably connected to an electronic control unit (ECU) 3 mounted on the moving object 2. The ECU 3 is a unit that performs electronic control of the moving object 2. In this embodiment, the information processing device 10 is capable of receiving CAN (Controller Area Network) data such as the speed and moving direction of the moving object 2 from the ECU 3.

[0021] Next, the hardware configuration of the information processing device 10 will be described.

[0022] FIG. 2 is a diagram illustrating an example of a hardware configuration of the information processing device 10. As shown in FIG.

[0023] The information processing device 10 is, for example, a computer, and includes a CPU (Central Processing Unit) 10A, a ROM (Read Only Memory) 10B, a RAM (Random Access Memory) 10C, and an I / F (Interface) 10D. The CPU 10A, ROM 10B, RAM 10C, and I / F 10D are interconnected by a bus 10E, and have a hardware configuration similar to that of a typical computer.

[0024] The CPU 10A is a calculation device that controls the information processing device 10. The CPU 10A corresponds to an example of a hardware processor. The ROM 10B stores programs and the like that realize various processes by the CPU 10A. The RAM 10C stores data necessary for various processes by the CPU 10A. The I / F 10D is an interface that is connected to the imaging unit 12, the detection unit 14, the display unit 16, the ECU 3, etc., and is used to send and receive data.

[0025] A program for executing information processing executed by the information processing device 10 of this embodiment is provided by being pre-installed in a ROM 10B or the like. The program executed by the information processing device 10 of this embodiment may be provided by being recorded on a recording medium in a format that can be installed on the information processing device 10 or in a format that can be executed. The recording medium is a computer-readable medium. Examples of the recording medium include a CD (Compact Disc)-ROM, a flexible disk (FD), a CD-R (Recordable), a DVD (Digital Versatile Disk), a USB (Universal Serial Bus) memory, and an SD (Secure Digital) card.

[0026] Next, a functional configuration of the information processing device 10 according to this embodiment will be described. The information processing device 10 uses VSLAM processing to simultaneously estimate peripheral position information of the moving object 2 and self-position information of the moving object 2 from images captured by the image capturing unit 12. The information processing device 10 stitches together multiple spatially adjacent captured images to generate and display a composite image (overhead image) overlooking the periphery of the moving object 2. In this embodiment, the image capturing unit 12 is used as the detection unit 14.

[0027] Fig. 3 is a diagram showing an example of the functional configuration of the information processing device 10. In Fig. 3, in order to clarify the data input / output relationship, an image capturing unit 12 and a display unit 16 are also shown in addition to the information processing device 10.

[0028] The information processing device 10 includes an acquisition unit 20 , a selection unit 21 , a VSLAM processing unit 24 , a distance conversion unit 27 , an action plan formulation unit 28 , a projection shape determination unit 29 , and an image generation unit 37 .

[0029] Some or all of the above-mentioned multiple units may be realized by, for example, causing a processing device such as CPU 10A to execute a program, that is, by software. Also, some or all of the above-mentioned multiple units may be realized by hardware such as an IC (Integrated Circuit), or may be realized by a combination of software and hardware.

[0030] The acquisition unit 20 acquires captured images from the imaging unit 12. That is, the acquisition unit 20 acquires captured images from each of the front imaging unit 12A, the left imaging unit 12B, the right imaging unit 12C, and the rear imaging unit 12D.

[0031] The acquisition unit 20 sends the acquired captured image to the projection transformation unit 36 ​​and the selection unit 21 every time it acquires a captured image.

[0032] The selection unit 21 selects a detection area of ​​the detection point. In this embodiment, the selection unit 21 selects at least one of the multiple imaging units 12 (imaging units 12A to 12D), thereby selecting the detection area.

[0033] The VSLAM processing unit 24 generates second information including position information of the three-dimensional objects around the moving body 2 and position information of the moving body 2 based on the image of the surroundings of the moving body 2. That is, the VSLAM processing unit 24 receives the captured image from the selection unit 21, performs VSLAM processing using the captured image to generate environment map information, and outputs the generated environment map information to the determination unit 30.

[0034] More specifically, the VSLAM processing unit 24 includes a matching unit 240 , a storage unit 241 , a self-position estimation unit 242 , a three-dimensional reconstruction unit 243 , and a correction unit 244 .

[0035] The matching unit 240 performs a feature extraction process and a matching process between multiple captured images (multiple captured images in different frames) captured at different times. In detail, the matching unit 240 performs a feature extraction process from these multiple captured images. The matching unit 240 performs a matching process for identifying corresponding points between the multiple captured images captured at different times, using the feature amounts between the multiple captured images. The matching unit 240 outputs the matching process result to the storage unit 241.

[0036] The self-position estimation unit 242 estimates the self-position relative to the captured image by projective transformation or the like using the multiple matching points acquired by the matching unit 240. Here, the self-position includes information on the position (three-dimensional coordinates) and tilt (rotation) of the image capturing unit 12. The self-position estimation unit 242 stores the self-position information as point cloud information in the environment map information 241A.

[0037] The three-dimensional reconstruction unit 243 performs perspective projection transformation processing using the movement amount (translation amount and rotation amount) of the self-position estimated by the self-position estimation unit 242, and determines the three-dimensional coordinates (coordinates relative to the self-position) of the matching point. The three-dimensional reconstruction unit 243 stores the peripheral position information, which is the determined three-dimensional coordinates, in the environmental map information 241A as point cloud information.

[0038] As a result, new peripheral position information and new self-position information are sequentially added to the environment map information 241A as the moving object 2 on which the image capturing unit 12 is mounted moves.

[0039] The storage unit 241 stores various types of data. The storage unit 241 is, for example, a semiconductor memory element such as a RAM or a flash memory, a hard disk, an optical disk, or the like. The storage unit 241 may be a storage device provided outside the information processing device 10. The storage unit 241 may also be a storage medium. Specifically, the storage medium may store or temporarily store programs and various types of information downloaded via a LAN (Local Area Network), the Internet, or the like.

[0040] The environmental map information 241A is information in which point cloud information, which is peripheral position information calculated by the three-dimensional restoration unit 243, and point cloud information, which is self-position information calculated by the self-position estimation unit 242, are registered in a three-dimensional coordinate space with a predetermined position in the real space as the origin (reference position). The predetermined position in the real space may be determined based on, for example, a predetermined condition.

[0041] For example, the predetermined position used in the environmental map information 241A is the self-position of the moving object 2 when the information processing device 10 executes the information processing of this embodiment. For example, assume a case where information processing is executed at a predetermined timing, such as a parking scene of the moving object 2. In this case, the information processing device 10 may set the self-position of the moving object 2 when it determines that the predetermined timing has been reached as the predetermined position. For example, the information processing device 10 may determine that the predetermined timing has been reached when it determines that the behavior of the moving object 2 has become behavior indicative of a parking scene. Examples of behavior indicative of a parking scene due to backing up include when the speed of the moving object 2 drops below a predetermined speed, when the gear of the moving object 2 is shifted into reverse gear, or when a signal indicating the start of parking is received by a user's operation instruction, etc. Note that the predetermined timing is not limited to a parking scene.

[0042] Fig. 4 is a schematic diagram of an example in which information on a specific height is extracted from the environment map information 241A. As shown in Fig. 4, the environment map information 241A is information in which point cloud information, which is the position information (peripheral position information) of each detection point P, and point cloud information, which is the self-position information of the self-position S of the moving object 2, are registered at corresponding coordinate positions in the three-dimensional coordinate space. Note that Fig. 4 shows the self-position S of self-positions S1 to S3 as an example. The larger the value of the number following S, the closer the self-position S is to the current timing.

[0043] The correction unit 244 corrects the peripheral position information and self-position information registered in the environmental map information 241A using, for example, the least squares method, so that the sum of the differences in distance in three-dimensional space between previously calculated three-dimensional coordinates and newly calculated three-dimensional coordinates for points that have been matched multiple times between multiple frames is minimized. Note that the correction unit 244 may also correct the amount of movement (translation amount and rotation amount) of the self-position used in the process of calculating the self-position information and peripheral position information.

[0044] The timing of the correction process by the correction unit 244 is not limited. For example, the correction unit 244 may execute the correction process at a predetermined timing. The predetermined timing may be determined based on a predetermined condition, for example. Note that in this embodiment, a case where the information processing device 10 is configured to include the correction unit 244 will be described as an example. However, the information processing device 10 may not be configured to include the correction unit 244.

[0045] Distance conversion unit 27 converts the relative positional relationship between the self-position and the surrounding three-dimensional objects, which can be known from the environmental map information, into the absolute value of the distance from the self-position to the surrounding three-dimensional objects, generates detection point distance information of the surrounding three-dimensional objects, and outputs it to action plan formulation unit 28. Here, the detection point distance information of the surrounding three-dimensional objects is information obtained by offsetting the self-position to coordinates (0,0,0) and converting the measured distance (coordinates) to each of the calculated multiple detection points P into, for example, meters. In other words, information on the self-position of moving object 2 is included as the coordinates (0,0,0) of the origin in the detection point distance information.

[0046] The distance conversion unit 27 uses, for example, vehicle status information, such as the speed data of the moving object 2, included in the CAN data sent from the ECU 3. For example, in the case of the environment map information 241A shown in FIG. 4, the relative positional relationship between the self-position S and multiple detection points P can be known, but the absolute value of the distance has not been calculated. Here, the distance between the self-position S3 and the self-position S2 can be calculated based on the inter-frame period for calculating the self-position and the speed data during that period based on the vehicle status information. Because the relative positional relationship in the environment map information 241A is similar to that in real space, knowing the distance between the self-position S3 and the self-position S2 also makes it possible to calculate the absolute values ​​of the distances from the self-position S to all other detection points P. That is, the distance conversion unit 27 uses the actual speed data of the moving object 2 included in the CAN data to convert the relative positional relationship between the self-position and a surrounding three-dimensional object into the absolute value of the distance from the self-position to the surrounding three-dimensional object.

[0047] The vehicle state information included in the CAN data can be associated with the environmental map information output from the VSLAM processing unit 24 using time information. In addition, when the detection unit 14 acquires distance information of the detection point P, the distance conversion unit 27 may be omitted.

[0048] The behavior plan formulation unit 28 formulates a behavior plan for the moving body 2 based on second information including position information (detection point distance information) of surrounding three-dimensional objects of the moving body 2, and generates first information including planned self-position information of the moving body 2 and position information of surrounding three-dimensional objects based on the planned self-position information of the moving body 2.

[0049] Fig. 5 is a schematic diagram showing an example of the functional configuration of the action plan formulation unit 28. As shown in Fig. 5, the action plan formulation unit 28 includes a planning processing unit 28A, a planned map information generation unit 28B, and a PID control unit 28C.

[0050] For example, in response to an instruction from the driver to select the automatic parking mode, the planning processing unit 28A executes planning processing based on the detection point distance information received from the distance conversion unit 27. Here, the planning processing executed by the planning processing unit 28A is processing to formulate a parking route from the current position of the mobile object 2 to a parking completion position in order to park the mobile object 2 in a parking area, a planned self-position of the mobile object 2 after a unit time that will be the nearest target point along the parking route, and actuator target values ​​such as the nearest accelerator and turning angle for reaching the nearest target point.

[0051] FIG. 6 is a schematic diagram illustrating an example of a parking route plan generated by the planning processing unit 28A. As shown in FIG. 6, the parking route plan is information including a route from the current position L1 to a parking completion position via planned via points L2, L3, L4, and L5 when backing into a parking area PA. The planned via points L2, L3, L4, and L5 are currently planned via points L2, L3, L4, and L5 for the mobile object 2 per unit time. Therefore, the distances between the current position L1 and the planned via points L2, L3, L4, and L5 may vary depending on the moving speed of the mobile object 2. Here, the unit time is, for example, a time interval corresponding to the frame rate of the VSLAM processing. The planned via point L2 is also the planned self-position relative to the current position L1. Furthermore, the parking route plan is updated based on the latest detection point distance information at the position where the mobile object 2 is located after a unit time has elapsed as the mobile object 2 moves from the current position L1 toward the planned via point L2. In this case, the position after the unit time has elapsed may coincide with the planned route point L2 relative to the current position L1, or may be a position displaced from the planned route point L2.

[0052] There is no particular limitation on the specific calculation method for the parking route plan described above, and any method can be used as long as it can acquire information including the planned self-position where the moving body 2 should be located after the next unit time has elapsed, every time a unit time has elapsed.

[0053] Furthermore, in this embodiment, the case where the behavior plan formulation unit 28 includes a planning processing unit 28A and sequentially formulates behavior plans in the planning processing unit 28A is exemplified. In contrast, the behavior plan formulation unit 28 may not include the planning processing unit 28A and may sequentially acquire behavior plans for the moving object 2 formulated externally.

[0054] The planned map information generating unit 28B offsets the origin (current self-position) of the detection point distance information to the planned self-position unit time later. FIG. 7 is a schematic diagram for explaining the planned map information generated by the planned map information generating unit 28B. In FIG. 7, detection point distance information (multiple point cloud information) around the moving object 2 is shown as the planned map information. For the sake of explanation, FIG. 7 also adds an area R1, an area R2, a trajectory T1, and positions L1 and L2. The point cloud in area R1 corresponds to another moving object (car1) located next to the parking area PA where the moving object 2 should be parked. The point cloud in area R2 corresponds to a pillar near the parking area PA. The trajectory T1 indicates the trajectory of the moving object 2 when it moves forward from the right side to the left side of the drawing, stops temporarily, and then backs up and parks into the parking area PA. Position L1 indicates the current self-position of the moving object 2, and position L2 indicates the planned self-position of the moving object 2 at a timing unit time in the future. The planned map information generating unit 28B offsets the detection point distance information, which has the current position L1 as the origin, with the planned own position L2 as the origin, to generate planned map information seen from the planned own position after a unit time.

[0055] The planned map information generating unit 28B offsets the origin (current self-position) of the updated detection point distance information to the planned self-position unit 30 each time the self-position of the moving body 2 and the position information of the surrounding three-dimensional objects of the moving body 2 are updated by the VSLAM processing. The planned map information generating unit 28B generates planned map information whose origin is offset to the planned self-position unit 30, and sends it to the determining unit 30.

[0056] The PID control unit 28C performs PID (Proportional Integral Differential) control based on the actuator target values ​​formulated by the planning processing unit 28A, and outputs actuator control values ​​for controlling actuators such as the accelerator and turning angle. For example, the PID control unit 28C updates the actuator control values ​​and outputs them to the actuators every time the planning processing unit 28A updates the actuator target values. The PID control unit 28C is an example of a control information generation unit.

[0057] 3, based on the first information, projection shape determination unit 29 determines the shape of a projection surface for generating an overhead image by projecting an image acquired by image capturing device 12 mounted on moving object 2. Projection shape determination unit 29 is an example of a projection shape determination unit.

[0058] Here, the projection surface is a three-dimensional surface onto which an image of the periphery of the moving object 2 is projected as an overhead image. The image of the periphery of the moving object 2 is a captured image of the periphery of the moving object 2, which is a captured image captured by each of the image capturing units 12A to 12D. The projection shape of the projection surface is a three-dimensional (3D) shape virtually formed in a virtual space corresponding to the real space. The image projected onto the projection surface may be the same image as the image used by the VSLAM processing unit 24 when generating the second information, or may be an image acquired at a different time or an image subjected to different image processing. In this embodiment, the determination of the projection shape of the projection surface executed by the projection shape determination unit 29 is referred to as a projection shape determination process.

[0059] Specifically, the projection shape determination unit 29 includes a determination unit 30, a deformation unit 32, and a virtual viewpoint line of sight determination unit .

[0060] [Configuration example of the determination unit 30] An example of the detailed configuration of the determination unit 30 shown in FIG. 3 will be described below.

[0061] Fig. 8 is a schematic diagram showing an example of the functional configuration of the determination unit 30. As shown in Fig. 8, the determination unit 30 includes an extraction unit 305, a nearest neighbor identification unit 307, a reference projection surface shape selection unit 309, a scale determination unit 311, an asymptotic curve calculation unit 313, a shape determination unit 315, and a boundary region determination unit 317.

[0062] The extraction unit 305 extracts detection points P that exist within a specific range from among the multiple detection points P for which the measurement distances have been received from the distance conversion unit 27, and generates a specific height extraction map. The specific range is, for example, a range from the road surface on which the moving object 2 is placed to a height equivalent to the vehicle height of the moving object 2. Note that the range is not limited to this range.

[0063] The extraction unit 305 extracts detection points P within the range and generates a specific height extraction map, thereby making it possible to extract detection points P of, for example, objects that obstruct the progress of the moving body 2 or objects located adjacent to the moving body 2.

[0064] Then, the extraction unit 305 outputs the generated specific height extraction map to the nearest neighbor identification unit 307.

[0065] The nearest neighbor identification unit 307 divides the periphery of the planned self-position S' of the moving body 2 into specific ranges (for example, angular ranges) using the specific height extraction map, and for each range, identifies the detection point P closest to the planned self-position S' of the moving body 2, or multiple detection points P in order of proximity to the planned self-position S' of the moving body 2, and generates neighbor point information. In this embodiment, an example will be described in which the nearest neighbor identification unit 307 identifies multiple detection points P in order of proximity to the planned self-position S' of the moving body 2 for each range, and generates neighbor point information.

[0066] The nearest neighbor specifying unit 307 outputs the measured distance of the detection point P specified for each range as neighbor point information to the reference projection surface shape selecting unit 309, scale determining unit 311, asymptotic curve calculating unit 313, and boundary region determining unit 317.

[0067] The reference projection plane shape selection unit 309 selects the shape of the reference projection plane based on the neighboring point information.

[0068] FIG. 9 is a schematic diagram showing an example of the reference projection surface 40. The reference projection surface will be described with reference to FIG. 9. The reference projection surface 40 is, for example, a projection surface having a shape that serves as a reference when changing the shape of the projection surface. The shape of the reference projection surface 40 is, for example, a bowl shape, a cylindrical shape, etc. Note that FIG. 9 shows an example of a bowl-shaped reference projection surface 40.

[0069] The bowl-shaped container has a bottom surface 40A and a side wall surface 40B, one end of which is continuous with the bottom surface 40A and the other end of which is open. The width of the horizontal cross section of the side wall surface 40B increases from the bottom surface 40A toward the open end of the other end. The bottom surface 40A is, for example, circular. Here, a circular shape includes a perfect circle and other circular shapes such as an ellipse. The horizontal cross section is an orthogonal plane perpendicular to the vertical direction (arrow Z direction). The orthogonal plane is a two-dimensional plane along the arrow X direction, which is perpendicular to the arrow Z direction, and the arrow Y direction, which is perpendicular to the arrow Z direction and the arrow X direction. Hereinafter, the horizontal cross section and the orthogonal plane may be referred to as the XY plane. The bottom surface 40A may have a shape other than a circle, such as an egg shape.

[0070] The cylindrical shape is a shape consisting of a circular bottom surface 40A and a side wall surface 40B that is continuous with the bottom surface 40A. The side wall surface 40B that constitutes the cylindrical reference projection surface 40 has a cylindrical shape with an opening at one end that is continuous with the bottom surface 40A and an open other end. However, the side wall surface 40B that constitutes the cylindrical reference projection surface 40 has a shape in which the diameter in the XY plane is approximately constant from the bottom surface 40A side toward the opening at the other end. The bottom surface 40A may have a shape other than a circle, such as an egg shape.

[0071] In this embodiment, the case where the shape of the reference projection plane 40 is bowl-shaped as shown in Fig. 9 will be described as an example. The reference projection plane 40 is a three-dimensional model virtually formed in a virtual space, with the bottom surface 40A being a surface that substantially coincides with the road surface below the moving object 2 and the center of the bottom surface 40A being the planned self-position S' of the moving object 2.

[0072] The reference projection surface shape selection unit 309 selects the shape of the reference projection surface 40 by reading one specific shape from multiple types of reference projection surfaces 40. For example, the reference projection surface shape selection unit 309 selects the shape of the reference projection surface 40 based on the positional relationship and distance between the intended self-position and surrounding three-dimensional objects. The shape of the reference projection surface 40 may also be selected based on an operational instruction from the user. The reference projection surface shape selection unit 309 outputs shape information of the determined reference projection surface 40 to the shape determination unit 315. In this embodiment, as described above, an embodiment in which the reference projection surface shape selection unit 309 selects a bowl-shaped reference projection surface 40 will be described as an example.

[0073] The scale determination unit 311 determines the scale of the reference projection plane 40 of the shape selected by the reference projection plane shape selection unit 309. For example, the scale determination unit 311 determines to reduce the scale when the distance from the expected self-position S' to a nearby point is shorter than a predetermined distance. The scale determination unit 311 outputs scale information of the determined scale to the shape determination unit 315.

[0074] The asymptotic curve calculation unit 313 calculates an asymptotic curve of the peripheral position information for the planned self-position based on the planned map information. The asymptotic curve calculation unit 313 uses the distance from the planned self-position S' to the nearest detection point P for each range from the planned self-position S' received from the nearest neighbor identification unit 307, and outputs asymptotic curve information of the calculated asymptotic curve Q to the shape determination unit 315 and the virtual viewpoint line of sight determination unit 34.

[0075] FIG. 10 is an explanatory diagram of an asymptotic curve Q generated by the determination unit 30. Here, the asymptotic curve is an asymptotic curve of a plurality of detection points P in the planned map information. FIG. 10 shows an example in which the asymptotic curve Q is displayed in a projected image obtained by projecting a captured image onto a projection surface when the moving object 2 is viewed from above. For example, it is assumed that the determination unit 30 has identified three detection points P in descending order of proximity to the planned self-position S' of the moving object 2. In this case, the determination unit 30 generates the asymptotic curve Q of these three detection points P.

[0076] The asymptotic curve calculation unit 313 may obtain a representative point located at the center of gravity of a plurality of detection points P for each specific range (for example, an angular range) on the reference projection plane 40, and calculate an asymptotic curve Q for the representative point for each of the plurality of ranges. Then, the asymptotic curve calculation unit 313 outputs asymptotic curve information of the calculated asymptotic curve Q to the shape determination unit 315. The asymptotic curve calculation unit 313 may output the asymptotic curve information of the calculated asymptotic curve Q to the virtual viewpoint line of sight determination unit 34.

[0077] The shape determination unit 315 enlarges or reduces the reference projection plane 40, which has a shape indicated by the shape information received from the reference projection plane shape selection unit 309, to the scale of the scale information received from the scale determination unit 311. Then, the shape determination unit 315 determines, as the projection shape, a shape obtained by deforming the reference projection plane 40 after enlarging or reducing it so that the shape conforms to the asymptotic curve information of the asymptotic curve Q received from the asymptotic curve calculation unit 313.

[0078] Here, the determination of the projection shape will be described in detail. FIG. 11 is a schematic diagram showing an example of the projection shape 41 determined by the determination unit 30. As shown in FIG. 11, the shape determination unit 315 determines, as the projection shape 41, a shape obtained by deforming the reference projection plane 40 into a shape that passes through a detection point P that is closest to the intended self-position S' of the moving object 2, which is the center of the bottom surface 40A of the reference projection plane 40. A shape that passes through the detection point P means that the deformed side wall surface 40B is a shape that passes through the detection point P. The intended self-position S' is determined by the behavior plan formulation unit 28.

[0079] That is, the shape determination unit 315 identifies the detection point P that is closest to the planned self-position S' among the multiple detection points P registered in the planned map information. In detail, the XY coordinates of the center position (planned self-position S') of the moving object 2 are set to (X, Y) = (0, 0). Then, the shape determination unit 315 determines the X 2 +Y 2 The detected point P where the value of is the smallest is identified as the detected point P closest to the expected self-position S'. Then, the shape determination unit 315 determines, as the projected shape 41, a shape obtained by deforming the side wall surface 40B of the reference projection plane 40 so that it passes through the detected point P.

[0080] More specifically, the shape determination unit 315 determines the deformed shape of the bottom surface 40A and the partial area of ​​the side wall surface 40B as the projected shape 41 so that, when the reference projection plane 40 is deformed, the partial area of ​​the side wall surface 40B becomes a wall surface passing through the detection point P closest to the expected self-position S′ of the moving object 2. The deformed projected shape 41 is, for example, a shape that is raised from a rising line 44 on the bottom surface 40A in a direction toward the center of the bottom surface 40A from the viewpoint of the XY plane (planar view). "Raising" means, for example, bending or folding the side wall surface 40B and the partial area of ​​the bottom surface 40A in a direction toward the center of the bottom surface 40A so that the angle formed between the side wall surface 40B of the reference projection plane 40 and the bottom surface 40A becomes smaller. Note that in the raised shape, the rising line 44 may be located between the bottom surface 40A and the side wall surface 40B, and the bottom surface 40A may remain undeformed.

[0081] The shape determination unit 315 determines to deform the specific region on the reference projection plane 40 so as to project it to a position that passes through the detection point P when viewed from the viewpoint (planar view) of the XY plane. The shape and range of the specific region may be determined based on predetermined criteria. Then, the shape determination unit 315 determines to deform the reference projection plane 40 so that the distance from the expected self-position S' increases continuously from the projected specific region toward the region other than the specific region on the side wall surface 40B.

[0082] For example, it is preferable to determine the projected shape 41 so that the outer periphery of the cross section along the XY plane has a curved shape, as shown in Fig. 11. Note that the outer periphery of the cross section of the projected shape 41 is, for example, a circle, but may have a shape other than a circle.

[0083] The shape determination unit 315 may determine, as the projected shape 41, a shape obtained by deforming the reference projection plane 40 so that the shape follows an asymptotic curve. The shape determination unit 315 generates an asymptotic curve of a predetermined number of detection points P in a direction away from the detection point P closest to the expected self-position S' of the moving object 2. The number of detection points P may be more than one. For example, the number of detection points P is preferably three or more. In this case, the shape determination unit 315 preferably generates an asymptotic curve of a plurality of detection points P that are located at positions that are at least a predetermined angle away from the expected self-position S'. For example, the shape determination unit 315 may determine, as the projected shape 41, a shape obtained by deforming the reference projection plane 40 so that the shape follows the generated asymptotic curve Q for the asymptotic curve Q shown in FIG. 10.

[0084] The shape determination unit 315 may divide the periphery of the expected self-position S' of the moving body 2 into specific ranges, and for each range, identify the detection point P closest to the moving body 2, or multiple detection points P in order of proximity to the moving body 2. Then, the shape determination unit 315 may determine, as the projection shape 41, a shape obtained by deforming the reference projection plane 40 so as to become a shape that passes through the detection points P identified for each range, or a shape that follows the asymptotic curve Q of the identified multiple detection points P.

[0085] Then, the shape determination unit 315 outputs the projection shape information of the determined projection shape 41 to the deformation unit 32.

[0086] Returning to FIG. 3, the deformation unit 32 deforms the projection plane based on the projection shape information determined using the planned map information received from the determination unit 30. That is, the deformation unit 32 deforms the projection plane using three-dimensional point cloud data whose origin is offset to the planned self-position S' after a unit time has elapsed (for example, the next frame) based on the action plan. This deformation of the reference projection plane is performed, for example, using the detected point P closest to the planned self-position S' of the moving object 2 as a reference. The deformation unit 32 outputs the deformed projection plane information to the projection conversion unit 36.

[0087] Furthermore, for example, the deformation unit 32 deforms the reference projection plane into a shape that follows the asymptotic curve of a predetermined number of detection points P in order of proximity to the expected self-position S′ of the moving object 2 based on the projection shape information.

[0088] The virtual viewpoint line of sight determining unit determines virtual viewpoint line of sight information based on the expected self-position S' and the asymptotic curve information, and sends it to the projection transformation unit .

[0089] Determination of virtual viewpoint line-of-sight information will be described with reference to FIGS. 10 and 11. The virtual viewpoint line-of-sight determination unit 34 determines, for example, a direction passing through a detection point P closest to the intended self-position S' of the moving object 2 and perpendicular to the deformed projection plane as the line-of-sight direction. Furthermore, the virtual viewpoint line-of-sight determination unit 34 fixes the line-of-sight direction L, for example, and determines the coordinates of the virtual viewpoint O as an arbitrary Z coordinate and arbitrary X and Y coordinates in a direction away from the asymptotic curve Q toward the intended self-position S'. In this case, the X and Y coordinates may be coordinates of a position farther from the asymptotic curve Q than the intended self-position S'. The virtual viewpoint line-of-sight determination unit 34 then outputs virtual viewpoint line-of-sight information indicating the virtual viewpoint O and the line-of-sight direction L to the projection transformation unit 36. As shown in FIG. 10, the line-of-sight direction L may be a direction from the virtual viewpoint O toward the position of the vertex W of the asymptotic curve Q.

[0090] The image generation unit 37 generates the moving object 2 and its overhead image using the projection plane. Specifically, the image generation unit 37 includes a projection conversion unit 36 ​​and an image synthesis unit 38.

[0091] The projection conversion unit 36 ​​generates a projection image by projecting the captured image acquired from the image capturing unit 12 onto the deformed projection surface based on the deformed projection surface information and the virtual viewpoint line of sight information. The projection conversion unit 36 ​​converts the generated projection image into a virtual viewpoint image and outputs it to the image synthesis unit 38. Here, the virtual viewpoint image is an image obtained by viewing the projection image in an arbitrary direction from a virtual viewpoint.

[0092] With reference to FIG. 11, the projection image generation process by the projection transformation unit 36 ​​will be described in detail. The projection transformation unit 36 ​​projects the captured image onto the deformed projection surface 42. The projection transformation unit 36 ​​then generates a virtual viewpoint image (not shown), which is an image obtained by viewing the captured image projected onto the deformed projection surface 42 from an arbitrary virtual viewpoint O in a line of sight direction L. The position of the virtual viewpoint O may be set, for example, to the planned self-position S' of the moving object 2 (used as the basis for the projection surface transformation process). In this case, the X and Y coordinate values ​​of the virtual viewpoint O may be set to the X and Y coordinate values ​​of the planned self-position S' of the moving object 2. Furthermore, the Z coordinate value (vertical position) of the virtual viewpoint O may be set to the Z coordinate value of the detection point P closest to the planned self-position S' of the moving object 2. The line of sight direction L may be determined, for example, based on a predetermined criterion.

[0093] The line of sight direction L may be, for example, a direction from the virtual viewpoint O toward the detection point P that is closest to the expected self-position S' of the moving object 2. The line of sight direction L may also be a direction that passes through the detection point P and is perpendicular to the deformed projection plane 42. Virtual viewpoint line of sight information indicating the virtual viewpoint O and the line of sight direction L is created by the virtual viewpoint line of sight determination unit 34.

[0094] The image synthesis unit 38 generates a synthesized image by extracting a part or all of the virtual viewpoint images. For example, the image synthesis unit 38 performs a process of stitching together a plurality of virtual viewpoint images (here, four virtual viewpoint images corresponding to the imaging units 12A to 12D) in the boundary region between the imaging units.

[0095] The image synthesis unit 38 outputs the generated synthetic image to the display unit 16. The synthetic image may be a bird's-eye view image with a virtual viewpoint O above the moving object 2, or may be an image in which the moving object 2 is displayed semi-transparently with a virtual viewpoint O inside the moving object 2.

[0096] (Projection surface transformation processing based on action planning) Next, a flow of the projection surface deformation process based on a behavior plan executed by the information processing device 10 according to this embodiment will be described. This projection surface deformation process based on a behavior plan does not perform the projection surface deformation process based on the self-position of the moving object 2 obtained by the VSLAM process, but performs the projection surface deformation process based on the planned self-position for a certain period of time ahead (future) obtained by the behavior plan.

[0097] 12 is a flowchart showing an example of the flow of the projection surface deformation process based on the action plan. Note that the overall flow of the detailed overhead image generation process executed by the information processing device 10 will be described in detail later.

[0098] First, a captured image is acquired (step Sa). VSLAM processing unit 24 generates environmental map information by VSLAM processing using the captured image, and distance conversion unit 27 acquires detection point distance information (step Sb).

[0099] The planning processing unit 28A formulates an action plan based on the detection point distance information (step Sc).

[0100] The planned map information generating unit 28B generates planned map information based on the planned self-position information and the detection point distance information acquired from the planning processing unit 28A (step Sd).

[0101] The determination unit 30 determines the shape of the projection surface using the planned map information (step Se).

[0102] The deformation unit 32 executes a projection surface deformation process based on the projection shape information (step Sf).

[0103] The processes from step Sa to step Sf are sequentially and repeatedly executed until the driving assistance process using the overhead image is completed, for example.

[0104] FIG. 13 is a flowchart showing an example of the flow of a process for generating an overhead image, including a projection surface deformation process based on an action plan, executed by the information processing device 10.

[0105] The acquisition unit 20 acquires the captured images for each direction from the image capturing unit 12 (step S2). The selection unit 21 selects the captured images as the detection area (step S4).

[0106] The matching unit 240 extracts features and performs matching processing (step S6) using the multiple captured images selected in step S4 and captured by the imaging unit 12 at different capture times. The matching unit 240 also registers information on corresponding points between the multiple captured images at different capture times, which have been identified by the matching processing, in the storage unit 241.

[0107] The self-position estimation unit 242 reads the matching points and the environment map information 241A (peripheral position information and self-position information) from the storage unit 241 (step S8). The self-position estimation unit 242 estimates the self-position relative to the captured image by projective transformation or the like using the multiple matching points acquired from the matching unit 240 (step S10), and registers the calculated self-position information in the environment map information 241A (step S12).

[0108] The three-dimensional restoration unit 243 reads the environment map information 241A (peripheral position information and self-position information) (step S14). The three-dimensional restoration unit 243 performs perspective projection transformation processing using the movement amount (translation amount and rotation amount) of the self-position estimated by the self-position estimation unit 242, determines the three-dimensional coordinates of the matching point (coordinates relative to the self-position), and registers them as peripheral position information in the environment map information 241A (step S18).

[0109] The correction unit 244 reads the environment map information 241A (peripheral position information and self-position information). The correction unit 244 corrects (step S20) the peripheral position information and self-position information already registered in the environment map information 241A using, for example, the least squares method, so that the total difference in distance in three-dimensional space between three-dimensional coordinates calculated previously and newly calculated three-dimensional coordinates for points that have been matched multiple times across multiple frames is minimized, and updates the environment map information 241A.

[0110] The distance conversion unit 27 acquires the speed data (host vehicle speed) of the moving object 2 included in the CAN data received from the ECU 3 of the moving object 2 (step S22). The distance conversion unit 27 uses the speed data of the moving object 2 to convert the coordinate distance between the point clouds included in the environment map information 241A into an absolute distance in meters, for example. The distance conversion unit 27 also offsets the origin of the environment map information to the host position S of the moving object 2 to generate detection point distance information indicating the distance from the moving object 2 to each of the plurality of detection points P (step S26). The distance conversion unit 27 outputs the detection point distance information to the action plan formulation unit 28.

[0111] The planning processing unit 28A executes planning processing and formulates a parking route from the current position of the mobile body 2 to the completion of parking in order to park the mobile body 2 in the parking area, the planned self-position of the mobile body 2 after a unit time which will be the nearest target point along the parking route, and target values ​​of actuators such as the nearest accelerator and turning angle to reach the nearest target point (step S28).

[0112] The planned map information generation unit 28B generates planned map information by offsetting the origin (current self-position S) of the detection point distance information to the planned self-position S' of the moving body 2 predicted after the elapse of a unit time, and sends it to the extraction unit 305 (step S30).

[0113] The PID control unit 28C performs PID control based on the actuator target values ​​formulated by the planning processing unit 28A, and sends actuator control values ​​to the actuators (step S31).

[0114] The extraction unit 305 extracts the detection points P that exist within a specific range from the detection point distance information (step S32).

[0115] The nearest neighbor identification unit 307 divides the periphery of the planned self-position S' of the moving body 2 into specific ranges, and for each range, identifies the detection point P closest to the planned self-position S' of the moving body 2, or multiple detection points P in order of closest to the planned self-position S' of the moving body 2, and extracts the distance between the planned self-position S' and the nearest object (step S33). The nearest neighbor identification unit 307 outputs the measured distance d of the detection point P identified for each range (the measured distance between the planned self-position S' of the moving body 2 and the nearest object) to the reference projection surface shape selection unit 309, the scale determination unit 311, the asymptotic curve calculation unit 313, and the boundary region determination unit 317.

[0116] The reference projection plane shape selection unit 309 selects the shape of the reference projection plane 40 (step S 34 ), and outputs shape information of the selected reference projection plane 40 to the shape determination unit 315 .

[0117] The scale determination unit 311 determines the scale of the reference projection plane 40 of the shape selected by the reference projection plane shape selection unit 309 (step S ), and outputs scale information of the determined scale to the shape determination unit 315.

[0118] The asymptotic curve calculation unit 313 calculates an asymptotic curve (step S38), and outputs it to the shape determination unit 315 and the virtual viewpoint line of sight determination unit 34 as asymptotic curve information.

[0119] The shape determination unit 315 determines a projection shape, which is how to deform the shape of the reference projection plane, based on the scale information and the asymptotic curve information (step S40). The shape determination unit 315 outputs projection shape information of the determined projection shape 41 to the deformation unit 32.

[0120] The transformation unit 32 transforms the shape of the reference projection plane based on the projection shape information (step S42), and outputs the transformed transformed projection plane information to the projection conversion unit .

[0121] The virtual viewpoint line-of-sight determination unit 34 determines virtual viewpoint line-of-sight information based on the expected self-position S' and the asymptotic curve information (step S44). The virtual viewpoint line-of-sight determination unit 34 outputs the virtual viewpoint line-of-sight information indicating the virtual viewpoint O and the line-of-sight direction L to the projection transformation unit 36.

[0122] The projection conversion unit 36 ​​generates a projection image by projecting the captured image acquired from the image capturing unit 12 onto the deformed projection surface based on the deformed projection surface information and the virtual viewpoint line of sight information. The projection conversion unit 36 ​​converts the generated projection image into a virtual viewpoint image (step S46) and outputs it to the image synthesis unit 38.

[0123] The boundary area determination unit 317 determines the boundary area based on the distance from the intended self-position S' specified for each range to the nearest object. That is, the boundary area determination unit 317 determines the boundary area as an overlap area of ​​spatially adjacent peripheral images based on the position of the object nearest to the intended self-position S' of the moving object 2 (step S48). The boundary area determination unit 317 outputs the determined boundary area to the image synthesis unit 38.

[0124] The image synthesis unit 38 generates a synthesized image by joining spatially adjacent virtual viewpoint images together using a boundary region (step S50). In the boundary region, the spatially adjacent virtual viewpoint images are blended at a predetermined ratio.

[0125] The display unit 16 displays the composite image (step S52).

[0126] The information processing device 10 determines whether to end the information processing (step S54). For example, the information processing device 10 makes the determination in step S54 by determining whether or not a signal indicating completion of parking of the moving object 2 has been received from the ECU 3 or the planning processing unit 28A. Alternatively, for example, the information processing device 10 may make the determination in step S54 by determining whether or not an instruction to end the information processing has been received through an operational instruction by the user.

[0127] If the determination in step S54 is negative (step S54: No), the processes from step S2 to step S54 are repeatedly executed. On the other hand, if the determination in step S54 is positive (step S54: Yes), this routine ends.

[0128] When returning from step S54 to step S2 after performing the correction process of step S20, the subsequent correction process of step S20 may be omitted. Also, when returning from step S54 to step S2 without performing the correction process of step S20, the subsequent correction process of step S20 may be performed.

[0129] Next, the operation and effect of the information processing device 10 according to the embodiment will be described using a comparative example.

[0130] The information processing device 10 according to the embodiment includes a VSLAM processing unit 24, a behavior plan formulation unit 28, and a shape determination unit 315 that is part of a projection shape determination unit 29. The VSLAM processing unit 24 generates second information (environment map information) including position information of three-dimensional objects in the vicinity of the moving object 2 and position information of the moving object 2, based on an image of the vicinity of the moving object 2. The behavior plan formulation unit 28 generates first information including planned self-location information of the moving object 2 and position information of three-dimensional objects in the vicinity based on the planned self-location information, based on the behavior plan information of the moving object. The projection shape determination unit 29 determines the shape of a projection surface onto which an image acquired from the imaging unit 12 is projected to generate an overhead image, based on the first information.

[0131] Therefore, the information processing device 10 generates planned map information for calculating the distance to the detection point based on the planned self-position formulated by the action plan formulation unit 28, rather than the self-position obtained by VSLAM processing, and uses this to determine the shape of the projection surface for generating an overhead image.

[0132] 14 and 15 are diagrams for explaining the projection plane deformation processing executed by an information processing device according to a comparative example. Here, a case will be described in which the detection point distance information output from the distance conversion unit 27 is input to the determination unit 30 without going through the action plan formulation unit 28. FIG. 14 is a diagram showing a situation in which the moving object 2 is reversed and parked in a parking area PA located between the pillar and the car 1, as seen from above. FIG. 15 is a schematic diagram showing an example of detection point distance information based on the self-position K1 obtained by the VSLAM processing.

[0133] 14, it is assumed that the moving object 2 moves backward from position K1 to positions K2, K3, K4, and K5. In this case, the information processing device according to the comparative example starts VSLAM processing, for example, when the moving object 2 reaches position K1, and generates and displays an overhead image based on the result of projection surface deformation processing using the moving object's own position K1 as a reference.

[0134] However, by the time VSLAM processing is performed based on a peripheral image acquired when the moving object 2 is located at position K1 and an overhead image based on the results of the projection surface deformation processing based on the self-position K1 is displayed, the moving object 2 has already moved from position K1 to position K2. Therefore, when the overhead image based on the results of the projection surface deformation processing based on the self-position K1 is actually displayed, the moving object 2 is not actually located at position K1. For example, at the self-position K2, the display unit 16 displays an overhead image based on the shape of the projection surface determined based on the past self-position K1. In this way, when the overhead image is displayed, the projection surface shape is calculated using distance information calculated based on a past point in time, which can result in an unnatural overhead image.

[0135] Furthermore, when the moving object 2 reverses into a parking space along the route shown in FIG. 14, the vehicle speed between positions K1 and K5 is not constant. For example, at K1, the moving object 2 accelerates as it starts to reverse, and then reverses at a constant speed. At K2, the moving object 2 decelerates as it approaches the pillar and car 1. At K3, the turning of the moving object 2 is controlled and the moving object 2 decelerates so that the longitudinal direction of the parking position PA and the direction of the moving object 2's reverse become parallel, while preventing the moving object 2 from coming into contact with the pillar and car 1. At K4, the moving object 2 accelerates as the direction of the reverse of the moving object 2 becomes parallel to the longitudinal direction of the parking position PA. At K5, the moving object 2 decelerates to stop at the parking position PA. In this way, the vehicle speed of the moving object 2 continues to change. As a result, image fluctuation appears in the overhead image using the projection surface shape based on distance information calculated based on a past point in time. Furthermore, if the moving object 2 moves forward once due to, for example, a change in the steering wheel while the moving object 2 is traveling from position K3 to position K4, further fluctuations in the image may occur.

[0136] In contrast, the information processing device 10 according to the embodiment performs projection shape deformation based on expected self-position information formulated by the behavior plan formulation unit 28 and also used to determine actuator control values. This reduces the difference between the actual position of the moving object 2 at the time when the overhead image is displayed on the display unit 16 and the self-position of the moving object 2 in the distance information used to deform the projection surface shape of the overhead image. This reduces unnatural fluctuations in the projection surface shape. As a result, when the projection surface of the overhead image is successively deformed in accordance with three-dimensional objects around the moving object, a more natural overhead image can be provided than in the past.

[0137] Furthermore, the information processing device 10 according to the embodiment generates environmental map information including self-position information and peripheral position information by VSLAM processing using images acquired by the acquisition unit 20. The behavior plan formulation unit 28 generates planned map information based on the planned self-position of the moving object 2, based on the self-position information and peripheral position information. Therefore, planned map information based on the planned self-position of the moving object 2 can be generated with a relatively simple configuration that uses only images from the imaging unit 12.

[0138] The information processing device 10 according to the embodiment generates control values ​​for actuators such as accelerator, brake, gear, and turning, which are third information related to the control of the moving object 2, based on the behavior plan information of the moving object 2. Therefore, it is possible to link the movement control of the moving object 2 with the deformation of the projection surface based on the expected self-position of the moving object 2. As a result, it is possible to provide continuous and natural bird's-eye views of the moving object 2 as it moves.

[0139] (Variation 1) The extent to which the planned self-position in the future is used as a reference for executing the projection surface deformation process based on the action plan can be adjusted arbitrarily by changing the planned self-position used as a reference when generating the planned map information.

[0140] (Variation 2) In the above embodiment, an action plan is formulated in response to an instruction from the driver to select the automatic parking mode, and the projection surface deformation process is executed based on the action plan. However, the projection surface deformation process based on the action plan is not limited to the automatic parking mode or the automatic driving mode, and can also be used to support the driver with an overhead image in a semi-automatic driving mode, a manual driving mode, etc. Furthermore, it can be used to support the driver with an overhead image not only in reverse parking but also in parallel parking, etc.

[0141] (Second embodiment) The information processing device 10 according to the second embodiment executes projection surface deformation processing based on an action plan using not only data obtained by VSLAM processing but also data acquired from at least one external sensor. Note that, for the sake of concrete explanation, the following description will be given assuming that the information processing system 1 includes a millimeter wave radar, a sonar, and a GPS sensor as external sensors.

[0142] Fig. 16 is a schematic diagram showing an example of the functional configuration of the information processing device 10 according to the second embodiment. As shown in Fig. 16, data from external sensors such as a millimeter wave radar, a sonar, and a GPS sensor is input to an action plan formulation unit 28.

[0143] Fig. 17 is a schematic diagram showing an example of the functional configuration of the behavior plan formulation unit 28 of the information processing device 10 according to the second embodiment. As shown in Fig. 17, the behavior plan formulation unit 28 includes a surrounding situation grasping unit 28D, a planning processing unit 28A, a planned map information generating unit 28B, and a PID control unit 28C.

[0144] The surrounding situation grasping unit 28D uses data from the VSLAM processing unit 24 and data from the millimeter-wave radar, sonar, and GPS sensor to perform moving object detection processing, as well as wide-area localization processing and SLAM processing, to generate self-location information and surrounding position information with higher accuracy than in the first embodiment. Note that the self-location information and surrounding position information include distance information obtained by converting the distance between the moving object 2 and a three-dimensional object around the moving object 2 into, for example, meters. Also, the wide-area localization processing refers to processing that uses data acquired from, for example, a GPS sensor to acquire self-location information of the moving object 2 over a wider range than the self-location information acquired by VSLAM processing.

[0145] The planning processing unit 28A executes planning processing based on the self-position information and peripheral position information from the surrounding situation grasping unit 28D. Here, the planning processing executed by the planning processing unit 28A includes parking route planning processing, wide-area route planning processing, expected self-position calculation processing according to the route plan, and actuator target value calculation processing. The wide-area route planning processing is a route plan for when the mobile object 2 travels on roads or the like and moves through a wide area.

[0146] The planned map information generating unit 28B generates planned map information using the surrounding position information generated by the surrounding situation grasping unit 28D and the planned self-position information formulated by the planning processing unit 28A, and sends it to the determining unit 30.

[0147] The PID control unit 28C performs PID control based on the actuator target values ​​formulated by the planning processing unit 28A, and outputs actuator control values ​​for controlling actuators such as the accelerator and turning angle.

[0148] The information processing device 10 according to the second embodiment described above uses not only data obtained by VSLAM processing but also data from millimeter-wave radar, sonar, and GPS sensors to grasp the surrounding situation with higher accuracy and executes projection surface deformation processing based on an action plan. Therefore, in addition to the information processing device 10 according to the first embodiment, it is possible to realize driving assistance using even more accurate overhead images.

[0149] (Third embodiment) An information processing device 10 according to the third embodiment performs projection surface deformation processing based on an action plan using images acquired by the image capturing unit 12 and data acquired from at least one external sensor. For the sake of concrete explanation, the following description will be given assuming that the information processing system 1 includes external sensors such as LiDAR, millimeter-wave radar, sonar, and a GPS sensor. Alternatively, the information processing device 10 may perform projection surface deformation processing based only on the images acquired by the image capturing unit 12 and the action plan.

[0150] Fig. 18 is a schematic diagram showing an example of the functional configuration of an information processing device 10 according to the third embodiment. As shown in Fig. 18, an image acquired by the photographing unit 12 is input to a behavior plan formulation unit 28 via an acquisition unit 20. In addition, data from external sensors such as LiDAR, millimeter wave radar, sonar, and GPS sensors is input to the behavior plan formulation unit 28.

[0151] Fig. 19 is a schematic diagram showing an example of the functional configuration of the behavior plan formulation unit 28 of the information processing device 10 according to the third embodiment. As shown in Fig. 19, the behavior plan formulation unit 28 includes a surrounding situation grasping unit 28D, a planning processing unit 28A, a planned map information generating unit 28B, and a PID control unit 28C.

[0152] The surrounding situation grasping unit 28D performs moving object detection processing, localization processing, and SLAM processing (including VSLAM processing) using the images acquired by the imaging unit 12 and data from the LiDAR, millimeter wave radar, sonar, and GPS sensor, and generates self-position information and surrounding position information. Note that the self-position information and surrounding position information include distance information obtained by converting the distance between the moving object 2 and a three-dimensional object around the moving object 2 into, for example, meters.

[0153] The planning processing unit 28A executes planning processing based on the self-position information and peripheral position information from the surrounding situation grasping unit 28D. Here, the planning processing executed by the planning processing unit 28A includes parking route planning processing, wide-area route planning processing, expected self-position calculation processing according to the route plan, and actuator target value calculation processing.

[0154] The planned map information generating unit 28B generates planned map information using the surrounding position information generated by the surrounding situation grasping unit 28D and the planned self-position information formulated by the planning processing unit 28A, and sends it to the determining unit 30.

[0155] The PID control unit 28C performs PID control based on the actuator target values ​​formulated by the planning processing unit 28A, and outputs actuator control values ​​for controlling actuators such as the accelerator and turning angle.

[0156] The information processing device 10 according to the third embodiment described above uses the images acquired by the image capturing unit 12 and data from the LiDAR, millimeter-wave radar, sonar, and GPS sensors to grasp the surrounding situation with higher accuracy and executes projection surface deformation processing based on an action plan. Therefore, in addition to the information processing device 10 according to the first embodiment, it is possible to realize driving assistance using even more accurate overhead images.

[0157] Although the embodiments and modifications have been described above, the information processing device, information processing method, and information processing program disclosed herein are not limited to the above-described embodiments, and the components can be modified and embodied in each implementation stage without departing from the spirit of the invention. Furthermore, various inventions can be created by appropriately combining multiple components disclosed in the above-described embodiments and modifications. For example, some components may be deleted from all of the components shown in the embodiments.

[0158] The information processing device 10 of the above embodiment and each modified example can be applied to various devices. For example, the information processing device 10 of the above embodiment and each modified example can be applied to a surveillance camera system that processes images obtained from a surveillance camera, or an in-vehicle system that processes images of the surrounding environment outside the vehicle. [Explanation of symbols]

[0159] 10. Information processing equipment 12, 12A~12D Photography Department 14 Detector 20 Acquisition Department 21 Selection section 24 VSLAM processing unit 27 Distance conversion section 28 Action Planning Department 28A Planning Processing Section 28B Planned map information generation unit 28C PID control unit 28D Surrounding Situation Assessment Department 29 Projection shape determining section 30 Decision Section 32 Deformed part 34 Virtual viewpoint line of sight determination unit 36 Projection transformation unit 37 Image generation unit 38 Image synthesis unit 240 Matching Department 241 Storage section 241A Environmental Map Information 242 Self-position estimation part 243 3D Reconstruction Unit 244 Correction Unit 305 Extraction part 307 Nearest neighbor identification part 309 Reference projection surface shape selection section 311 Scale determination unit 313 Asymptotic curve calculation part 315 Shape determination unit

Claims

1. a behavior plan formulation unit that generates first information based on behavior plan information of the moving object, the first information including planned self-location information indicating a planned self-location of the moving object along a route of the moving object and position information of a surrounding three-dimensional object based on the planned self-location information; a projection shape determination unit that determines a shape of a projection surface onto which a first image acquired by an image capturing device mounted on the moving object is projected to generate an overhead image, based on the first information; An image processing device comprising:

2. the behavior plan formulation unit generates the first information based on second information including position information of a three-dimensional object in the vicinity of the moving object and position information of the moving object, and the behavior plan information; The image processing device according to claim 1 .

3. The second information is information generated by VSLAM processing using a second image of the periphery of the moving object. The image processing device according to claim 2 .

4. The first image is a different image from the second image. The image processing device according to claim 3 .

5. The second information is information generated by SLAM processing using data acquired from at least one external sensor.

5. The image processing device according to claim 2.

6. a control information generating unit configured to generate third information related to control of the moving object based on the action plan information of the moving object and the second information; The moving object is controlled based on the third information. The image processing device according to any one of claims 2 to 5.

7. the projection shape determination unit determines a shape of the projection plane based on distance information between position information of the peripheral three-dimensional object and the expected self-position. The image processing device according to any one of claims 1 to 6.

8. the projection shape determination unit determines the shape of the projection plane based on the peripheral three-dimensional object at a position closest to the expected self-position of the moving object. The image processing device according to claim 7 .

9. 1. A computer-implemented image processing method comprising: generating first information including predetermined self-position information indicating a predetermined self-position of the moving body on a route of the moving body and position information of a surrounding three-dimensional object based on the predetermined self-position information, based on behavior plan information of the moving body; determining a shape of a projection surface onto which a first image acquired by an image capturing device mounted on the moving body is projected to generate an overhead image, based on the first information; An image processing method comprising:

10. On the computer, generating first information including predetermined self-location information indicating a predetermined self-location of the moving object on a route based on behavior plan information of the moving object and position information of a surrounding three-dimensional object based on the predetermined self-location information; determining, based on the first information, the shape of a projection surface onto which a first image acquired by an image capturing device mounted on the moving object is projected to generate an overhead image; An image processing program for executing the above.

Images