Information processing device, information processing method, and information processing program

Abstract

An information processing device (10) comprises, in one aspect, a map information generation unit (22). With regard to map information (23B) including first surrounding position information, which is information on the position of an object positioned in the surroundings of a moving body (2), the map information generation unit (22) initializes a range pertaining to detection by a sensor mounted on the moving body (2) and adds second surrounding position information acquired from the sensor to the range.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and an information processing program.

Background Art

[0002] There is a technique for acquiring position information such as the positional relationship between a moving object and an object around the moving object using a sensor such as a sonar. Further, there is a technique for acquiring (estimating) position information indicating the position of the moving object and the position of the object around the moving object by performing Visual SLAM (Simultaneous Localization and Mapping) processing using the acquired image around the moving object. Further, there is a technique for deforming the shape of a projection plane for generating an aerial view image around the moving object using the acquired position information (map information) such as the position of the moving object and the position of the object around the moving object.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Patent Document 3

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, regarding the acquisition of map information, for example, when a moving object (such as a pedestrian) around a moving object is detected by a sonar, the position information of the moving object is accumulated in accordance with the movement of the moving object. At this time, the position information is accumulated as if there is a three-dimensional object such as a wall along the trajectory of the moving object. When the position information is accumulated as if there is a three-dimensional object, the shape of the projection plane may not be appropriately deformed.

[0005] In one aspect, the present invention aims to provide an information processing device, an information processing method, and an information processing program capable of generating highly accurate map information. [Means for solving the problem]

[0006] In one embodiment, the information processing device disclosed in this application includes a map information generation unit. The map information generation unit initializes a range for detection by a sensor mounted on the moving body in map information which includes first surrounding position information which is information on the positions of objects located around the moving body, and adds second surrounding position information obtained from the sensor to the range. [Effects of the Invention]

[0007] According to one embodiment of the information processing device disclosed in this application, it is possible to generate highly accurate map information. Thus, according to one embodiment of the information processing device disclosed in this application, for example, the shape of the projection surface can be appropriately deformed using the highly accurate map information. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 shows an example of the overall configuration of an information processing system according to the embodiment. [Figure 2] Figure 2 shows an example of the hardware configuration of the information processing device according to the present invention. [Figure 3] Figure 3 shows an example of the functional configuration of an information processing device according to the embodiment. [Figure 4] Figure 4 is a flowchart showing an example of the procedure for map information generation according to the embodiment. [Figure 5] Figure 5 is a schematic diagram showing an example of map information according to the embodiment. [Figure 6] Figure 6 is a diagram illustrating an example of map information when the moving object moves along the direction of travel from the position shown in Figure 5, according to an embodiment. [Figure 7]Figure 7 is a diagram illustrating an example of map information when the moving object moves along the direction of travel from the position shown in Figure 6, according to an embodiment. [Figure 8] Figure 8 shows an example of map information when a moving object is reverse-parked between two vehicles, according to an embodiment. [Figure 9] Figure 9 shows an example of map information in reverse parking, which differs from that in Figure 8, relating to an embodiment. [Figure 10] Figure 10 is a diagram relating to an embodiment, showing an example of map information in which the moving object has moved along the backward direction from its position in Figure 9. [Figure 11] Figure 11 is a schematic diagram showing an example of a reference projection plane according to the embodiment. [Figure 12] Figure 12 is a schematic diagram showing an example of a projected shape according to the embodiment. [Figure 13] Figure 13 is an explanatory diagram of the asymptotic curve according to the embodiment. [Figure 14] Figure 14 is a schematic diagram showing an example of the configuration of the determination unit according to the embodiment. [Figure 15] Figure 15 is a schematic diagram showing an example of map information according to the embodiment. [Figure 16] Figure 16 is a flowchart illustrating an example of the image processing procedure performed by the information processing device according to the embodiment. [Modes for carrying out the invention]

[0009] Hereinafter, embodiments of the information processing apparatus, information processing method, and information processing program disclosed in this application will be described in detail with reference to the attached drawings. Note that the following embodiments are not intended to limit the disclosed technology. Furthermore, each embodiment can be appropriately combined as long as the processing content is not inconsistent.

[0010] FIG. 1 is a diagram showing an example of the overall configuration of the information processing system 1 of the present embodiment. The information processing system 1 includes an information processing apparatus 10, a photographing unit 12, a detection unit 14, and a display unit 16. The information processing apparatus 10, the photographing unit 12, the detection unit 14, and the display unit 16 are connected so as to be able to exchange data or signals.

[0011] In the present embodiment, the information processing apparatus 10, the photographing unit 12, the detection unit 14, and the display unit 16 will be described by taking as an example a form in which they are mounted on the moving body 2.

[0012] The moving body 2 is an object that can move. The moving body 2 is, for example, a vehicle, a flyable object (a manned aircraft, an unmanned aircraft (e.g., a UAV (Unmanned Aerial Vehicle), a drone)), a robot, a ship, or the like. Also, the moving body 2 is, for example, a moving body that advances through a driving operation by a person or a moving body that can automatically advance (autonomous advancement) without a driving operation by a person. In the present embodiment, a case where the moving body 2 is a vehicle will be described as an example. The vehicle is, for example, a two-wheeled vehicle, a three-wheeled vehicle, a four-wheeled vehicle, or the like. In the present embodiment, a case where the vehicle is a four-wheeled vehicle will be described as an example.

[0013] Note that not all of the information processing apparatus 10, the photographing unit 12, the detection unit 14, and the display unit 16 are limited to a form in which they are mounted on the moving body 2. The information processing apparatus 10 may be mounted on a stationary object. The stationary object is, for example, an object fixed to the ground. Specifically, the stationary object is an object that cannot move or an object in a state of being stationary with respect to the ground. Also, the information processing apparatus 10 may be mounted on a cloud server that executes processing on the cloud.

[0014] The imaging unit 12 photographs the area around the moving object 2 and acquires captured image data. Hereafter, the captured image data will simply be referred to as the captured image. The imaging unit 12 is, for example, a digital camera capable of shooting video. Note that "shooting" refers to converting the image of a subject formed by an optical system such as a lens into an electrical signal. The imaging unit 12 outputs the captured image to the information processing device 10. In this embodiment, the imaging unit 12 is assumed to be a monocular fisheye camera (for example, with a field of view of 195 degrees).

[0015] In this embodiment, a configuration in which four imaging units 12 are mounted on a mobile body 2—a forward imaging unit 12A, a left imaging unit 12B, a right imaging unit 12C, and a rear imaging unit 12D—will be described as an example. Each of the multiple imaging units 12 (forward imaging unit 12A, left imaging unit 12B, right imaging unit 12C, and rear imaging unit 12D) captures a subject in a different imaging area E (forward imaging area E1, left imaging area E2, right imaging area E3, and rear imaging area E4) and acquires an image. That is, the multiple imaging units 12 have different imaging directions. Furthermore, the imaging directions of these multiple imaging units 12 are pre-adjusted so that at least a portion of the imaging area E overlaps with adjacent imaging units 12. Also, in Figure 1, for the sake of explanation, the imaging area E is shown in the size shown in Figure 1, but in reality it will include an area further away from the mobile body 2.

[0016] Furthermore, the four forward-facing units 12A, left-facing unit 12B, right-facing unit 12C, and rear-facing unit 12D are just examples, and there is no limit to the number of units 12. For example, if the mobile body 2 has a long, narrow shape like a bus or truck, one unit 12 can be placed at the front, rear, front of the right side, rear of the right side, front of the left side, and rear of the left side of the mobile body 2, for a total of six units 12. In other words, the number and placement of the units 12 can be arbitrarily set depending on the size and shape of the mobile body 2.

[0017] The detection unit 14 detects the positional information of each of the multiple detection points around the moving object 2. In other words, the detection unit 14 detects the positional information of each of the detection points in the detection area DA. A detection point refers to each of the points in real space that are individually observed by the detection unit 14. The detection points correspond to the positions of, for example, three-dimensional objects around the moving object 2.

[0018] The position information of a detection point refers to information indicating the position of the detection point in real space (three-dimensional space). For example, the position information of a detection point includes the distance from the detection unit 14 (i.e., the position of the moving body 2) to the detection point, and the direction of the detection point relative to the detection unit 14. These distances and directions can be represented, for example, by position coordinates indicating the relative position of the detection point relative to the detection unit 14, by position coordinates indicating the absolute position of the detection point, or by a vector.

[0019] The detection unit 14 may be, for example, a 3D (Three-Dimensional) scanner, a 2D (Two-Dimensional) scanner, a distance sensor (millimeter-wave radar, laser sensor), a sonar sensor that detects objects using sound waves, or an ultrasonic sensor. The laser sensor may be, for example, a three-dimensional LiDAR (Laser imaging Detection and Ranging) sensor. The detection unit 14 may also be a device using technologies such as a stereo camera or motion stereo, which measures distance from images captured by a monocular camera, such as SfM (Structure from Motion) technology. Alternatively, multiple imaging units 12 may be used as the detection unit 14. Alternatively, one of the multiple imaging units 12 may be used as the detection unit 14.

[0020] Furthermore, in this embodiment, the detection unit 14 is described assuming it is a sonar sensor, but it is not limited to this, and various known sensors for distance measurement can be used as the detection unit 14. The detection unit 14 may also be a distance sensor mounted on the rear of the moving body 2. The rear of the moving body 2 corresponds, for example, to the direction opposite to the reference direction on the moving body 2. If the moving body 2 is a vehicle, the reference direction corresponds to the direction in front of the driver (forward direction). In addition, the distance sensor as the detection unit 14 may be further arranged on the side of the moving body 2. Moreover, the multiple sensors are not limited to the above description, and can be mounted at any location on the moving body 2 as long as they can detect the area around the moving body 2. If the multiple sensors are sensors that can only measure distance, the detection unit 14 may calculate the position information of a planar object by triangulation using the multiple distance information output from the multiple sensors.

[0021] Furthermore, in this embodiment, multiple detection units 14 are mounted on the mobile body 2. In this case, the multiple sensors (14A, 14B, 14C, 14D) are arranged in an array on the exterior of the mobile body 2, for example, as shown in Figure 1. In this embodiment, a configuration in which four detection units 14 are mounted on the mobile body 2—a left rear detection unit 14A, a rear left detection unit 14B, a rear right detection unit 14C, and a right rear detection unit 14D—will be described as an example.

[0022] Multiple detection units 14 (left rear detection unit 14A, rear left detection unit 14B, rear right detection unit 14C, right rear detection unit 14D) each detect the positional information of multiple detection points included in detection areas DA (left rear detection area DA1, rear left detection area DA2, rear right detection area DA3, right rear detection area DA4) in different directions. That is, the range for detecting objects (hereinafter referred to as the detection range) of the multiple detection units 14 is different from one another. Furthermore, the detection directions of these multiple detection units 14 are pre-adjusted so that at least a portion of the detection area DA overlaps with adjacent detection units 14. Also, in Figure 1, for the sake of explanation, the detection area DA is shown with the size shown in Figure 1, but in reality it may include an area further away from the moving object 2. In addition, the detection range shown in Figure 1 is the range encompassed by multiple detection areas (DA1 to DA4), but for the sake of simplicity of explanation below, it will be assumed to be in a fan shape.

[0023] Furthermore, the four left rear detection units 14A, rear left detection unit 14B, rear right detection unit 14C, and right rear detection unit 14D are just examples, and there is no limit to the number of detection units 14. For example, if the mobile body 2 has a vertically elongated shape like a bus or truck, an additional detection unit 14 can be placed on the front, the front of the right side, and the front of the left side of the mobile body 2, for a total of seven detection units 14. In other words, the number and placement of the detection units 14 can be arbitrarily set depending on the size and shape of the mobile body 2.

[0024] The display unit 16 displays various types of information. The display unit 16 is, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display.

[0025] In this embodiment, the information processing device 10 is communicatively connected to an electronic control unit (ECU) 3 mounted on the mobile body 2. The ECU 3 is a unit that electronically controls the mobile body 2. In this embodiment, the information processing device 10 is capable of receiving CAN (Controller Area Network) data such as the speed and direction of movement of the mobile body 2 from the ECU 3.

[0026] Next, the hardware configuration of the information processing device 10 will be described.

[0027] Figure 2 shows an example of the hardware configuration of the information processing device 10.

[0028] The information processing device 10 includes a CPU (Central Processing Unit) 10A, a ROM (Read Only Memory) 10B, a RAM (Random Access Memory) 10C, and an I / F (Interface) 10D, and is, for example, a computer. The CPU 10A, ROM 10B, RAM 10C, and I / F 10D are interconnected by a bus 10E, resulting in a hardware configuration that utilizes a typical computer.

[0029] The CPU 10A is an arithmetic unit that controls the information processing device 10. The CPU 10A corresponds to an example of a hardware processor. The ROM 10B stores programs and other data that implement various processes performed by the CPU 10A. The RAM 10C stores data necessary for various processes performed by the CPU 10A. The I / F 10D is an interface for sending and receiving data to and from the imaging unit 12, detection unit 14, display unit 16, and ECU 3, etc.

[0030] The program for executing the information processing performed by the information processing device 10 of this embodiment is provided pre-installed in a ROM 10B or the like. Alternatively, the program executed by the information processing device 10 of this embodiment may be provided as a file in a format installable or executable by the information processing device 10, recorded on a recording medium. The recording medium is a medium readable by a computer. Examples of recording media include CD (Compact Disc)-ROM, flexible disk (FD), CD-R (Recordable), DVD (Digital Versatile Disk), USB (Universal Serial Bus) memory, and SD (Secure Digital) card.

[0031] Next, the functional configuration of the information processing device 10 according to this embodiment will be described. The information processing device 10 initializes the detection range DA related to detection by the sensor (detection unit 14) mounted on the mobile body 2 in map information which includes first surrounding position information which is information on the positions of objects located around the mobile body, and adds the position information of the detection point (second surrounding position information) acquired from the sensor to the detection range DA. The information processing device 10 connects a plurality of spatially adjacent captured images to generate and display a composite image (overhead view image) that provides an overhead view of the area around the mobile body 2.

[0032] Figure 3 shows an example of the functional configuration of the information processing device 10. In addition to the information processing device 10, Figure 3 also shows the imaging unit 12, detection unit 14, and display unit 16, etc., in order to clarify the data input / output relationships.

[0033] The information processing device 10 comprises an acquisition unit 20, a map information generation unit 22, a determination unit 30, a transformation unit 32, a virtual viewpoint line-of-sight determination unit 34, a projection transformation unit 36, and an image synthesis unit 38.

[0034] Some or all of the above-mentioned parts may be implemented by having a processing unit such as a CPU 10A execute a program, that is, by software. Alternatively, some or all of the above-mentioned parts may be implemented by hardware such as an IC (Integrated Circuit), or by using a combination of software and hardware.

[0035] The acquisition unit 20 acquires captured images from the imaging unit 12. For example, the acquisition unit 20 acquires captured images from the front imaging unit 12A, the left imaging unit 12B, the right imaging unit 12C, and the rear imaging unit 12D. Each time the acquisition unit 20 acquires a captured image, it outputs the acquired image to the projection conversion unit 36.

[0036] The acquisition unit 20 acquires CAN data such as travel distance and turning angle from the moving body 2. Each time the acquisition unit 20 acquires CAN data, it outputs the acquired CAN data to the map information generation unit 22. Specifically, the acquisition unit 20 outputs the acquired CAN data to the self-position estimation unit 221.

[0037] The acquisition unit 20 acquires location information of the detection points from the detection unit 14. For example, the acquisition unit 20 acquires location information (second peripheral location information) of the detection points from each of the left rear detection unit 14A, the rear left detection unit 14B, the rear right detection unit 14C, and the right rear detection unit 14D. The second peripheral location information corresponds to the location information of objects located within the detection range around the moving object 2. Each time the acquisition unit 20 acquires location information for each of the multiple detection points, it outputs the acquired location information to the map information generation unit 22. Specifically, the acquisition unit 20 outputs the acquired location information to the second offset adjustment unit 227.

[0038] The map information generation unit 22 includes a self-position estimation unit 221, a first offset adjustment unit 223, an initialization unit 225, a second offset adjustment unit 227, an addition unit 229, a storage unit 231, and a correction unit 233. In this embodiment, the correction unit 233 may be omitted as a modification. The map information generation unit 22 initializes the range for detection by sensors mounted on the mobile body 2 in map information including first surrounding position information, which is information on the positions of objects located around the mobile body, and adds second surrounding position information acquired from the sensors to this range. The components of the map information generation unit 22 will be described below.

[0039] The self-position estimation unit 221 estimates the position of the mobile body 2 in the global coordinate system of the map information, for example, by odometry based on CAN data. Odometry is a method that calculates the amount of movement of each wheel by performing calculations on CAN data such as the rotation angle of the wheels and the rotation angle of the steering wheel of the mobile body 2, and estimates the position of the mobile body 2 (self-position information) from the cumulative calculation. The self-position estimation unit 221 estimates the movement vector of the mobile body 2 (also called relative movement amount information) by odometry based on CAN data, for example, based on the position of the mobile body 2 at the position information acquisition interval (i.e., the time interval between the acquisition of temporally adjacent position information).

[0040] Furthermore, if a LiDAR sensor is used in the detection unit 14, the self-position estimation unit 221 may estimate the position of the moving object 2 using LiDAR SLAM. Also, if a depth camera such as a ToF (Time-of-Flight) sensor is used in the detection unit 14, the self-position estimation unit 221 may estimate the position of the moving object 2 using, for example, Depth SLAM.

[0041] The first offset adjustment unit 223 reads detection range information 23A related to the detection unit 14 from the storage unit 231. The detection range information 23A is, for example, information indicating the range (detection range) of detection of three-dimensional objects by the detection unit 14. The detection range information 23A is set in advance by inspections based on the performance (specifications: horizontal angle, maximum measurement distance, etc.) of various sensors used as the detection unit 14, and the mounting angle of the detection unit 14 to the moving body 2, and is stored in the storage unit 231. The detection range may be the entire area in which the detection unit 14 can detect an object, or it may be a part of the object detection range of the detection unit 14 that is close to the moving body 2. In addition, the detection range information 23A may be adjusted as appropriate based on, for example, the speed information of the moving body 2 based on CAN data.

[0042] The first offset adjustment unit 223 adjusts the offset of the detection range of the detection point by the detection unit 14 in the global coordinate system based on the detection range information 23A and the movement vector related to the detection unit 14. The offset of the detection range of the detection unit 14 is, for example, the deviation of the detection range from the origin of the global coordinate system and the direction of the detection range. The first offset adjustment unit 223 outputs the adjusted first offset as first coordinate information to the initialization unit 225.

[0043] The initialization unit 225 reads map information 23B from the storage unit 231. Map information 23B is, for example, information that geographically shows the surrounding conditions of the mobile body 2. Map information 23B is information in which a point cloud, which is the position information of each detected point, is registered in a three-dimensional coordinate space (global coordinate system) with a predetermined position in real space as the origin (reference position). The map information 23B may also contain the self-position information of the mobile body 2. The information (point cloud) of the detected points registered in map information 23B corresponds to the first surrounding position information. That is, the first surrounding position information is information about the positions of objects located around the mobile body 2. The first surrounding position information (and the second surrounding position information) may be obtained, for example, from a detection unit 14 implemented by a distance sensor mounted on the rear of the mobile body 2.

[0044] The initialization unit 225 identifies the detection range of the detection unit 14 in the map information 23B based on the map information 23B and the first coordinate information. The update range targeted for initialization by the initialization unit 225 corresponds to the detection range of the detection unit 14 in the map information 23B based on the position (self-position information) of the moving object 2. The initialization unit 225 initializes (resets) the update range in the map information 23B. For example, as initialization of the update range, the initialization unit 225 deletes the surrounding position information included in the update range from the map information 23B among the first surrounding position information. At this time, since the initialization unit 225 deletes the information included in the update range, it may also be called a deletion unit. The initialization unit 225 may also pad the data in the update range with zeros as initialization of the update range (zero padding, zero reset). Furthermore, the initialization unit 225 may determine and delete surrounding location information to be deleted as part of the initialization (surrounding location information to be deleted) based on the detection range information (detection range information 23A) and the relative amount of movement of the moving object 2 with respect to the origin in the map information 23B (movement vector information). The initialization unit 225 outputs the map information after initialization (hereinafter referred to as the map information after initialization) to the addition unit 229.

[0045] The second offset adjustment unit 227 adjusts the offset of the detected point's position in the global coordinate system based on the position information of the detected point (e.g., the detection distance in each direction for each sonar) and the movement vector. The offset of the detected point's position refers to, for example, the deviation of the detected point's position from the origin of the global coordinate system and the orientation of the detected point. The second offset adjustment unit 227 outputs the adjusted offset (second offset) as second coordinate information to the addition unit 229.

[0046] The additional unit 229 identifies the detection range of the detection unit 14 in the initialized map information based on the initialized map information and the second coordinate information. If the moving object 2 is stopped, the detection range identified by the initialization unit 225 is applied to the initialized map information, and the identification of the detection range by the additional unit 229 may be omitted. The additional unit 229 adds the second surrounding position information acquired by the sensor (detection unit 14) to the detection range in the map information 23B. That is, the additional unit 229 registers the second surrounding position information in the detection range in the initialized map information. This enables the updating of the map information 23B. The additional unit 229 may also update the self-position of the moving object 2 in the initialized map information based on the movement vector output from the self-position estimation unit 221. The additional unit 229 stores the updated map information as new map information in the storage unit 231.

[0047] As a result, new second surrounding location information is sequentially added to the map information 23B as the second surrounding location information is acquired from the detection unit 14 mounted on the mobile body 2. For example, when the mobile body 2 is moving, the map information 23B is sequentially updated in accordance with the movement of the mobile body 2.

[0048] The storage unit 231 stores various types of data. The storage unit 26 is, for example, a semiconductor memory element such as RAM or flash memory, a hard disk, or an optical disk. The storage unit 231 may also be a storage device located outside the information processing device 10. Furthermore, the storage unit 231 may be a storage medium. Specifically, the storage medium may be a medium on which programs and various types of information have been downloaded and stored or temporarily stored via a LAN (Local Area Network) or the Internet.

[0049] The correction unit 233 corrects the position information of each of the multiple detection points registered in the map information 23B, and the self-position information of the moving object 2. The correction unit 233 corrects the position information and self-position information registered in the map information 23B using the position information of each of the corresponding detection points P, which was acquired at an acquisition time later than the acquisition time of the detection point.

[0050] For example, the correction unit 233 corrects the location information of the detected point (first surrounding location information) registered in the map information 23B using the location information of the corresponding detected point (second surrounding location information) detected again by the detection unit 14. At this time, the correction unit 233 may further correct the first surrounding location information and the self-location information using various parameters used to calculate the location information of each detected point P, which are registered in the map information 23B. Through this correction process, the correction unit 233 corrects the error in the first surrounding location information of the detected point registered in the map information 23B. The correction unit 233 may correct the first surrounding location information of the detected point P registered in the map information 23B using, for example, the least squares method. The cumulative error in the first surrounding location information of the detected point P is corrected by the correction process by the correction unit 233. Note that the information processing device 10 may be configured without the correction unit 233.

[0051] The following is an example of a case where a process to generate map information 23B (hereinafter referred to as the map information generation process) is performed at a predetermined time, such as when the mobile object 2 is parking. For example, the information processing device 10 determines that the predetermined time has been reached when it determines that the behavior of the mobile object 2 has become a behavior indicating a parking scene. Behavior indicating a parking scene by reversing includes, for example, when the speed of the mobile object 2 falls below a predetermined speed, when the gear of the mobile object 2 is put into reverse gear, or when a signal indicating the start of parking is received by the user's operation instruction. Note that the predetermined time is not limited to parking scenes.

[0052] Figure 4 is a flowchart illustrating an example of the map information generation process. The map information generation process will be explained using Figures 4 through 10.

[0053] (Map information generation process) As the mobile body 2 moves, the acquisition unit 20 acquires CAN data from the mobile body 2. The self-position estimation unit 221 estimates the self-position information of the mobile body 2 using an odometry method based on the CAN data. Relative movement amount information is acquired, for example, by calculating a movement vector using an odometry method based on CAN data such as the speed and rudder angle of the mobile body 2. As a result, relative movement amount information (movement vector) of the mobile body 2 is acquired based on the CAN data (step S1).

[0054] The first offset adjustment unit 223 reads detection range information 23A from the storage unit 231. The first offset adjustment unit 223 adjusts the first offset of the detection range based on the detection range information 23A and the relative movement amount information. The initialization unit 225 identifies the detection range in the map information 23B based on the adjusted first offset (first coordinate information) and the map information 23B. As a result, the detection range in the map information 23B, including the first surrounding position information, is identified based on the relative movement amount information and the detection range information 23A (step S2). The identified detection range is information indicating a predetermined range around the moving object relative to the moving object 2, and corresponds to the update range that is the target of initialization by the initialization unit 225. That is, the detection range corresponding to the update range corresponds to the information around the moving object based on the position of the moving object 2 (self-position information).

[0055] The initialization unit 225 initializes the information of the detection points included in the update range in the map information 23B. For example, as initialization of the information of the detection points included in the update range, the initialization unit 225 deletes the surrounding location information included in the detection range from the first surrounding location information from the map information 23B that includes the first surrounding location information. As a result, the map information 23B that includes the first surrounding location information is initialized in the update range (step S3). That is, through the processing in steps S2 and S3, the surrounding location information located in the update range from the first surrounding location information already stored in the map information 23B is deleted using the relative movement amount information and the update range. As a result, initialized map information is generated in which the data of the update range has been initialized.

[0056] The acquisition unit 20 acquires location information of the detection point (second surrounding location information) from the detection unit 14 as the moving body 2 moves (step S4). When the moving body 2 is stationary, the acquisition unit 20 acquires location information of the detection point (second surrounding location information) from the detection unit 14, for example, at the above time intervals.

[0057] The second offset adjustment unit 227 adjusts the second offset of the detection point's position based on the second surrounding position information and the relative movement amount. The addition unit 229 identifies the detection range of the detection unit 14 in the initialized map information based on the adjusted second offset (second coordinate information) and the initialized map information. The addition unit 229 adds the second surrounding position information to the detection range in the map information 23B. As a result, the second surrounding position information acquired using the relative movement amount information is added to the map information 23B. As a result, the map information 23B is updated (step S5).

[0058] The additional unit 229 outputs the updated map information 23B to the determination unit 30 and the storage unit 231 (step S6). The output destination of the updated map information 23B is not limited to the above, and it may also be output to various components that utilize the updated map information 23B.

[0059] If the map information generation process is not completed (No in step S7), the processes from step S1 onwards are repeated. That is, when the information processing device 10 determines that the map information generation process should continue (No in step S7), the deletion of surrounding location information within the updated range and the addition of newly acquired surrounding location information (second surrounding location information) from the map information 23B are repeatedly performed. When the map information generation process is completed (Yes in step S7), the procedure for map information generation is completed. The determination of completion of the map information generation process is, for example, by stopping the power supply to the mobile body 2 (for example, if the mobile body 2 is a vehicle, by stopping the operating power supply). Note that the determination of completion of the map information generation process is not limited to stopping the power supply to the mobile body 2, and can be set arbitrarily.

[0060] Figure 5 is a schematic diagram showing an example of map information 23B. For the sake of explanation, the map information 23B shown in Figure 5 shows the moving object 2, the direction of travel TD of the moving object 2, the detection range DA by the detection unit 14, and the string "car" indicating another car. As shown in Figure 5, map information 23B is information in which point cloud information, which is the position information (first surrounding position information) of each detection point P, is registered at the corresponding coordinate positions in the three-dimensional coordinate space (global coordinate system) as the moving object 2 moves.

[0061] Figure 6 shows an example of map information 23B when moving along the direction of travel TD from the position of the moving object 2 shown in Figure 5. For the sake of explanation, the map information 23B shown in Figure 6 shows the moving object 2, the direction of travel TD of the moving object 2, the detection range DA by the detection unit 14, and the string "car" indicating another car.

[0062] Furthermore, as shown in Figure 6, the detection point BP detected in Figure 5 is initially deleted from the map information 23B by initialization by the initialization unit 225, but is added to the map information 23B by the addition unit 229 along with other detection points (white circles shown in Figure 5). As shown in Figure 6, the detection unit 14 (multiple sensors) installed behind the mobile body 2 acquires the position information of objects (cars) around the mobile body 2, and the acquired position information is stored in the map information 23B as first surrounding position information.

[0063] Figure 7 shows an example of map information 23B when moving along the direction of travel TD from the position of the moving object 2 shown in Figure 6. For the sake of explanation, the map information 23B shown in Figure 7 shows the moving object 2, the direction of travel TD of the moving object 2, the detection range DA by the detection unit 14, and the string "car" indicating other cars. As shown in Figure 7, even when the position information (detection point) PP goes outside the detection range DA, it is updated based on the information from the odometry method using CAN data, and is therefore stored in the map information 23B as first surrounding position information.

[0064] Figures 5 to 7 illustrate an example using a detection unit 14 (sensor) mounted on the rear of the mobile body 2, but the embodiment is not limited to this. For example, a detection unit 14 (sensor) installed in front of or to the side of the mobile body P vehicle may be used to acquire positional information of objects around the mobile body 2 when the mobile body 2 is moving forward.

[0065] Figure 8 shows an example of map information 23B when the moving body 2 is moved from its position shown in Figure 7 to the left side relative to the direction of travel TD, and then reverse-parked between two vehicles (car1 and car2). For the sake of explanation, Figure 8 shows the pedestrian PDS, as a moving object, traversing the rear of the moving body 2 from the rear right to the left side.

[0066] In other words, for the sake of explanation, the map information 23B in Figure 8 shows, for the sake of explanation, a moving object 2, a pedestrian PDS, the direction of movement DM of the pedestrian PDS, the detection range DA by the detection unit 14, and the string car(car1, car2, car) indicating other vehicles.

[0067] As shown in Figure 8, the pedestrian PDS moves along the direction of movement DM. If the map information 23B in the update range is not initialized at this time, detection points will accumulate along with the movement of the pedestrian PDS. That is, detection points will exist at positions after the pedestrian PDS has passed, and the information of detection points accumulated in the map information 23B will differ from the actual positions of pedestrian PDS around the moving object 2. On the other hand, in this embodiment, the detection points of pedestrian PDS are sequentially deleted by the initialization unit 225 and are not accumulated in the map information 23B. That is, as shown in Figure 8, the detection range DA is always updated to the latest state, so past detection points of pedestrian PDS do not remain in the map information 23B. Therefore, the difference between the information of detection points accumulated in the map information 23B and the actual positions of pedestrian PDS around the moving object 2 is suppressed. The detection points PP of past stationary objects detected by the detection range DA are sequentially updated and accumulated using an odometry method with CAN data, and thus remain in the map information 23B as first surrounding position information. Furthermore, the objects detected within the detection range DA and updated to the latest state are not limited to pedestrian PDS; they may also be other moving objects or stationary objects.

[0068] Figure 9 shows an example of map information 23B in reverse parking, different from Figure 8. For ease of explanation, Figure 9 shows the three-dimensional object TDO, the moving body 2, the direction of travel (reverse direction) of the moving body 2 TD, and the detection range DA by the detection unit 14. The first surrounding position information PP in Figure 9 shows map information already stored in map information 23B. As shown in Figure 9, in map information 23B, the first surrounding information PP is indicated by a white circle.

[0069] Figures 5 to 9 show how the range in which the sensor acquires position information of objects surrounding the moving object 2 coincides with the update range in which position information is deleted. The range in which position information is acquired and the update range do not have to strictly coincide. For example, the sensor may acquire position information that is further away than the update range. In this case, the update range is set to be included in the range in which position information is acquired. Alternatively, the update range may be set to extend further than the range in which the sensor acquires position information.

[0070] Furthermore, in Figures 5 to 9, the map information 23B is updated while acquiring information on the relative movement of the moving object 2. However, the deletion and addition of surrounding position information may also be performed while the moving object 2 is stationary. In this case, for example, even if a pedestrian PDS crosses behind the stationary moving object 2, it is possible to suppress the accumulation of position information along the pedestrian PDS's trajectory in the map information 23B.

[0071] Figure 10 shows an example of map information 23B after the mobile body 2 has moved further along the backward direction TD from its position in Figure 9. For ease of explanation, Figure 10 shows the three-dimensional object TDO, the mobile body 2, the direction of movement (backward direction) of the mobile body 2 TD, and the detection range DA by the detection unit 14. In the map information 23B in Figure 10, the first surrounding position information PP is shown as a white circle, and the second surrounding position information P is shown as a black circle. The position information corresponding to the black circles shown in Figure 10 is deleted from the map information 23B by the initialization unit 225 and added to the map information 23B by the addition unit 229 as the mobile body 2 moves.

[0072] Returning to Figure 3, let's explain the determination unit 30. The determination unit 30 uses map information 23B and self-position information to determine the projection shape of the projection surface. That is, the determination unit 30 uses the position information of the detected point P (first surrounding position information and second surrounding position information) stored in the map information 23B and the self-position information to determine the projection shape of the projection surface. The projection surface is a three-dimensional surface for projecting the surrounding image of the moving object 2.

[0073] In other words, the projection surface is a virtual three-dimensional surface onto which the captured image 50 of the area around the moving object is projected. The projected shape of the projection surface has a three-dimensional (3D) shape that is virtually formed in a virtual space corresponding to real space. The surrounding image of the moving object 2 is a captured image of the area around the moving object 2. In this embodiment, the surrounding image of the moving object 2 is a captured image taken by each of the imaging units 12A to 12D.

[0074] Furthermore, if self-position information is registered in map information 23B, the determination unit 30 determines the shape of the projection plane based on the map information 23B. The determination unit 30 determines the shape obtained by deforming the reference projection plane according to the position information of the detected point P registered in map information 23B as the projection shape.

[0075] Figure 11 is a schematic diagram showing an example of a reference projection plane 40. The reference projection plane 40 is, for example, a projection plane with a shape that serves as a reference when changing the shape of a projection plane. The shape of the reference projection plane 40 can be, for example, a bowl shape, a cylinder shape, etc.

[0076] A bowl shape is a shape having a base surface 40A and side walls 40B, where one end of the side wall surface 40B is continuous with the base surface 40A and the other end is open. The width of the horizontal cross-section of the side wall surface 40B increases from the base surface 40A side toward the open end. The base surface 40A is, for example, circular. Here, circular shape includes shapes other than perfect circles, such as ellipses. A horizontal cross-section is an orthogonal plane perpendicular to the vertical direction (arrow Z direction). An orthogonal plane is a two-dimensional plane along the arrow X direction perpendicular to the arrow Z direction, and the arrow Y direction perpendicular to both the arrow Z direction and the arrow X direction. In the following, the horizontal cross-section and orthogonal plane may be referred to as the XY plane. Note that the base surface 40A may be a shape other than a circle, such as an egg shape.

[0077] A cylindrical shape consists of a circular base surface 40A and side wall surfaces 40B continuous with the base surface 40A. The side wall surfaces 40B that constitute the reference projection plane 40 of the cylindrical shape are cylindrical in shape, with one end opening continuous with the base surface 40A and the other end open. However, the side wall surfaces 40B that constitute the reference projection plane 40 of the cylindrical shape have a shape in which the diameter of the XY plane is approximately constant from the base surface 40A side toward the opening side of the other end. The base surface 40A may be a shape other than a circle, such as an egg shape.

[0078] In this embodiment, the case where the shape of the reference projection plane 40 is bowl-shaped will be described as an example. The reference projection plane 40 is a three-dimensional model virtually formed in a virtual space, with its bottom surface 40A being a surface that substantially coincides with the road surface below the moving body 2, and the center of the bottom surface 40A being the self-position S of the moving body 2. The self-position S corresponds to the self-position information.

[0079] The determination unit 30 determines the shape obtained by deforming the reference projection plane 40 to a shape that passes through the detection point P closest to the moving body 2 as the projected shape. The shape that passes through the detection point P means that the deformed side wall surface 40B has a shape that passes through the detection point P.

[0080] Figure 12 is a schematic diagram showing an example of the projected shape 41. The determination unit 30 determines the projected shape 41 by deforming the reference projection plane 40 to a shape that passes through the detection point P closest to the self-position S of the moving body 2, which is the center of the bottom surface 40A of the reference projection plane 40. This self-position S is the latest self-position S calculated by the self-position estimation unit 24, i.e., the latest position of the moving body 2.

[0081] The determination unit 30 identifies the detection point P closest to its own position S from among the multiple detection points P registered in the map information 23B. Specifically, the XY coordinates of the center position (self position S) of the moving body 2 are set to (X,Y)=(0,0). The determination unit 30 then identifies the detection point P where the value of X2+Y2 is the minimum as the detection point P closest to its own position S. The determination unit 30 then determines the shape obtained by deforming the side wall surface 40B of the reference projection plane 40 so that it passes through the detection point P as the projection shape 41.

[0082] Specifically, the determination unit 30 determines the deformed shape of a portion of the bottom surface 40A and the side surface 40B as the projected shape 41, such that when the reference projection plane 40 is deformed, a portion of the side surface 40B becomes a wall surface passing through the detection point P closest to the moving body 2. The deformed projected shape 41 is, for example, a shape that rises from the rising line 44 on the bottom surface 40A toward the center of the bottom surface 40A. To "rise" means, for example, to bend or fold a portion of the side surface 40B and the bottom surface 40A toward the center of the bottom surface 40A, such that the angle between the side surface 40B and the bottom surface 40A of the reference projection plane 40 becomes a smaller angle.

[0083] The determination unit 30 determines that a specific region on the reference projection plane 40 should be deformed so that it protrudes to a position passing through the detection point P from the viewpoint (plan view) of the XY plane. The shape and range of the specific region may be determined based on predetermined criteria. The determination unit 30 then determines that the reference projection plane 40 should be deformed so that the distance from its own position S increases continuously from the protruding specific region toward the region on the side wall surface 40B other than the specific region.

[0084] More specifically, as shown in Figure 12, it is preferable to determine the projected shape 41 such that the outer periphery of the cross-section along the XY plane is curved. The outer periphery of the cross-section of the projected shape 41 is, for example, circular, but may be a shape other than circular.

[0085] The determination unit 30 may also determine the projection shape 41 as a shape obtained by deforming the reference projection plane 40 so that it follows the asymptotic curve. The asymptotic curve is the asymptotic curve of multiple detection points P. The determination unit 30 generates a predetermined number of asymptotic curves of multiple detection points P in the direction away from the detection point P closest to the self-position S of the moving body 2. The number of these detection points P may be multiple. For example, it is preferable that the number of these detection points P be three or more. In this case, it is also preferable that the determination unit 30 generates asymptotic curves of multiple detection points P located at positions that are at a predetermined angle or more away from the self-position S.

[0086] Figure 13 is an explanatory diagram of the asymptotic curve Q. Figure 13 shows an example in which the asymptotic curve Q is shown on a projected image 51 obtained by projecting a captured image onto a projection plane when the moving object 2 is viewed from above in a bird's-eye view. For example, suppose the determination unit 30 identifies three detection points P in order of proximity to the self-position S of the moving object 2. In this case, the determination unit 30 generates an asymptotic curve Q of these three detection points P. Then, the determination unit 30 can determine the shape obtained by deforming the reference projection plane 40 so that it follows the shape of the generated asymptotic curve Q as the projected shape 41.

[0087] The determination unit 30 may divide the area around the self-position S of the moving body 2 into specific angular ranges, and for each angular range, it may identify the detection point P closest to the moving body 2, or multiple detection points P in order of proximity to the moving body 2. The determination unit 30 may then determine the projection shape 41 as the shape obtained by deforming the reference projection plane 40 so that it is a shape that passes through the identified detection point P or a shape that follows the asymptotic curve Q of the multiple identified detection points P for each angular range.

[0088] Next, we will describe an example of the detailed configuration of the determination unit 30.

[0089] Figure 14 is a schematic diagram showing an example of the configuration of the determination unit 30. The determination unit 30 comprises an extraction unit 30A, a nearest neighbor identification unit 30B, a reference projection plane shape selection unit 30C, a scale determination unit 30D, an asymptotic curve calculation unit 30E, and a shape determination unit 30F.

[0090] The extraction unit 30A uses the self-position information of the mobile body 2 and the map information 23B to extract detection points P that are within a specific range from among the multiple detection points P included in the map information 23B. Here, the map information 23B includes distance information from the mobile body 2 to the detection points P. The specific range is, for example, the range from the road surface on which the mobile body 2 is positioned to a height equivalent to the height of the mobile body 2. However, this range is not limited to this range. By the extraction unit 30A extracting detection points P within this range, for example, detection points P of objects that obstruct the movement of the mobile body 2 can be extracted. The extraction unit 30A outputs the distance information of each extracted detection point P to the nearest neighbor identification unit 30B. The extraction unit 30A outputs the current self-position information of the mobile body 2 to the virtual viewpoint line of sight determination unit 34. If the distance information from the mobile body 2 to the detection points P is not included in the map information 23B, the distance information may be converted using the map information 23B and the self-position information and input to the extraction unit 30A.

[0091] The nearest neighbor identification unit 30B divides the area around the moving body 2's own position S into specific angular ranges, and for each angular range, identifies the detection point P closest to the moving body 2, or multiple detection points P in order of proximity to the moving body 2. The nearest neighbor identification unit 30B identifies the detection points P using distance information received from the extraction unit 30A. In this embodiment, one example described is a configuration in which the nearest neighbor identification unit 30B identifies multiple detection points P in order of proximity to the moving body 2 for each angular range.

[0092] The nearest point identification unit 30B outputs distance information of the detected points P identified for each angular range to the reference projection plane shape selection unit 30C, the scale determination unit 30D, and the asymptotic curve calculation unit 30E.

[0093] The reference projection plane shape selection unit 30C selects the shape of the reference projection plane 40. The reference projection plane shape selection unit 30C selects the shape of the reference projection plane 40 by reading a specific shape from the storage unit 231, which stores multiple types of reference projection plane shapes 40. For example, the reference projection plane shape selection unit 30C selects the shape of the reference projection plane 40 based on the positional relationship and distance information between its own position and surrounding three-dimensional objects. Alternatively, the shape of the reference projection plane 40 may be selected based on user instructions. The reference projection plane shape selection unit 30C outputs the determined shape information of the reference projection plane 40 to the shape determination unit 30F. In this embodiment, as described above, the reference projection plane shape selection unit 30C will be described as selecting a bowl-shaped reference projection plane 40 as an example.

[0094] The scale determination unit 30D determines the scale of the reference projection plane 40 of the shape selected by the reference projection plane shape selection unit 30C. The scale determination unit 30D makes decisions such as reducing the scale when there are multiple detection points P within a predetermined distance range from its own position S. The scale determination unit 30D outputs the scale information of the determined scale to the shape determination unit 30F.

[0095] The asymptotic curve calculation unit 30E uses the distance information of the detection point P closest to its own position S for each angular range from its own position S, received from the nearest neighbor identification unit 30B, to calculate the asymptotic curve Q and outputs the asymptotic curve information of the calculated asymptotic curve Q to the shape determination unit 30F and the virtual viewpoint line of sight determination unit 34. The asymptotic curve calculation unit 30E may also calculate the asymptotic curve Q of the detection point P accumulated for each of the multiple parts of the reference projection plane 40. The asymptotic curve calculation unit 30E may then output the asymptotic curve information of the calculated asymptotic curve Q to the shape determination unit 30F and the virtual viewpoint line of sight determination unit 34.

[0096] The shape determination unit 30F enlarges or reduces the reference projection plane 40 of the shape indicated by the shape information received from the reference projection plane shape selection unit 30C to the scale information received from the scale determination unit 30D. Then, the shape determination unit 30F determines the shape of the enlarged or reduced reference projection plane 40 as the projected shape 41, which is deformed to conform to the asymptotic curve information of the asymptotic curve Q received from the asymptotic curve calculation unit 30E. The shape determination unit 30F outputs the projected shape information of the determined projected shape 41 to the deformation unit 32.

[0097] Returning to Figure 3, let's continue the explanation. Next, we will explain the deformation unit 32. The deformation unit 32 deforms the reference projection plane 40 into the projection shape 41 indicated by the projection shape information received from the determination unit 30. That is, the deformation unit 32 deforms the projection plane on which the captured image of the area around the moving object is projected, based on the map information 23B. Specifically, the deformation unit 32 deforms the projection plane based on the first and second surrounding position information that were not initialized in the map information 23B.

[0098] Through this deformation process, the deformation unit 32 generates a deformed projection surface 42, which is the deformed reference projection surface 40 (see Figure 12). That is, the deformation unit 32 deforms the reference projection surface 40 using the position information of the detected point P stored in the map information 23B and the self-position information of the moving body 2. In detail, for example, the deformation unit 32 deforms the reference projection surface 40 into a curved shape that passes through the detected point P closest to the moving body 2, based on the projection shape information. Through this deformation process, the deformation unit 32 generates the deformed projection surface 42.

[0099] For example, the deformation unit 32 deforms the reference projection plane 40 to a shape that follows the asymptotic curve Q of a predetermined number of detection points P in order of proximity to the moving body 2, based on the projection shape information. Preferably, the deformation unit 32 deforms the reference projection plane 40 using the position information of the detection points P acquired before the first time point and the self-position information of its own position S.

[0100] Here, the first time is the most recent time when the detection unit 14 detects the position information of the detection point P, or any time prior to the most recent time. For example, the detection point P acquired before the first time includes the position information of a specific object in the vicinity of the moving body 2, while the detection point P acquired at the first time does not include the position information of a specific object in the vicinity. The determination unit 30 can determine the projected shape 41 in the same manner as described above using the position information of the detection point P acquired before the first time, which is included in the map information 23B. Then, the deformation unit 32 can generate the deformed projection surface 42 in the same manner as described above using the projected shape information of the projected shape 41.

[0101] In this case, for example, even if the position information of the detection point P detected by the detection unit 14 at a first time does not include the position information of the detection point P detected in the past, the deformation unit 32 can still generate a deformation projection surface 42 corresponding to the detection point P detected in the past.

[0102] Next, the projection conversion unit 36 ​​will be described. The projection conversion unit 36 ​​generates a projected image 51 by projecting the captured image acquired from the imaging unit 12 onto the deformed projection surface 42, which is the reference projection surface 40 deformed by the deformation unit 32. In detail, the projection conversion unit 36 ​​receives deformed projection surface information of the deformed projection surface 42 from the deformation unit 32. Deformed projection surface information is information that indicates the deformed projection surface 42. The projection conversion unit 36 ​​projects the captured image acquired from the imaging unit 12 via the acquisition unit 20 onto the deformed projection surface 42 indicated by the received deformed projection surface information. Through this projection process, the projection conversion unit 36 ​​generates the projected image 51. The projection conversion unit 36 ​​converts the projected image 51 into a virtual viewpoint image. A virtual viewpoint image is an image of the projected image 51 viewed from a virtual viewpoint in any direction.

[0103] The projection transformation unit 36 ​​will be explained using Figure 12. The projection transformation unit 36 ​​projects the captured image 50 onto the deformable projection surface 42. The projection transformation unit 36 ​​then generates a virtual viewpoint image (not shown) which is an image of the captured image 50 projected onto the deformable projection surface 42 as viewed from an arbitrary virtual viewpoint O in the line of sight direction L. The position of the virtual viewpoint O can be, for example, the latest self-position S of the moving object 2. In this case, the XY coordinate values ​​of the virtual viewpoint O can be set to the XY coordinate values ​​of the latest self-position S of the moving object 2. Also, the Z coordinate value (vertical position) of the virtual viewpoint O can be set to the Z coordinate value of the detection point P closest to the self-position S of the moving object 2. The line of sight direction L may be determined, for example, based on a predetermined criterion.

[0104] The line of sight direction L may be, for example, the direction from the virtual viewpoint O toward the detection point P closest to the self-position S of the moving object 2. Alternatively, the line of sight direction L may be a direction that passes through the detection point P and is perpendicular to the deformation projection plane 42. Virtual viewpoint line of sight information indicating the virtual viewpoint O and the line of sight direction L is created by the virtual viewpoint line of sight determination unit 34.

[0105] Returning to Figure 3, let's continue the explanation. The virtual viewpoint line of sight determination unit 34 determines the virtual viewpoint line of sight information, for example, in the following procedure. The virtual viewpoint line of sight determination unit 34 determines the line of sight direction L as a direction that passes through the detection point P closest to the self-position S of the moving object 2 and is perpendicular to the deformation projection plane 42. Alternatively, the virtual viewpoint line of sight determination unit 34 may fix the direction of the line of sight direction L and determine the coordinates of the virtual viewpoint O as an arbitrary Z coordinate and arbitrary XY coordinates in the direction away from the asymptotic curve Q toward the self-position S. In this case, the XY coordinates may be coordinates at a position further away from the asymptotic curve Q than the self-position S. The virtual viewpoint line of sight determination unit 34 then outputs the virtual viewpoint line of sight information indicating the virtual viewpoint O and the line of sight direction L to the projection transformation unit 36. As shown in Figure 13, the line of sight direction L may be the direction toward the vertex W of the asymptotic curve Q from the virtual viewpoint O.

[0106] The projection transformation unit 36 ​​receives virtual viewpoint line-of-sight information from the virtual viewpoint line-of-sight determination unit 34. By receiving this virtual viewpoint line-of-sight information, the projection transformation unit 36 ​​identifies the virtual viewpoint O and the line-of-sight direction L. The projection transformation unit 36 ​​then generates a virtual viewpoint image from the captured image 50 projected onto the deformed projection surface 42, which is the image viewed from the virtual viewpoint O in the line-of-sight direction L. The virtual viewpoint image corresponds, for example, to an image in which the captured image 50 can be viewed along the line-of-sight direction L from the virtual viewpoint O. The projection transformation unit 36 ​​outputs the virtual viewpoint image to the image synthesis unit 38.

[0107] The image synthesis unit 38 generates a composite image by extracting part or all of the virtual viewpoint image. For example, the image synthesis unit 38 determines the width of the overlapping portion of multiple captured images 50 included in the virtual viewpoint image, stitches the captured images 50 together, and performs blending to determine which captured images 50 to display in the overlapping portion. This generates a composite image 54. The image synthesis unit 38 then outputs the composite image 54 to the display unit 16. The composite image 54 may be a bird's-eye view image with the virtual viewpoint O above the moving object 2, or the virtual viewpoint O may be inside the moving object 2, with the moving object 2 displayed semi-transparently.

[0108] Next, we will explain an example of the image processing flow performed by the information processing device 10. Figure 16 is a flowchart showing an example of the image processing procedure performed by the information processing device 10.

[0109] (Image processing) The acquisition unit 20 acquires the captured image 50 from the shooting unit 12 (step S10). The acquisition unit 20 outputs the acquired captured image 50 to the projection conversion unit 36.

[0110] The acquisition unit 20 acquires the position information of each of the multiple detection points P from the detection unit 14. The acquisition unit 20 also acquires CAN data from the ECU 3. Based on the position information of each of the multiple detection points P and the CAN data, the map information 23B is generated by the map information generation process described above (step S11).

[0111] The determination unit 30 acquires the generated map information 23B (step S12).

[0112] The extraction unit 30A extracts detection points P that are within a specific range from among the detection points P. The nearest detection point identification unit 30B uses the distance information of each extracted detection point P to identify multiple detection points P in order of proximity to the moving object 2 for each angular range (direction) around the moving object 2. In this way, the nearest detection point to the moving object 2 is identified for each direction within the specific range (step S13).

[0113] The reference projection plane shape selection unit 30C selects the shape of the reference projection plane 40 (step S14). As described above, the reference projection plane shape selection unit 30C will be described as selecting a bowl-shaped reference projection plane 40 as an example. In addition, the selection of the reference projection plane may be based on the self-position information of the moving body 2, the position information of objects around the moving body 2 (first peripheral position information and / or second peripheral position information), distance information, etc., and the shape of the reference projection plane 40 to be used for image processing may be selected from multiple types of reference projection plane shapes 40.

[0114] The scale determination unit 30D determines the scale of the reference projection plane 40 of the shape selected in step S14 (step S15).

[0115] The asymptotic curve calculation unit 30E calculates an asymptotic curve Q using the distance information of each of the multiple detection points P for each angular range identified in step S13 (step S16).

[0116] The shape determination unit 30F enlarges or reduces the reference projection plane 40 of the shape selected in step S14 to the scale determined in step S15. Then, the shape determination unit 30F deforms the enlarged or reduced reference projection plane 40 so that it follows the asymptotic curve Q calculated in step S16. The shape determination unit 30F determines this deformed shape as the projection shape 41 (step S17).

[0117] The deformation unit 32 deforms the reference projection plane 40 selected by the reference projection plane shape selection unit 30C into the projection shape 41 determined by the determination unit 30 (step S18). Through this deformation process, the deformation unit 32 generates a deformed projection plane 42, which is the deformed reference projection plane 40 (see Figure 12).

[0118] The virtual viewpoint line of sight determination unit 34 determines the virtual viewpoint line of sight information (step S19). For example, the virtual viewpoint line of sight determination unit 34 determines the self-position S of the moving object 2 as the virtual viewpoint O, and the direction toward the vertex W of the asymptotic curve Q from the virtual viewpoint O as the line of sight direction L. Specifically, the virtual viewpoint line of sight determination unit 34 determines the direction toward the vertex W of the asymptotic curve Q for a specific angular range from among the asymptotic curve Q calculated for each angular range in step S16 as the line of sight direction L.

[0119] The projection conversion unit 36 ​​projects the captured image 50 acquired in step S10 onto the deformed projection surface 42 generated in step S17. The projection conversion unit 36 ​​then converts the projected image 51 into a virtual viewpoint image by projecting the captured image 50 onto the deformed projection surface 42 along the line of sight direction L from the virtual viewpoint O determined in step S19. In other words, the projection conversion unit 36 ​​converts the captured image 50 projected onto the deformed projection surface 42 into a virtual viewpoint image using the virtual viewpoint line of sight information (step S20).

[0120] The image synthesis unit 38 generates a composite image 54 by extracting part or all of the virtual viewpoint image generated in step S20 (step S21). The virtual viewpoint image includes overlapping portions of multiple captured images 50 (hereinafter referred to as overlapping portions). The image synthesis unit 38 performs, for example, determining the width of the overlapping portions, stitching together the captured images 50, and blending to determine which captured images 50 to display in the overlapping portions.

[0121] The image synthesis unit 38 performs display control to output the generated composite image 54 to the display unit 16 (step S22). As a result, the generated composite image 54 is displayed on the display unit 16.

[0122] Next, the information processing device 10 determines whether or not to terminate the image processing (step S23). For example, the information processing device 10 makes the determination in step S23 by determining whether or not it has received a signal from the ECU 3 indicating that the operation of the mobile body 2 has stopped (for example, the engine has stopped). Alternatively, for example, the information processing device 10 may make the determination in step S23 by determining whether or not it has received an instruction to terminate the image processing based on an operation instruction from the user.

[0123] If a negative judgment is determined in step S23 (step S23: No), the processing in steps S10 to S22 above is repeated. If an affirmative judgment is determined in step S23 (step S23: Yes), this image processing routine ends.

[0124] As described above, the information processing device 10 according to the embodiment includes a map information generation unit 22 that initializes the detection range (detection range) related to detection by a sensor (detection unit 14) mounted on the moving body 2 in map information 23B which includes first peripheral position information, which is information on the positions of objects located around the moving body, and adds second peripheral position information acquired from the sensor to the detection range. For example, as initialization of data in the detection range DA, the map information generation unit 22 deletes the peripheral position information included in the detection range from the first peripheral position information in map information 23B.

[0125] As a result, according to the information processing device 10 of the embodiment, the surrounding location information included in the detection range in the map information 23B can be updated in accordance with the acquisition of location information. In other words, according to the information processing device 10 of the embodiment, in the area around the moving object 2 (vehicle) in the map information 23B, the deletion and addition of surrounding location information is performed sequentially, and the map information 23B that reflects the latest situation around the moving object can be generated. In other words, according to the information processing device 10 of the embodiment, the location information within the update range in the map information 23B can always be kept up-to-date.

[0126] Therefore, according to the information processing device 10 according to the embodiment, as shown in Figure 8, even if a moving object such as a pedestrian PDS passes through the detection range, it is possible to suppress the accumulation of positional information along the trajectory of the moving object (pedestrian, etc.) near the moving object in the map information 23B. As a result, according to the information processing device 10 according to the embodiment, it is possible to generate highly accurate map information 23B.

[0127] Furthermore, the information processing device 10 according to the embodiment further includes a deformation unit 32 that deforms the projection surface (selected reference projection surface 40) on which the captured image 50 of the area around the moving object is projected, based on the map information 23B generated by the map information generation process. For example, the information processing device 10 according to the embodiment deforms the projection surface based on first and second peripheral position information that were not initialized in the map information 23B. For example, as shown in Figure 10, first peripheral position information (white circle) PP and second peripheral position information (black circle) P are used as peripheral position information used in the projection surface deformation process.

[0128] As a result, the information processing device 10 according to the embodiment can deform the projection surface using highly accurate map information 23B, and by projecting the captured image 50 onto the appropriately deformed projection surface, a composite image (overhead view image) can be presented to the user. Therefore, the information processing device 10 according to the embodiment can present the user with a natural overhead view image with improved visibility.

[0129] Furthermore, multiple sensors related to the information processing device 10 according to the embodiment are mounted on the mobile body 2. For example, multiple sensors related to the information processing device 10 according to the embodiment are arranged in an array on the exterior of the mobile body 2. Alternatively, the sensors related to the information processing device 10 according to the embodiment may be distance sensors mounted on the rear of the mobile body 2, and the first peripheral position information may be obtained from the distance sensors. Additionally, sensors related to the information processing device 10 according to the embodiment may be further arranged on the side of the mobile body 2 as distance sensors. Furthermore, if the detection unit 14 is implemented with multiple sensors, peripheral position information obtained by some (nearby) sensors may be subject to storage / deletion, while peripheral position information obtained by other (far) sensors may be used only for storage. As a result, the information processing device 10 according to the embodiment can generate highly accurate map information 23B for any direction around the mobile body 2.

[0130] Furthermore, in the information processing device 10 according to the embodiment, the detection range corresponding to the update range corresponds to information around the moving object based on the position of the moving object 2 (self-position information). In addition, according to the information processing device 10 according to the embodiment, the surrounding position information to be deleted is determined based on the detection range information and the relative amount of movement (movement vector) of the moving object 2 with respect to the origin in the map information 23B (relative movement amount information). Thus, according to the information processing device 10 according to the embodiment, the surrounding position information to be deleted or the update range can be determined based on the moving object 2, regardless of whether the moving object 2 is moving or stopped. For this reason, according to the information processing device 10 according to the embodiment, the surrounding position information included in the update range can always be kept up-to-date, regardless of whether the moving object 2 is moving or stopped.

[0131] In this embodiment, as an example of the use of map information 23B generated by the map information generation process, the deformation process of the projection surface in reverse parking was described. However, the use of map information 23B is not limited to the deformation process of the projection surface. For example, if the moving object 2 is a vehicle, the map information 23B may be used for autonomous driving or forward parking, etc.

[0132] When the technical concept of the embodiment is realized by an information processing method, the information processing method initializes the detection range of the sensor mounted on the mobile body 2 in the map information 23B, which includes first surrounding position information, which is information on the positions of objects located around the mobile body, and adds second surrounding position information acquired from the sensor to that range. The procedure and effects of the map information generation process performed by the information processing method are the same as in the embodiment, so a description is omitted.

[0133] When the technical concept of this embodiment is implemented using an information processing program, the program causes the computer to initialize the detection range of the sensor mounted on the mobile body 2 in the map information 23B, which includes first surrounding position information, which is information about the positions of objects located around the mobile body, and to add the second surrounding position information acquired from the sensor to that range. For example, the map information generation process can also be realized by installing the information processing program from a non-volatile storage medium to various server devices (processing devices) and then expanding them in memory. In this case, the program that can cause the computer to execute this method can also be stored and distributed on storage media such as magnetic disks (hard disks, etc.), optical disks (CD-ROMs, DVDs, etc.), and semiconductor memory. The processing procedures and effects in the information processing program are the same as in the embodiment, so a description is omitted.

[0134] According to this configuration, accurate map information 23B can be generated by sequentially updating location information within the update range. As a result, according to one embodiment of the information processing device disclosed in this application, for example, the shape of the projection surface can be appropriately deformed using the accurate map information 23B.

[0135] Although embodiments and various modifications have been described above, the information processing apparatus 10, information processing method, and information processing program disclosed in this application are not limited to the above embodiments, etc., and the components can be modified and implemented in each implementation stage, etc., without departing from the gist of the invention. Furthermore, various inventions can be formed by appropriately combining the multiple components disclosed in the above embodiments and various modifications, etc. For example, some components may be deleted from all the components shown in the embodiments. [Explanation of symbols]

[0136] 1. Information Processing System 10 Information Processing Devices 12, 12A~12D Photography Department 14, 14A~14D Detection section 20 Acquisition Department 22 Map Information Generation Unit 23A Detection range information 23B Map Information 30 Decision Section 30A extraction part 30B Nearest neighbor identification part 30C Reference Projection Plane Shape Selection Section 30D scale determination unit 30E Asymptotic curve calculation section 30F Shape determining section 32 Deformed part 34 Virtual viewpoint line of sight determination unit 36 Projection Transformation Unit 38 Image Synthesis Unit 221 Self-position estimation part 223 First offset adjustment section 225 Initialization section 227 Second offset adjustment section 229 Additional section 231 Storage section 233 Correction section

Claims

1. An acquisition unit acquires information on the relative amount of movement of the moving body with respect to the origin in map information which includes first peripheral position information which is information on the positions of objects located around the moving body, and acquires second peripheral position information which is included in the range for detection by sensors mounted on the moving body as the moving body moves. A first offset adjustment unit that identifies the range in the map information based on the range and the relative amount of movement information, A deletion unit that deletes the surrounding location information included in the range from the first surrounding location information from the map information, A second offset adjustment unit that identifies the range in the map information from which the surrounding location information has been deleted, based on the second surrounding location information, the relative amount of movement, and the map information from which the surrounding location information has been deleted. An additional unit that adds the second surrounding location information to the area identified in the map information from which the aforementioned surrounding location information has been deleted, in accordance with the movement of the moving object, A map information generation unit having, Equipped with, The second peripheral position information includes position information of the detection point of the moving body within the range, Information processing device.

2. The system further includes a deformation unit that deforms the projection surface on which the captured image of the area around the moving object is projected, based on the aforementioned map information. The information processing apparatus according to claim 1.

3. The deformation unit deforms the projection plane based on the first and second peripheral position information that were not initialized in the map information. The information processing apparatus according to claim 2.

4. The aforementioned sensors are mounted in multiple locations on the moving body. The information processing apparatus according to claim 1.

5. Multiple sensors are arranged in an array on the exterior of the moving body. The information processing apparatus according to claim 4.

6. The sensor is a distance sensor mounted on the rear of the moving body, The first surrounding position information is obtained from the distance sensor. The information processing apparatus according to claim 1.

7. The sensor is further positioned to the side of the moving body as a distance sensor. The information processing apparatus according to claim 6.

8. Information is obtained regarding the relative amount of movement of the moving object with respect to the origin in map information which includes first peripheral position information, which is information about the positions of objects located around the moving object. The second peripheral position information, which is included in the detection range of the sensor mounted on the moving body, is acquired as the moving body moves. Based on the aforementioned range and the information on the relative amount of movement, the aforementioned range in the map information is identified. The surrounding location information included in the range of the first surrounding location information is deleted from the map information. Based on the second surrounding location information, the relative amount of movement, and the map information from which the surrounding location information has been removed, the range in the map information from which the surrounding location information has been removed is identified. The system includes adding the second surrounding location information to the area identified in the map information from which the aforementioned surrounding location information has been deleted, in accordance with the movement of the moving object. The second peripheral position information includes position information of the detection point of the moving body within the range, Information processing methods.

9. On the computer, Information is obtained regarding the relative amount of movement of the moving object with respect to the origin in map information which includes first peripheral position information, which is information about the positions of objects located around the moving object. The second peripheral position information, which is included in the detection range of the sensor mounted on the moving body, is acquired as the moving body moves. Based on the aforementioned range and the information on the relative amount of movement, the aforementioned range in the map information is identified. The surrounding location information included in the range of the first surrounding location information is deleted from the map information. Based on the second surrounding location information, the relative amount of movement, and the map information from which the surrounding location information has been removed, the range in the map information from which the surrounding location information has been removed is identified. An information processing program for causing the second surrounding location information to be added to the range identified in the map information from which the aforementioned surrounding location information has been deleted, in accordance with the movement of the moving object, The second peripheral position information includes position information of the detection point of the moving body within the range, Information processing program.

Images