Information processing device, information processing method, and information processing program

Abstract

In one embodiment, an information processing device (10) is provided with a map information generation unit (22). In map information (23B) including first peripheral position information, a map information generation unit (22) initializes a range relating to detection by a sensor mounted on a moving body (2), and adds second peripheral position information acquired from the sensor to the range, the first peripheral position information is information on the position of an object located in the periphery of the moving body.

Description

Technical Field

[0001] The present invention relates to an information processing device, an information processing method, and an information processing program. Background Art

[0002] There is a technology that uses sensors such as sonar to obtain position information such as the positional relationship between a moving body and objects around the moving body. In addition, there is a technology that uses the acquired image of the surroundings of the moving body to perform Visual SLAM (Simultaneous Localization and Mapping) processing to obtain (infer) position information indicating the position of the moving body and the positions of the objects around the moving body. In addition, there is also a technology that uses the acquired position information (map information) such as the position of the moving body and the positions of the objects around the moving body to deform the shape of the projection surface used to generate a bird's-eye view image of the surroundings of the moving body.

[0003] Patent document 1: International Publication No. 2021 / 111531;

[0004] Patent document 2: U.S. Patent Application Publication No. 2021 / 0140934;

[0005] Patent document 3: U.S. Patent Application Publication No. 2021 / 0323539.

[0006] However, in acquiring map information, for example, when a moving object (pedestrians, etc.) around a moving object is detected by sonar, the position information of the moving object is accumulated in accordance with the movement of the moving object. In this case, the position information is accumulated in a manner that a three-dimensional object such as a wall exists along the trajectory of the moving object. If the position information is accumulated in a manner that a three-dimensional object exists, the shape of the projection surface may not be properly deformed. Summary of the invention

[0007] In one aspect, an object of the present invention is to provide an information processing device, an information processing method, and an information processing program that can generate map information with high accuracy.

[0008] The information processing device disclosed in the present application includes a map information generating unit in a technical solution. The map information generating unit initializes a range related to detection based on a sensor mounted on the moving body in map information including first surrounding position information, and adds second surrounding position information acquired from the sensor to the range, wherein the first surrounding position information is information about the position of an object located around the moving body.

[0009] According to a technical solution of the information processing device disclosed in the present application, it is possible to generate map information with good accuracy. Therefore, according to a technical solution of the information processing device disclosed in the present application, for example, it is possible to use map information with good accuracy to appropriately deform the shape of the projection surface. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 It is a diagram showing an example of the overall configuration of an information processing system according to an embodiment.

[0011] Figure 2 It is a diagram showing an example of the hardware configuration of the information processing device according to the embodiment.

[0012] Figure 3 It is a diagram showing an example of the functional configuration of the information processing device according to the embodiment.

[0013] Figure 4 This is a flowchart showing an example of the procedure of the map information generation process according to the embodiment.

[0014] Figure 5 This is a schematic diagram showing an example of map information according to an embodiment.

[0015] Figure 6 It is a method for expressing the implementation mode. Figure 5 FIG. 1 is a diagram showing an example of map information when the position of a moving object moves along the traveling direction.

[0016] Figure 7 It is a method for expressing the implementation mode. Figure 6 FIG. 1 is a diagram showing an example of map information when the position of a moving object moves along the traveling direction.

[0017] Figure 8 This is a diagram showing an example of map information when a moving object is backed up and parked between two vehicles according to the embodiment.

[0018] Fig. 9 It is the same as the embodiment Figure 8 A diagram showing an example of different map information during reverse parking.

[0019] Fig.10 The moving body of the embodiment is Fig. 9 A diagram showing an example of map information in which the position of a moving object in FIG. 1 is moved in a backward direction.

[0020] Fig.11 It is a schematic diagram showing an example of a reference projection plane according to the embodiment.

[0021] Fig.12 It is a schematic diagram showing an example of a projection shape according to the embodiment.

[0022] Fig.13 It is an explanatory diagram of an asymptotic curve of an embodiment.

[0023] Fig.14 It is a schematic diagram showing an example of the structure of the determination unit according to the embodiment.

[0024] Fig.15 This is a schematic diagram showing an example of map information according to an embodiment.

[0025] Fig.16 This is a flowchart showing an example of the steps of image processing executed by the information processing device according to the embodiment. DETAILED DESCRIPTION

[0026] Hereinafter, the information processing device, information processing method and information processing program disclosed in the present application will be described in detail with reference to the accompanying drawings. In addition, the following embodiments do not limit the disclosed technology. Moreover, each embodiment can be appropriately combined within the scope of not making the processing content contradictory.

[0027] Figure 1 1 is a diagram showing an example of the overall configuration of the information processing system 1 of the present embodiment. The information processing system 1 includes an information processing device 10, an imaging unit 12, a detection unit 14, and a display unit 16. The information processing device 10, the imaging unit 12, the detection unit 14, and the display unit 16 are connected to each other so as to be able to exchange data or signals.

[0028] In the present embodiment, a mode in which the information processing device 10 , the imaging unit 12 , the detection unit 14 , and the display unit 16 are mounted on the mobile object 2 will be described as an example.

[0029] The mobile body 2 refers to an object that can move. The mobile body 2 is, for example, a vehicle, an object that can fly (a manned aircraft, an unmanned aircraft (for example, UAV (Unmanned Aerial Vehicle), unmanned aerial vehicle), a robot, a ship, etc. In addition, the mobile body 2 is, for example, a mobile body that moves through human driving operations, and a mobile body that can automatically move (autonomous movement) without human driving operations. In the present embodiment, the case where the mobile body 2 is a vehicle is described as an example. The vehicle is, for example, a two-wheeled vehicle, a three-wheeled vehicle, a four-wheeled vehicle, etc. In the present embodiment, the case where the vehicle is a four-wheeled vehicle is described as an example.

[0030] In addition, the information processing device 10, the imaging unit 12, the detection unit 14, and the display unit 16 are not limited to being all mounted on the mobile body 2. The information processing device 10 may also be mounted on a stationary object. For example, a stationary object is an object fixed to the ground. Specifically, a stationary object is an object that cannot move or an object that is stationary relative to the ground. In addition, the information processing device 10 may also be mounted on a cloud server that performs processing on the cloud.

[0031] The photographing unit 12 photographs the surroundings of the moving body 2 and obtains photographed image data. Hereinafter, the photographed image data will be referred to as photographed images for explanation. The photographing unit 12 is, for example, a digital camera capable of photographing dynamic images. In addition, photographing refers to converting an image of a subject formed by an optical system such as a lens into an electrical signal. The photographing unit 12 outputs the photographed image to the information processing device 10. In addition, in the present embodiment, the photographing unit 12 is assumed to be a single-lens fisheye camera (for example, a field of view of 195 degrees) for explanation.

[0032] In the present embodiment, a method in which four photographing units 12, namely, a front photographing unit 12A, a left photographing unit 12B, a right photographing unit 12C, and a rear photographing unit 12D are mounted on a mobile body 2 is described as an example. The plurality of photographing units 12 (the front photographing unit 12A, the left photographing unit 12B, the right photographing unit 12C, and the rear photographing unit 12D) respectively photograph subjects in photographing areas E (the front photographing area E1, the left photographing area E2, the right photographing area E3, and the rear photographing area E4) in different directions to acquire photographed images. That is, the photographing directions of the plurality of photographing units 12 are different from each other. In addition, the photographing directions of these plurality of photographing units 12 are pre-adjusted so as to overlap at least a portion of the photographing area E between adjacent photographing units 12. In addition, in Figure 1 For the sake of convenience, Figure 1 The size shown indicates the imaging area E, but actually includes an area far from the moving object 2 .

[0033] In addition, the four front camera units 12A, the left camera unit 12B, the right camera unit 12C, and the rear camera unit 12D are just examples, and the number of the camera units 12 is not limited. For example, when the mobile body 2 has a longitudinal shape such as a bus or a truck, one camera unit 12 may be arranged at the front, rear, front of the right side, rear of the right side, front of the left side, and rear of the left side of the mobile body 2, respectively, and a total of six camera units 12 may be used. That is, the number and arrangement positions of the camera units 12 may be arbitrarily set according to the size and shape of the mobile body 2.

[0034] The detection unit 14 detects position information of each of a plurality of detection points around the moving body 2. In other words, the detection unit 14 detects position information of each of the detection points in the detection area DA. The detection points represent individual points in the real space that are individually observed by the detection unit 14. The detection points correspond to the positions of three-dimensional objects around the moving body 2, for example.

[0035] The position information of the detection point refers to information indicating the position of the detection point in the real space (three-dimensional space). The position information of the detection point is, for example, information indicating the distance from the detection unit 14 (i.e., the position of the moving body 2) to the detection point and the direction of the detection point with respect to the detection unit 14. These distances and directions can be represented by, for example, position coordinates indicating the relative position of the detection point with respect to the detection unit 14, position coordinates indicating the absolute position of the detection point, or vectors.

[0036] The detection unit 14 is, for example, a 3D (Three-Dimensional) scanner, a 2D (Two Dimensional) scanner, a distance sensor (millimeter wave radar, laser sensor), a sonar sensor that detects objects through sound waves, an ultrasonic sensor, etc. The laser sensor is, for example, a three-dimensional LiDAR (Laser imaging Detection and Ranging) sensor. In addition, the detection unit 14 may also be a device that uses a technology such as a motion stereo method that measures distance based on images captured by a stereo camera or a monocular camera, such as SfM (Structure from Motion) technology. In addition, a plurality of imaging units 12 may be used as the detection unit 14. In addition, one of the plurality of imaging units 12 may be used as the detection unit 14.

[0037] In addition, in the present embodiment, the description is made assuming that the detection unit 14 as a sensor is a sonar sensor, but it is not limited to this, and various known sensors for distance measurement can be used as the detection unit 14. In addition, the detection unit 14 as a sensor can also be a distance sensor mounted on the rear of the mobile body 2. The rear of the mobile body 2 is equivalent to the direction on the side opposite to the reference direction in the mobile body 2, for example. In the case where the mobile body 2 is a vehicle, the reference direction is equivalent to the direction on the front side of the driver (the forward direction). In addition, the distance sensor as the detection unit 14 can also be further configured on the side of the mobile body 2. In addition, the multiple sensors are not limited to the above description, as long as they can detect the surroundings of the mobile body 2, they can be mounted in any place in the mobile body 2. In the case where the multiple sensors are sensors that can only measure distances, the detection unit 14 can also calculate the position information of the object on the plane by triangulation using multiple distance information output from the multiple sensors.

[0038] In addition, in this embodiment, a plurality of detection units 14 are mounted on the mobile body 2. Figure 1 As shown, a plurality of sensors (14A, 14B, 14C, 14D) are arranged in an array, for example, in the exterior of the mobile body 2. In the present embodiment, the form in which the mobile body 2 is equipped with four detection units 14, namely, a left rear detection unit 14A, a rear left detection unit 14B, a rear right detection unit 14C, and a right rear detection unit 14D is described as an example.

[0039] The plurality of detection units 14 (left rear detection unit 14A, rear left detection unit 14B, rear right detection unit 14C, right rear detection unit 14D) each detects position information of a plurality of detection points contained in detection areas DA (left rear detection area DA1, rear left detection area DA2, rear right detection area DA3, right rear detection area DA4) in different directions. That is, the ranges (hereinafter referred to as detection ranges) of the plurality of detection units 14 related to the detection of an object are different from each other. In addition, the detection directions of the plurality of detection units 14 are pre-adjusted so as to overlap at least a portion of the detection area DA between adjacent detection units 14. In addition, in Figure 1 For the sake of convenience, Figure 1 The size shown indicates the detection area DA, but in reality, it may further include an area away from the moving object 2. Figure 1 The detection range shown is a range included in a plurality of detection areas (DA1 to DA4), but is hereinafter configured in a fan-shaped shape for simplicity of description.

[0040] In addition, the four left rear detection units 14A, rear left detection unit 14B, rear right detection unit 14C, and right rear detection unit 14D are examples, and the number of detection units 14 is not limited. For example, when the mobile body 2 has a longitudinal shape such as a bus or a truck, one detection unit 14 can be further arranged in front of the mobile body 2, in front of the right side, and in front of the left side, respectively, and a total of seven detection units 14 can be arranged. That is, according to the size and shape of the mobile body 2, the number and arrangement positions of the detection units 14 can be arbitrarily set.

[0041] The display unit 16 displays various information and is, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display.

[0042] In the present embodiment, the information processing device 10 is connected to an electronic control unit (ECU: Electronic Control Unit) 3 mounted on the mobile body 2 so as to be communicable. The ECU 3 is a unit that performs electronic control of the mobile body 2. In the present embodiment, the information processing device 10 can receive CAN (Controller Area Network) data such as the speed and moving direction of the mobile body 2 from the ECU 3.

[0043] Next, the hardware configuration of the information processing device 10 will be described.

[0044] Figure 2 It is a diagram showing an example of the hardware configuration of the information processing device 10 .

[0045] The information processing device 10 includes a CPU (Central Processing Unit) 10A, a ROM (Read Only Memory) 10B, a RAM (Random Access Memory) 10C, and an I / F (InterFace) 10D, and is, for example, a computer. The CPU 10A, ROM 10B, RAM 10C, and I / F 10D are interconnected via a bus 10E, and are configured using the hardware structure of a general computer.

[0046] The CPU 10A is a computing device for controlling the information processing device 10. The CPU 10A corresponds to an example of a hardware processor. The ROM 10B stores programs for implementing various processes based on the CPU 10A. The RAM 10C stores data required for various processes based on the CPU 10A. The I / F 10D is an interface for connecting to the imaging unit 12, the detection unit 14, the display unit 16, and the ECU 3, etc., and for sending and receiving data.

[0047] The program for executing the information processing executed by the information processing device 10 of the present embodiment is provided by preinstalling it in ROM10B or the like. In addition, the program executed by the information processing device 10 of the present embodiment may also be provided by recording it on a recording medium in a form that can be installed in the information processing device 10 or in an executable form. The recording medium is a medium that can be read by a computer. The recording medium is a CD (Compact Disc)-ROM, a floppy disk (FD), a CD-R (Recordable), a DVD (Digital Versatile Disk), a USB (Universal Serial Bus) memory, an SD (Secure Digital) card, etc.

[0048] Next, the functional structure of the information processing device 10 of the present embodiment is described. The information processing device 10 initializes the range (detection range) DA related to the detection based on the sensor (detection unit 14) mounted on the mobile body 2 in the map information including the first surrounding position information, and adds the position information (second surrounding position information) of the detection point acquired from the sensor to the detection range DA. The first surrounding position information is information about the position of the object located around the mobile body. The information processing device 10 combines a plurality of spatially adjacent captured images to generate and display a composite image (bird's-eye view image) of the surroundings of the mobile body 2.

[0049] Figure 3 is a diagram showing an example of the functional structure of the information processing device 10. Figure 3 In order to clarify the input and output relationship of data, in addition to the information processing device 10, the imaging unit 12, the detection unit 14, the display unit 16, etc. are also shown.

[0050] The information processing device 10 includes an acquisition unit 20 , a map information generation unit 22 , a determination unit 30 , a deformation unit 32 , a virtual viewpoint line of sight determination unit 34 , a projection conversion unit 36 ​​, and an image synthesis unit 38 .

[0051] Part or all of the above-mentioned parts may be realized by, for example, making a processing device such as CPU10A execute a program, that is, by software. In addition, part or all of the above-mentioned parts may be realized by hardware such as IC (Integrated Circuit), or by software and hardware.

[0052] The acquisition unit 20 acquires captured images from the imaging unit 12. For example, the acquisition unit 20 acquires captured images from the front imaging unit 12A, the left imaging unit 12B, the right imaging unit 12C, and the rear imaging unit 12D. Each time the acquisition unit 20 acquires a captured image, it outputs the acquired captured image to the projection conversion unit 36.

[0053] The acquisition unit 20 acquires CAN data such as the moving distance and the rotation angle from the moving object 2, and outputs the acquired CAN data to the map information generation unit 22 each time the acquisition unit 20 acquires CAN data. Specifically, the acquisition unit 20 outputs the acquired CAN data to the own position estimation unit 221.

[0054] The acquisition unit 20 acquires the position information of the detection point from the detection unit 14. For example, the acquisition unit 20 acquires the position information of the detection point (second peripheral position information) from the left rear detection unit 14A, the rear left detection unit 14B, the rear right detection unit 14C, and the right rear detection unit 14D. The second peripheral position information is equivalent to the position information of the object within the detection range around the mobile body 2. Each time the acquisition unit 20 acquires the position information of each of the plurality of detection points, the acquired position information is output to the map information generation unit 22. Specifically, the acquisition unit 20 outputs the acquired position information to the second offset adjustment unit 227.

[0055] The map information generation unit 22 includes a self-position estimation unit 221, a first offset adjustment unit 223, an initialization unit 225, a second offset adjustment unit 227, an addition unit 229, a storage unit 231, and a correction unit 233. In addition, as a modified example of the present embodiment, the correction unit 233 may be omitted. The map information generation unit 22 initializes a range related to detection by a sensor mounted on the mobile body 2 in the map information including the first surrounding position information, and adds the second surrounding position information acquired from the sensor to the range. The first surrounding position information is information on the position of an object located around the mobile body. The following describes each component in the map information generation unit 22.

[0056] The own position estimation unit 221 estimates the position of the mobile body 2 in the global coordinate system of the map information, for example, by using odometry based on CAN data. Odometry is a method of calculating the movement amount of each wheel by calculating CAN data such as the rotation angle of the wheels and the rotation angle of the steering wheel in the mobile body 2, and estimating the position (own position information) of the mobile body 2 based on the cumulative calculation. The own position estimation unit 221 estimates the movement vector (also called relative movement amount information) of the mobile body 2, for example, based on the position of the mobile body 2 in the acquisition interval of the position information (i.e., the time interval related to the acquisition of the position information adjacent in time) by using odometry based on CAN data.

[0057] In addition, when LiDAR is used as the sensor in the detection unit 14, the own position estimation unit 221 can also estimate the position of the moving body 2 through LiDAR SLAM. In addition, when a depth camera such as a ToF (Time-of-Flight) sensor is used as the sensor in the detection unit 14, the own position estimation unit 221 can also estimate the position of the moving body 2 through Depth SLAM, for example.

[0058] The first offset adjustment unit 223 reads the detection range information 23A related to the detection unit 14 from the storage unit 231. The detection range information 23A is, for example, information indicating the range (detection range) of the detection unit 14 for detecting a three-dimensional object. The detection range information 23A is pre-set based on the performance (specifications: horizontal angle, maximum distance measurement, etc.) of various sensors used as the detection unit 14, and the installation angle of the detection unit 14 toward the moving body 2, and is stored in the storage unit 231. In addition, the detection range may be the entire area where the object can be detected by the detection unit 14, or may be a partial area close to the moving body 2 in the detection range of the detection unit 14 for the object. In addition, the detection range information 23A may be appropriately adjusted based on, for example, the speed information of the moving body 2 based on the CAN data.

[0059] The first offset adjustment unit 223 adjusts the offset of the detection range of the detection unit 14 for the detection point in the global coordinate system based on the detection range information 23A and the movement vector related to the detection unit 14. The offset of the detection range of the detection unit 14 is, for example, the offset of the detection range relative to the origin of the global coordinate system and the direction of the detection range. The first offset adjustment unit 223 outputs the adjusted first offset to the initialization unit 225 as the first coordinate information.

[0060] The initialization unit 225 reads the map information 23B from the storage unit 231. The map information 23B is, for example, information that geographically represents the surrounding conditions of the mobile body 2. The map information 23B is information that registers the position information of each detection point, that is, a point group, in a three-dimensional coordinate space (global coordinate system) with a specified position in the actual space as the origin (reference position). In addition, the own position information of the mobile body 2 may also be registered in the map information 23B. The information (point group) of the detection point registered in the map information 23B corresponds to the first surrounding position information. That is, the first surrounding position information is information on the position of an object located around the mobile body 2. In addition, the first surrounding position information (and the second surrounding position information) may also be obtained from the detection unit 14 implemented by a distance sensor mounted on the rear of the mobile body 2, for example.

[0061] The initialization unit 225 determines the detection range of the detection unit 14 in the map information 23B based on the map information 23B and the first coordinate information. The update range, which is the object of initialization based on the initialization unit 225, corresponds to the detection range of the detection unit 14 in the map information 23B based on the position (self-position information) of the mobile body 2. The initialization unit 225 initializes (resets) the update range in the map information 23B. For example, as initialization in the update range, the initialization unit 225 deletes the peripheral position information included in the update range in the first peripheral position information from the map information 23B. At this time, the initialization unit 225 deletes the information included in the update range, so it can also be called a deletion unit. In addition, as initialization in the update range, the initialization unit 225 can also fill the data in the update range with zeros (zero filling, zero filling, zero reset). In addition, the initialization unit 225 can also determine and delete the peripheral position information (peripheral position information to be deleted) as the initialization deletion based on the information of the detection range (detection range information 23A) and the information (relative movement amount information) of the relative movement amount (movement vector) of the mobile body 2 relative to the origin in the map information 23B. The initialization unit 225 outputs the initialized map information (hereinafter referred to as initialized map information) to the adding unit 229 .

[0062] The second offset adjustment unit 227 adjusts the offset of the position of the detection point in the global coordinate system based on the position information of the detection point (for example, the detection distance in each direction of each sonar) and the movement vector. The offset of the position of the detection point refers to, for example, the offset of the position of the detection point relative to the origin of the global coordinate system and the direction of the detection point. The second offset adjustment unit 227 outputs the adjusted offset (second offset) as the second coordinate information to the appending unit 229.

[0063] The appending unit 229 determines the detection range of the detection unit 14 in the initialized map information based on the initialized map information and the second coordinate information. In addition, when the mobile body 2 stops, the detection range determined by the initialization unit 225 can also be applied to the initialized map information, and the determination of the detection range by the appending unit 229 can be omitted. The appending unit 229 appends the second peripheral position information acquired by the sensor (detection unit 14) to the detection range in the map information 23B. That is, the appending unit 229 registers the second peripheral position information to the detection range in the initialized map information. In this way, the map information 23B is updated. In addition, the appending unit 229 can also update the own position of the mobile body 2 in the initialized map information based on the movement vector output from the own position inference unit 221. The appending unit 229 stores the updated map information as new map information in the storage unit 231.

[0064] Thus, as the second surrounding position information is acquired from the detection unit 14 mounted on the moving body 2, new second surrounding position information is sequentially added to the map information 23B. For example, when the moving body 2 moves, the map information 23B is sequentially updated as the moving body 2 moves.

[0065] The storage unit 231 stores various data. The storage unit 26 is, for example, a semiconductor memory element such as a RAM, a flash memory, a hard disk, an optical disk, etc. In addition, the storage unit 231 may also be a storage device provided outside the information processing device 10. In addition, the storage unit 231 may also be a storage medium. Specifically, the storage medium may also download programs and various information via a LAN (Local Area Network), a network, etc. and store or temporarily store them.

[0066] The correction unit 233 corrects the position information of each of the plurality of detection points registered in the map information 23B and the own position information of the mobile body 2. The correction unit 233 corrects the position information registered in the map information 23B and the own position information using the position information of each corresponding detection point P acquired at an acquisition time later than the acquisition time of the detection point.

[0067] For example, the correction unit 233 uses the position information (second peripheral position information) of the corresponding detection point detected again by the detection unit 14 to correct the position information (first peripheral position information) of the detection point registered in the map information 23B. At this time, the correction unit 233 may further use various parameters used in the calculation of the position information of each detection point P registered in the map information 23B to correct the first peripheral position information and its own position information. Through this correction process, the correction unit 233 corrects the error of the first peripheral position information of the detection point registered in the map information 23B. The correction unit 233 only needs to correct the first peripheral position information of the detection point P registered in the map information 23B, for example, by using the least square method. Through the correction process based on the correction unit 233, the cumulative error of the first peripheral position information of the detection point P is corrected. In addition, the information processing device 10 may also be a structure that does not have a correction unit 233.

[0068] As an example, it is assumed that a process of generating map information 23B at a predetermined time such as a parking scene of the mobile body 2 (hereinafter referred to as a map information generation process) is executed. For example, when the information processing device 10 determines that the behavior of the mobile body 2 is a behavior indicating a parking scene, it determines that the predetermined time has arrived. The behavior indicating a parking scene based on backing up is, for example, a case where the speed of the mobile body 2 is below a predetermined speed, a case where the gear position of the mobile body 2 is put into a reverse gear, a case where a signal indicating the start of parking is received by an operation instruction of the user, etc. In addition, the predetermined time is not limited to the parking scene.

[0069] Figure 4 This is a flowchart showing an example of the steps of the map information generation process. Figures 4 to 10 , the steps of map information generation processing, etc. are explained.

[0070] (Map information generation process)

[0071] As the moving body 2 moves, the acquisition unit 20 acquires CAN data from the moving body 2. The own position estimation unit 221 estimates the own position information of the moving body 2 by using the odometry method of the CAN data. For example, the relative movement amount information is acquired by calculating the movement vector by using the odometry method based on the speed, steering angle, etc. of the moving body 2. Based on the above, the relative movement amount information (movement vector) of the moving body 2 is acquired based on the CAN data (step S1).

[0072] The first offset adjustment unit 223 reads the detection range information 23A from the storage unit 231. The first offset adjustment unit 223 adjusts the first offset of the detection range based on the detection range information 23A and the relative movement amount information. The initialization unit 225 determines the detection range in the map information 23B based on the adjusted first offset (first coordinate information) and the map information 23B. In summary, based on the relative movement amount information and the detection range information 23A, the detection range in the map information 23B including the first peripheral position information is determined (step S2). The determined detection range is information indicating a prescribed range of the surroundings of the moving body relative to the moving body 2, and corresponds to the object of initialization based on the initialization unit 225, that is, the update range. That is, the detection range corresponding to the update range corresponds to the information of the surroundings of the moving body based on the position of the moving body 2 (the self-position information).

[0073] The initialization unit 225 initializes the information of the detection points included in the update range in the map information 23B. For example, as the initialization of the information of the detection points included in the update range, the initialization unit 225 deletes the peripheral position information included in the detection range in the first peripheral position information from the map information 23B including the first peripheral position information. Thus, the map information 23B including the first peripheral position information in the update range is initialized (step S3). That is, through the processing of steps S2 and S3, the peripheral position information located in the update range of the first peripheral position information already accumulated in the map information 23B is deleted using the relative movement amount information and the update range. Thus, the initialized map information in which the data of the update range is initialized is generated.

[0074] As the moving body 2 moves, the acquisition unit 20 acquires the position information of the detection point (second peripheral position information) from the detection unit 14 (step S4). When the moving body 2 stops, the acquisition unit 20 acquires the position information of the detection point (second peripheral position information) from the detection unit 14, for example, at the above time intervals.

[0075] The second offset adjustment unit 227 adjusts the second offset of the position of the detection point based on the second peripheral position information and the relative movement amount. The appending unit 229 determines the detection range of the detection unit 14 in the initialized map information based on the adjusted second offset (second coordinate information) and the initialized map information. The appending unit 229 appends the second peripheral position information to the detection range in the map information 23B. Thus, the acquired second peripheral position information is appended to the map information 23B using the relative movement amount information. In summary, the map information 23B is updated (step S5).

[0076] The adding unit 229 outputs the updated map information 23B to the determining unit 30 and the storage unit 231 (step S6). The output destination of the updated map information 23B is not limited to the above, and the updated map information 23B may be output to various components that utilize the updated map information 23B.

[0077] If the map information generation process is not completed (No in step S7), the process after step S1 is repeated. That is, if the information processing device 10 determines that the map information generation process is to be continued (No in step S7), the deletion of the surrounding position information within the update range and the addition of the newly acquired surrounding position information (second surrounding position information) are repeatedly performed from the map information 23B. If the map information generation process is completed (Yes in step S7), the map information generation process is completed. The determination of the end of the map information generation process is, for example, the stop of the power supply in the moving body 2 (for example, when the moving body 2 is a vehicle, the stop of the operating power supply). In addition, the determination of the end of the map information generation process is not limited to the stop of the power supply in the moving body 2, and can be set arbitrarily.

[0078] Figure 5 23B is a schematic diagram showing an example of map information 23B. Figure 5 In the map information 23B shown, for the sake of convenience, the moving body 2, the moving direction TD of the moving body 2, the detection range DA by the detection unit 14, and the character string car indicating other vehicles are shown. Figure 5 As shown, as the mobile body 2 moves, the map information 23B is information in which the position information (first peripheral position information) of each detection point P, namely, point group information, is registered at the corresponding coordinate positions in the three-dimensional coordinate space (global coordinate system).

[0079] Figure 6 It means from Figure 5 FIG. 2 is a diagram showing an example of map information 23B when the position of the mobile body 2 is moving along the travel direction TD. Figure 6 In the illustrated map information 23B, for convenience of explanation, the moving object 2 , the traveling direction TD of the moving object 2 , the detection range DA of the detection unit 14 , and the character string car indicating another vehicle are illustrated.

[0080] In addition, if Figure 6 As shown, in Figure 5 The detection point BP detected in the map information 23B is deleted once by the initialization of the initialization unit 225, but is added to the map information 23B by the addition unit 229 together with other detection points ( Figure 5 The white circle shown in FIG. 23B is added to the map information 23B. Figure 6 As shown, the position information of the objects (cars) around the moving body 2 is acquired by the detection unit 14 (plural sensors) provided behind the moving body 2, and the acquired position information is accumulated in the map information 23B as the first surrounding position information.

[0081] Figure 7 It means from Figure 6 FIG. 2 is a diagram showing an example of map information 23B when the position of the mobile body 2 is moving along the travel direction TD. Figure 7 In the map information 23B shown, for the sake of convenience, the moving body 2, the moving direction TD of the moving body 2, the detection range DA of the detection unit 14, and the character string car representing other vehicles are shown. Figure 7 As shown, even if the position information (detection point) PP is outside the detection range DA, it is updated based on the information of the odometry method using the CAN data, and is therefore accumulated in the map information 23B as the first surrounding position information.

[0082] exist Figures 5 to 7 In the above, as an example, the detection unit 14 (sensor) mounted on the rear of the moving body 2 is used for explanation, but the embodiment is not limited to this. For example, the detection unit 14 (sensor) installed in front of or on the side of the vehicle of the moving body P can also be used to obtain the position information of the objects around the moving body 2 when the moving body 2 moves forward.

[0083] Figure 8 This means that the front of the moving body 2 is moved from Figure 7 The figure shows an example of map information 23B in which the position of the moving object 2 is moved to the left relative to the traveling direction TD and the moving object 2 is backed up and parked between two vehicles (car1 and car2). Figure 8 , for convenience of explanation, a situation is shown in which a pedestrian PDS as a moving body crosses the rear of the moving body 2 from the right side to the left side behind the moving body 2.

[0084] That is, in Figure 8 In the map information 23B, for convenience of explanation, the moving body 2, the pedestrian PDS, the moving direction DM of the pedestrian PDS, the detection range DA of the detection unit 14, and the character string car (car1, car2, car) indicating other vehicles are shown.

[0085] like Figure 8 As shown, the pedestrian PDS moves along the moving direction DM. At this time, without initializing the map information 23B in the update range, the detection points are accumulated along the movement of the pedestrian PDS. That is, there is a detection point at the position after the pedestrian PDS passes, and the information of the detection point accumulated in the map information 23B is different from the position of the pedestrian PDS around the actual moving body 2. On the other hand, in this embodiment, the detection points of the pedestrian PDS are deleted one by one by the initialization unit 225, so they are not accumulated in the map information 23B. That is, as Figure 8 As shown in FIG. 2 , the detection range DA is always updated to the latest state, so the detection points of the past pedestrian PDS do not remain in the map information 23B. Therefore, the difference between the information of the detection points accumulated in the map information 23B and the position of the surrounding pedestrian PDS of the actual moving body 2 is suppressed. In addition, the detection points PP of the past stationary body detected by the detection range DA are sequentially updated and accumulated by the distance measurement method using CAN data, and thus remain in the map information 23B as the first surrounding position information. In addition, the object detected by the detection range DA and updated to the latest state is not limited to the pedestrian PDS, and it can also be other moving bodies or stationary objects.

[0086] Fig. 9 It means Figure 8 An example of map information 23B in different reverse parking situations is shown in FIG. Fig. 9 , for convenience of explanation, a three-dimensional object TDO, a moving object 2 , a traveling direction (backward direction) TD of the moving object 2 , and a detection range DA of the detection unit 14 are shown. Fig. 9 The first peripheral position information PP in the map information 23B represents the map information already accumulated in the map information 23B. Fig. 9 As shown, in the map information 23B, the first peripheral information PP is represented by a white circle.

[0087] exist Figures 5 to 9 , a situation is shown in which the range in which the sensor acquires the position information of the surrounding objects of the mobile body 2 is consistent with the update range in which the position information is deleted. The range in which the position information is acquired and the update range may not be strictly consistent. For example, the sensor may acquire position information farther than the update range. In this case, the update range is set to be included in the range in which the position information is acquired. In addition, the update range may also be set to be farther than the range in which the sensor acquires the position information.

[0088] In addition, Figures 5 to 9 In the example, the map information 23B is updated while the information of the relative movement amount of the moving body 2 is obtained, but the surrounding position information can also be deleted and added when the moving body 2 is stopped. In this case, for example, even if the pedestrian PDS crosses behind the stopped moving body 2, the position information along the trajectory of the pedestrian PDS can be suppressed from being accumulated in the map information 23B.

[0089] Fig.10 It means that the moving object 2 is Fig. 9 An example of map information 23B is shown in which the position of the mobile body 2 in FIG. Fig.10 In FIG. 1 , for convenience of explanation, a three-dimensional object TDO, a moving object 2, a moving direction (backward direction) TD of the moving object 2, and a detection range DA of the detection unit 14 are shown. Fig.10 In the map information 23B in FIG. 1 , the first peripheral position information PP is represented by a white circle, and the second peripheral position information P is represented by a black circle. Fig.10 The position information corresponding to the black circle shown is deleted from the map information 23B by the initialization unit 225 and added to the map information 23B by the addition unit 229 .

[0090] return Figure 3 , the determination unit 30 is described. The determination unit 30 uses the map information 23B and the own position information to determine the projection shape of the projection surface. That is, the determination unit 30 uses the position information (first peripheral position information and second peripheral position information) of the detection point P accumulated in the map information 23B and the own position information to determine the projection shape of the projection surface. The projection surface refers to a three-dimensional surface for projecting the peripheral image of the mobile body 2.

[0091] In other words, the projection plane is a virtual three-dimensional plane on which the captured image 50 of the surroundings of the moving body is projected. The projection shape of the projection plane has a three-dimensional (3D) shape virtually formed in a virtual space corresponding to the actual space. The surrounding image of the moving body 2 refers to the captured image of the surroundings of the moving body 2. In the present embodiment, the surrounding image of the moving body 2 is the captured image captured by the capturing units 12A to 12D, respectively.

[0092] When the own position information is registered in the map information 23B, the decision unit 30 decides the shape of the projection surface based on the map information 23B. The decision unit 30 decides the shape obtained by deforming the reference projection surface according to the position information of the detection point P registered in the map information 23B as the projection shape.

[0093] Fig.11: is a schematic diagram showing an example of the reference projection plane 40. The reference projection plane 40 is, for example, a projection plane of a shape that serves as a reference when the shape of the projection plane is changed. The shape of the reference projection plane 40 is, for example, a bowl shape, a cylindrical shape, or the like.

[0094] The bowl-shaped shape refers to a shape having a bottom surface 40A and a side wall surface 40B, one end of the side wall surface 40B is continuous with the bottom surface 40A, and the other end is open. The width of the horizontal section of the side wall surface 40B increases from the bottom surface 40A side toward the open side of the other end. The bottom surface 40A is, for example, circular. Here, the circular shape refers to a circular shape other than a perfect circle, such as a perfect circle, an ellipse, etc. The horizontal section refers to an orthogonal plane orthogonal to the vertical direction (arrow Z direction). The orthogonal plane is a two-dimensional plane along the arrow X direction orthogonal to the arrow Z direction, and the arrow Y direction orthogonal to the arrow Z direction and the arrow X direction. In the following, the horizontal section and the orthogonal plane are sometimes referred to as the XY plane for explanation. In addition, the bottom surface 40A may also be a shape other than a circular shape such as an oval.

[0095] The cylindrical shape refers to a shape composed of a circular bottom surface 40A and a side wall surface 40B continuous with the bottom surface 40A. In addition, the side wall surface 40B constituting the cylindrical reference projection surface 40 is a cylindrical shape with an opening at one end continuous with the bottom surface 40A and an opening at the other end. Among them, the side wall surface 40B constituting the cylindrical reference projection surface 40 is a shape in which the diameter of the XY plane is substantially constant from the bottom surface 40A side toward the opening side of the other end. In addition, the bottom surface 40A may also be a shape other than a circular shape such as an oval.

[0096] In this embodiment, a case where the reference projection plane 40 is in a bowl shape is described as an example. The reference projection plane 40 is a three-dimensional model virtually formed in a virtual space in which a bottom surface 40A is set to be substantially consistent with the road surface below the moving body 2 and the center of the bottom surface 40A is set to the self-position S of the moving body 2. The self-position S is equivalent to the self-position information.

[0097] The determination unit 30 determines, as the projection shape, a shape obtained by deforming the reference projection plane 40 into a shape passing through the detection point P closest to the moving body 2. The shape passing through the detection point P means a shape in which the deformed side wall surface 40B passes through the detection point P.

[0098] Fig.12 : is a schematic diagram showing an example of the projection shape 41. The determination unit 30 determines the projection shape 41 as a shape obtained by deforming the reference projection plane 40 into a shape passing through the center of the bottom surface 40A of the reference projection plane 40, that is, the detection point P closest to the self-position S of the moving body 2. The self-position S is the latest self-position S calculated by the self-position estimation unit 24, that is, the latest position of the moving body 2.

[0099] The determination unit 30 determines the detection point P closest to the self-position S among the plurality of detection points P registered in the map information 23B. Specifically, the XY coordinates of the center position (self-position S) of the mobile body 2 are set to (X, Y) = (0, 0). Furthermore, the determination unit 30 determines the detection point P whose value of X2+Y2 represents the minimum value as the detection point P closest to the self-position S. Furthermore, the determination unit 30 determines the shape obtained by deforming the side wall surface 40B of the reference projection surface 40 through the detection point P as the projection shape 41.

[0100] Specifically, the determination unit 30 determines the deformed shape of the bottom surface 40A and the area of ​​the part of the side wall surface 40B as the projection shape 41 so that when the reference projection surface 40 is deformed, the area of ​​the part of the side wall surface 40B becomes a wall surface passing through the detection point P closest to the moving body 2. The deformed projection shape 41 is, for example, a shape that rises from the rising line 44 on the bottom surface 40A toward the direction close to the center of the bottom surface 40A. The rising, for example, means that the side wall surface 40B and the part of the bottom surface 40A are bent or folded toward the direction close to the center of the bottom surface 40A so that the angle between the side wall surface 40B of the reference projection surface 40 and the bottom surface 40A becomes a smaller angle.

[0101] The determination unit 30 determines that the determination area in the reference projection surface 40 is deformed so as to protrude toward a position passing through the detection point P from the viewpoint (overhead view) of the XY plane. The shape and range of the determination area can also be determined based on a predetermined reference. Furthermore, the determination unit 30 determines the shape obtained by deforming the reference projection surface 40 in such a manner that the distance from the own position S becomes larger from the protruding determination area toward the area outside the determination area in the side wall surface 40B.

[0102] In detail, it is preferred that Fig.12 As shown, the projection shape 41 is determined so that the shape of the outer periphery of the cross section along the XY plane becomes a curved shape. The shape of the outer periphery of the cross section of the projection shape 41 is, for example, a circular shape, but may be a shape other than a circular shape.

[0103] In addition, the determination unit 30 may determine the shape obtained by deforming the reference projection plane 40 into a shape along an asymptotic curve as the projection shape 41. The asymptotic curve is an asymptotic curve of a plurality of detection points P. The determination unit 30 generates an asymptotic curve of a plurality of detection points P of a predetermined number from the detection point P closest to the own position S of the moving body 2 toward a direction away from it. The number of the detection points P only needs to be a plurality. For example, the number of the detection points P is preferably 3 or more. In addition, in this case, it is preferred that the determination unit 30 generates an asymptotic curve of a plurality of detection points P located at positions separated by more than a predetermined angle when viewed from the own position S.

[0104] Fig.13 It is an explanatory diagram of the asymptotic curve Q. Fig.13 This is an example in which the asymptotic curve Q is shown in the projection image 51 obtained by projecting the captured image onto the projection plane when the moving body 2 is viewed from above. For example, it is assumed that the decision unit 30 determines three detection points P in the order of approaching the own position S of the moving body 2. In this case, the decision unit 30 generates the asymptotic curve Q of the three detection points P. Then, the decision unit 30 may determine the shape obtained by deforming the reference projection plane 40 into a shape along the generated asymptotic curve Q as the projection shape 41.

[0105] Furthermore, the determination unit 30 may divide the area around the own position S of the moving body 2 for each determined angle range, and determine the detection point P closest to the moving body 2 for each of the angle ranges, or determine a plurality of detection points P in the order of approaching the moving body 2. Furthermore, the determination unit 30 may determine, as the projection shape 41, a shape obtained by deforming the reference projection plane 40 into a shape passing through the determined detection point P or a shape along an asymptotic curve Q of the determined plurality of detection points P for each of the angle ranges.

[0106] Next, an example of a detailed configuration of the determination unit 30 will be described.

[0107] Fig.14 : is a schematic diagram showing an example of the structure of the determination unit 30. The determination unit 30 includes an extraction unit 30A, a nearest neighbor identification unit 30B, a reference projection surface shape selection unit 30C, a scale determination unit 30D, an asymptotic curve calculation unit 30E, and a shape determination unit 30F.

[0108] The extraction unit 30A uses the own position information of the mobile body 2 and the map information 23B to extract the detection points P that exist within a certain range from the plurality of detection points P included in the map information 23B. Here, the map information 23B includes information on the distance from the mobile body 2 to the detection point P. The certain range is, for example, a range from the road surface where the mobile body 2 is arranged to a height equivalent to the vehicle height of the mobile body 2. In addition, the range is not limited to this range. The extraction unit 30A extracts the detection points P within the range, thereby, for example, being able to extract the detection points P of objects that become obstacles to the movement of the mobile body 2. The extraction unit 30A outputs the distance information of each of the extracted detection points P to the nearest determination unit 30B. The extraction unit 30A outputs the current own position information of the mobile body 2 to the virtual viewpoint line of sight determination unit 34. In addition, when the distance information from the mobile body 2 to the detection point P is not included in the map information 23B, the distance information may be converted using the map information 23B and the own position information and input to the extraction unit 30A.

[0109] The nearest neighbor specifying unit 30B divides the area around the own position S of the mobile body 2 for each specified angle range, specifies the detection point P closest to the mobile body 2 for each angle range, or specifies a plurality of detection points P in the order of approaching the mobile body 2. The nearest neighbor specifying unit 30B specifies the detection point P using the distance information received from the extracting unit 30A. In the present embodiment, a method in which the nearest neighbor specifying unit 30B specifies a plurality of detection points P in the order of approaching the mobile body 2 for each angle range is described as an example.

[0110] The nearest neighbor specifying unit 30B outputs the distance information of the detection point P specified for each angle range to the reference projection surface shape selecting unit 30C, the scale determining unit 30D, and the asymptotic curve calculating unit 30E.

[0111] The reference projection plane shape selection unit 30C selects the shape of the reference projection plane 40. The reference projection plane shape selection unit 30C selects the shape of the reference projection plane 40 by reading a determined shape from the storage unit 231 storing a plurality of types of shapes of the reference projection plane 40. For example, the reference projection plane shape selection unit 30C selects the shape of the reference projection plane 40 based on the positional relationship between its own position and the surrounding three-dimensional objects, distance information, etc. In addition, the shape of the reference projection plane 40 can also be selected by the user's operation instruction. The reference projection plane shape selection unit 30C outputs the shape information of the determined reference projection plane 40 to the shape determination unit 30F. In the present embodiment, as described above, the reference projection plane shape selection unit 30C selects the shape of the bowl-shaped reference projection plane 40 as an example for explanation.

[0112] The scale determination unit 30D determines the scale of the reference projection surface 40 of the shape selected by the reference projection surface shape selection unit 30C. The scale determination unit 30D makes a decision to reduce the scale, for example, when there are a plurality of detection points P within a range of a predetermined distance from the own position S. The scale determination unit 30D outputs the scale information of the determined scale to the shape determination unit 30F.

[0113] The asymptotic curve calculation unit 30E uses each of the distance information of the detection points P closest to the own position S in each angle range from the own position S received from the nearest determination unit 30B, and outputs the asymptotic curve information of the calculated asymptotic curve Q to the shape determination unit 30F and the virtual viewpoint line of sight determination unit 34. In addition, the asymptotic curve calculation unit 30E may calculate the asymptotic curve Q of the detection points P accumulated according to a plurality of parts of the reference projection plane 40. Furthermore, the asymptotic curve calculation unit 30E may output the asymptotic curve information of the calculated asymptotic curve Q to the shape determination unit 30F and the virtual viewpoint line of sight determination unit 34.

[0114] The shape determination unit 30F enlarges or reduces the reference projection surface 40 of the shape represented by the shape information received from the reference projection surface shape selection unit 30C to the scale of the scale information received from the scale determination unit 30D. Furthermore, the shape determination unit 30F determines the shape obtained by deforming the enlarged or reduced reference projection surface 40 into a shape based on the asymptotic curve information of the asymptotic curve Q received from the asymptotic curve calculation unit 30E as the projection shape 41. The shape determination unit 30F outputs the projection shape information of the determined projection shape 41 to the deformation unit 32.

[0115] return Figure 3 , the description continues. Next, the deformation unit 32 is described. The deformation unit 32 deforms the reference projection plane 40 into a projection shape 41 represented by the projection shape information received from the determination unit 30. That is, the deformation unit 32 deforms the projection plane on which the captured image of the surrounding of the moving body is projected based on the map information 23B. Specifically, the deformation unit 32 deforms the projection plane based on the first surrounding position information and the second surrounding position information that are not initialized in the map information 23B.

[0116] Through this deformation process, the deformation unit 32 generates a deformed reference projection surface 40, that is, a deformed projection surface 42 (see Fig.12 ). That is, the deformation unit 32 deforms the reference projection plane 40 using the position information of the detection point P accumulated in the map information 23B and the own position information of the moving body 2. In detail, for example, the deformation unit 32 deforms the reference projection plane 40 into a curved surface shape passing through the detection point P closest to the moving body 2 based on the projection shape information. Through this deformation processing, the deformation unit 32 generates a deformed projection plane 42.

[0117] For example, the deformation unit 32 deforms the reference projection plane 40 into a shape of an asymptotic curve Q along a plurality of detection points P whose number is predetermined in the order of approaching the moving body 2 based on the projection shape information. In addition, it is preferred that the deformation unit 32 deforms the reference projection plane 40 using the position information of the detection points P and the own position information of the own position S acquired before the first moment.

[0118] Here, the first moment refers to the latest moment at which the position information of the detection point P is detected by the detection unit 14, or any moment past the latest moment. For example, the detection point P acquired before the first moment includes the position information of a certain object located in the vicinity of the mobile body 2, and the detection point P acquired at the first moment does not include the position information of the certain object located in the vicinity. The decision unit 30 uses the position information of the detection point P acquired before the first moment included in the map information 23B to determine the projection shape 41 in the same manner as described above. Furthermore, the deformation unit 32 uses the projection shape information of the projection shape 41 to generate the deformation projection surface 42 in the same manner as described above.

[0119] In this case, for example, even if the position information of the detection point P detected by the detection unit 14 at the first moment does not include the position information of the detection point P detected in the past from that moment on, the deformation unit 32 can generate the deformation projection surface 42 corresponding to the detection point P detected in the past.

[0120] Next, the projection conversion unit 36 ​​will be described. The projection conversion unit 36 ​​generates a projection image 51 obtained by projecting the captured image acquired from the imaging unit 12 onto the reference projection plane 40 deformed by the deformation unit 32, that is, the deformed projection plane 42. In detail, the projection conversion unit 36 ​​receives the deformed projection plane information of the deformed projection plane 42 from the deformation unit 32. The deformed projection plane information refers to information indicating the deformed projection plane 42. The projection conversion unit 36 ​​projects the captured image acquired from the imaging unit 12 via the acquisition unit 20 onto the deformed projection plane 42 indicated by the received deformed projection plane information. Through this projection processing, the projection conversion unit 36 ​​generates the projection image 51. The projection conversion unit 36 ​​converts the projection image 51 into a virtual viewpoint image. The virtual viewpoint image refers to an image obtained by visually confirming the projection image 51 from a virtual viewpoint in an arbitrary direction.

[0121] use Fig.12 The projection conversion unit 36 ​​is described. The projection conversion unit 36 ​​projects the captured image 50 onto the deformed projection surface 42. Moreover, the projection conversion unit 36 ​​generates an image obtained by visually confirming the captured image 50 projected onto the deformed projection surface 42 from an arbitrary virtual viewpoint O along the line of sight direction L, namely a virtual viewpoint image (not shown). The position of the virtual viewpoint O may be, for example, the latest self-position S of the moving body 2. In this case, the value of the XY coordinate of the virtual viewpoint O may be set to the value of the XY coordinate of the latest self-position S of the moving body 2. In addition, the value of the Z coordinate (position in the vertical direction) of the virtual viewpoint O may be set to the value of the Z coordinate of the detection point P closest to the self-position S of the moving body 2. The line of sight direction L may also be determined based on a predetermined reference, for example.

[0122] The line of sight direction L may be, for example, a direction from the virtual viewpoint O toward the detection point P closest to the own position S of the moving body 2. In addition, the line of sight direction L may be a direction passing through the detection point P and perpendicular to the deformation projection surface 42. The virtual viewpoint line of sight information indicating the virtual viewpoint O and the line of sight direction L is generated by the virtual viewpoint line of sight determination unit 34.

[0123] return Figure 3, continue the explanation. The virtual viewpoint line of sight determining unit 34 determines the virtual viewpoint line of sight information, for example, according to the following steps. The virtual viewpoint line of sight determining unit 34 determines the direction that passes through the detection point P closest to the own position S of the moving body 2 and is perpendicular to the deformation projection surface 42 as the line of sight direction L. In addition, the virtual viewpoint line of sight determining unit 34 may also fix the direction of the line of sight direction L, and determine the coordinates of the virtual viewpoint O to be any Z coordinate, or any XY coordinate in the direction from the asymptotic curve Q to away from the own position S. In this case, the XY coordinates may also be the coordinates of a position farther from the asymptotic curve Q than the own position S. Moreover, the virtual viewpoint line of sight determining unit 34 outputs the virtual viewpoint line of sight information representing the virtual viewpoint O and the line of sight direction L to the projection conversion unit 36. In addition, as Fig.13 As shown, the sight line direction L may also be the direction from the virtual viewpoint O toward the vertex W of the asymptotic curve Q.

[0124] The projection conversion unit 36 ​​receives the virtual viewpoint line of sight information from the virtual viewpoint line of sight determination unit 34. The projection conversion unit 36 ​​determines the virtual viewpoint O and the line of sight direction L by receiving the virtual viewpoint line of sight information. Furthermore, the projection conversion unit 36 ​​generates a virtual viewpoint image, which is an image visually recognized from the virtual viewpoint O along the line of sight direction L, based on the captured image 50 projected onto the deformation projection surface 42. The virtual viewpoint image corresponds to, for example, an image in which the captured image 50 can be visually recognized from the virtual viewpoint O along the line of sight direction L. The projection conversion unit 36 ​​outputs the virtual viewpoint image to the image synthesis unit 38.

[0125] The image synthesis unit 38 generates a synthetic image obtained by extracting a part or all of the virtual viewpoint image. For example, the image synthesis unit 38 determines the width of the overlapping portion of the plurality of captured images 50 included in the virtual viewpoint image, performs a pasting process on the captured images 50, and determines a mixing process of the captured images 50 displayed in the overlapping portion. Thus, a synthetic image 54 is generated. Furthermore, the image synthesis unit 38 outputs the synthetic image 54 to the display unit 16. In addition, the synthetic image 54 may be a bird's-eye view image in which the top of the moving body 2 is set as the virtual viewpoint O, or the inside of the moving body 2 is set as the virtual viewpoint O, and the moving body 2 is displayed semi-transparently.

[0126] Next, an example of the flow of image processing executed by the information processing device 10 will be described. Fig.16 : is a flowchart showing an example of the steps of image processing executed by the information processing device 10 .

[0127] (Image Processing)

[0128] The acquisition unit 20 acquires the captured image 50 from the imaging unit 12 (step S10 ). The acquisition unit 20 outputs the acquired captured image 50 to the projection conversion unit 36 ​​.

[0129] The acquisition unit 20 acquires position information of each of the plurality of detection points P from the detection unit 14. The acquisition unit 20 also acquires CAN data from the ECU 3. The map information 23B is generated by the above-mentioned map information generation process based on the position information of each of the plurality of detection points P and the CAN data (step S11).

[0130] The determination unit 30 acquires the generated map information 23B (step S12 ).

[0131] The extraction unit 30A extracts the detection points P that exist within the specified range from among the detection points P. The nearest identification unit 30B identifies a plurality of detection points P in the order of proximity to the moving object 2 for each angular range (azimuth) around the moving object 2, using the distance information of each of the extracted detection points P. Thus, the nearest detection point to the moving object 2 is identified for each azimuth in the specified range (step S13).

[0132] The reference projection surface shape selection unit 30C selects the shape of the reference projection surface 40 (step S14). As described above, the reference projection surface shape selection unit 30C selects the bowl-shaped reference projection surface 40 as an example for explanation. In addition, regarding the selection of the reference projection surface, the shape of the reference projection surface 40 used for image processing can also be selected from a plurality of types of shapes of the reference projection surface 40 based on the self-position information of the moving body 2, the position information of the objects around the moving body 2 (the first peripheral position information and / or the second peripheral position information), the distance information, etc.

[0133] The scale determination unit 30D determines the scale of the reference projection plane 40 of the shape selected in step S14 (step S15 ).

[0134] The asymptotic curve calculation unit 30E calculates the asymptotic curve Q using each of the distance information of the plurality of detection points P for each angle range determined in step S13 (step S16 ).

[0135] The shape determination unit 30F enlarges or reduces the reference projection plane 40 of the shape selected in step S14 to the scale determined in step S15. Furthermore, the shape determination unit 30F deforms the enlarged or reduced reference projection plane 40 into a shape along the asymptotic curve Q calculated in step S16. The shape determination unit 30F determines the deformed shape as the projection shape 41 (step S17).

[0136] The deformation unit 32 deforms the reference projection surface 40 selected by the reference projection surface shape selection unit 30C into the projection shape 41 determined by the determination unit 30 (step S18). Through this deformation process, the deformation unit 32 generates the deformed reference projection surface 40, that is, the deformed projection surface 42 (see Fig.12 ).

[0137] The virtual viewpoint sight line determination unit 34 determines the virtual viewpoint sight line information (step S19). For example, the virtual viewpoint sight line determination unit 34 sets the own position S of the mobile body 2 as the virtual viewpoint O, and determines the direction from the virtual viewpoint O toward the position of the vertex W of the asymptotic curve Q as the sight line direction L. Specifically, the virtual viewpoint sight line determination unit 34 determines the direction toward the vertex W of the asymptotic curve Q of a certain angle range among the asymptotic curves Q calculated for each angle range in step S16 as the sight line direction L.

[0138] The projection conversion unit 36 ​​projects the captured image 50 acquired in step S10 onto the deformed projection surface 42 generated in step S17. Furthermore, the projection conversion unit 36 ​​projects the captured image 50 onto the deformed projection surface 42 from the virtual viewpoint O determined in step S19 along the line of sight direction L, thereby converting the projected image 51 into a virtual viewpoint image. That is, the projection conversion unit 36 ​​converts the captured image 50 projected onto the deformed projection surface 42 into a virtual viewpoint image using the virtual viewpoint line of sight information (step S20).

[0139] The image synthesis unit 38 generates a synthesized image 54 by extracting part or all of the virtual viewpoint image generated in step S20 (step S21). The virtual viewpoint image includes an overlapping portion (hereinafter referred to as an overlapping portion) of a plurality of captured images 50. The image synthesis unit 38 performs, for example, a determination of the width of the overlapping portion, a bonding process of the captured images 50, and a determination of a blending process of the captured images 50 displayed in the overlapping portion.

[0140] The image synthesis unit 38 performs display control to output the generated synthesized image 54 to the display unit 16 (step S22 ). As a result, the generated synthesized image 54 is displayed on the display unit 16 .

[0141] Next, the information processing device 10 determines whether to end the image processing (step S23). For example, the information processing device 10 makes the determination in step S23 by determining whether a signal indicating the stop of the movement of the mobile body 2 (for example, the stop of the engine, etc.) is received from the ECU 3. In addition, for example, the information processing device 10 may also make the determination in step S23 by determining whether an instruction to end the image processing is received through an operation instruction from a user, etc.

[0142] If a negative determination is made in step S23 (step S23: No), the processing from step S10 to step S22 is repeated. If a positive determination is made in step S23 (step S23: Yes), the image processing routine is terminated.

[0143] As described above, the information processing device 10 of the embodiment includes a map information generating unit 22, which initializes a range (detection range) related to detection by a sensor (detection unit 14) mounted on the mobile body 2 in the map information 23B including the first surrounding position information, and adds the second surrounding position information acquired from the sensor to the detection range. The first surrounding position information is information about the position of an object located around the mobile body. For example, as the initialization of the data in the detection range DA, the map information generating unit 22 deletes the surrounding position information included in the detection range in the first surrounding position information from the map information 23B.

[0144] Thus, according to the information processing device 10 of the embodiment, it is possible to update the surrounding position information included in the detection range in the map information 23B according to the acquisition of the position information. That is, according to the information processing device 10 of the embodiment, the surrounding position information of the mobile body 2 (vehicle) in the map information 23B is deleted and added in sequence, and the map information 23B reflecting the latest situation of the surrounding of the mobile body can be generated. In other words, according to the information processing device 10 of the embodiment, the position information within the update range in the map information 23B can always be kept in the latest situation.

[0145] Therefore, according to the information processing device 10 of the embodiment, as Figure 8 As shown, even if a moving object such as a pedestrian PDS passes through the detection range, the accumulation of position information along the trajectory of a moving object (pedestrian, etc.) near the moving object into the map information 23B can be suppressed. Thus, according to the information processing device 10 of the embodiment, it is possible to generate high-precision (accurate) map information 23B.

[0146] In addition, the information processing device 10 of the embodiment further includes a deformation unit 32, which deforms the projection surface (the selected reference projection surface 40) on which the captured image 50 of the surroundings of the moving object is projected based on the map information 23B generated by the map information generation process. For example, the information processing device 10 of the embodiment deforms the projection surface based on the first surrounding position information and the second surrounding position information that are not initialized in the map information 23B. For example, Fig.10 As shown in FIG. 1 , as the peripheral position information used in the deformation process of the projection surface, the first peripheral position information (white circle) PP and the second peripheral position information (black circle) P are used.

[0147] Thus, according to the information processing device 10 of the embodiment, the projection surface can be deformed using the accurate (correct) map information 23B, so that the captured image 50 can be projected onto the appropriately deformed projection surface, and a synthetic image (bird's-eye view image) can be presented to the user. Thus, according to the information processing device 10 of the embodiment, a natural bird's-eye view image with improved visibility can be presented to the user.

[0148] In addition, a plurality of sensors related to the information processing device 10 of the embodiment are mounted on the mobile body 2. For example, a plurality of sensors related to the information processing device 10 of the embodiment are arranged in an array in the exterior of the mobile body 2. In addition, the sensor related to the information processing device 10 of the embodiment may be a distance sensor mounted on the rear of the mobile body 2, and the first peripheral position information may be obtained from the distance sensor. In addition, the sensor related to the information processing device 10 of the embodiment may be further arranged on the side of the mobile body 2 as a distance sensor. In addition, in the case where the detection unit 14 is implemented by a plurality of sensors, the peripheral position information obtained by a part of the sensors (for nearby) is used as the object of accumulation / deletion, and the peripheral position information obtained by other sensors (for distant) may also be used only for accumulation. Thus, according to the information processing device 10 of the embodiment, it is possible to generate map information 23B with good accuracy for any direction around the mobile body 2.

[0149] In addition, in the information processing device 10 of the embodiment, the detection range corresponding to the update range corresponds to the information about the surroundings of the moving body based on the position of the moving body 2 (self-position information). In addition, according to the information processing device 10 of the embodiment, the surrounding position information to be deleted is determined based on the information of the detection range and the information (relative movement amount information) of the relative movement amount (movement vector) of the moving body 2 relative to the origin in the map information 23B. Therefore, according to the information processing device 10 of the embodiment, regardless of whether the moving body 2 moves or not, it is possible to determine the surrounding position information or update range of the deletion object based on the moving body 2. Therefore, according to the information processing device 10 of the embodiment, regardless of whether the moving body 2 moves or stops, it is possible to maintain the surrounding position information always included in the update range in the latest state.

[0150] In this embodiment, as an example of the use of the map information 23B generated by the map information generation process, the deformation process of the projection surface during reverse parking is described, but the use of the map information 23B is not limited to the deformation process of the projection surface. For example, when the moving body 2 is a vehicle, the map information 23B can also be used for automatic driving or forward parking.

[0151] When the technical idea in the embodiment is realized by using an information processing method, the information processing method initializes a range related to detection by a sensor mounted on the mobile body 2 in the map information 23B including the first surrounding position information, and adds the second surrounding position information acquired from the sensor to the range, wherein the first surrounding position information is information on the position of an object located around the mobile body. The steps and effects of the map information generation process performed by the information processing method are the same as those in the embodiment, and thus the description thereof is omitted.

[0152] In the case where the technical idea of ​​the present embodiment is realized by an information processing program, the information processing program causes the computer to execute as follows: in the map information 23B containing the first peripheral position information, the range related to the detection of the sensor mounted on the mobile body 2 is initialized, and the second peripheral position information obtained from the sensor is added to the range, and the first peripheral position information is the information of the position of the object located around the mobile body. For example, by installing the information processing program from a non-volatile storage medium to various server devices (processing devices) and expanding them on the memory, the map information generation process can be realized. At this time, the program that enables the computer to execute the method can also be stored in a storage medium such as a disk (hard disk, etc.), an optical disk (CD-ROM, DVD, etc.), and a semiconductor memory for distribution. The processing steps and effects in the information processing program are the same as those in the embodiment, so the description is omitted.

[0153] According to this configuration, by sequentially updating the position information within the update range, it is possible to generate high-precision map information 23B. As a result, according to one embodiment of the information processing device disclosed in the present application, for example, the shape of the projection surface can be appropriately deformed using high-precision map information 23B.

[0154] The above describes the embodiments and variations, but the information processing device 10, information processing method, and information processing program disclosed in the present application are not directly limited to the above embodiments, etc., and the structural elements can be deformed and concretized within the scope of the main purpose in each implementation stage, etc. In addition, various inventions can be formed by appropriate combinations of multiple structural elements disclosed in the above embodiments and variations, etc. For example, some structural elements can be deleted from all the structural elements shown in the embodiments.

[0155] Description of Reference Numerals

[0156] 1…information processing system; 10…information processing device; 12, 12A~12D…shooting unit; 14, 14A~14D…detection unit; 20…acquisition unit; 22…map information generation unit; 23A…detection range information; 23B…map information; 30…determination unit; 30A…extraction unit; 30B…nearest neighbor determination unit; 30C…reference projection surface shape selection unit; 30D…scale determination unit; 30E…asymptotic curve calculation unit; 30F…shape determination unit; 32…deformation unit; 34…virtual viewpoint line of sight determination unit; 36…projection conversion unit; 38…image synthesis unit; 221…self-position estimation unit; 223…first offset adjustment unit; 225…initialization unit; 227…second offset adjustment unit; 229…addition unit; 231…storage unit; 233…correction unit.

Claims

1. An information processing device, wherein: A map information generating unit is provided, which initializes a range related to detection based on a sensor mounted on a mobile body in map information including first surrounding position information, and adds second surrounding position information obtained from the sensor to the range, wherein the first surrounding position information is information on the position of an object located around the mobile body.

2. The information processing device according to claim 1, wherein: As initialization of the range, the map information generating unit deletes the surrounding position information included in the range in the first surrounding position information from the map information.

3. The information processing device according to claim 1, wherein: The device further includes a deformation unit that deforms a projection surface on which a captured image of the periphery of the moving object is projected based on the map information.

4. The information processing device according to claim 3, wherein: The deformation unit deforms the projection surface based on the first surrounding position information and the second surrounding position information which are not initialized in the map information.

5. The information processing device according to claim 1, wherein: A plurality of the sensors are mounted on the moving object.

6. The information processing device according to claim 5, wherein: The plurality of sensors are arranged in an array on the exterior of the moving body.

7. The information processing device according to claim 1, wherein: The sensor is a distance sensor mounted on the rear of the moving body. The first peripheral position information is acquired from the distance sensor.

8. The information processing device according to claim 7, wherein: The sensor is further arranged on the side of the moving body as the distance sensor.

9. The information processing device according to any one of claims 1 to 8, wherein: The range corresponds to information about the periphery of the moving object based on the position of the moving object.

10. The information processing device according to claim 9, wherein: The map information generating unit determines the peripheral position information to be deleted as initialization based on the information of the range and the information of the relative movement amount of the moving object with respect to the origin in the map information.

11. An information processing method, wherein: In the map information including the first surrounding location information, Initialize the detection range of the sensor mounted on the moving object. adding the second surrounding position information acquired from the sensor to the range, The first peripheral position information is information on the position of an object located around the moving body.

12. An information processing program, wherein: Used to make the computer perform the following steps: In the map information including the first surrounding position information, a range related to detection by a sensor mounted on the mobile body is initialized. adding the second surrounding position information acquired from the sensor to the range, The first peripheral position information is information on the position of an object located around the moving body.

Images