Map generation system, map generation method and self-position estimation device

Abstract

To provide a map generation system and a map generation method capable of forming a 3D map that reproduces a landscape of an optional viewpoint with high accuracy.SOLUTION: A map generation system includes a photographing part (3) for photographing a landscape around a vehicle, a position attitude sensor (4) for detecting a present position of the vehicle and detecting an attitude of the vehicle, a vehicle speed acquisition part (5) for acquiring vehicle speed information of the vehicle, a state estimation part (61) for estimating a three-dimensional position and an attitude angle of the vehicle on the basis of present position information showing the present position and attitude information showing the attitude detected by the position attitude acquisition part and the vehicle speed information acquired by the vehicle speed acquisition part, and a 3D map data generation part (63) for generating 3D map data that reproduces a landscape of an optional viewpoint from the three-dimensional position and the attitude angle estimated by the state estimation part and time series data of the imaging information of an imaging part.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to a map generation system that generates three-dimensional (hereinafter referred to as 3D) map data, a map generation method, and a self-location estimation device that estimates a self-location based on the 3D map data. [Background technology]

[0002] For example, Patent Document 1 proposes a technology in which 3D information about the surroundings of a vehicle is acquired using a camera, feature points in the image are extracted from the 3D information, and 3D map data about the surroundings of the vehicle and the vehicle's self-position are estimated based on the feature points. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2018-139084 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the method of generating 3D map data using image feature points, such as that described in Patent Document 1, has the problem that, depending on the conditions, only map data with low accuracy can be obtained that is not suitable for use in self-location estimation. For example, when extracting feature points in an image based on large changes in brightness, if there are no areas where the brightness changes significantly, feature points cannot be extracted, and 3D map data cannot be generated with high accuracy. Furthermore, there is also the problem that map data using images before feature point extraction is highly distorted and cannot be used for self-location estimation as is.

[0005] An object of the present disclosure is to provide a map generation system and a map generation method that can generate a 3D map that reproduces a landscape from an arbitrary viewpoint with high accuracy. Another object of the present disclosure is to provide a self-location estimation device that can generate a 3D map that reproduces a landscape from an arbitrary viewpoint with high accuracy and perform self-location estimation with high accuracy. [Means for solving the problem]

[0006] A first aspect of the present disclosure is A map generation system that generates 3D map data of an area observed by a vehicle (2), an image capturing unit (3) for capturing an image of the surroundings of the vehicle as a captured image of the vehicle; a position and attitude sensor (4) for detecting the current position of the vehicle and detecting the attitude of the vehicle; a vehicle speed acquisition unit (5) for acquiring vehicle speed information of the vehicle; a state estimation unit (61) that estimates a 3D position and an attitude angle of the vehicle based on current position information indicating the current position and attitude information indicating the attitude detected by the position and attitude acquisition unit, and the vehicle speed information acquired by the vehicle speed acquisition unit; and a 3D map data generation unit (63) that generates 3D map data that reproduces a landscape from an arbitrary viewpoint from the 3D position and the attitude angle estimated by the state estimation unit and time-series data of the imaging information of the imaging unit.

[0007] In this way, the current position and attitude angle of the vehicle are detected with high accuracy based on the current position and attitude of the vehicle detected by the position and attitude angle sensor and the vehicle speed information acquired by the vehicle speed acquisition unit. 3D map data that reproduces the scenery from any viewpoint is then generated from the time-series data of the highly accurate current position and attitude angle and the vehicle's image information from the image capture unit. This makes it possible to generate highly accurate 3D map data.

[0008] A self-location estimation device according to a second aspect of the present disclosure includes, in addition to the map generation system according to the first aspect, a comparator (65) provided in the vehicle, which compares a 3D map included in the 3D map data generated by the 3D map data generation unit with the subject vehicle image captured by the imaging unit; and a self-position estimation unit (66) that is provided in the vehicle and performs self-position estimation to estimate the current position of the vehicle based on the comparison result of the comparator.

[0009] In this way, the 3D map included in the 3D map data generated by the 3D map data generation unit is compared with the image captured by the image capture unit, and self-location estimation can be performed based on the comparison result. Therefore, it is possible to create a 3D map that reproduces a landscape from any viewpoint with high accuracy, and to perform self-location estimation with high accuracy based on the high-accuracy 3D map.

[0010] A third aspect of the present disclosure is A map generation method for generating 3D map data of an area observed by a vehicle (2), comprising: Photographing the scenery around the vehicle as a subject vehicle photographed image by a photographing unit (3); Detecting the current position of the vehicle and detecting the attitude of the vehicle by a position and attitude sensor (4); Acquiring vehicle speed information of the vehicle by a vehicle speed acquisition unit (5); estimating a 3D position and an attitude angle of the vehicle based on current position information indicating the current position, attitude information indicating the attitude, and the vehicle speed information; and generating 3D map data that reproduces a landscape from an arbitrary viewpoint from the 3D position, the attitude angle, and time-series data of the host vehicle's image capture information from the image capture unit.

[0011] In this way, the current position and attitude angle of the vehicle are detected with high accuracy based on the current position and attitude of the vehicle detected by the position and attitude angle sensor and the vehicle speed information acquired by the vehicle speed acquisition unit. 3D map data that reproduces the scenery from any viewpoint is then generated from the time-series data of the highly accurate current position and attitude angle and the imaging information from the imaging unit. This makes it possible to generate highly accurate 3D map data.

[0012] The reference symbols in parentheses attached to each component indicate an example of the correspondence between the component and the specific components described in the embodiments described below. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is an overall view of a map generation system according to a first embodiment of the present disclosure. [Figure 2] FIG. 1 is a block diagram of a map generation system. [Figure 3] Schematic diagram of a deep neural network using the NeRF architecture. [Figure 4] FIG. 10 is a diagram showing a 3D map restored using 3D map data. [Figure 5] FIG. 10 is an overall view of a map generation system according to a second embodiment of the present disclosure. [Figure 6] FIG. 2 is a block diagram of a server. [Figure 7] FIG. 10 is a block diagram of a self-location estimation device according to a third embodiment of the present disclosure. [Figure 8] FIG. 2 is a block diagram of a comparator. [Figure 9] FIG. 10 is a block diagram of a comparator according to a modified example. [Figure 10] FIG. 10 is a block diagram of a comparator according to a modified example. [Figure 11] FIG. 10 is a block diagram of a self-location estimation device according to a fourth embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. In the following, including other embodiments described below, identical or equivalent parts will be denoted by the same reference numerals.

[0015] (First embodiment) A first embodiment of the present disclosure will be described. A map generation system 1 shown in Fig. 1 is mounted on a vehicle 2 and generates 3D map data of an area observed by the vehicle 2, for example, 3D map data for self-location estimation. Specifically, the map generation system 1 of this embodiment is configured to include an image sensor 3, a position and attitude angle sensor 4, a vehicle speed sensor 5, and an information processing device 6. The image sensor 3, the position and attitude angle sensor 4, the vehicle speed sensor 5, and the information processing device 6 are each connected to each other so as to be able to communicate directly or via an in-vehicle LAN (Local Area Network).

[0016] The map generation system 1 generates a 3D map when, for example, the driver presses a 3D map generation switch (not shown) to issue a command to generate a 3D map in an area for which 3D map generation is desired. For example, it is preferable to generate a 3D map using the map generation system 1 in an area where map data used by a navigation system or the like does not exist, such as an indoor parking lot. In this way, 3D map data can be generated by focusing on an area where 3D mapping is effective, i.e., a specific area. However, the timing of 3D map generation is arbitrary and is not limited to when a command to generate a 3D map is issued, and may be automatically performed while the host vehicle 2 is traveling. Furthermore, even for an area for which 3D map generation has been performed once, 3D map generation may be repeated to update the area.

[0017] The image sensor 3 constitutes a photographing unit and is attached, for example, under the roof or in front of the rearview mirror of the vehicle 2 (hereinafter referred to as the host vehicle 2). The image sensor 3 captures the scenery around the host vehicle 2 as a host vehicle image and generates image information indicating the host vehicle image. For example, the image sensor 3 is used to capture images of obstacles that may impede vehicle travel and are used to create 3D map data. Potential obstacles include static targets, such as road structures, as well as dynamic targets, such as pedestrians and other vehicles, which are moving objects around the host vehicle 2. However, static targets are the objects considered to be obstacles in 3D map generation. Therefore, when generating a 3D map, it is preferable to capture images using the image sensor 3 at a time when no dynamic targets are present. However, if a dynamic target is captured, the dynamic target may be excluded from the image information, and 3D map data may be generated based on information about only static targets. Here, a perimeter monitoring camera capturing an image of a predetermined area around the host vehicle 2 is used as the image sensor 3.

[0018] The perimeter monitoring camera captures images of the surroundings of the vehicle 2 and outputs the captured image information to the information processing device 6. The perimeter monitoring camera may be a front camera that captures the scenery in front of the vehicle, or multiple cameras that capture the scenery behind, to the left, right, and sides of the vehicle. For example, the perimeter monitoring camera may be a front camera that captures only a predetermined range in front of the vehicle 2, or the front camera may be rotated around a predetermined center point to capture ranges in various directions as rotational images. Multiple cameras may also be used to simultaneously capture multiple areas. The imaging range of each camera is within a predetermined angle in the vertical direction and a predetermined angle in the horizontal direction relative to the camera's imaging center, and can be determined based on the mounting height and mounting direction of the camera on the vehicle 2, i.e., the orientation of the imaging center.

[0019] The position and attitude angle sensor 4 is a sensor that functions as a position sensor and an attitude angle sensor. As a function of the position sensor, the position and attitude angle sensor 4 detects the current position of the host vehicle 2 and transmits a detection signal indicating the current position of the host vehicle 2 to the information processing device 6. The position and attitude angle sensor 4 receives a position information signal from, for example, a Global Navigation Satellite System (GNSS) to obtain the current position of the host vehicle 2, that is, current position information indicating the three-dimensional position, and transmits this to the information processing device 6.

[0020] Furthermore, as an attitude angle sensor, the position and attitude angle sensor 4 detects the attitude of the host vehicle 2 and transmits attitude information indicating the attitude to the information processing device 6. Specifically, the position and attitude angle sensor 4 detects physical quantities indicating the attitude of the host vehicle 2 to calculate the attitude angle and transmits detection signals indicating the physical quantities to the information processing device 6. Here, the attitude angle refers to the attitude and angle of an object in 3D measurement. In this embodiment, the attitude angle represents the attitude and angle of the host vehicle 2 in 3D space, in other words, its direction and tilt. For example, the position and attitude angle sensor 4 can be a six-axis sensor consisting of three-axis acceleration sensors in the front-to-back, left-to-right, and up-to-down directions of the vehicle and three-axis angular velocity sensors for roll, pitch, and yaw. The tilt of the host vehicle 2 can be determined based on the acceleration of the three axes, and the attitude angle can be determined by integrating each angular velocity. Therefore, the detection signals of the acceleration and angular velocity from the position and attitude angle sensor 4 are transmitted to the information processing device 6 as attitude information.

[0021] The vehicle speed sensor 5 corresponds to a vehicle speed acquisition unit, and outputs a detection signal corresponding to the vehicle speed of the host vehicle 2, and transmits the detection signal to the information processing device 6. Note that, although the detection signal of the vehicle speed sensor 5 is used as the vehicle speed information of the host vehicle 2 here, the vehicle speed information may be acquired from another device. For example, the detection signal of a wheel speed sensor may be used, or vehicle speed information may be input to the information processing device 6 from another electronic control unit (hereinafter referred to as ECU) that handles vehicle speed information.

[0022] The information processing device 6 corresponds to a control unit for generating 3D map data. The information processing device 6 constitutes an ECU that functions as various control units for realizing 3D map generation in the map generation system 1, and is constituted by a microcomputer equipped with a CPU, ROM, RAM, I / O, etc. Here, the information processing device 6 is constituted by one ECU, but it may also be constituted by a combination of multiple ECUs.

[0023] 2, in this embodiment, when generating 3D map data, the information processing device 6 receives as input imaging information that is the detection result of the image sensor 3 and detection signals from the position and attitude angle sensor 4 and the vehicle speed sensor 5. Then, the information processing device 6 performs various controls for generating 3D map data based on the input imaging information and detection signals.

[0024] Specifically, the information processing device 6 has a state estimation unit 61, a correlation unit 62, a 3D map data generation unit 63, and a 3D map data storage unit 64.

[0025] The state estimation unit 61 receives detection signals from the position and attitude angle sensor 4 and the vehicle speed sensor 5, estimates the vehicle speed based on the detection signal from the vehicle speed sensor 5, and estimates the current position and attitude based on the estimated vehicle speed and the detection signal from the position and attitude angle sensor 4. In other words, the state estimation unit 61 estimates the current position represented by the latitude and longitude information of the vehicle 2, the attitude of the vehicle 2 in 3D space, and the vehicle speed of the vehicle 2 as the state of the vehicle 2. Note that the current position can be estimated based on GNSS position information signals when they can be received. Furthermore, in places where GNSS position information signals cannot be received, such as indoor parking lots, the current position can be estimated with high accuracy by estimating the amount and direction of movement from a place where GNSS position information signals could be received based on the vehicle speed and the detection signal from the position and attitude angle sensor 4.

[0026] The association unit 62 receives the current position, attitude, and vehicle speed estimated by the state estimation unit 61 and the imaging information of the image sensor 3, and based on these, associates the current position, attitude, and vehicle speed with each image included in the imaging information to create time series data. Specifically, the association unit 62 links the imaging timing of each image included in the imaging information with the position, attitude, and vehicle speed of the vehicle 2 at that imaging timing. Furthermore, the association unit 62 sorts the associated images and the current position, attitude, and vehicle speed of the vehicle 2 by imaging timing and lists them in chronological order to create time series data.

[0027] The 3D map data generation unit 63 generates a 3D map that reproduces a scene from an arbitrary viewpoint from the time-series data of the current position, posture, and vehicle speed generated by the association unit 62. For example, the 3D map data generation unit 63 generates 3D map data that serves as a 3D model using an AI (Artificial Intelligence) technology called NeRF (Neural Radiance Field) that constructs a three-dimensional image from images. Note that the applied technology is not limited to NeRF as long as it can reproduce a scene from an arbitrary viewpoint.

[0028] Structure from Motion (SfM) is an approach to AI that attempts to roughly estimate location from the image data it represents, using only image data as input. Using image data of the scene to be 3D modeled from multiple viewpoints, a deep neural network based on NeRF architecture is trained to learn the color (RGB) and transparency (σ) of each coordinate point in 3D space, generating a highly accurate 3D model. Specifically, as shown in Figure 3, a deep neural network is trained using the camera's orientation coordinates (x, y, z) represented by the x-, y-, and z-axes when the image information was acquired, and the orientation (θ, φ) expressed by the Euler angles θ and φ. The deep neural network then outputs the color (RGB) and transparency (σ) of each coordinate point in 3D space. However, SfM is highly dependent on the environment and prone to drift, which degrades input accuracy. While highly accurate 3D models can be generated for small areas, it is difficult to generate highly accurate 3D models for large areas. For example, in the case of a method such as Patent Document 1, which uses only a captured image to extract feature points contained in the image, the position and orientation of the camera are estimated based on the feature points and used as input to a deep neural network. With this method, the accuracy of the input deteriorates, causing distortion in the generated 3D map and making it impossible to generate a highly accurate 3D model.

[0029] On the other hand, in this embodiment, the current position and attitude angle can be estimated with high accuracy based on the detection signals of the vehicle speed sensor 5 and the position and attitude angle sensor 4. These current position and attitude angle are used as the input attitude coordinates (x, y, z) and attitude (θ, φ) in the deep neural network shown in Fig. 2, respectively, to obtain the color (RGB) and transparency (σ) of each coordinate point in the 3D space.

[0030] In this way, the highly accurate current position and attitude angle obtained based on the detection signals of the vehicle speed sensor 5 and the position and attitude angle sensor 4 can be used as input to the deep neural network. This realizes high accuracy in the camera position and attitude, and as a result, it becomes possible to achieve high accuracy in the color (RGB) and transparency (σ) of each coordinate point in the output 3D space. Furthermore, since the color (RGB) and transparency (σ) of each coordinate point in the 3D space become 3D map data, it becomes possible to generate highly accurate 3D map data.

[0031] The 3D map data storage unit 64 accumulates the 3D map data generated by the 3D map data generation unit 63 and stores it as a 3D model of the area for which the 3D map data was generated, for example, as a file of AI model parameters that output color (RGB) and transparency (σ) for each xyz coordinate. Therefore, as shown in Fig. 4, it is possible to use the 3D map data stored in the 3D map data storage unit 64 to restore a 3D map of the area for which the 3D map data was generated on, for example, the display device 7.

[0032] It should be noted that 3D map data is created for the location for which a generation command has been issued, and therefore 3D map data is created for each location for which a generation command has been issued.

[0033] The map generation system 1 according to this embodiment is configured as described above. The map generation system 1 configured in this manner is used to realize a map generation method, thereby generating three-dimensional map data.

[0034] Specifically, first, when an instruction to create 3D map data is issued for an area for which 3D map data generation is desired, the position and attitude angle sensor 4 detects the current position and attitude of the vehicle 2, and transmits current position information and attitude information indicating these to the information processing device 6. At the same time, the image sensor 3 captures an image of the scenery around the vehicle 2, and transmits the captured image information to the information processing device 6.

[0035] Next, in the information processing device 6, the state estimation unit 61 estimates the current position, attitude angle, and vehicle speed based on the current position information, attitude information, and vehicle speed information. The association unit 62 then inputs the current position, attitude, and vehicle speed estimated by the state estimation unit 61 and the imaging information from the image sensor 3, and creates time-series data by associating the current position, attitude, and vehicle speed with each image included in the imaging information based on this. The 3D map data generation unit 63 then generates 3D map data that reproduces a landscape from an arbitrary viewpoint from the time-series data of the current position, attitude, and vehicle speed created by the association unit 62. In this way, 3D map data is generated, and each time 3D map data is generated, it is stored in the 3D map data storage unit 64, thereby creating a 3D model for each area for which 3D map data was generated.

[0036] In this map generation method, the current position and attitude angle of the host vehicle 2 are detected with high accuracy based on the current position and attitude detected by the position and attitude angle sensor 4 and the vehicle speed acquired by the vehicle speed sensor 5. Then, 3D map data that reproduces a scene from an arbitrary viewpoint is generated from the highly accurate current position and attitude angle and time-series data of the image information from the image sensor 3. Specifically, the highly accurate current position and attitude angle are used as input to a neural network to obtain image generation information at coordinate points in 3D space, specifically color (RGB) and transparency (σ). In this way, 3D map data having, as a data file, parameters of an AI model for reproducing image generation information at each coordinate point in 3D space can be obtained, making it possible to generate highly accurate 3D map data.

[0037] In other words, 3D map data can be generated by a map generation method that uses the current position and attitude angle of the vehicle 2 based on the detection signals of the position and attitude angle sensor 4 and the detection signals of the vehicle speed sensor 5, making it possible to generate highly accurate 3D map data. The 3D map data generated in this way can be restored to an image and virtually displayed as the surrounding scenery when the user is in any location, or can be used as matching data for self-position estimation, which will be described later.

[0038] (Second embodiment) A second embodiment of the present disclosure will be described. This embodiment is different from the first embodiment in that some of the functions of the information processing device 6 are performed by an external server, but the rest is the same as the first embodiment, so only the parts that are different from the first embodiment will be described.

[0039] 5, the map generation system 1 of this embodiment also includes an image sensor 3, a position and attitude angle sensor 4, a vehicle speed sensor 5, and an information processing device 6 provided in the host vehicle 2. In addition, the map generation system 1 of this embodiment includes a communicator 8 such as a DCM (Data Communication Module) that enables the host vehicle 2 to communicate with the outside, and a server 10 provided outside the host vehicle 2. By communicating with the server 10 using the communicator 8, the server 10 is caused to perform some of the functions of the information processing device 6 described in the first embodiment.

[0040] Specifically, the information processing device 6 of this embodiment has the state estimation unit 61 and the association unit 62 described in the first embodiment, and as shown in Fig. 6, the server 10 is provided with a 3D map data generation unit 11 and a 3D map data storage unit 12. With this configuration, the state estimation unit 61 estimates the current position and attitude of the vehicle 2, and then transmits information related to the estimation to the server 10 via wireless communication from the communicator 8. Then, the 3D map data generation unit 11 and the 3D map data storage unit 12 each generate 3D map data like the 3D map data generation unit 63 of the first embodiment, and then store the 3D map data like the 3D map data storage unit 64.

[0041] In this way, even if the server 10 generates and stores 3D map data, the same effects as in the first embodiment can be obtained. Furthermore, 3D map data is large in volume, and generating the 3D map data places a heavy processing load on the user. For this reason, by using an external server 10 instead of having the information processing device 6 of the user's vehicle 2 handle all of the 3D map data generation, it is possible to reduce the storage capacity and processing load. Note that, although the server 10 is provided with the 3D map data storage unit 12 here, the server 10 may be provided with only the 3D map data generation unit 11, and the generated 3D map data may be transmitted to the user's vehicle 2 via wireless communication and stored in the 3D map data storage unit 64 of the user's vehicle 2.

[0042] (Third embodiment) A third embodiment of the present disclosure will be described. This embodiment performs self-location estimation and the like using the 3D map data described in the first and second embodiments. In the first and second embodiments, the map generation system 1 that generates 3D map data was described. However, when a function for performing self-location estimation is provided, a self-location estimation device is configured by adding a configuration required for self-location estimation to the map generation system 1. Since the rest is the same as in the first and second embodiments, only the parts that differ from the first and second embodiments will be described. Note that, here, as in the first embodiment, a structure in which a 3D map data generation unit 63 and a 3D map data storage unit 64 are provided within an information processing device 6 will be described as an example.

[0043] As shown in FIG. 7, the information processing device 6 of this embodiment includes a state estimation unit 61, a correspondence unit 62, a 3D map data generation unit 63, and a 3D map data storage unit 64, as well as a comparator 65, a self-position estimation unit 66, and a self-attitude estimation unit 67.

[0044] When the 3D map data is generated as in the first embodiment, the self-position estimation and the self-orientation estimation are performed based on the 3D map data stored in the 3D map data storage unit 64 and the detection signal of the image sensor 3.

[0045] Here, the self-position means the current position of the vehicle 2 in 3D space, and the self-attitude means the attitude angle of the vehicle 2. As in the first and second embodiments, when generating 3D map data, not only the imaging information of the image sensor 3 but also the detection signals of the position and attitude angle sensor 4 and the vehicle speed sensor 5 are used. When 3D map data is generated in this way, the self-position and self-attitude can be estimated by comparing the 3D map data with the imaging information of the vehicle 2 while it is traveling.

[0046] The comparator 65 compares the image indicated by the imaging information transmitted from the image sensor 3, i.e., the image captured by the host vehicle 2 while the vehicle is traveling, with the 3D map in the stored 3D map data. The comparator 65 then extracts a 3D map that matches the image captured by the host vehicle or a 3D map that is similar to the image captured by the host vehicle from the 3D map data. The comparison by the comparator 65 may be performed such that, for each frame of the image captured by the host vehicle, all of the images in the frame are compared with the 3D map data, or only a portion of the images may be compared with the 3D map data.

[0047] 8, the comparator 65 is configured to include a first feature extraction unit 65a, a second feature extraction unit 65b, and a comparison unit 65c, and performs the comparison by comparing a portion of the image captured by the host vehicle in a frame with the 3D map data. This reduces the amount of data to be compared, thereby reducing the processing load required for the comparison and reducing the amount of data.

[0048] Specifically, the first feature extraction unit 65a extracts feature portions from the 3D map in the 3D map data, and the second feature extraction unit 65b extracts feature portions from the image captured by the vehicle. Features include, for example, distinctive points, objects, and other distinctive parts in the image. Examples include parts that differ from the surrounding brightness or color, or distinctive shapes such as squares or circles. The comparison unit 65c compares the feature portions in the 3D map extracted by the first feature extraction unit 65a with the feature portions in the image captured by the vehicle extracted by the second feature extraction unit 65b to determine correspondence, i.e., match the feature portions. For example, the comparator 65 selects from the 3D map those that match and / or are similar to the image indicated by the imaging information.

[0049] The self-position estimation unit 66 and the self-attitude estimation unit 67 perform self-position estimation and self-attitude estimation based on the comparison result of the comparator 65. That is, if there is a 3D map in the 3D map data that matches the image captured by the host vehicle, it is possible to estimate from what position and in what attitude the matching 3D map was captured. In other words, the position of the host vehicle 2 can be estimated relatively from the matching 3D map. Even if there is no matching 3D map in the 3D map data, it is possible to estimate from what position and in what attitude the image information captured by the host vehicle 2 was based on the difference between the image captured by the host vehicle and an approximate 3D map, for example, the amount of deviation. For example, based on the approximate 3D map, it is possible to estimate that the image captured by the host vehicle 2 was captured when the host vehicle 2 was slightly to the left, or when the host vehicle 2 was slightly tilted. Therefore, corrections based on the estimation results can be made to perform self-position estimation and self-attitude estimation. Furthermore, since imaging information is continuously transmitted from the image sensor 3, it is also possible to repeatedly correct the vehicle's position and attitude based on multiple frames of images captured by the vehicle itself, thereby estimating the vehicle's position and attitude. This makes it possible to construct a highly accurate 3D map that reproduces the scenery from any viewpoint, and to perform highly accurate estimation of the vehicle's position and attitude based on that highly accurate 3D map.

[0050] Such self-position estimation and self-attitude estimation can be used for automatic driving control, such as automatic parking of a vehicle. Obstacles around the vehicle 2 can be estimated based on the 3D map data, and the current position of the vehicle 2 can be obtained by self-position estimation. Therefore, for example, the information processing device 6 calculates a travel route that avoids the obstacles, and the vehicle 2 is controlled to trace that travel route. By performing such vehicle driving control, it becomes possible to perform automatic driving while avoiding obstacles in an area for which 3D map data has been generated. Furthermore, since the self-attitude estimation can estimate how the vehicle 2 is tilted, the driving and braking force control of each wheel of the vehicle 2 can be performed based on the estimation result, thereby further improving the stability of vehicle driving.

[0051] Although the case where vehicle position estimation based on 3D map data is performed only based on images captured by the vehicle has been described above, the attitude angle and vehicle speed obtained from the detection signals of the position and attitude angle sensor 4 and the vehicle speed sensor 5 may also be used as additional information. This makes it possible to estimate the vehicle position and attitude with higher accuracy. Furthermore, the use of additional information also makes it possible to reduce the processing load of the vehicle position estimation and attitude estimation.

[0052] Furthermore, while the first embodiment has been described above in which the 3D map data generation unit 63 and the 3D map data storage unit 64 are provided in the information processing device 6 of the host vehicle 2, the same applies to the second embodiment in which they are provided in the server 10. In this case, the server 10 is provided with a comparator 65 and a self-position estimation unit 66, and when performing self-position estimation, the information processing device 6 transmits image information from the image sensor 3 to the server 10 via the communicator 8. The server 10 can then perform self-position estimation by matching the 3D map data stored in the 3D map data storage unit 12 with the image information transmitted from the image sensor 3. Furthermore, if the self-position estimation result is to be used for vehicle driving control such as automatic parking, the self-position estimation result may be returned from the server 10, acquired via the communicator 8, and used for vehicle driving control of the host vehicle 2.

[0053] (Modification of the third embodiment) In the third embodiment, the comparator 65 is configured with a first feature extraction unit 65a, a second feature extraction unit 65b, and a comparison unit 65c, and is configured to extract and compare feature portions from images. Alternatively, the comparator 65 may be configured as shown in Fig. 9 or 10. Furthermore, the comparator 65 may be configured to directly compare the RGB values of images without going through a feature extraction unit. The configurations shown in Figs. 9 and 10 are not limited to cases where the comparator 65 is provided on the host vehicle 2 side, and can, of course, also be applied to cases where the comparator is provided on the server 10 side.

[0054] In FIG. 9, the comparator 65 is configured with a first segmentation unit 65d, a second segmentation unit 65e, and a comparison unit 65f. The first segmentation unit 65d segments each area of the 3D map in the 3D map data, and the second segmentation unit 65e segments each area of the image captured by the host vehicle. In the segmentation, the areas are divided into groups with specific characteristics, such as groups of buildings and groups of signs. The comparison unit 65c compares each area of the 3D map segmented by the first segmentation unit 65d with each area of the image captured by the host vehicle segmented by the second segmentation unit 65e. The comparison unit 65c then performs correspondence by comparison, i.e., matches the segmented images, and selects an image from the 3D map that matches and / or is similar to the image indicated by the host vehicle image information. When performing such a comparison for each region, it is possible to perform a comparison somewhere within the region, for example, at a predetermined point at the corner of a building ahead, compared to a point comparison, which makes it possible to perform a comparison that is robust even when conditions change, thereby increasing reliability. In this way, the comparator 65 may perform a comparison using segmentation images.

[0055] In FIG. 10, the comparator 65 is configured with a first depth estimation unit 65g, a second depth estimation unit 65h, and a comparison unit 65i. The first depth estimation unit 65g estimates the depth of each point on the 3D map in the 3D map data, and the second depth estimation unit 65h estimates the depth of each point on the image captured by the vehicle. The depth estimation estimates the depth of each point in the image, for example, for each pixel, i.e., the distance from the shooting location to each point. The comparison unit 65i compares a 3D map including depth information for each point on the 3D map estimated by the first depth estimation unit 65g with an image including depth information for each point on the image captured by the vehicle estimated by the second depth estimation unit 65h. The comparison unit 65i then performs correspondence by comparison, i.e., matches images including depth information, and selects an image from the 3D map that matches and / or is similar to the image indicated by the vehicle image information. In this way, the comparator 65 may perform the comparison using an image that includes depth information.

[0056] (Fourth embodiment) A fourth embodiment of the present disclosure will be described. This embodiment performs brightness correction on 3D map data in comparison with the first to third embodiments, and is otherwise similar to the first to third embodiments, so only the differences from the first to third embodiments will be described. Note that the following description will be given taking as an example a case where self-position estimation and self-posture estimation are performed using 3D map data stored as in the third embodiment.

[0057] As shown in FIG. 11, in this embodiment, the information processing device 6 is provided with a correction unit 68, which performs brightness correction on the 3D map data stored in the 3D map data storage unit 64, and the comparator 65 performs comparison based on the corrected 3D map data.

[0058] The correction unit 68 performs brightness correction according to the time of day. For example, if the imaging information used to generate the 3D map data is captured at a different time of day from the time of day when it was captured, the appearance of each part of the image may change, even if it is the same image. The appearance of each part of the image may change between daytime and nighttime, and even during the daytime, the appearance of each part of the image may change depending on the time of day due to, for example, the shadow of a neighboring building. For this reason, the correction unit 68 performs brightness correction on the 3D map data according to the time of day.

[0059] In this way, by comparing the brightness-corrected 3D map data with the image captured by the vehicle itself, it is possible to estimate the vehicle's position and attitude with higher accuracy.

[0060] The correction according to the time period by the correction unit 68 may be performed each time the comparator 65 performs a comparison, or may be performed by storing 3D map data for each time period in the correction unit 68. In the former case, the correction unit 68 acquires the time at the timing of the comparison, corrects the 3D map data according to the time period to which that time belongs, and the comparator 65 compares the corrected 3D map data. In the latter case, since the correction unit 68 stores 3D map data for each time period, the comparator 65 acquires the time at the timing of the comparison, and selects 3D map data for the time period to which that time belongs from the 3D map data stored by the correction unit 68 for comparison.

[0061] (Other embodiments) Although the present disclosure has been described based on the above-described embodiment, it is not limited to the embodiment and encompasses various modifications and modifications within the equivalent range. In addition, various combinations and forms, as well as other combinations and forms including only one element, more than one element, or less than one element, are also within the scope and spirit of the present disclosure.

[0062] For example, in each of the above embodiments, after 3D map data has been generated, new 3D map data may be generated to update the data. That is, the host vehicle 2 may be driven multiple times in the same location, and each time new driving data is obtained, 3D map data may be generated based on the imaging information, current position, attitude angle, and vehicle speed, and the previously generated 3D map data may be updated with the newly generated 3D map data.

[0063] In addition, in each of the above embodiments, the position and attitude angle sensor 4 receives a GNSS position information signal to acquire current position information of the vehicle, and the six-axis sensor acquires detection signals of triaxial acceleration and triaxial angular velocity. However, these may be separate sensors, or various information may be acquired from another ECU that handles current position information, triaxial acceleration, and triaxial angular velocity information, such as a navigation ECU or an ECU related to vehicle driving. Even when various information is acquired from another ECU, it can be said that the current position information, triaxial acceleration, and triaxial angular velocity acquired by the position and attitude angle sensor 4 are used simply by passing through the respective ECUs. Similarly, although vehicle speed information is acquired based on a detection signal from the vehicle speed sensor 5, the vehicle speed information may be acquired from a detection signal of a wheel speed sensor, or from another ECU that handles vehicle speed information.

[0064] Although an indoor parking lot has been used as an example of a case where 3D map data is generated only for a specific area, i.e., only for a specific area, by narrowing down the area to be 3D mapped, other locations may also be used. Furthermore, even within an indoor parking lot, the area to be 3D mapped may be further narrowed down within the range to be 3D mapped, such as a point near the entrance, a point near the parking space, or an intermediate point between them. To narrow down the area to be 3D mapped, a command to generate a 3D map may be issued only to the narrowed area, or only the area to be 3D mapped may be captured from the beginning based on the current position and attitude angle of the vehicle 2. Alternatively, only the area to be 3D mapped may be extracted from the captured image data of the vehicle and then used for 3D mapping. Alternatively, a 3D map may be generated for the entire captured area, and then only the data corresponding to the narrowed area may be stored. This allows 3D map data to be generated for only the most effective area, and the 3D map data may be smaller in size, thereby reducing storage capacity.

[0065] The controller and methods described herein may be implemented by a special-purpose computer configured with a processor and memory programmed to perform one or more functions embodied in a computer program. Alternatively, the controller and methods described herein may be implemented by a special-purpose computer configured with a processor configured with one or more dedicated hardware logic circuits. Alternatively, the controller and methods described herein may be implemented by one or more special-purpose computers configured with a processor and memory programmed to perform one or more functions in combination with a processor configured with one or more hardware logic circuits. Furthermore, the computer program may be stored as instructions executed by a computer on a computer-readable non-transitory storage medium.

[0066] (Aspects of the present disclosure) The present disclosure described above can be understood from the following viewpoints, for example.

[0067] [First viewpoint] A map generation system that generates three-dimensional map data of an area observed by a vehicle (2), an image capturing unit (3) for capturing an image of the surroundings of the vehicle as a captured image of the vehicle; a position and attitude sensor (4) for detecting the current position of the vehicle and detecting the attitude of the vehicle; a vehicle speed acquisition unit (5) for acquiring vehicle speed information of the vehicle; a state estimation unit (61) that estimates a three-dimensional position and an attitude angle of the vehicle based on current position information indicating the current position and attitude information indicating the attitude detected by the position and attitude acquisition unit, and the vehicle speed information acquired by the vehicle speed acquisition unit; a 3D map data generation unit (63) that generates 3D map data that reproduces a landscape from an arbitrary viewpoint, based on the 3D position and the attitude angle estimated by the state estimation unit and time-series data of the image of the vehicle captured by the imaging unit. [Second perspective] a communication device (8) mounted on the vehicle and performing communication with the outside of the vehicle; a server (10) provided outside the vehicle; the photographing unit, the position and orientation sensor, the vehicle speed acquisition unit, and the state estimation unit are mounted on the vehicle, and the 3D map data generation unit is provided in the server; The map generation system according to a first aspect, wherein information regarding the three-dimensional position and the attitude angle of the vehicle is transmitted to the server via the communication device, and the server generates the three-dimensional map data using the three-dimensional map data generation unit based on the information transmitted to the server. [Third Perspective] a map generation system according to the first aspect; a comparator (65) provided in the vehicle, which compares a 3D map included in the 3D map data generated by the 3D map data generation unit with the image captured by the imaging unit; a self-position estimation unit (66) provided in the vehicle and configured to perform self-position estimation to estimate the current position of the vehicle based on the comparison result of the comparator. [Fourth viewpoint] a map generation system according to the second aspect; a comparator (65) provided in the server for comparing a 3D map included in the 3D map data generated by the 3D map data generation unit with the image captured by the image capture unit; a self-position estimation unit (66) provided in the server and configured to perform self-position estimation to estimate the current position of the vehicle based on the comparison result of the comparator. [Fifth viewpoint] The comparator a first feature extraction unit (65a) that extracts a feature portion that is a characteristic part within the three-dimensional map; a second feature extraction unit (65b) that extracts a feature portion that is a characteristic portion in the image captured by the imaging unit; a comparison unit (65c) that compares the feature portion extracted by the first feature extraction unit with the feature portion extracted by the second feature extraction unit, The self-location estimation device according to the third or fourth aspect, wherein the self-location estimation is performed based on a comparison result by the comparison unit. [Sixth viewpoint] The comparator a first segmentation unit (65d) that performs segmentation of an area within the three-dimensional map; a second segmentation unit (65e) that performs segmentation of an area in the subject vehicle captured image of the imaging unit; a comparison unit (65f) that compares the area in the 3D map segmented by the first segmentation unit with the area in the subject vehicle captured image segmented by the second segmentation unit, The self-location estimation device according to the third or fourth aspect, wherein the self-location estimation is performed based on a comparison result by the comparison unit. [Seventh viewpoint] The comparator a first depth estimation unit (65g) that estimates the depth of points within the three-dimensional map; a second depth estimation unit (65h) that estimates the depth of points in the subject vehicle captured image of the imaging unit; a comparison unit (65i) that compares a 3D map including depth information of points in the 3D map whose depths are estimated by the first depth estimation unit with an image including depth information of points in the subject vehicle captured image whose depths are estimated by the second depth estimation unit, The self-location estimation device according to the third or fourth aspect, wherein the self-location estimation is performed based on a comparison result by the comparison unit. [Eighth viewpoint] a correction unit (68) that corrects the brightness of the three-dimensional map data according to a time period; The self-position estimation device according to any one of the third to seventh aspects, wherein the comparator compares the 3D map in the 3D map data corrected by the correction unit during a time period that includes the time when the image was captured by the imaging unit with the image of the vehicle captured by the imaging unit. [Ninth viewpoint] the photographing unit is a front camera that photographs a scene ahead of the vehicle, The map generation system according to the first or second aspect, wherein the position and attitude angle sensor detects the vehicle based on a GNSS position information signal, and detects the attitude of the vehicle using a six-axis sensor consisting of three-axis acceleration sensors in the forward / backward, left / right, and up / down directions of the vehicle, and three-axis angular velocity sensors in the roll, pitch, and yaw directions. [10th viewpoint] the photographing unit includes a plurality of cameras that photograph a scene around the vehicle, The map generation system according to the first or second aspect, wherein the position and attitude angle sensor detects the vehicle based on a GNSS position information signal, and detects the attitude of the vehicle using a six-axis sensor consisting of three-axis acceleration sensors in the forward / backward, left / right, and up / down directions of the vehicle, and three-axis angular velocity sensors in the roll, pitch, and yaw directions. [11th viewpoint] A map generation method for generating three-dimensional map data of an area observed by a vehicle (2), comprising: An imaging unit (3) captures images of the scenery around the vehicle to obtain image information of the vehicle itself; Detecting the current position of the vehicle and detecting the attitude of the vehicle by a position and attitude sensor (4); Acquiring vehicle speed information of the vehicle by a vehicle speed acquisition unit (5); estimating a three-dimensional position and an attitude angle of the vehicle based on current position information indicating the current position, attitude information indicating the attitude, and the vehicle speed information; generating three-dimensional map data that reproduces a landscape from an arbitrary viewpoint from the three-dimensional position, the attitude angle, and time-series data of the host vehicle's image capture information from the imaging unit. [12th viewpoint] The map generating method according to an eleventh aspect, wherein the generating of the three-dimensional map data is performed by narrowing down the area for generating the 3D map to a specific area. [13th viewpoint] The map generating method according to the eleventh or twelfth aspect, wherein, in generating the 3D map data, when a new three-dimensional position and attitude angle of the vehicle are estimated when the vehicle is driven multiple times in the same location, new 3D map data is generated based on the newly estimated three-dimensional position and attitude angle of the vehicle, and previously generated 3D map data is updated with the newly generated 3D map data. [Explanation of symbols]

[0068] 1...map generation system, 2...vehicle, 3...image sensor, 4...position and attitude angle sensor, 5...vehicle speed sensor, 6...information processing device, 7...display device, 8...communication device, 10...server, 11, 63...3D map data generation unit, 12, 64...3D map data storage unit, 61...state estimation unit, 62...association unit, 65...comparison unit, 66...self-position estimation unit, 67...self-attitude estimation unit, 68...correction unit

Claims

1. A map generation system that generates three-dimensional map data of an area observed by a vehicle (2), an image capturing unit (3) for capturing an image of the surroundings of the vehicle as a subject vehicle image; a position and attitude sensor (4) for detecting the current position of the vehicle and detecting the attitude of the vehicle; a vehicle speed acquisition unit (5) for acquiring vehicle speed information of the vehicle; a state estimation unit (61) that estimates a three-dimensional position and an attitude angle of the vehicle based on current position information indicating the current position and attitude information indicating the attitude detected by the position and attitude acquisition unit, and the vehicle speed information acquired by the vehicle speed acquisition unit; a three-dimensional map data generation unit (63) that generates three-dimensional map data that reproduces a landscape from an arbitrary viewpoint, based on the three-dimensional position and the attitude angle estimated by the state estimation unit and time-series data of images captured by the imaging unit of the vehicle.

2. a communication device (8) mounted on the vehicle and performing communication with the outside of the vehicle; a server (10) provided outside the vehicle; the imaging unit, the position and orientation sensor, the vehicle speed acquisition unit, and the state estimation unit are mounted on the vehicle, and the 3D map data generation unit is provided on the server; 2. The map generation system according to claim 1, wherein information about the three-dimensional position and the attitude angle of the vehicle is transmitted to the server via the communication device, and the server generates the three-dimensional map data using the three-dimensional map data generation unit based on the information transmitted to the server.

3. A map generation system according to claim 1; a comparator (65) provided in the vehicle for comparing a three-dimensional map included in the three-dimensional map data generated by the three-dimensional map data generating unit with an image captured by the imaging unit; a self-position estimation unit (66) provided in the vehicle and performing self-position estimation to estimate the current position of the vehicle based on the comparison result of the comparator.

4. A map generation system according to claim 2; a comparator (65) provided in the server for comparing a 3D map included in the 3D map data generated by the 3D map data generating unit with the image captured by the imaging unit; a self-position estimation unit (66) provided in the server and configured to perform self-position estimation to estimate the current position of the vehicle based on the comparison result of the comparator.

5. The comparator a first feature extraction unit (65a) that extracts a feature portion that is a characteristic part within the three-dimensional map; a second feature extraction unit (65b) that extracts a feature portion that is a characteristic portion in the image captured by the imaging unit; a comparison unit (65c) that compares the feature portion extracted by the first feature extraction unit with the feature portion extracted by the second feature extraction unit, The self-position estimation device according to claim 3 , wherein the self-position estimation is performed based on a comparison result from the comparison unit.

6. The comparator a first segmentation unit (65d) for segmenting an area within the three-dimensional map; a second segmentation unit (65e) that performs segmentation of an area in the subject vehicle captured image of the imaging unit; a comparison unit (65f) that compares the area in the 3D map segmented by the first segmentation unit with the area in the subject vehicle captured image segmented by the second segmentation unit, The self-position estimation device according to claim 3 , wherein the self-position estimation is performed based on a comparison result from the comparison unit.

7. The comparator a first depth estimation unit (65g) that performs depth estimation of points within the three-dimensional map; a second depth estimation unit (65h) that estimates the depth of points in the subject vehicle captured image of the imaging unit; a comparison unit (65i) that compares a three-dimensional map including depth information of points in the three-dimensional map whose depths are estimated by the first depth estimation unit with an image including depth information of points in the image captured by the vehicle whose depths are estimated by the second depth estimation unit, The self-position estimation device according to claim 3 , wherein the self-position estimation is performed based on a comparison result from the comparison unit.

8. a correction unit (68) that corrects the brightness of the three-dimensional map data according to a time period; 5. The self-position estimation device according to claim 3, wherein the comparator compares the 3D map in the 3D map data corrected by the correction unit for a time period that includes a time when the image was captured by the imaging unit with the image of the vehicle captured by the imaging unit.

9. the photographing unit is a front camera that photographs a scene ahead of the vehicle, 3. The map generation system according to claim 1, wherein the position and attitude angle sensor detects the vehicle based on a position information signal from a Global Navigation Satellite System (GNSS), and detects the attitude of the vehicle using a six-axis sensor consisting of three-axis acceleration sensors in the forward / backward, left / right, and up / down directions of the vehicle, and three-axis angular velocity sensors in the roll, pitch, and yaw directions.

10. the photographing unit includes a plurality of cameras that photograph a scene around the vehicle, 3. The map generation system according to claim 1, wherein the position and attitude angle sensor detects the vehicle based on a position information signal from a Global Navigation Satellite System (GNSS), and detects the attitude of the vehicle using a six-axis sensor consisting of three-axis acceleration sensors in the forward / backward, left / right, and up / down directions of the vehicle, and three-axis angular velocity sensors in the roll, pitch, and yaw directions.

11. A map generation method for generating three-dimensional map data of an area observed by a vehicle (2), comprising: An imaging unit (3) captures images of the scenery around the vehicle to obtain image information of the vehicle itself; Detecting the current position of the vehicle and detecting the attitude of the vehicle using a position and attitude sensor (4); Acquiring vehicle speed information of the vehicle by a vehicle speed acquisition unit (5); estimating a three-dimensional position and an attitude angle of the vehicle based on current position information indicating the current position, attitude information indicating the attitude, and the vehicle speed information; generating three-dimensional map data that reproduces a landscape from an arbitrary viewpoint from the three-dimensional position, the attitude angle, and time-series data of the host vehicle's image capture information from the imaging unit.

12. The map generating method according to claim 11 , wherein the generating of the three-dimensional map data is performed by narrowing down an area for generating a 3D map to a specific area.

13. 13. The map generating method according to claim 11, wherein, in generating the three-dimensional map data, when a new three-dimensional position and attitude angle of the vehicle are estimated when the vehicle is caused to travel a plurality of times in the same place, new three-dimensional map data is generated based on the newly estimated three-dimensional position and attitude angle of the vehicle, and previously generated three-dimensional map data is updated with the newly generated three-dimensional map data.

Images