Information processing device and information processing method

Abstract

To enable a user to confirm without performing automatic traveling, that there is a higher probability not to enable performing the automatic traveling in a place where teaching traveling has been performed.SOLUTION: An information processing device includes: an extraction section for receiving image information being the information obtained by photographing the periphery of a mobile body when teaching traveling is performed, so as to extract information to be used for estimating an own position obtained from the received image information; and an output control section for outputting to an output section, instruction information different from information in a case where the amount of the information to be used in own position estimation is the second amount greater than the first amount when the amount of the information to be used in the own position estimation extracted by the extraction section is the first amount. A start position of the teaching traveling is the start position of automatic traveling, and an end position of the teaching traveling is a parking target position. When the amount of the information to be used in the own position estimation is the second amount, the output control section notifies a user of the normal end of the teaching traveling, and the instruction information is output before performing the automatic traveling based on traveling route data.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus and an information processing method.

Background Art

[0002] Conventionally, a technique for creating map data based on the characteristics of objects existing around a vehicle during travel has been disclosed. Then, by comparing the created map data with an image acquired by a photographing device mounted on the vehicle, self-position estimation, which is a process of estimating where the vehicle is located in the map data, is executed, and a technique for performing automatic driving along the traveled route has been disclosed.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] By the way, generally, in such a technique, the user manually drives the vehicle in advance (hereinafter referred to as teaching driving) at a place where automatic driving is to be performed, and map data and driving route data indicating the driving route necessary for self-position estimation for performing automatic driving at that place are created. After that, automatic driving is realized by using the created map data and driving route data.

[0005] During automatic driving, the map data used for self-position estimation is data that stores the three-dimensional positions of feature points of objects existing in the actual scene. During automatic driving, feature points are extracted from an image captured by the vehicle, and it is collated which of the feature points in the map data the extracted feature points are. At this time, if the number of feature points that match the map data is a predetermined number or more, it is determined that the self-position of the vehicle has been estimated.

[0006] However, when a training run is performed to create map data of the surrounding environment in an area where the characteristics of the vehicle's environment are insufficient, such as an area surrounded by objects without any irregularities or color changes, the number of feature points included in the created map data will be insufficient for the predetermined number required for self-localization, making it highly likely that self-localization will not be possible. However, with conventional technology, users could not confirm that there was a high probability that autonomous driving would not be possible in the area where the training run was performed until they actually performed autonomous driving.

[0007] The problem that this disclosure aims to solve is to provide an information processing device and information processing method that allows a user to confirm, without performing autonomous driving, that there is a high probability that autonomous driving will not be possible in a location where a training run has been conducted. [Means for solving the problem]

[0008] The information processing device disclosed herein is an information processing device that performs automatic driving based on driving path data indicating a driving path when a user manually drives a vehicle for training purposes, and comprises: an extraction unit that receives image information which is information taken of the surroundings of a moving object during the training drive and extracts information used for self-position estimation obtained from the received image information; and an output control unit that outputs different instruction information to an output unit when the amount of information used for self-position estimation extracted by the extraction unit is a first amount, and when the amount of information used for self-position estimation is a second amount which is greater than the first amount, the starting position of the training drive is the starting position of the automatic driving, the ending position of the training drive is the parking target position, and the output control unit notifies the user of the successful completion of the training drive when the amount of information used for self-position estimation is the second amount, and the instruction information is output before the automatic driving based on the driving path data is performed. The second quantity is a sufficient amount of information to perform the self-localization, and the first quantity is a smaller amount of information than the second quantity. . [Effects of the Invention]

[0009] According to the information processing device and information processing method relating to this disclosure, a user can confirm, without performing autonomous driving, that there is a high probability that autonomous driving will not be possible in a location where a training run has been conducted. [Brief explanation of the drawing]

[0010] [Figure 1] Figure 1 shows an example of a mobile body according to Embodiment 1. [Figure 2] Figure 2 is a schematic diagram showing an example of the installation position of the imaging device according to Embodiment 1. [Figure 3] Figure 3 is a diagram showing an example of a hardware configuration diagram of an information processing device according to Embodiment 1. [Figure 4] Figure 4 is a block diagram showing an example of the functional configuration of a mobile body according to Embodiment 1. [Figure 5] Figure 5 is a schematic diagram showing an example of an application scenario for the mobile body according to Embodiment 1. [Figure 6] Figure 6 is a schematic diagram showing an example of an instruction image according to Embodiment 1. [Figure 7] Figure 7 is a flowchart showing an example of the procedure for processing information about the teacher's run according to Embodiment 1. [Figure 8] Figure 8 is a flowchart showing an example of the information processing procedure for automated driving according to Embodiment 1. [Figure 9] Figure 9 is a flowchart showing an example of the procedure for processing information during training runs, relating to Modification 2. [Figure 10] Figure 10 is a flowchart showing an example of the information processing procedure for automated driving according to Modification Example 2. [Figure 11] Figure 11 is a block diagram showing an example of the functional configuration of a mobile body according to Embodiment 2. [Figure 12] Figure 12 is a flowchart showing an example of the procedure for processing information about the teacher's run according to Embodiment 2. [Figure 13] Figure 13 is a flowchart showing an example of the information processing procedure for automated driving according to Embodiment 2. [Modes for carrying out the invention]

[0011] <Embodiment 1> With reference to the accompanying drawings, embodiments of an information processing apparatus, an information processing method, and a moving body according to the present disclosure will be described.

[0012] FIG. 1 is a diagram showing an example of the moving body 10 of the present embodiment.

[0013] The moving body 10 includes an information processing apparatus 20, an output unit 10A, an input unit 10B, an internal sensor 10C, a photographing device 10D, a drive control unit 10F, and a drive unit 10G.

[0014] The information processing apparatus 20 is, for example, a dedicated or general-purpose computer. In the present embodiment, an example in which the information processing apparatus 20 is mounted on the moving body 10 will be described.

[0015] The moving body 10 is a movable object. In the present embodiment, the moving body 10 is an object on which a user can ride. The moving body 10 is, for example, a vehicle. The vehicle is a two-wheeled motor vehicle, a three-wheeled motor vehicle, or a four-wheeled motor vehicle, etc. Further, the moving body 10 is, for example, a moving body that advances through a driving operation by a person, or a moving body that can automatically advance (autonomous travel) without a driving operation by a person. In the present embodiment, an example in which the moving body 10 is a vehicle will be described.

[0016] The output unit 10A outputs information. In the present embodiment, the output unit 10A outputs information such as instruction information generated by the information processing apparatus 20. Details of the instruction information will be described later.

[0017] The output unit 10A has a display function for displaying information. Note that the output unit 10A may further have a communication function for transmitting information to an external device or the like, a sound output function for outputting sound, a function for lighting or flashing light, and the like. For example, the output unit 10A includes at least one of a display unit 10K, a communication unit 10H, a speaker 10I, and a lighting unit 10J. In the present embodiment, an example in which the output unit 10A includes the communication unit 10H, the speaker 10I, the lighting unit 10J, and the display unit 10K will be described.

[0018] The communication unit 10H transmits information to other devices. For example, the communication unit 10H transmits information to other devices via a known communication line. The speaker 10I outputs sound. The lighting unit 10J is a light that turns on or flashes light. The display unit 10K displays information. The display unit 10K is, for example, a known organic EL (Electro Luminescence) display, a liquid crystal display (Liquid Crystal Display), or a projection device.

[0019] The output unit 10A can be installed in a location where a user riding on the mobile vehicle 10 can see the information output from the output unit 10A.

[0020] For example, the mobile unit 10 is equipped with multiple display units (hereinafter collectively referred to as "display units 10K"). These multiple display units 10K are arranged at different positions within the interior of the mobile unit 10. The number of display units 10K provided on the mobile unit 10 may be one or more, and is not limited to multiple units. In this embodiment, a configuration in which the mobile unit 10 is equipped with one display unit 10K will be described as an example.

[0021] In detail, the display unit 10K has its orientation pre-adjusted so that the user U riding on the mobile body 10 can see the display surface.

[0022] Returning to Figure 1, let's continue the explanation. The input unit 10B receives instructions or information input from user U. The input unit 10B is, for example, at least one of an instruction input device that receives input through user U's operation input, and a microphone that receives voice input. The instruction input device is, for example, a pointing device such as a button, mouse or trackball, or a keyboard. The instruction input device may also be an input function on a touch panel integrated with the display unit 10K.

[0023] The internal sensor 10C is a sensor that observes information about the mobile body 10 itself. The internal sensor 10C detects the position of the mobile body 10, the speed of the mobile body 10, or the acceleration of the mobile body 10, etc.

[0024] The internal sensor 10C is, for example, an inertial measurement unit (IMU), a velocity sensor, or a GPS (Global Positioning System).

[0025] The imaging device 10D is a sensor that observes the area around the mobile body 10. The imaging device 10D may be mounted on the mobile body 10 or mounted on the outside of the mobile body 10. The outside of the mobile body 10 refers to, for example, another mobile body or external device.

[0026] The area surrounding the mobile body 10 is the region within a predetermined range from the mobile body 10. This range is the observable range of the imaging device 10D. This range can be set in advance.

[0027] The imaging device 10D observes the area around the moving object 10 and acquires surrounding information. The surrounding information includes at least one of the following: an image of the area around the moving object 10, and information indicating the distance and direction between the moving object 10 and objects in the area surrounding the moving object 10.

[0028] The imaging device 10D obtains captured image data (hereinafter referred to as "captured image") through imaging. This imaging device is a digital camera, a stereo camera, etc. The captured image is digital image data in which a pixel value is defined for each pixel.

[0029] In this embodiment, the surrounding information acquired by the imaging device 10D will be described as, for example, an image of the area around the moving object 10. Hereafter, the image of the area around the moving object 10 will be referred to as the surrounding image.

[0030] The imaging device 10D is pre-adjusted in terms of its installation position and field of view so that it can capture images of the area around the mobile body 10. In this embodiment, the mobile body 10 is equipped with multiple imaging devices 10D with different imaging directions.

[0031] Figure 2 is a schematic diagram showing an example of the installation location of the imaging devices 10D. For example, the mobile body 10 is equipped with four imaging devices 10D. However, the number of imaging devices 10D installed on the mobile body 10 is not limited to four.

[0032] The imaging device 10D only needs to be installed in a position and number that allows it to acquire images in approximately the entire area (for example, 360°) centered on the moving body 10 in the horizontal plane, and is not limited to the installation positions and number shown in Figure 2.

[0033] Returning to Figure 1, let's continue the explanation. The drive unit 10G is a drive device mounted on the mobile body 10. The drive unit 10G can be, for example, an engine, a motor, or wheels.

[0034] The drive control unit 10F controls the drive unit 10G in order to automatically drive the mobile body 10. The drive control unit 10F controls the drive unit 10G based on information obtained from the internal sensor 10C or the imaging device 10D, or information received from the information processing device 20. The drive control unit 10F controls the accelerator amount, brake amount, steering angle, etc., of the mobile body 10. For example, the drive control unit 10F controls the mobile body 10 to enter a path indicated by the information received from the information processing device 20, and to stop or drive.

[0035] Next, the hardware configuration of the information processing device 20 will be described. Figure 3 is an example of a hardware configuration diagram of the information processing device 20.

[0036] The information processing device 20 has a CPU (Central Processing Unit) 11A, ROM (Read Only Memory) 11B, RAM (Random Access Memory) 11C, and I / F 11D, etc., all interconnected by a bus 11E, and has a hardware configuration that uses a typical computer.

[0037] The CPU 11A is an arithmetic unit that controls the information processing device 20 of this embodiment. The ROM 11B stores programs and the like that realize the processing performed by the CPU 11A. The RAM 11C stores data necessary for processing performed by the CPU 11A. The I / F 11D is an interface for sending and receiving data.

[0038] The program for executing the information processing performed by the information processing device 20 in this embodiment is provided pre-installed in ROM 11B or the like. Alternatively, the program executed by the information processing device 20 in this embodiment may be provided as a file in a format installable or executable by the information processing device 20, stored in a computer-readable storage medium (e.g., flash memory).

[0039] Next, the functional configuration of the mobile body 10 will be described. Figure 4 is a block diagram showing an example of the functional configuration of the mobile body 10.

[0040] The mobile unit 10 comprises an information processing device 20, an output unit 10A, an input unit 10B, an internal sensor 10C, an imaging device 10D, a drive control unit 10F, and a drive unit 10G.

[0041] The information processing device 20, output unit 10A, input unit 10B, internal sensor 10C, imaging device 10D, and drive control unit 10F are connected via bus 10L to exchange data or signals. The drive control unit 10F is connected to the drive unit 10G to exchange data or signals.

[0042] The information processing device 20 includes a storage unit 20B and a processing unit 20A. The processing unit 20A and the storage unit 20B are connected via a bus 10L to enable the exchange of data or signals. Furthermore, the output unit 10A, the input unit 10B, the internal sensor 10C, the imaging device 10D, and the drive control unit 10F are connected to the processing unit 20A via the bus 10L to enable the exchange of data or signals.

[0043] At least one of the following components may be connected to the processing unit 20A by wire or wirelessly: the storage unit 20B, the output unit 10A (communication unit 10H, speaker 10I, illumination unit 10J, display unit 10K), the input unit 10B, the internal sensor 10C, the imaging device 10D, and the drive control unit 10F. Alternatively, at least one of the following components may be connected to the processing unit 20A via a network: the storage unit 20B, the output unit 10A (communication unit 10H, speaker 10I, illumination unit 10J, display unit 10K), the input unit 10B, the internal sensor 10C, the imaging device 10D, and the drive control unit 10F.

[0044] The storage unit 20B stores data. The storage unit 20B is, for example, a semiconductor memory element such as RAM (Random Access Memory) or flash memory, a hard disk, or an optical disk. The storage unit 20B may also be a storage device provided outside the information processing device 20. Furthermore, the storage unit 20B may store or temporarily store programs or information downloaded via a LAN (Local Area Network) or the Internet. The storage unit 20B may also be composed of multiple storage media.

[0045] The mobile device operates in one of several driving modes according to user instructions. The driving modes include a training mode, in which the user performs a training run to create map data of the area around the mobile device, and an automatic driving mode, in which the mobile device operates automatically in the area where the training run was performed.

[0046] The processing unit 20A includes a teacher driving processing unit 20A1 that performs processing in teacher driving mode and an automatic driving processing unit 20A2 that performs processing in automatic driving mode. The teacher driving processing unit 20A1 is a processing unit 20A that executes teacher driving mode. The automatic driving processing unit 20A2 is a processing unit 20A that executes automatic driving mode.

[0047] The teacher driving processing unit 20A1 includes an acquisition unit 20C, an extraction unit 20D, a creation unit 20E, a determination unit 20F, a specification unit 20G, an output control unit 20H, and a reception unit 20I.

[0048] The acquisition unit 20C, extraction unit 20D, creation unit 20E, determination unit 20F, identification unit 20G, output control unit 20H, and reception unit 20I are implemented by, for example, one or more processors. For example, each of the above units may be implemented by having a processor such as a CPU execute a program, i.e., by software. Each of the above units may be implemented by a processor such as a dedicated IC (Integrated Circuit), i.e., by hardware. Each of the above units may be implemented by using both software and hardware. When multiple processors are used, each of the multiple processors may implement one of the multiple units, or it may implement two or more of the multiple units.

[0049] The processor implements each of the above-mentioned parts by reading and executing the program stored in the memory unit 20B. Alternatively, instead of storing the program in the memory unit 20B, the processor may be configured to directly incorporate the program into its circuitry. In this case, the processor implements each of the above-mentioned parts by reading and executing the program incorporated into its circuitry.

[0050] The acquisition unit 20C acquires peripheral information from the imaging device 10D. As described above, in this embodiment, the imaging device 10D obtains peripheral images 40, which are images of the area around the moving object 10, as peripheral information. Therefore, the acquisition unit 20C acquires peripheral images 40 of the moving object 10 from the imaging device 10D.

[0051] The imaging device 10D acquires surrounding images 40 at predetermined intervals in a time series. Each time the imaging device 10D acquires a surrounding image 40, it outputs the acquired surrounding image 40 to the teacher driving processing unit 20A1. Therefore, the acquisition unit 20C of the teacher driving processing unit 20A1 acquires the surrounding images 40 sequentially from the imaging device 10D.

[0052] The extraction unit 20D extracts feature points around the path traveled by the moving object 10 by analyzing the surrounding image 40 acquired by the acquisition unit 20C from the imaging device 10D.

[0053] The method for extracting feature points by the extraction unit 20D can be any known method, and the extraction method is not limited.

[0054] The extraction unit 20D assigns identification information to the extracted feature points and stores the identification information in the storage unit 20B in association with information indicating the position and range of the feature points in real space. The extraction unit 20D can use position information of the moving body 10 obtained from the internal sensor 10C to extract information indicating the position and range of the feature points in real space.

[0055] The creation unit 20E creates map data containing the location information of the feature points extracted by the extraction unit 20D.

[0056] The map data created by the creation unit 20E is information used for self-localization. For each of the multiple feature points in the real-world scene obtained in advance over a wide area (including the area around the moving object 10), the three-dimensional position in real space and the feature quantity, which is data representing the characteristics of the feature point extracted from image information captured during map data creation, are stored in association. The feature points stored as map data are, for example, parts (e.g., corners) from the image information of objects that can serve as landmarks in the real-world scene (e.g., buildings, signs, or billboards) from which characteristic image patterns can be obtained. In addition, pre-stored landmarks such as position markers may be used as position information in the real-world scene. The multiple feature points of the map data are stored in a way that allows them to be individually identified, for example, by identification numbers.

[0057] The three-dimensional real-space locations of feature points stored in map data are represented in a three-dimensional Cartesian coordinate system (X, Y, Z) based on, for example, latitude, longitude, and height. The three-dimensional real-space locations of feature points are stored, for example, through measurements using the principle of triangulation from camera images taken at multiple locations, or through measurements using LIDAR (Light Detection and Ranging) or stereo cameras.

[0058] The feature quantities used for feature points stored in map data include brightness and density from image information, as well as SIFT (Scale Invariant Feature Transform) features or SURF (Speeded Up Robust Features) features. Furthermore, even for feature points at the same three-dimensional location, the feature quantity data for each feature point may be stored separately based on the shooting position and direction of the imaging device used to capture that feature point. Additionally, the feature quantity data for feature points stored in map data may be stored in association with the image information of the object possessing that feature point.

[0059] The method used by the creation unit 20E to create map data can be any known method, and the method of creating map data is not limited. For example, the creation unit 20E creates map data containing the location information of feature points by creating location information that indicates the location of the feature points extracted by the extraction unit 20D.

[0060] Figure 5 is a schematic diagram showing an example of an application scenario for the mobile body 10 of this embodiment. For example, consider a scenario where user U is performing a training run of the mobile body 10 from point A to point B with the aim of teaching the mobile body 10 the route to automatically travel along an arrow from point A to point B. Point A is, for example, the position where user U disembarks from the mobile body 10, and also the starting position for unmanned automatic travel of the mobile body 10. Point B is, for example, the target parking position for the mobile body 10. Driver U1 is an example of user U riding in the mobile body 10.

[0061] In this case, the imaging device 10D attached to the mobile body 10 photographs the area around the mobile body 10, and the acquisition unit 20C acquires a surrounding image 40 of that area. Therefore, the extraction unit 20D acquires the surrounding image 40 of the area around the mobile body 10 from the acquisition unit 20C. The extraction unit 20D then analyzes the surrounding image 40 to extract feature points contained in the surrounding image 40. Furthermore, the creation unit 20E creates map data containing the location information of the feature points extracted by the extraction unit 20D. The creation unit 20E stores the created map data in the map data storage unit 20B3.

[0062] Returning to Figure 4, the explanation continues. The determination unit 20F extracts a portion of the path traveled by the moving object 10 according to predetermined conditions, and determines whether the number of extracted feature points in each extracted path (hereinafter referred to as the "partial path") is insufficient to meet the predetermined number required for self-localization. If the number of extracted feature points is greater than the predetermined number, it is considered a location where self-localization is likely to succeed. The predetermined conditions include, for example, the vehicle moving a predetermined distance (e.g., 20 cm), or the continuous extraction of a predetermined number or more of the same feature points in temporally continuous images acquired by the imaging device 10D.

[0063] The determination unit 20F repeatedly performs determinations for each partial path and identifies locations where the number of feature points is insufficient compared to a predetermined number.

[0064] The identification unit 20G performs self-position estimation to determine the position of the mobile body 10 when it is performing autonomous driving. The identification unit 20G compares the map data created by the creation unit 20E with the image acquired by the imaging device 10D mounted on the mobile body 10. This allows the identification unit 20G to determine where the mobile body 10 is located within the created map data.

[0065] Next, the output control unit 20H will be described. The output control unit 20H outputs information to the output unit 10A. In this embodiment, the output control unit 20H outputs instruction information to the output unit 10A.

[0066] The instruction information differs from the information provided when the determination unit 20F determines that the number of feature points is insufficient compared to a predetermined number. The instruction information may also indicate that the number of feature points is insufficient compared to a predetermined number when the determination unit 20F determines that the number of feature points is insufficient. Furthermore, the instruction information may indicate the location where the number of feature points is insufficient when the determination unit 20F determines that the number of feature points is insufficient. Additionally, the instruction information may indicate the location where an object containing many features (hereinafter referred to as a feature object) or a location marker, which is a landmark pre-stored in the teacher driving processing unit 20A1, should be placed in order to add location information to the map data when the determination unit 20F determines that the number of feature points is insufficient. The location where the feature object or location marker is placed is a location where, if the teacher driving is performed again with the feature object or location marker in place, the shortage of feature points is likely to be resolved. In other words, that location is an effective location for resolving the shortage of feature points.

[0067] In this embodiment, one example of the instruction information is described as indicating the location where a feature or location marker should be placed in order to add location information to the map data, when the determination unit 20F determines that the number of feature points is insufficient to meet a predetermined number.

[0068] Instructional information may be represented by an image, an audio, or text. A combination of these may also be used. When the instructional information is an image, it will be referred to as the instructional image.

[0069] Figure 6 is a schematic diagram showing an example of an instruction image 52. As shown in Figure 6, the instruction image 52 is an image that indicates the location where a feature or location marker should be placed in order to add location information to map data around the mobile object 10.

[0070] The output control unit 20H generates an instruction image 52 using the map data created by the creation unit 20E, the information obtained by the determination unit 20F from determining whether the number of feature points for each partial route is insufficient compared to a predetermined number, and the location information of the moving object 10 estimated by the identification unit 20G.

[0071] For example, if the determination unit 20F determines that the number of feature points is insufficient compared to a predetermined number, the output control unit 20H generates an instruction image 52 indicating the location where the number of feature points is insufficient. More specifically, if the determination unit 20F determines that the number of feature points is insufficient, the output control unit 20H generates an instruction image 52 indicating the location where a feature object or position marker should be placed in the area where the number of feature points is insufficient.

[0072] Here, a partial path in which the number of feature points is insufficient to a predetermined number required to estimate the self-position of the mobile body 10 is hereinafter referred to as an automatically traversable path RP. As shown in Figure 6, it is preferable for the output control unit 20H to generate an instruction image 52 that highlights the automatically traversable path RP. For example, the automatically traversable path RP is either a framed image which is an image indicating the automatically traversable path RP, or a color image which is an image in which the color of the automatically traversable path RP is shown in a color that attracts the attention of the user U. The color that attracts attention is, for example, yellow or red, but is not limited to these colors. By showing the user the instruction information using an image, the user can know the specific location of the automatically traversable path RP.

[0073] Then, in response to the instruction that the teacher's run has ended, the output control unit 20H outputs the generated instruction image 52 to the output unit 10A.

[0074] In this embodiment, the output control unit 20H displays the instruction image 52 on one or more of the multiple display units 10K provided on the mobile body 10.

[0075] Furthermore, when using voice-based instruction information, the output control unit 20H should output the instruction voice from speaker 10I. For example, the output control unit 20H should output an instruction voice such as, "There are areas where autonomous driving is difficult. Ending the teacher's run," from speaker 10I. An instruction image may also be displayed.

[0076] Returning to Figure 4, let's continue the explanation. Next, we will explain the reception unit 20I. When the determination unit 20F determines that the number of feature points is insufficient compared to a predetermined number, the reception unit 20I receives an input of an instruction to retry the teacher run in the instruction image 52.

[0077] For example, suppose the instruction image 52 shown in Figure 6 is displayed on the display unit 10K. If the determination unit 20F determines that the number of feature points in the instruction image 52 is insufficient compared to a predetermined number, User U places a feature object or position marker at the location where the number of feature points is insufficient, and then inputs an instruction to retry the training run.

[0078] User U inputs a command to retry the teacher run in the instruction image 52, for example, by operating the input unit 10B. More specifically, if the input unit 10B is a touch panel, User U inputs a command to retry the teacher run by touching the retry button displayed on the touch panel. The reception unit 20I receives the input of the command to retry the teacher run by receiving the information instructed by User U from the input unit 10B.

[0079] The reception unit 20I may also accept input for instructions to retry the training run by analyzing the gestures of user U or the voice emitted by user U.

[0080] For example, user U may emit a voice message indicating that they desire a teacher run. In this case, the reception unit 20I can receive an input for a retry of the teacher run by analyzing the voice data collected by the microphone, which serves as the input unit 10B, using a known voice analysis method.

[0081] In response to an instruction to retry the teacher run, the reception unit 20I outputs information indicating the instruction to retry the teacher run to the automatic driving processing unit 20A2.

[0082] The automatic driving processing unit 20A2 performs self-position estimation based on the map data stored in the memory unit 20B and performs automatic driving.

[0083] The memory unit 20B includes a teacher driving program 20B1, an automatic driving program 20B2, and a map data storage unit 20B3.

[0084] When the teacher driving processing unit 20A1 reads the teacher driving program 20B1, processing in teacher driving mode is executed based on the teacher driving program 20B1. When the automatic driving processing unit 20A2 reads the automatic driving program 20B2, processing in automatic driving mode is executed based on the automatic driving program 20B2. The map data storage unit 20B3 stores the map data created by the creation unit 20E.

[0085] Next, an example of the procedure for processing information about the teacher run in Embodiment 1, which is performed by the mobile unit 10, will be described. Figure 7 is a flowchart of an example of the procedure for processing information about the teacher run in Embodiment 1, which is performed by the mobile unit 10.

[0086] First, the acquisition unit 20C acquires a surrounding image 40 from the imaging device 10D (step S101). Next, the extraction unit 20D extracts feature points from the surrounding image 40 acquired in step S101 (step S103). In other words, it extracts information used for self-localization from the surrounding image 40 acquired in step S101.

[0087] In step S105, the creation unit 20E adds map data containing the location information of the feature points extracted in step S103 to the map data storage unit 20B3 (step S105).

[0088] In step S107, the determination unit 20F extracts partial routes from the created map data according to predetermined conditions (step S107). The predetermined conditions include, for example, that a predetermined number or more of identical feature points are continuously extracted within temporally continuous images acquired by the imaging device 10D.

[0089] Next, the determination unit 20F stores whether the number of feature points for each partial path is insufficient compared to a predetermined number, that is, whether it is less than the predetermined number (step S109).

[0090] In step S111, the input unit 10B determines whether the teacher run has ended or not, as instructed by the user (step S111). If the teacher run has ended (step S111: Yes), proceed to step S113. If the teacher run has not ended (step S111: No), return to step S101. Note that any known method may be used to recognize that the teacher run has ended, and the method is not limited. For example, the system may recognize that the teacher run has ended by receiving an instruction from the user to end the teacher run via the input unit 10B. Alternatively, the system may recognize that the teacher run has ended when the shift lever is placed in the parking range.

[0091] Next, the determination unit 20F determines whether the number of feature points in each sub-path is insufficient compared to a predetermined number required to estimate its own position (step S113). In other words, it determines whether the amount of information used for self-position estimation is insufficient compared to a predetermined value. If the number of feature points in any of the sub-paths of the route that has been run as a teacher is insufficient compared to the predetermined number, that is, if it is greater than the predetermined number (step S113: No), the process proceeds to step S115. In step S115, the user is notified that the teacher run has been completed successfully, and this routine ends. When notifying the user of the completion of the teacher run, it may also be notified that automatic driving in automatic driving mode is possible, along with the completion of the teacher run. If one or more of the extracted paths from the route that has been run as a teacher has insufficient number of feature points compared to a predetermined number, that is, if it is less than the predetermined number (step S113: Yes), the process proceeds to step S117.

[0092] If one or more of the extracted paths have fewer feature points than a predetermined number, that is, fewer than the predetermined number (step S113: Yes), the output control unit 20H outputs instruction information to the output unit 10A (step S117). The instruction information is, for example, information that the display unit 10K uses to notify that there are areas in the instruction image 52 where automatic driving is likely not possible. This allows the user to know that there are areas where automatic driving is likely not possible after this training run. In other words, the user can know that automatic driving is likely not possible without even attempting automatic driving. This instruction information may also be announced by voice. If the instruction information is announced by voice, the driver U1 can know that automatic driving is likely not possible without looking at the display unit 10K. This allows the driver U1 to know that automatic driving is likely not possible even when looking at an area other than the display unit 10K.

[0093] Next, the input unit 10B determines whether or not an instruction to retry the teacher run has been entered (step S119). For example, if the display unit 10K is a touch panel, the input unit 10B is a retry button displayed on this touch panel, and the user U enters an instruction to retry the teacher run by touching the retry button on the display unit 10K. If an instruction to retry the teacher run has been entered (step S119: Yes), the system returns to step S101. If an instruction to retry the teacher run has not been entered (step S119: No), the system proceeds to step S121.

[0094] In step S121, it is determined whether a predetermined time has elapsed since the instruction information was announced (step S117). The predetermined time is, for example, 5 minutes. If it is determined that the predetermined time has elapsed (step S121: Yes), the process proceeds to step S123. In step S123, the user is notified of an abnormal termination indicating that the teacher run did not complete successfully, and this routine is terminated. Note that when notifying the user of the completion of the teacher run, the notification method should be different in step S115 and step S123. If it is determined that the predetermined time has not elapsed (step S121: No), the process returns to step S119.

[0095] Next, an example of the information processing procedure for automated driving in Embodiment 1, performed by the mobile unit 10, will be described. Figure 8 is a flowchart of an example of the information processing procedure for automated driving in Embodiment 1, performed by the mobile unit 10.

[0096] First, the automatic driving processing unit 20A2 estimates the initial position based on the map data stored in the map data storage unit 20B3 and the captured image (step S200). The estimated initial position is taken as the self-position (step S201). Next, the automatic driving processing unit 20A2 performs automatic driving from the self-position estimated in step S201 (step S203). In step S205, the automatic driving processing unit 20A2 determines whether or not the mobile body 10 has arrived at the destination (step S205). If it is determined that the mobile body 10 has not arrived at the destination (step S205: No), the process returns to step S201. If it is determined that the mobile body 10 has not arrived at the destination (step S205: No), the process from step S201 to step 205 is repeated. In the repeated step S201, the self-position is estimated based on the map data stored in the map data storage unit 20B3 and the captured image. Furthermore, if it is determined in step S205 that the mobile unit 10 has arrived at its destination (step S205: Yes), this routine terminates.

[0097] In this way, even without actually performing autonomous driving tests, users can learn that there is a high probability that autonomous driving will not be possible, thus improving user convenience.

[0098] <Example 1> In the training run shown in Figure 7, if the number of feature points falls short of a predetermined number, step S117 notifies the user that there are likely to be areas where automatic driving is not possible. However, since the user cannot identify the areas where the number of feature points is short of the predetermined number, there is a possibility that they will not be able to smoothly supplement the number of feature points and complete the training run in a state where automatic driving is possible. Therefore, in step S117, an instruction image as shown in Figure 6 may be used to notify the user that the areas where the number of feature points is short of the predetermined number are places where feature objects should be placed. In this way, the user can know where to place the feature objects. If the user knows where to place the feature objects, they can place the feature objects at those locations, thereby creating an environment where automatic driving is possible even in areas where automatic driving was not possible during the training run.

[0099] <Modification 2> In the teacher run in Modification 1, if the number of feature points is insufficient, the partial path where the number of feature points is insufficient relative to a predetermined number is notified as a place where a feature object should be placed. However, instead of a feature object, a location marker may be notified as a place where a location marker should be placed. Figure 9 is a flowchart showing an example of the information processing procedure for the teacher run in Modification 2 performed by the mobile body 10. The information processing procedure for the teacher run in Modification 2 shown in Figure 9 is described below.

[0100] First, the determination unit 20F extracts a partial path based on predetermined conditions (step S301). These predetermined conditions include, for example, that a predetermined number or more of identical feature points are continuously extracted within temporally continuous images acquired by the imaging device 10D. Next, the acquisition unit 20C acquires a surrounding image 40 of the partial path extracted in step S301 (step S303).

[0101] In step S305, the extraction unit 20D determines whether or not it has extracted position markers from the surrounding image 40 of the partial path acquired in step S303 (step S305). In other words, it determines whether or not it has extracted information to be used for self-localization from the surrounding image 40 of the partial path acquired in step S303. If no position markers are extracted from the surrounding image 40 of the partial path (step S305: No), the process proceeds to step S311. If position markers are extracted from the surrounding image 40 of the partial path (step S305: Yes), the process proceeds to step S307.

[0102] In step S307, the creation unit 20E stores map data containing the location information of the location markers extracted in step S307 in the map data storage unit 20B3 (step S307). If it is the second time or later, the location of the location marker extracted this time is added to the stored map data.

[0103] Next, the determination unit 20F determines whether the number of position markers in the sub-path is insufficient compared to a predetermined number required to estimate its own position (step S309). In other words, it determines whether the amount of information used for self-position estimation is insufficient compared to a predetermined value. If the number of position markers in the sub-path is not insufficient compared to a predetermined number required to estimate its own position, that is, if it is more than the predetermined number (step S309: No), the process proceeds to step S317. If the number of position markers in the sub-path is insufficient compared to a predetermined number required to estimate its own position, that is, if it is less than the predetermined number (step S309: Yes), the process proceeds to step S311.

[0104] In step S311, the extraction unit 20D extracts feature points from the surrounding image 40 of the partial path acquired in step S303 (step S311). Next, the creation unit 20E adds map data containing the location information of the feature points extracted in step S311 to the map data storage unit 20B3 (step S313). Next, in step S315, the determination unit 20F stores whether the number of feature points of the partial path is insufficient compared to a predetermined number, that is, whether it is less than the predetermined number (step S315).

[0105] In step S317, it is determined whether the processing from steps S301 to S315 has been performed on the surrounding images 40 of all partial paths up to the end point of the teacher's run (step S317). Note that any known method can be used to recognize that the teacher's run has been reached, and the method is not limited. For example, the system may recognize that the teacher's run has been reached by receiving an instruction from the user to end the teacher's run via the input unit 10B. Alternatively, the system may recognize that the teacher's run has been reached when the shift lever is placed in the parking range.

[0106] If it is determined that the surrounding images 40 have been processed for all parts of the route up to the end point of the teacher's run (Step S317: Yes), proceed to Step S319. If it is determined that the surrounding images 40 have not been processed for all parts of the route up to the end point of the teacher's run (Step S317: No), return to Step S301 above.

[0107] In step S319, it is determined whether there is a subpath among all subpaths where the number of position markers is insufficient compared to a predetermined number that also has a shortage of feature points (step S319). If there is no subpath among all subpaths where the number of position markers is insufficient compared to a predetermined number that also has a shortage of feature points (step S319: No), the process proceeds to step S321. In step S321, the user is notified that the training run has been completed successfully, and this routine ends. When notifying the user of the completion of the training run, it may also be notified that automatic driving in automatic driving mode is possible, along with the completion of the training run. If there is a subpath among all subpaths where the number of position markers is insufficient compared to a predetermined number that also has a shortage of feature points, that is, if the route traveled in the training run includes a subpath where both the number of position markers and the number of feature points are insufficient compared to a predetermined number (step S319: Yes), the process proceeds to step S323.

[0108] In step S323, the output control unit 20H outputs instruction information to the output unit 10A (step S323). The instruction information is, for example, information that the display unit 10K uses to notify the location of a feature object or a location marker.

[0109] Next, the input unit 10B determines whether or not an instruction to retry the teacher run has been input (step S325). For example, if the display unit 10K is a touch panel, the input unit 10B is a retry button displayed on this touch panel, and the user U inputs an instruction to retry the teacher run by touching the retry button on the display unit 10K. If an instruction to retry the teacher run has been input (step S325: Yes), the system returns to step S301. If an instruction to retry the teacher run has not been input (step S325: No), the system proceeds to step S327.

[0110] In step S327, it is determined whether a predetermined time has elapsed since the instruction information was announced (step S323). The predetermined time is, for example, 5 minutes. If it is determined that the predetermined time has elapsed (step S327: Yes), the process proceeds to step S329. In step S329, the user is notified of an abnormal termination indicating that the teacher run did not complete successfully, and this routine is terminated. Note that when notifying the user of the completion of the teacher run, the notification method should be different in step S321 and step S329. If it is determined that the predetermined time has not elapsed (step S329: No), the process returns to step S325.

[0111] Next, an example of the information processing procedure for automated driving in Modification 2 performed by the mobile body 10 will be described. Figure 10 is a flowchart of an example of the information processing procedure for automated driving in Modification 2 performed by the mobile body 10.

[0112] First, the automatic driving processing unit 20A2 performs an initial position estimation (step S401). Next, in step S403, the automatic driving processing unit 20A2 determines whether or not there is a position marker around the moving body 10 (step S403). If it is determined that there is a position marker around the moving body 10 (step S403: Yes), the process proceeds to step S405. In step S405, the automatic driving processing unit 20A2 identifies the position of the position marker from the map data stored in the map data storage unit 20B3 (step S405). Next, in step S407, the automatic driving processing unit 20A2 estimates its own position from the position of the position marker identified in step S405 (step S407).

[0113] If, in step S403, it is determined that there are no position markers around the moving body 10 (step S403: No), the process proceeds to step S409. In step S409, the automatic driving processing unit 20A2 identifies the location of feature points from the map data stored in the map data storage unit 20B3 (step S409). Next, in step S411, the automatic driving processing unit 20A2 estimates its own position from the location of the feature points identified in step S409 (step S411). Then, the automatic driving processing unit 20A2 performs automatic driving based on the self-position estimated in step S407 or step S411 (step S412).

[0114] In step S413, the automatic driving processing unit 20A2 determines whether or not the destination has been reached (step S413). If it determines that the destination has not been reached (step S413: No), it returns to step S401. If it determines that the destination has been reached (step S413: Yes), this routine terminates.

[0115] As described above, the information processing device 20 of this embodiment comprises an extraction unit 20D, a determination unit 20F, and an output control unit 20H. The extraction unit 20D receives image information, which is information captured around a moving object, and extracts feature points from the received image information. The determination unit 20F determines whether the number of feature points extracted by the extraction unit 20D is insufficient compared to a predetermined number. If the determination unit 20F determines that the number of feature points is insufficient compared to a predetermined number, the output control unit 20H outputs different instruction information (instruction image 52) than when the number of feature points is sufficient.

[0116] In conventional technology, when a training run is performed to create map data of the surrounding environment in an area where the features of the vehicle's surroundings are insufficient, such as an area surrounded by objects without any irregularities or color changes, the number of feature points included in the created map data is likely to be insufficient to estimate the vehicle's own position, making self-position estimation highly unlikely. However, with conventional technology, users could not confirm the high probability that autonomous driving would be impossible in the area where the training run was performed until they actually performed autonomous driving.

[0117] On the other hand, the information processing device 20 of this embodiment outputs a different instruction image 52 when the number of feature points around the moving object is insufficient compared to a predetermined number, based on the teacher's driving test. Therefore, when the determination unit 20F determines that the number of feature points is insufficient compared to a predetermined number, the information processing device 20 can output an instruction image 52 indicating that the number of feature points is insufficient. For example, information indicating locations where the number of feature points is insufficient compared to a predetermined number can be output without the user U actually performing automatic driving.

[0118] Therefore, with the information processing device 20 of this embodiment, the user can confirm, without performing automatic driving, that there is a high probability that automatic driving is not possible in a location where training driving has been conducted. Furthermore, since it may be difficult for the user to determine what kind of object contains many features, position markers prepared in advance for training driving are provided to the user, and by using the position markers, the user can change an environment where automatic driving is likely to be impossible into an environment where automatic driving is possible in a simpler way.

[0119] <Embodiment 2> In Embodiment 1, the amount of information used for self-localization was the number of feature points. In Embodiment 2, the information used for self-localization is information obtained through a tracking process, which involves tracking identical feature points extracted from continuous image information. The amount of information used for self-localization is the number of tracked feature points as a result of this tracking process. Embodiment 2 will be described in detail below, but components identical to those in Embodiment 1 will be denoted by the same reference numerals and their descriptions will be omitted.

[0120] Figure 11 is a block diagram showing an example of the functional configuration of the mobile body 10 in Embodiment 2. In Embodiment 2, the matching processing unit 20J, the tracking processing unit 20K, and the creation unit 20E differ from the configuration shown in Figure 4 of Embodiment 1.

[0121] The matching processing unit 20J uses the similarity of features to determine whether feature points extracted from consecutive images are identical (hereinafter, this process is referred to as the matching process). In other words, it determines whether feature point a1 in frame A, which is a past image, and feature point b1 in frame B, which is the current image, are the same feature point among the images used in the matching process. Any known method can be used for the matching method, which is the method for determining whether feature points are the same between frames, and the matching method is not limited.

[0122] The tracking processing unit 20K performs tracking on the feature points that were matched in the matching process. The tracking processing unit 20K determines whether the distance between the position X estimated in frame B of feature point a1 extracted from frame A and the position Y of feature point b1 extracted from frame B is less than a predetermined value (hereinafter, this process will be referred to as the tracking process). If the distance between the estimated position X and the extracted position Y is less than a predetermined value, the feature point is considered to have been tracked. The predetermined value is, for example, 5 pixels. Note that any known method may be used for tracking, and the tracking method is not limited. For example, by estimating the amount of movement of the imaging device from the speed of the moving object 10 or the tire angle, it is possible to estimate the position of a feature point in image B from the position of a feature point in image A. The position of a feature point is, for example, the coordinate on the image. If the number of tracked feature points is greater than a predetermined number, it is considered to be a location where self-localization is likely to succeed.

[0123] The creation unit 20E creates map data containing the location information of the tracked feature points. The creation unit 20E may also exclude feature points that were not tracked from the location data of the feature points extracted by the extraction unit 20D.

[0124] Figure 12 is a flowchart showing an example of the procedure for processing information about the training run in Embodiment 2, which is performed by the mobile unit 10.

[0125] First, the acquisition unit 20C acquires a surrounding image 40 from the imaging device 10D (step S501). Next, the extraction unit 20D extracts feature points from the surrounding image 40 acquired in step S501 (step S503). In other words, it extracts information used for self-localization from the surrounding image 40 acquired in step S501.

[0126] In step S505, the matching processing unit 20J performs a matching process between feature points extracted from the current image and feature points extracted from past images that are continuous with the current image (step S505). Next, the tracking processing unit 20K performs a tracking process on the feature points that were matched in step S505 (step S507).

[0127] In step S509, the creation unit 20E creates map data containing the location information of the feature points tracked in step S507 and stores it in the map data storage unit 20B3 (step S509). In subsequent steps of step S509, the newly created map data is added to the already stored map data.

[0128] In step S511, the determination unit 20F extracts partial routes from the map data according to predetermined conditions (step S511). The predetermined conditions include, for example, that a predetermined number or more of identical feature points are continuously extracted within temporally continuous images acquired by the imaging device 10D.

[0129] Next, the determination unit 20F stores whether the number of feature points for each extracted subpath is insufficient compared to a predetermined number, that is, whether it is less than or more than the predetermined number (step S513).

[0130] In step S515, the input unit 10B determines whether or not the user has given an instruction to end the teacher run (step S515).

[0131] Next, the determination unit 20F determines whether the number of tracked feature points for each sub-path is insufficient compared to a predetermined number required to estimate its own position (step S517). In other words, it determines whether the amount of information used for self-position estimation is insufficient compared to a predetermined value. If the number of feature points for any of the sub-paths is not insufficient compared to the predetermined number, that is, if it is greater than the predetermined number (step S517: No), the process proceeds to step S519. In step S519, the user is notified that the training run has been completed successfully, and this routine ends. When notifying the user of the completion of the training run, it may also be notified that automatic driving in automatic driving mode is possible, along with the completion of the training run. If the number of tracked feature points for any one or more of the extracted paths is insufficient compared to a predetermined number, that is, if it is less than the predetermined number (step S517: Yes), the process proceeds to step S521.

[0132] If one or more of the extracted paths have fewer tracked feature points than a predetermined number, that is, fewer than the predetermined number (step S517: Yes), the output control unit 20H outputs instruction information to the output unit 10A indicating that there is a high probability that automatic driving is not possible in some areas (step S521). The instruction information is, for example, information that the display unit 10K uses to notify that automatic driving is not possible in the instruction image 52. This allows the user to know that automatic driving is not possible after this training run. In other words, the user can know that automatic driving is not possible without attempting automatic driving. This instruction information may also be announced by voice. If the instruction information is announced by voice, the driver U1 can know that automatic driving is not possible without looking at the display unit 10K. This allows the driver U1 to know that automatic driving is not possible when looking at an area other than the display unit 10K.

[0133] Next, the input unit 10B determines whether or not an instruction to retry the teacher run has been input (step S523). For example, if the display unit 10K is a touch panel, the input unit 10B is a retry button displayed on this touch panel, and the user U inputs an instruction to retry the teacher run by touching the retry button on the display unit 10K. If an instruction to retry the teacher run has been input (step S523: Yes), the system returns to step S501. If an instruction to retry the teacher run has not been input (step S523: No), the system proceeds to step S525.

[0134] In step S525, it is determined whether a predetermined time has elapsed since the instruction information was broadcast (step S521). The predetermined time is, for example, 5 minutes. If it is determined that the predetermined time has elapsed (step S525: Yes), this routine is terminated. If it is determined that the predetermined time has not elapsed (step S525: No), the process returns to step S523.

[0135] Next, an example of the information processing procedure for automated driving in Embodiment 2, performed by the mobile unit 10, will be described. Figure 13 is a flowchart showing an example of the information processing procedure for automated driving in Embodiment 2, performed by the mobile unit 10.

[0136] First, the automatic driving processing unit 20A2 estimates its own position based on the map data stored in the map data storage unit 20B3 and the captured image (step S601). Next, the automatic driving processing unit 20A2 performs automatic driving based on the position estimated in step S601 (step S603). In step S605, the automatic driving processing unit 20A2 determines whether the mobile body 10 has arrived at its destination (step S605). If it determines that the mobile body 10 has not arrived at its destination (step S605: No), it returns to step S601. If it determines that the mobile body 10 has arrived at its destination (step S605: Yes), this routine terminates.

[0137] In this way, users can learn that autonomous driving is highly unlikely to be possible, even without actually performing the test.

[0138] <Other Embodiments> In Modification 1, partial paths where the number of feature points is insufficient to a predetermined number are indicated as locations where feature objects should be placed, and in Modification 2, locations where position markers should be placed instead of feature objects are indicated. However, partial paths where the number of feature points is insufficient to a predetermined number may also be indicated as locations where either feature objects, position markers, or both feature objects and position markers should be placed. This increases the degree of freedom in the objects to be placed and further improves user convenience.

[0139] Furthermore, in this embodiment, an example was described in which the information processing device 20 is mounted on the mobile body 10. However, the information processing device 20 may also be mounted outside the mobile body 10. In this case, the information processing device 20 should be configured to communicate with each of the electronic devices such as the internal sensor 10C mounted on the mobile body 10 via a network.

[0140] In the embodiment described above, the program for executing the information processing is configured as a module that includes each of the multiple functional units. In actual hardware, for example, the CPU (processor circuit) reads the information processing program from ROM or HDD and executes it, thereby loading each of the multiple functional units onto RAM (main memory) and generating each of the multiple functional units onto RAM (main memory). It is also possible to implement some or all of the multiple functional units using dedicated hardware such as ASIC (Application Specific Integrated Circuit) or FPGA (Field-Programmable Gate Array).

[0141] Although embodiments have been described above, these embodiments are presented as examples only and are not intended to limit the scope of this disclosure. The novel embodiments described above can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. The embodiments described above are included in the scope or spirit of this disclosure and are included in the scope of the invention and its equivalents as described in the claims. [Explanation of Symbols]

[0142] 10 Mobile Units 10A output section 10B Input Section 10C Internal Sensor 10D imaging device 10F Drive Control Unit 10G drive unit 10H Communications Department 10I Speaker 10J lighting section 10K display 10L bus 11A CPU 11B ROM 11C RAM 11D I / F 11E Bus 20 Information Processing Devices 20A Processing Unit 20A1 Teacher Driving Processing Unit 20A2 Automatic Driving Processing Unit 20B Storage section 20B1 Teacher Run Program 20B2 Automated Driving Program 20B3 Map data storage unit 20C Acquisition Department 20D extraction part 20E Creation Department 20F Judgment section 20G specific part 20H Output Control Unit 20I Reception Department 20J Matching Processing Unit 20K Tracking Processing Unit

Claims

1. In an information processing device that performs automatic driving based on driving path data showing the driving path when a user manually drives the vehicle, During the aforementioned teacher run, an extraction unit receives image information, which is information captured around the moving object, and extracts information used for self-position estimation from the received image information. An output control unit outputs to the output unit different instruction information when the amount of information used for self-position estimation extracted by the extraction unit is a first amount, compared to when the amount of information used for self-position estimation is a second amount greater than the first amount. Equipped with, The starting position of the teacher's driving is the starting position of the automatic driving. The end position of the aforementioned teacher's run is the parking target position. If the amount of information used for self-position estimation is equal to the second amount, the output control unit notifies the user that the training run has been completed successfully. The instruction information is output before the automatic driving based on the driving route data is performed. The second quantity is the amount of information sufficient to perform the self-localization estimation. The first quantity is a smaller amount of information than the second quantity. Information processing device.

2. The first quantity is the amount by which the amount of information used for self-localization is insufficient compared to a predetermined value. The information processing apparatus according to claim 1.

3. The second quantity is the amount of information used for self-localization that satisfies a predetermined value. The information processing apparatus according to claim 1.

4. The aforementioned instruction information is information indicating that the amount of information used for self-position estimation is insufficient compared to the predetermined value, when the amount of information used for self-position estimation is insufficient compared to the predetermined value. The information processing apparatus according to claim 2.

5. The aforementioned instruction information is information indicating that the amount of information used for self-position estimation satisfies the predetermined value when the amount of information used for self-position estimation satisfies the predetermined value. The information processing apparatus according to claim 3.

6. The output control unit, If the amount of information used for self-localization is insufficient compared to the predetermined value, an instruction voice indicating information different from that when the amount of information used for self-localization is sufficient is output as the instruction information. The information processing apparatus according to claim 2.

7. The output control unit, If the amount of information used for self-localization satisfies the predetermined value, an instruction voice indicating that the amount of information used for self-localization satisfies the predetermined value is output as the instruction information. The information processing apparatus according to claim 3.

8. The output control unit, If the amount of information used for self-localization is insufficient compared to the predetermined value, an instruction image showing different information than that when the amount of information used for self-localization is sufficient is output as the instruction information. The information processing apparatus according to claim 2.

9. The output control unit, If the amount of information used for self-localization satisfies the predetermined value, an instruction image indicating that the amount of information used for self-localization satisfies the predetermined value is output as the instruction information. The information processing apparatus according to claim 3.

10. The amount of information used for the aforementioned self-localization is represented by the number of feature points. An information processing apparatus according to any one of claims 1 to 9.

11. In an information processing method for performing automated driving based on driving path data that shows the driving path when a user manually drives a vehicle, Information used for self-localization is extracted from image information, which is information captured around the moving object during the aforementioned teacher run. When the amount of information extracted for self-position estimation around the moving object is a first amount, the output unit outputs different instruction information than when the amount of information used for self-position estimation is a second amount greater than the first amount. Includes, The starting position of the teacher's driving is the starting position of the automatic driving. The end position of the aforementioned teacher's run is the parking target position. If the amount of information used for self-position estimation is the second amount, the user is notified of the successful completion of the training run. The instruction information is output before the automatic driving based on the driving route data is performed. The second quantity is the amount of information sufficient to perform the self-localization estimation. The first quantity is a smaller amount of information than the second quantity. Information processing methods.

12. The first quantity is the amount by which the amount of information used for self-localization is insufficient compared to a predetermined value. The information processing method according to claim 11.

13. The second quantity is the amount of information used for self-localization that satisfies a predetermined value. The information processing method according to claim 11.

14. The aforementioned instruction information is information indicating that the amount of information used for self-position estimation is insufficient compared to the predetermined value, when the amount of information used for self-position estimation is insufficient compared to the predetermined value. The information processing method according to claim 12.

15. The aforementioned instruction information is information indicating that the amount of information used for self-position estimation satisfies the predetermined value when the amount of information used for self-position estimation satisfies the predetermined value. The information processing method according to claim 13.

16. In the step of outputting the instruction information to the output unit, If the amount of information used for self-localization is insufficient compared to the predetermined value, an instruction voice indicating information different from that when the amount of information used for self-localization is sufficient is output as the instruction information. The information processing method according to claim 12.

17. In the step of outputting the instruction information to the output unit, If the amount of information used for self-localization satisfies the predetermined value, an instruction voice indicating that the amount of information used for self-localization satisfies the predetermined value is output as the instruction information. The information processing method according to claim 13.

18. In the step of outputting the instruction information to the output unit, If the amount of information used for self-localization is insufficient compared to the predetermined value, an instruction image showing different information than that when the amount of information used for self-localization is sufficient is output as the instruction information. The information processing method according to claim 12.

19. In the step of outputting the instruction information to the output unit, If the amount of information used for self-localization satisfies the predetermined value, an instruction image indicating that the amount of information used for self-localization satisfies the predetermined value is output as the instruction information. The information processing method according to claim 13.

20. The amount of information used for the aforementioned self-localization is represented by the number of feature points. The information processing method according to any one of claims 11 to 19.

Images