Parking support method and parking support device

Abstract

To provide a parking support method and a parking support device in a parking support system capable of performing self-position estimation of a vehicle with high accuracy even when changes occur in the surrounding environment when performing automatic parking.SOLUTION: In a first parking and a second parking performed after the first parking, a parking support method obtains map information existing around a vehicle, and displays differential information indicating a difference between first map information on the first parking and second map information on the second parking when the first map information and the second map information are different. The map information indicates the coordinate positions of a group of feature points extracted from an image taken of the surroundings of the vehicle. In the differential information, the coordinate positions of feature point groups that exist in the first map information but do not exist in the second map information are highlighted.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a parking assistance method and a parking assistance device. [Background technology]

[0002] In recent years, there has been a demand for a parking assistance function that performs automatic parking (hereinafter referred to as automatic parking) without involving steering operation or accelerator / brake operation by the driver.

[0003] For example, in Patent Document 1, when a driver drives a vehicle, the vehicle generates automatic driving information for automatic driving of the vehicle based on an image of the surrounding environment. Furthermore, when performing automatic driving, the vehicle calculates vehicle position information based on the image of the surrounding environment and the automatic driving information, and performs automatic parking. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2017-138664 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, the technology described in Patent Document 1 has problems. In the technology described in Patent Document 1, automatic parking is performed based on information when the driver drives and parks the vehicle. In this case, the environment around the vehicle may be different between when the driver drives and parks the vehicle and when automatic parking is performed thereafter. For example, it is not uncommon for an object (such as a bicycle) that was not present when the driver drove and parked the vehicle to be present when the vehicle is subsequently parked automatically.

[0006] In this case, the accuracy of the vehicle's self-location estimation during automatic parking may decrease, and as a result, the accuracy of the automatic parking itself may also decrease. On the other hand, the change in the environment around the vehicle during parking may be temporary. For example, if the above-mentioned object is a bicycle, it is quite likely that it will be gone or its position will have changed the next time the vehicle is parked. Therefore, even if the surrounding environment is updated to that grasped during automatic parking, the accuracy of the vehicle's self-location estimation may still decrease.

[0007] The present disclosure has been made in consideration of the above-mentioned problems, and has as its main objective to provide a parking assistance method and a parking assistance device in a parking assistance system that is capable of estimating the vehicle's self-position with high accuracy even when changes occur in the surrounding environment when performing automatic parking. [Means for solving the problem]

[0008] A parking assistance method according to the present disclosure is a parking assistance method used in a parking assistance system that estimates a self-position of a vehicle from map information present around the vehicle and performs parking assistance based on the self-position, the method acquiring the map information present around the vehicle in a first parking and a second parking performed after the first parking, and when the first map information in the first parking and the second map information in the second parking are different, displaying difference information indicating a difference between the first map information and the second map information, the map information being information indicating coordinate positions of a group of feature points extracted from an image capturing an image of the periphery of the vehicle, and the difference information being information that highlights and displays the coordinate positions of a group of feature points that are present in the first map information but not in the second map information.

[0009] A parking assistance device according to the present disclosure is a parking assistance device used in a parking assistance system that estimates a self-position of a vehicle from map information present around the vehicle and performs parking assistance based on the self-position, and includes: an acquisition unit that acquires the map information present around the vehicle in a first parking and a second parking performed after the first parking; and a display unit that displays difference information indicating a difference between the first map information and the second map information when the first map information in the first parking and the second map information in the second parking are different, wherein the map information is information indicating the coordinate positions of a group of feature points extracted from an image captured around the vehicle, and the difference information is information that highlights and displays the coordinate positions of a group of feature points that are present in the first map information but not in the second map information. Effect of the Invention

[0010] With the parking assistance method and parking assistance device according to the present disclosure, when performing automatic parking, it is possible to estimate the vehicle's own position with high accuracy even if changes occur in the surrounding environment. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of the parking assistance system according to the first embodiment. [Diagram 2] FIG. 2 is a flowchart illustrating an example of an information processing procedure according to the first embodiment. [Diagram 3] 3(a) to (c) are diagrams showing an example of map information generated in the information processing device of embodiment 1, where (a) is a diagram showing an example of map information recorded in a memory unit, (b) is a diagram showing an example of map information generated in an automatic driving mode, and (c) is a diagram showing an example of a state in which the self-position is estimated from the map information in (a) and the map information in (b). [Figure 4] FIG. 4 is a diagram showing an example of a display form of difference information according to the first embodiment. [Diagram 5] FIG. 5 is a diagram showing an example of a display form of difference information according to the first embodiment. [Figure 6] FIG. 6 is a diagram showing an example of a display form of difference information according to the first embodiment. [Figure 7] FIG. 7 is a diagram showing an example of a display form of difference information in the modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] (Embodiment 1) A parking assistance system according to a first embodiment will be described with reference to FIGS.

[0013] The parking assistance system according to the first embodiment is a system that performs parking assistance to realize automatic parking, which performs parking of a vehicle based on prerecorded route information. Specifically, the vehicle drives in either a teacher driving mode or an automatic driving mode according to the user's instructions. The teacher driving mode is a parking mode performed as a preliminary step to automatic parking, and is a mode in which the vehicle is parked by the user's operation. In the supervised driving mode, while the vehicle is parked in a parking space by the user's operation, the parking assistance device creates map information of the surroundings of the vehicle to be used for estimating the vehicle's self-position based on information obtained from the sensor device or the imaging device 50, and sequentially records the created map information until parking is completed. Furthermore, the parking assistance device records the route information traveled until parking of the vehicle is completed together with the creation of the map information. The automatic driving mode is a mode in which the vehicle is parked automatically without user operation. In the automatic driving mode, the surrounding map information is generated from the image captured by the imaging device 50, as in the teacher driving mode. Next, the surrounding map information recorded in the teacher driving mode is collated, the vehicle's own position is estimated from the position information of the recorded map information, and the vehicle is parked according to the route information recorded based on the estimated position. Next, each member constituting the parking assistance system according to the first embodiment will be described.

[0014] (Configuration of parking assistance system 10) First, the configuration of a parking assistance system 10 according to the first embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the configuration of the parking assistance system 10 according to the first embodiment.

[0015] The parking assistance system 10 is for performing automatic parking of a vehicle as described above, and includes an information processing device 20, an information display device 30, an automatic driving control device 40, an imaging device 50, and a drive device 60. The parking assistance system 10 may be mounted inside the vehicle, or may be connected to the vehicle via a separate wireless network and provided entirely or partially outside the vehicle. The configurations of the information processing device 20 and the information display device 30 will be described in detail below.

[0016] (Configuration of information processing device 20) The information processing device 20 includes a processing unit 210 and a storage unit 220, and generates map information consisting of image feature point information such as various three-dimensional structures and road surface textures around the vehicle for use in estimating the vehicle's own position, or OGM information from image features of three-dimensional structures, from captured images of the surroundings of the vehicle captured by the imaging devices 50 installed on the front, rear, left and right sides of the vehicle. Furthermore, the map information newly generated from the captured images in which the captured positions around the vehicle are updated as the vehicle travels is associated with the amount of movement (distance and direction) of the vehicle estimated from the steering angle sensor and wheel speed pulse sensor, which are on-board sensors, and stored as one piece of map information. Furthermore, the information processing device 20 generates difference information indicating the difference between the first map information generated in the first parking and recorded in the storage unit 220 of the information processing device 20 and the second map information newly generated in the second parking. Furthermore, the information processing device 20 updates the stored map information as necessary. The first parking may be performed by the driver driving the vehicle, and the second parking may be performed by automatic driving. Alternatively, the first parking and the second parking may be performed by automatic driving by repeating parking in the automatic driving mode.

[0017] (Processing unit 210) A more detailed configuration of the processing unit 210 will be described below. The processing unit 210 performs information processing for generating the above-mentioned difference information. The processing unit 210 includes an image acquisition unit 211, a map information generation unit 212, a difference information generation unit 213, an output control unit 214, and a self-position estimation unit 215. Each unit constituting the processing unit 210 will be described below.

[0018] (Image acquisition unit 211) The image acquisition unit 211 acquires captured images of the vehicle's surroundings captured by the imaging device 50 mounted on the vehicle. The captured images are digital image data in which a pixel value is specified for each pixel. Typically, the imaging device 50 mounted on the vehicle is composed of four imaging devices 50 installed on the front, rear, left and right of the vehicle. Note that the image acquisition unit 211 may acquire captured images of the vehicle's surroundings captured by an imaging device 50 such as an infrastructure camera, not limited to the imaging device 50 mounted on the vehicle.

[0019] (Map information generating unit 212) The map information generating unit 212 generates map information using the captured image acquired by the image acquiring unit 211. The map information is information used to calculate the vehicle's own position. The vehicle's own position is estimated by matching a feature amount in the map information stored in advance with a feature amount in the map information generated based on the captured image. In the first embodiment, an example will be described in which the map information is held in an OGM (Occupied Grid Map) format in which the presence or absence of a three-dimensional obstacle or the presence or absence of a three-dimensional obstacle is set for each section (also called a grid) divided into a grid defined by a certain size (for example, 5 cm or 10 cm) is set.

[0020] In addition, in response to information from the information display device 30, the map information generation unit 212 updates the map information recorded in the storage unit 220 by reflecting difference information in the map information recorded in the storage unit 220.

[0021] (Difference information generation unit 213) The difference information generating unit 213 generates difference information using the second map information generated by the map information generating unit 212 and the first map information recorded in advance. In the first embodiment, the difference information is information for notifying the user of a difference when a difference exists. As an example, the difference information is information indicating a set of sections in which the total number of sets of sections in which a difference exists is equal to or greater than a predetermined value, when comparing the first map information generated in advance and recorded in the storage unit 220 of the information processing device 20 with the second map information newly generated in the automatic driving mode.

[0022] (Output control unit 214) The output control unit 214 outputs the difference information generated by the difference information generation unit 213 to the information display device 30.

[0023] (Self-position estimation unit 215) The self-position estimation unit 215 collates the map information recorded in the storage unit 220, and estimates the vehicle's own position from the position information of the map information.

[0024] (Storage unit 220) The storage unit 220 is a recording medium for storing map information. The map information stored in the storage unit 220 is initially generated in the teacher driving mode. Thereafter, when an instruction is received from the user, the map information stored in the storage unit 220 is updated based on the map information generated in the automatic driving mode.

[0025] Examples of the recording medium that can be used include tapes such as magnetic tapes and cassette tapes, magnetic disks such as floppy disks (registered trademark) and hard disks, disks including optical disks such as CD-ROMs (Compact Disc Read-Only Memory), MO disks (Magneto-Optical discs), MDs (Mini Discs), DVDs (Digital Versatile Discs: registered trademark), CD-Rs (CD Recordable), and Blu-ray Discs (registered trademark), cards such as IC cards (including memory cards) and optical cards, semiconductor memories such as mask ROMs, EPROMs (Erasable Programmable Read-Only Memory), EEPROMs (Electrically Erasable and Programmable Read-Only Memory: registered trademark), and flash ROMs, and logic circuits such as PLDs (Programmable logic devices) and FPGAs (Field Programmable Gate Arrays).

[0026] The storage unit 220 does not necessarily have to be provided inside the information processing device 20, and may be provided outside the information processing device 20 as necessary.

[0027] (Configuration of information display device 30) Next, the information display device 30 constituting the parking assistance system 10 will be described. The information display device 30 displays the difference information generated by the information processing device 20 in a form that is visible to the user. The information display device 30 may be directly connected to the information processing device 20 or may be connected via a network. For example, an in-vehicle display device that functions as an HMI for a car navigation system or a parking assistance system can be cited as an example of a device that is directly connected to the information processing device 20. For example, a portable information terminal such as a smartphone or a tablet can be cited as an example of a device that is connected to the information processing device 20 via a network.

[0028] The information display device 30 includes a difference information acquisition section 310 , a difference information processing section 320 , a display section 330 , an input section 340 , and an instruction section 350 .

[0029] (Difference information acquisition unit 310) The difference information acquisition unit 310 acquires the difference information generated and output by the information processing device 20 .

[0030] (Difference information processing unit 320) The difference information processing unit 320 performs processing to display the difference information acquired by the difference information acquisition unit 310 on the display unit 330 in a form visible to the user. In this embodiment, display in a form visible to the user specifically means display using an image. Furthermore, when the difference information processing unit 320 receives an instruction from the input unit 340 to update the map information recorded in the storage unit 220, it instructs the instruction unit 350 to reflect, in the map information recorded in the storage unit 220, the section sets in which the total number of sets of sections in which differences exist is equal to or greater than a predetermined value.

[0031] (Display section 330) The display unit 330 is a display device that notifies the user of information by displaying an image. The notified information is information for notifying the user of the difference information acquired by the difference information acquisition unit 310 and an inquiry as to whether or not to update the map information recorded in the storage unit 220. Specific examples of the display unit 330 include display devices such as a liquid crystal display or an organic EL display.

[0032] (Input unit 340) The input unit 340 is an input device for receiving instructions from a user. Examples of the input unit 340 include a touch panel that is integrated with the display unit 330, or a microphone that receives voice input.

[0033] (Instruction section 350) The instruction unit 350 instructs the information processing device 20 to update the map information recorded in the memory unit 220 by reflecting, among the difference information of the map information, the information of the section set for which an update instruction has been received from the user, in the map information recorded in the memory unit 220.

[0034] (Configuration of the automatic driving control device 40) The automatic driving control device 40 includes an automatic driving information registration unit 410 and an automatic driving control unit 420.

[0035] (Autonomous driving information registration unit 410) The automatic driving information registration unit 410 operates in the teacher driving mode. In the teacher driving mode, the automatic driving information registration unit 410 records information about the route traveled until the vehicle is parked.

[0036] (Automatic driving control unit 420) The automatic driving control unit 420 operates in the automatic driving mode. The automatic driving control unit 420 controls the drive unit 60 to automatically drive the vehicle based on the image of the vehicle surroundings captured by the imaging device 50 in the automatic driving mode and the self-position estimated by the self-position estimation unit 215, and in accordance with the route information recorded in the automatic driving information registration unit 410.

[0037] (Configuration of imaging device 50) The installation position and the angle of view of the imaging device 50 are adjusted in advance so that the imaging device 50 can capture images of the surroundings of the vehicle. In the first embodiment, the vehicle is provided with a plurality of imaging devices 50 with different imaging directions.

[0038] (Configuration of driving device 60) The driving device 60 controls the driving unit to automatically drive the vehicle. More specifically, the driving device 60 controls the acceleration amount, braking amount, steering angle, etc. of the vehicle so that the vehicle automatically travels on the route at a specified speed based on the vehicle's own position and the route information recorded in the teacher driving mode.

[0039] (Procedure for processing information in the parking assistance system 10) Next, the procedure of information processing executed in the parking assistance system 10 will be described below with reference to Fig. 2. Fig. 2 is a flowchart showing the procedure of information processing executed in the parking assistance system 10.

[0040] First, the image acquisition unit 211 acquires images of the surroundings of the vehicle captured by the imaging device 50 while the vehicle is traveling (step S101).

[0041] Next, the map information generating unit 212 generates map information using the captured image acquired in step S101, and estimates the vehicle's own position (step S102). A specific example of the generated map information will be described below.

[0042] 3(a) to 3(c) are diagrams showing an example of map information generated by the map information generating unit 212. As shown in FIG. 3(a) to 3(c), in the map information in the OGM format, the space around the vehicle to be parked is divided into sections of a certain size, and the presence or absence of a three-dimensional obstacle is set for each section, or the presence or absence of a three-dimensional obstacle is unknown. The sections with diagonal lines represent sections where a three-dimensional obstacle exists, and the white sections represent sections where a three-dimensional obstacle does not exist. FIG. 3(a) is a diagram showing an example as a partial area of ​​the map information recorded in the storage unit 220, FIG. 3(b) is a diagram showing an example of map information generated from a captured image acquired at a certain point during the vehicle's travel in the automatic travel mode, and FIG. 3(c) is a diagram showing an example of a state in which the self-position 110 is estimated with respect to the map information in FIG. 3(a) from the map information in FIG. 3(a) and the map information in FIG. 3(b). In addition, the section surrounded by a thick frame in Figure 3 (a) and Figure 3 (c) is the parking position 100. In addition, the size of the section in Figure 3 is shown as a schematic large size, but in reality, the size of the section is further subdivided, and the map information is shown in finer sections.

[0043] In the automatic driving mode, the map information generating unit 212 generates map information based on the captured image acquired by the image acquiring unit 211, and calculates the position of the section in which the vehicle exists, which is the vehicle's own position, in the generated map information. The map information generating unit 212 reads out the map information (FIG. 3(b)) generated in the automatic driving mode and the map information (FIG. 3(a)) recorded in the storage unit 220, performs matching, searches for the position where the total value of the matching error is the smallest, and determines the vehicle's own position. The result of the self-position calculated by the matching process is shown in FIG. 3(c). The vehicle is driven to follow the route based on the self-position 110 estimated by the self-position estimating unit 215 and the route information stored in the automatic driving information registering unit 410, and automatic parking is performed. In addition, the matching process may involve calculating the amount of movement of the vehicle from its previously calculated self-position to the position where the current captured image is to be acquired using on-board sensors (e.g., a steering angle sensor and a wheel speed pulse sensor, etc.), and searching the surrounding area based on the position obtained by adding the amount of movement to the previous self-position, thereby reducing the amount of processing required for the search and reducing errors in calculating the self-position. The number of captured images used for matching is not limited to one, and map information obtained from four captured images taken of the front, rear, left and right sides of the vehicle may be matched. This makes it possible to obtain the vehicle's own position robustly against differences in position accuracy, differences in the illuminance environment around the vehicle, and changes in the placement of three-dimensional obstacles. In the matching process, the estimation result of the self-location is calculated not necessarily as an integer multiple of the section location, but as a decimal point location, thereby making it possible to increase the location resolution.

[0044] The difference information generating unit 213 generates difference information by comparing the second map information generated by the map information generating unit 212 with the first map information previously recorded in the storage unit 220 (step S103). Specifically, the first map information previously recorded and used in step S102 is compared with the second map information generated in the automatic driving mode, and a set of sections in which the total number of sets of sections having a difference is equal to or greater than a predetermined value is detected as an area in which the difference is considered to be a difference in map information rather than within the error range, and difference information is generated. In other words, information is generated that indicates a change in the environment around the vehicle between when the map information recorded in the storage unit 220 was generated during parking and when the vehicle is traveling. Furthermore, in order to display the difference information on the information display device, the difference information generating section 213 transmits to the output control section 214 the difference information in association with the captured image from which the difference information was detected.

[0045] The output control unit 214 outputs the difference information generated by the difference information generating unit 213 and the captured image associated with the difference information to the information display device 30 (step S104). The processes up to this step are executed in the information processing device 20. The processes in the information display device 30 will be described below with continued reference to FIG. 2. The processes in the information processing device 20 and the processes in the information display device 30 may be processed continuously as one unit, or may be processed with a time interval therebetween.

[0046] The difference information acquisition unit 310 of the information display device 30 acquires the difference information output in step S104 (step S105).

[0047] The difference information processing section 320 generates an image for displaying the difference information acquired by the difference information acquisition section 310 in a form that can be visually recognized by the user (step S106).

[0048] The difference information processing unit 320 displays the image generated by the difference information processing unit 320 on the display unit 330, and notifies the user whether or not to update the map information recorded in the storage unit 220 (step S107). For example, the difference information processing unit 320 controls the display unit 330 to output an image indicating whether or not to update the map information recorded in the storage unit 220 by reflecting, in the map information recorded in the storage unit 220, information indicating a section set in which the total number of sections in which differences exist is equal to or greater than a predetermined value, together with the image generated by the difference information processing unit 320.

[0049] The input unit 340 receives an instruction from the user as to whether or not to update the map information recorded in the storage unit 220 (step S109). This makes it possible to determine whether or not to reflect changes in the environment around the vehicle in the map information based on the user's judgment. If an instruction to update the map information recorded in the storage unit 220 is received from the user (step S109: Yes), the process proceeds to step S110. If an instruction to update the map information recorded in the storage unit 220 is not received from the user (step S109: No), the operation of the parking assistance system 10 is terminated.

[0050] The instruction unit 350 instructs the information processing device 20 to update the map information recorded in the storage unit 220, and ends the operation of the parking assistance system 10 (step S110).

[0051] In this way, the parking assistance system 10 allows the user to determine whether the change in the captured image around the vehicle is temporary, and determines whether to reflect the change in the environment around the vehicle in the map information based on the user's determination. Therefore, the parking assistance system 10 can estimate the vehicle's self-position with high accuracy even if the surrounding environment changes when performing automatic parking, by creating map information that accurately reflects the change in the environment around the vehicle every time parking is performed.

[0052] It is preferable that the user be notified as to whether or not to update the map information recorded in the storage unit 220 immediately after parking is completed and the vehicle is stopped.

[0053] Further, although the case where the storage unit 220 stores one piece of map information has been described as an example, the storage unit 220 may store a plurality of pieces of map information.

[0054] Furthermore, in a third parking performed after the second parking, map information around the vehicle may be obtained, and if the second map information generated in the second parking and recorded in the memory unit 220 of the information processing device 20 differs from the third map information newly generated in the third parking, difference information showing the difference between the second map information and the third map information may be displayed.

[0055] (Display form 1) Next, the display form when the difference information is displayed to the user on the display unit 330 will be specifically described below with reference to Figures 4 to 6. As described above, the difference information in the first embodiment is information indicating a set of sections where the total number of sets of sections where there is a difference is equal to or greater than a predetermined value when comparing the first map information recorded in advance in the storage unit 220 with the second map information newly generated in the autonomous driving mode.

[0056] FIG. 4 is a diagram showing where a set of sections in which the total number of sets of sections in which differences exist is equal to or greater than a predetermined value (hereinafter, referred to as a set of sections that is a difference) is located with respect to the vehicle's driving route. In this display form, the position of the set of sections that is a difference (also referred to as a difference position) is specified by a relative position with respect to the vehicle's driving route or the parking position 100, and the difference position is displayed surrounded by a thick frame. Furthermore, the position of the set of sections that is a difference is specified by a relative position with respect to the vehicle's driving route or the parking position 100, and the difference position is displayed as a newly added object using text information. As for the vehicle's driving route, the route traveled from the parking start point to the parking end point is displayed using an arrow line. By displaying the difference information to the user in this form, the user can specifically know the position where the environment around the vehicle has changed. In Fig. 4(a), a travel route, a solid line surrounded by a thick frame representing a difference information position, and dotted lines representing a parking start position and a parking end position are shown in an aerial photograph of a three-dimensional obstacle taken from above. However, since it is usually difficult to create an aerial photograph from an image captured by the imaging device 50 mounted on a vehicle, the image displayed on the information display device may be composed of only the travel route, the solid line surrounded by a thick frame, and the dotted line, as shown in Fig. 4(b).

[0057] In response to the above-mentioned notification, the user may use, for example, input unit 340, which is an input function of a touch panel provided integrally with display unit 330, to select whether or not to update.

[0058] (Display form 2) Fig. 5 is a diagram showing where a set of sections in which the total number of sets of sections in which differences exist is equal to or greater than a predetermined value is located relative to the vehicle's travel route. In addition to the display format of Fig. 4, the section in which the total number of sets of sections in which differences exist is equal to or greater than a predetermined value is displayed in a color different from other sections. By displaying difference information to the user in this format, the user can specifically know the location where the environment around the vehicle has changed. In Fig. 5(a), in an aerial photograph showing a three-dimensional obstacle viewed from above, the driving route, the filled-in areas using a different color from other sections to indicate the difference information positions, the solid lines surrounded by thick frames indicating the difference information positions, and the dotted lines indicating the parking start position and the parking end position are shown. However, since it is usually difficult to create an aerial photograph from an image captured by the imaging device 50 mounted on a vehicle, the image displayed on the information display device may be composed only of the driving route, the filled-in areas using a different color from other sections to indicate the difference information positions, the solid lines surrounded by thick frames, and the dotted lines, as shown in Fig. 5(b).

[0059] (Display form 3) FIG. 6 is a diagram showing a section set in which the total number of sections having differences is equal to or greater than a predetermined value, superimposed on a captured image of the periphery of a vehicle. As an example, a range including a predetermined area in which the section set is located is cut out from the captured image and displayed on the captured image. The section set that is the difference is displayed surrounded by a thick frame. In addition, the section set that is the difference is displayed by text information as a newly added object. By displaying the difference information to the user in this form, the position where the environment around the vehicle has changed can be specifically known. The captured image may be a composite image that is composited based on captured images captured by a plurality of imaging devices 50 mounted on the vehicle. In order to make it easier for the user to recognize the difference information, the display image of FIG. 4 or FIG. 5 may be displayed side by side in addition to the display image of FIG. 6. Although the display forms 1 to 3 have been described taking the case where a new object is added as an example, the difference information can be displayed using a similar display form even if an object has disappeared.

[0060] (Processing when there are multiple map information represented as differential information) In the examples shown in Figs. 4 to 6, the map information represented as the difference information is one, but the map information represented as the difference information may be a plurality of pieces. That is, a plurality of objects may be added, or a plurality of objects may be lost. Also, an added object and a lost object may exist simultaneously. This allows the map information to reflect changes in the environment around the vehicle in more detail. When a plurality of pieces of difference information exist, in the flowchart of Fig. 2, after repeating steps S109 and S110, the user is asked whether or not all the pieces of difference information have been updated, and the process is terminated.

[0061] When there are multiple pieces of map information represented as difference information, they may be notified to the user individually or collectively. When notifying the user individually, one of the added or disappeared objects is notified, and the operation of accepting instructions from the user is repeated in sequence to process each piece of difference information. In this case, the user can examine each piece of map information represented as difference information one by one and determine whether or not to reflect it. When notifying the user collectively, instructions from the user for the added or disappeared object are accepted at once, and each piece of difference information is processed.

[0062] (Modification) In the above-mentioned parking assistance system 10, the map information is stored in the OGM format, in which the area around the parking route is divided into a grid of a certain size, and the presence or absence of a three-dimensional obstacle is set for each section, or the presence or absence of a three-dimensional obstacle is unknown. However, a modified example will be described below in which the map information is stored in another format.

[0063] As a modified example, the map information is information showing the coordinate positions of the feature points extracted from the captured image. In this case, the difference information is information for emphasizing and displaying coordinate positions where no corresponding feature points exist in a comparison between the first map information generated in advance and recorded in the storage unit 220 of the information processing device 20 and the second map information newly generated in the automatic driving mode. Each feature point of the feature points has a coordinate position expressed by (Xi, Yi) or (Xi, Yi, Zi) in a coordinate space in which the origin is set at an arbitrary point with respect to the environment around the parking and driving and the coordinate axes of a two-dimensional plane or three-dimensional space are set. Furthermore, in addition to the coordinate information of the feature points, it is also possible to associate the coordinates of the vehicle position at the time of extracting the feature points to form map information.

[0064] More specifically, the difference information is information for emphasizing and displaying the coordinate positions of a group of feature points that are present in the first map information generated in advance and recorded in the information processing device 20 but are not present in the newly generated second map information. Alternatively, the difference information is information for emphasizing and displaying the coordinate positions of a group of feature points that are not present in the first map information generated in advance and recorded in the information processing device 20 but are present only in the newly generated second map information. The former is a case where an object that did not exist when the parking was performed when the map information recorded in the information processing device 20 was generated is added, and the latter is a case where an object that existed when the parking was performed when the map information recorded in the information processing device 20 was generated is disappeared. In the former case, in response to an instruction from the user, the map information recorded in the storage unit 220 is updated by adding the coordinate positions of the feature points that exist only in the newly generated map information to the map information recorded in the storage unit 220. Also, in the latter case, in response to an instruction from the user, the map information recorded in the storage unit 220 is updated by deleting the coordinate positions of the feature points that do not exist in the newly generated map information.

[0065] (Display form 4) Next, a display form when displaying difference information to a user in the modified example will be specifically described below with reference to FIG. 7. FIG. 7 is a diagram showing coordinate positions where no corresponding feature points exist in a captured image of the periphery of a vehicle. The coordinate positions where no corresponding feature points exist are displayed surrounded by a thick frame. Furthermore, the user is notified of whether or not to update the map information recorded in the storage unit 220. In response to the above-mentioned notification, the user may select whether or not to update using, for example, the input unit 340, which is an input function in a touch panel provided integrally with the display unit 330. By displaying difference information to the user in this form, the user can specifically know the position where the environment around the vehicle has changed.

[0066] In the above-mentioned embodiment, the program for executing the information processing has a modular configuration including each of the above-mentioned multiple functional units, and as actual hardware, for example, a CPU (processor circuit) reads out and executes the information processing program from a ROM or HDD, whereby each of the above-mentioned multiple functional units is loaded onto a RAM (main memory), and each of the above-mentioned multiple functional units is generated on the RAM (main memory). It is also possible to realize a part or all of each of the above-mentioned multiple functional units using dedicated hardware such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array).

[0067] Although the embodiment has been described above, the embodiment is presented as an example and is not intended to limit the scope of the present disclosure. The embodiment can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the gist of the invention. The embodiment is included in the scope or gist of the present disclosure, and is included in the scope of the invention and its equivalents described in the claims. [Industrial Applicability]

[0068] The parking assistance system according to the present disclosure can be used, for example, for automatic parking assistance. [Explanation of symbols]

[0069] 10 Parking Assistance System 20 Information processing device 210 Processing section 211 Image acquisition unit 212 Map information generation unit 213 Difference information generation section 214 Output control section 215 Self-position estimation part 220 Storage section 30 Information display device 310 Difference information acquisition unit 320 Differential information processing section 330 Display section 340 Input section 350 Instruction section 40 Automatic Driving Control Device 410 Autonomous Driving Information Registration Department 420 Automatic Driving Control Unit 50 Imaging device 60 Drive unit

Claims

1. A parking assistance method for use in a parking assistance system that estimates a vehicle's own position from map information present around the vehicle and performs parking assistance based on the vehicle's own position, comprising: In a first parking and a second parking performed after the first parking, the map information existing around the vehicle is acquired; When the first map information in the first parking area and the second map information in the second parking area are different, difference information indicating a difference between the first map information and the second map information is displayed; the map information is information indicating coordinate positions of a group of feature points extracted from a captured image of an area around the vehicle, the difference information is information for highlighting and displaying coordinate positions of a group of feature points that are present in the first map information but not present in the second map information; Parking assistance methods.

2. the difference information is information displayed by a frame line, A difference position indicating a difference position between the first map information and the second map information based on a relative position with respect to a driving route or a parking position of the vehicle is indicated by the frame line. The parking assistance method according to claim 1 .

3. the difference information is information displayed by character information, A difference position indicating a difference position between the first map information and the second map information based on a relative position with respect to a driving route or a parking position of the vehicle is indicated by the character information.

3. A parking assistance method according to claim 1 or 2.

4. an image of a range including the difference position is cut out from a captured image of the periphery of the vehicle and displayed; 4. The parking assistance method according to claim 2 or 3.

5. The timing for displaying the difference information is after the second parking is completed. A parking assistance method according to any one of claims 1 to 4.

6. The timing for displaying the difference information is during driving in the second parking position. A parking assistance method according to any one of claims 1 to 5.

7. The first parking is performed by a driver driving the vehicle, The second parking is performed by automatic driving. A parking assistance method according to any one of claims 1 to 6.

8. The first parking and the second parking are performed by automatic driving. A parking assistance method according to any one of claims 1 to 6.

9. notifying a user of the vehicle of whether or not a difference between the first map information and the second map information should be reflected in the map information; reflecting a difference between the first map information and the second map information in the map information when a user of the vehicle selects to reflect the difference; A parking assistance method according to any one of claims 1 to 8.

10. notifying a user of the vehicle of whether or not a difference between the first map information and the second map information should be reflected in the map information; When a user of the vehicle has not selected to reflect the difference between the first map information and the second map information, the difference is not reflected in the map information. A parking assistance method according to any one of claims 1 to 9.

11. The display device for displaying the difference information is a portable information terminal, and connected to an information processing device; A parking assistance method according to claim 9 or 10.

12. A parking assistance device used in a parking assistance system that estimates a vehicle's own position from map information present around the vehicle and performs parking assistance based on the vehicle's own position, In the first parking and the second parking performed after the first parking, the surroundings of the vehicle an acquisition unit for acquiring the map information present in the vicinity; First map information for the first parking area, and second map information for the second parking area. When the first map information and the second map information are different, difference information indicating a difference between the first map information and the second map information is displayed. a display unit that displays the Equipped with the map information is information indicating coordinate positions of a group of feature points extracted from a captured image of an area around the vehicle, the difference information is information for highlighting and displaying coordinate positions of a group of feature points that are present in the first map information but not present in the second map information; Parking assistance device.

Images