Information processing apparatus, information processing method, information processing program, and computer-readable recording medium with information processing program recorded therein

Abstract

To provide an information processing apparatus for updating a defective part, in a stored road image, of a predetermined object included in the stored road image and a new road image, such as a white line, a speed display on a road, a cross walk, a stop position display, a signboard along a road, a sign, a signal, and architecture on the basis of the new road image.SOLUTION: A navigation apparatus functioning as an information processing apparatus has a control unit, a storage device, an input device, a display unit, an input and output interface device, a plurality of sensors, a camera, a GPS reception unit, and a data receiving and transmitting unit. The control unit acquires a new road image including objects relating to a road within a captured range, specifies a road image in which the same point as that of the new road image was captured among road images stored in storage means, and updates a defective part in the specified road image by the new road image.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present application relates to an information processing device or the like that updates road images stored in a storage device. [Background technology]

[0002] Conventionally, technologies that assist drivers in driving using captured images of the vehicle's forward direction have become widespread. For example, there is a technology that assists drivers in parking by displaying captured images of the rear of the vehicle on a display while parking. However, such parking assistance technologies have a problem in that the brightness of the captured images is low when parking at night or in dark places, making it difficult to distinguish parking space lines from the captured images, and therefore failing to provide sufficient assistance to the driver.

[0003] In response to this, Patent Document 1 discloses a technology that detects the outline of a parking space based on an image captured by a camera, and presents a pseudo parking space line to the driver by superimposing a figure based on the detected outline of the parking space on the captured image or extending the frame line of the parking space, thereby assisting the driver in parking operations. However, this technology does not correct the shape of the parking space itself if the shape is incomplete, such as if the parking space is interrupted halfway, so there is a risk that the driver will not be able to recognize the correct frame line.

[0004] Under such circumstances, Patent Document 2 discloses a technology in which, when the detected parking stall lines are incomplete, a previous image in which parking stall lines having the same shape as the detected parking stall lines are captured is read out, and the parking stall lines in the previous image are aligned with the parking stall lines in the image currently being captured and superimposed on the image currently being captured. This makes it possible to present to the driver an image in which the incomplete parking stall lines have been complemented by the previous image. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2003-63339 [Patent Document 2] Patent No. 4670463 Summary of the Invention [Problem to be solved by the invention]

[0006] However, the technology described in Patent Document 1 has a problem in that if the parking stall lines are incomplete even in the past images, the parking stall lines cannot be complemented using the past images.

[0007] The present application has been made in view of the above-mentioned problems, and one example of an object of the present application is to provide an information processing device or the like that can appropriately update information included in road images. [Means for solving the problem]

[0008] The invention described in claim 1 comprises an image acquisition means for acquiring a new road image including road-related objects, and an update means for updating missing portions of the road image of a specified object included in the road image stored in a storage means and the new road image based on the new road image.

[0009] The invention described in claim 2 comprises an image acquisition means for acquiring a new road image including road-related objects, and a complementation means for complementing missing portions of a road image stored in a storage means and a specified object included in the new road image based on the road image.

[0010] The invention described in claim 3 is an information processing method used by an information processing device, and includes an image acquisition step of acquiring a new road image including road-related objects, and an update step of updating missing portions of the road image of a specified object included in a road image stored in a storage means and the new road image based on the new road image.

[0011] The invention described in claim 4 is an information processing method used by an information processing device, and includes an image acquisition step of acquiring a new road image including road-related objects, and a complementation step of complementing missing portions of a road image stored in a storage means and a specified object included in the new road image based on the road image.

[0012] The invention described in claim 5 causes a computer to function as an image acquisition means for acquiring a new road image including road-related objects, and an update means for updating missing portions of the road image of a specified object included in the road image stored in a storage means and the new road image based on the new road image.

[0013] The invention described in claim 6 is a computer that stores the information processing program described in claim 5. It is a data-readable recording medium. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a block diagram of an information processing device 1 according to the present embodiment. [Figure 2] FIG. 2 is a diagram showing an example of a past road image in the present embodiment. [Figure 3] FIG. 2 is a diagram showing an example of a new road image in the present embodiment. [Figure 4] FIG. 10 is a diagram showing an example of a past road image updated with a new road image in this embodiment. [Figure 5] FIG. 2 is a diagram showing an example of a new road image supplemented with a past road image in this embodiment. [Figure 6] FIG. 2 is a block diagram of a navigation device NV according to the present embodiment. [Figure 7] 10 is a flowchart showing an example of image update processing by the control unit 211 of the navigation device NV of this embodiment. [Figure 8] FIG. 2 is a diagram showing an example of a white line on a new road image in this embodiment. [Figure 9]FIG. 2 is a diagram showing an example of a white line in a past road image according to the present embodiment. [Figure 10] 10A and 10B are diagrams illustrating an example of a difference white line calculated from a new road image and a previous road image in this embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0015] An embodiment of the present invention will be described with reference to FIGS. 1 to 5. FIG.

[0016] As shown in FIG. 1, the information processing device 1 includes an image acquisition unit 111A, a specification unit 111B, and an update unit 111C.

[0017] In this configuration, the image acquisition means 111A acquires a new road image that includes road-related objects within the image capture range.

[0018] The specifying means 111B specifies, from among road images stored in a storage means provided inside or outside the information processing device 1, a road image captured at the same point as the new road image acquired by the image acquiring means 111A.

[0019] The update means 111C updates part of the information included in the road image identified by the identification means 111B with the new road image acquired by the image acquisition means 111A.

[0020] As described above, according to the operation of the information processing device 1 of the embodiment, part of the information of the road image stored in the storage means is updated with a new road image, so that the information contained in the road image can be kept up to date.

[0021] For example, in a road image 500 shown in FIG. 2, among three white lines 501L, 501C, and 501R, if a portion of the white line 501L (an example of "a portion of the information of the road image") is hidden by a puddle 502, the portion of the white line 501L is updated with a new road image 600 captured at the same spot as the road image 500, as shown in FIG. 3. As a result, as shown in FIG. 4, the road image 500 is complemented with a white line portion 503 where the puddle 502 of the white line 501L was located. In other words, the information of the road image 500 can be enriched with the new road image 600. Furthermore, as will be described later, since the road image is used when complementing the new road image, enriching the information can improve the accuracy at the time of complementation.

[0022] On the other hand, when the new road image 600 shown in Fig. 3 is displayed on a display, if parts of the three white lines 601L, 601C, and 601R are concealed by dense fog 602, the parts of the white lines 601L, 601C, and 601R are complemented by the road image 500. This results in the new road image 600 shown in Fig. 5. This allows the user of the information processing device 1 to visually recognize the white line portions 603L, 603C, and 603R that are actually concealed by the dense fog 602 and cannot be seen. Note that in the new road image 600 shown in Fig. 5, the complemented white line portions 603L, 603C, and 603R are displayed so as to be distinguishable from the other portions, allowing the user to determine which parts have been complemented and making the white line portions more visible. [Example]

[0023] Next, a specific example corresponding to the above-described embodiment will be described.

[0024] The following embodiments will be described with reference to Figures 6 to 10. The embodiments described below are examples in which the present invention is applied to a navigation device (an example of an "information processing device").

[0025] [1. Configuration of the navigation device NV] First, the configuration of the navigation device NV according to this embodiment will be described.

[0026] As shown in FIG. 6, the navigation device NV is configured to include a control unit 211, a storage device 212 consisting of an HDD or the like, an input device 213 consisting of a keyboard or remote controller, a touch panel or the like, a display unit 214, a bus line 215, an input / output interface unit 220, a vehicle speed sensor 221, an angular velocity sensor 222, an acceleration sensor 223, a steering angle sensor 224, a GPS (Global Positioning System) receiving unit 225 connected to an antenna 226, a data transmitting / receiving unit 227, and a camera 228.

[0027] The vehicle speed sensor 221 detects the current speed of the vehicle using a speed detection process using, for example, a vehicle speed pulse acquired from an automobile equipped with the navigation device NV, and outputs speed data. The angular velocity sensor 222 detects, for example, the angular velocity of the vehicle, which is a change in direction, and outputs angular velocity data and relative direction data per unit time. The acceleration sensor 223 detects, for example, the acceleration of the vehicle in the longitudinal direction, and outputs acceleration data per unit time, etc. The steering angle sensor 224 detects the steering angle of the vehicle and outputs steering angle data, etc. The GPS receiver 225 receives navigation radio waves from GPS satellites and outputs GPS positioning data, such as latitude, longitude, and altitude data, absolute direction data of the vehicle's traveling direction, and GPS speed data. The data transmitter / receiver 227 performs processing related to sending and receiving data to and from a server device via a network.

[0028] The camera 228 is installed facing the outside of the automobile in which the navigation device NV is installed, and captures road images that include within their capture range objects related to the road the automobile is traveling on. In this embodiment, one camera 228 is installed inside the automobile, and captures the scenery ahead of the automobile (in the direction of travel).

[0029] The storage device 212 (an example of a "storage means") stores map image data for displaying a map on the display unit 214, map information used when searching for a route, road link information, and the like. The storage device 212 also stores road images captured by the camera 228 or other cameras as past road images. The storage device 212 stores the past road images in association with the location at the time of capture, road ID (identification information assigned to all roads), and image capture condition information indicating the traveling direction. The storage device 212 also stores various programs such as an operating system and application programs. The various programs and past road images may be acquired, for example, from a server device or the like via a network, or may be read from recording media such as a CD, DVD, or USB memory.

[0030] The input device 213 accepts an input operation from the user and transmits an operation signal indicating the operation content to the control unit 211 .

[0031] The display unit 214 displays various display data under the control of the control unit 211. The display unit 214 is configured to include a graphics controller 214a, a buffer memory 214b consisting of a memory such as a VRAM (Video RAM), and a display 214c consisting of a liquid crystal display or the like. In this configuration, the graphics controller 214a controls the entire display unit 214 based on control data sent from the control unit 211 via a bus line 215. The buffer memory 214b also temporarily stores image information that can be displayed immediately. Images are then displayed on the display 214c based on the image data output from the graphics controller 214a.

[0032] The control unit 211 is made up of a CPU 211a that controls the entire control unit 211, a ROM 211b in which control programs and the like that control the control unit 211 are stored in advance, and a RAM 211c that temporarily stores various data. The control unit 211 is connected to a vehicle speed sensor 221, an angular velocity sensor 222, an acceleration sensor 223, a steering angle sensor 224, and a GPS receiving unit 225 via a bus line 215 and an input / output interface unit 220, and controls the entire navigation device NV and the operations of various components such as a display unit 214 based on the speed data, angular velocity data, relative direction data, steering angle data, GPS positioning data, absolute direction data of the vehicle's traveling direction, acceleration data, and the like output from each of these sensors.

[0033] The control unit 211 acquires new road images captured by the camera 228 in real time, identifies past road images stored in the storage device 212 that were captured at the same location as the new road image acquired from the camera 228, and updates part of the information (in this embodiment, white lines) included in the identified past road image with the new road image acquired from the camera 228. Specifically, the control unit 211 compares the past road image with the new road image, and updates the information if a white line that does not exist in the past road image is detected. At this time, the control unit 211 stores information indicating the position at the time of capture, the road ID, the traveling direction, etc. in association with each other as an update record.

[0034] At the same time, the control unit 211 complements part of the information (white lines in this embodiment) included in the new road image acquired from the camera 228 with the previously identified past road image, and displays the complemented part on the display unit 214 in real time. At this time, the control unit 211 displays the complemented part so that it can be distinguished from the other parts. As a result, a real-time new road image with part of the information complemented is displayed on the display 214c of the display unit 214. Note that the control unit 211 updates the past road image or complements the new road image only when an update or complement is necessary. In other words, as described above, if part of the white lines is hidden for some reason, the control unit 211 updates or complements the hidden part. Furthermore, the control unit 211 acquires new road images from the camera 228 as moving images (for example, acquiring 30 images per second) and displays them on the display unit 214, so that the road images are displayed as moving images on the display 214c.

[0035] In this configuration, the CPU 211a of the control unit 211 is an example of an "image acquisition means," a "specification means," an "updating means," a "complementing means," and an "output means."

[0036] [2. Image update processing by the control unit 211 of the navigation device NV] 7 to 10, the image update process executed mainly by the control unit 211 will be described. Note that, here, the white line portion included in the road image is the object to be updated or complemented.

[0037] The control unit 211 acquires a new road image captured by the camera 228 (step S11). When acquiring a new road image, the control unit 211 acquires the image capturing condition information indicating the image capturing position (the vehicle position information output by the GPS receiving unit 225), the road ID of the road currently being traveled, and the traveling direction on that road (absolute direction data output by the GPS receiving unit 225).

[0038] Next, the control unit 211 detects white line portions in the new road image (step S12). As a detection method, a conventionally known general image processing method is applied. For example, white portions are extracted by referring to the brightness values ​​in the image, and linear areas within the white line portions are determined to be white line portions.

[0039] Next, the control unit 211 identifies a past road image corresponding to the new road image from among the past road images stored in the storage device 212 (step S13). Specifically, the control unit 211 searches using the shooting condition information indicating the position at the time of shooting, road ID, and traveling direction acquired in step S11 as a search key. However, depending on the accuracy of the GPS, it may not be possible to identify an appropriate past road image. Therefore, the control unit 211 performs processing according to the following procedure as a method of identifying a desired past road image. (1) Using the photographing condition information as a search key, a plurality of candidate past road images are narrowed down. (2) Calculate the similarity (not necessarily a numerical value) between each past road image narrowed down in (1) above and the new road image acquired in step S11 based on the image characteristics, and identify the past road image with the highest similarity.

[0040] Here, the process of (2) will be described in detail. First, the control unit 211 extracts image features from each of the previous road images and the new road image narrowed down in the process of (1), and searches for corresponding points of the features between the two images. Here, SIFT (Scale-Invariant Feature Transform) or the like can be used as the image features. A small number of corresponding points means that there are few similar features between the two images, and the appearance of the images also differs significantly. Therefore, the control unit 211 searches for the corresponding points described above for all previous road images narrowed down in the process of (1), and excludes from the candidates for identification those images for which the number of corresponding points is less than an empirically set threshold T. Next, the control unit 211 considers the positional relationship of the corresponding points of the image features extracted earlier from the remaining previous road images, and identifies one final previous road image. Specifically, the control unit 211 accumulates the differences in pixel positions on the images for all corresponding points between the new road image and the previous road image. The more similar the images look between the two images, the smaller this integrated value will be, and if the images look exactly the same, the integrated value will be 0. Therefore, the control unit 211 calculates the integrated value for each past road image that is a candidate for identification, and identifies the one with the smallest value as the past road image. However, it is preferable that the control unit 211 also performs threshold processing on the integrated value, and excludes past road images that look significantly different.

[0041] Next, the control unit 211 compares the white line portion detected in the processing in step S12 with the white line portion of each past road image stored in the storage device 212 (step S14), and determines whether there is a difference (step S15).

[0042] The processing of step S15 will now be described in detail with reference to Figures 8 to 10. The control unit 211 compares the white line portions of the new road image acquired in the processing of step S11 with the previous road image identified in the processing of step S13, and determines whether there is a difference between them. An example of this comparison method will be described below.

[0043] Here, a case will be described in which a new road image 700 shown in Fig. 8(A) is compared with a previous road image shown in Fig. 9(A). One white line 701 is detected in the new road image 700, whereas two white lines 801 and 802 are detected in the previous road image 800 shown in Fig. 9(A). Information about each white line is calculated from the angle of the white line on the image and the coordinates of the start and end points of the white line, as shown in Fig. 8(B) and Fig. 9(B).

[0044] In the above example, the control unit 211 searches for a white line having a similar angle in the previous road image 800 based on the angle of the white line 701 in the new road image 700. In this example, the angle of the white line 701 in the new road image 700 is 35°, and the angles of the white lines 801 and 802 in the previous road image 800 are the same as that of the white line 701, so the control unit 211 searches for the white lines 801 and 802. In this search, an error of ±ε may be allowed in the angle.

[0045] Next, the control unit 211 calculates the difference in the white line area between the white line 701 in the new road image 700 and the white lines 801 and 802 found in the previous road image 800, based on the starting point and end point. In this example, the white line 701 in the new road image 700 is from (0,480) to (500,200), and the white lines 801 and 802 in the previous road image 800 are divided lines with only a portion of the white line 701 removed. The control unit 211 calculates the range of the difference from the starting point and end point, thereby obtaining a difference white line 901 in the difference image 900 shown in FIG. 10.

[0046] If the differential white line 901 is obtained, the control unit 211 determines "YES" in the processing of step S15. Note that in this example, a case has been described in which an undetected portion of a white line exists in the previous road image 800, and a differential white line is obtained to fill in that portion. However, conversely, a differential white line is similarly obtained when an undetected portion of a white line exists in the new road image, and the control unit 211 also determines "YES" in the processing of step S15 when a differential white line is obtained. That is, the control unit 211 compares the new road image with the previous road image, and if an undetected portion exists in either one and a differential white line is obtained, it determines that there is a difference. Furthermore, if a differential white line is obtained, information about the differential white line is saved and used in the processing of step S16 or step S17, which will be described later.

[0047] If the control unit 211 determines in the processing of step S15 that there are no different portions (step S15: NO), it causes the display unit 214 to display the new road image acquired in step S11 as is (step S18), and ends the image update processing. On the other hand, if the control unit 211 determines that there are different portions (step S15: YES), it then updates the previous road image with the new road image (step S16). Specifically, if the different portion determined in the processing of step S15 is a white line portion that can be detected in the new road image but cannot be detected in the previous road image, the control unit 211 updates the white line portion that cannot be detected in the previous road image with a white line portion that is in the same position in the new road image (the differential white line that was saved in the processing of step S14).

[0048] On the other hand, if the difference portion determined in the processing of step S15 is a white line portion that can be detected in the previous road image but cannot be detected in the new road image, the control unit 211 complements the white line portion that cannot be detected in the new road image with a white line portion that is in the same position in the previous road image (the differential white line that was saved in the processing of step S14) (step S17). At this time, the control unit 211 complements the white line portion with, for example, green so that the complemented portion can be distinguished from other portions. After completing the processing of step S17, the control unit 211 causes the display unit 214 to display the new road image that has been complemented in the processing of step S17 (step S18), and ends the image update processing.

[0049] As described above, according to the operation of the navigation device NV of this embodiment, the control unit 211 acquires a new road image from the camera 228 that includes road-related objects in its photographed range (step S11), identifies a past road image from among the past road images stored in the memory device 212 that was captured at the same location as the new road image acquired from the camera 228 (step S13), and updates the white line portion included in the identified past road image with the new road image acquired from the camera 228 (step S16).

[0050] Therefore, the navigation device NV can supplement the white line portions of the past road images stored in the storage device 212 with the latest new road images.

[0051] Furthermore, the control unit 211 complements the white line portions included in the new road image acquired from the camera 228 with the previous road image identified in the processing of step S13 (step S17). This allows the white line portions of the new road image acquired from the camera 228 to be complemented with the previous road image and then displayed. If the white line portions are not displayed in the new road image acquired from the camera 228, it is highly likely that the user of the navigation device NV is not able to directly see the white line portions on the road. Therefore, by displaying a new road image with the white line portions complemented, the user can drive by estimating the positions of the white lines, thereby improving safety.

[0052] [3. Modifications] Next, modifications of the above embodiment will be described. The modifications described below can be combined as appropriate.

[0053] [3.1. Variation 1] In the above-described embodiment, the past road images are stored in the storage device 212, but instead, the past road images may be stored in a storage device of a server device outside the navigation device NV, and the image update process shown in Figure 7 may be performed while accessing the storage device via a network.

[0054] [3.2. Variation 2] In the above-described embodiment, the case where the object to be updated or supplemented is a white line on a road has been described, but the object is not limited to a white line, and any object included in a road image can be updated or supplemented. For example, speed signs, crosswalks, stop signs, signs, traffic lights, buildings, etc. on the road can be updated or supplemented. Furthermore, multiple types of objects can be updated or supplemented.

[0055] [3.3. Variation 3] In the above-described embodiment, the display device is the display 214c, but other display devices may be used instead. For example, a head-up display (HUD) may be used. A head-up display projects information directly into the human field of vision, allowing the driver to view new road images at any time without having to shift his or her line of sight to the display 214c of the navigation device NV. [Explanation of symbols]

[0056] 1. Information processing equipment 111A Image acquisition means 111B Specific means 111C Update method NV navigation device 211 Control Unit 211a CPU 211b ROM 211c RAM 212 Storage device 213 Input Device 214 Display Unit 214a graphics controller 214b Buffer memory 214c Display 215 Bus Line 220 Input / Output Interface Section 221 Vehicle speed sensor 222 Angular rate sensor 223 Accelerometer 224 Steering Angle Sensor 225 GPS receiver 226 Antenna 227 Data transmission and reception unit 228 Camera

Claims

[Claim 1] image acquisition means for acquiring new road images including road-related objects; an updating means for updating a missing portion of the predetermined object included in the road image stored in the storage means and the new road image based on the new road image; An information processing device comprising:

Images