Vehicle surrounding image display device and vehicle surrounding image display method

Abstract

To enable a driver to easily recognize a situation of a periphery of a vehicle.SOLUTION: A vehicle periphery image display device 100 comprises: a position detection unit 131 for detecting a position of a vehicle 1; a form selection unit 132 for selecting one display form DFA from among a plurality of display forms DF as a display form DS of a map image PG of a periphery of the position of the vehicle 1; a map image display unit 133 for displaying the map image PG in the one display form DFA on a display 61; an image acquisition unit 134 for acquiring an image of a periphery of the vehicle 1; a determination unit 135 for determining whether or not a distance LD from the vehicle 1 to a tollgate FS is equal to or less than a first distance L1; and a periphery image display unit 137 by which, in a case where the determination unit 135 determines that the distance LD is equal to or less than the first distance L1, the image of the periphery of the vehicle 1 is displayed on the display 61 in a direction corresponding to the map image PG in the one display form DFA.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a vehicle surrounding image display device and a vehicle surrounding image display method.

Background Art

[0002] Conventionally, a technique for displaying an image around a vehicle has been known. For example, in Patent Document 1, a storage unit that stores in advance facility information regarding a predetermined facility to be a display target of an overhead image and display conditions of the overhead image in the predetermined facility, and a display condition determination unit that determines whether or not the display conditions of the overhead image in the predetermined facility are met, and when it is determined that the display conditions are met by the display condition determination, an image conversion unit that generates an overhead image by converting an image from the captured image, and a display control unit that displays the overhead image generated by the image conversion unit on the display unit are described.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the technique described in Patent Document 1, the orientation of the displayed overhead image may not be appropriate. When the orientation of the overhead image is not appropriate, there is a possibility that it is difficult for the driver to recognize the situation around the vehicle displayed in the overhead image. An object of the present invention is to provide a vehicle surrounding image display device and a vehicle surrounding image display method that enable the driver to easily recognize the situation around the vehicle.

Means for Solving the Problems

[0005] To achieve the above objective, for example, the vehicle surrounding image display device of this embodiment includes: a position detection unit for detecting the position of a vehicle; a form selection unit for selecting one display form from a plurality of display forms as the display form of a map image of the area around the vehicle's position; a map image display unit for displaying the map image on a display in the one display form; an image acquisition unit for acquiring an image of the area around the vehicle; a determination unit for determining whether the distance from the vehicle to a specific facility is less than or equal to a first distance; and, if the determination unit determines that the distance is less than or equal to the first distance, a surrounding image display unit for displaying an image of the area around the vehicle on the display in an orientation corresponding to the map image in the one display form. The aforementioned specific facility is a facility that may come into contact with the vehicle as the vehicle passes over it. . [Effects of the Invention]

[0006] According to the vehicle surrounding image display device and vehicle surrounding image display method of the present invention, the image of the area around the vehicle is displayed in an orientation corresponding to the map image, so that the driver can easily recognize the situation around the vehicle. [Brief explanation of the drawing]

[0007] [Figure 1] This figure shows an example of the configuration of an in-vehicle device including a vehicle surrounding image display device. [Figure 2] This figure shows an example of a method for generating a 3D image using a surrounding image display unit. [Figure 3] This is a screen diagram showing an example of a map display screen that displays a map image in the first display mode. [Figure 4] This is a screen diagram showing an example of a surrounding display screen that displays a 3D image in an orientation corresponding to the map image in the first display mode. [Figure 5] This is a screen diagram showing an example of a surrounding display screen that displays an overhead view image in an orientation corresponding to the map image in the first display mode. [Figure 6] This is a screenshot showing an example of a map display screen that displays a map image in the second display mode. [Figure 7] This is a screen diagram showing an example of a surrounding display screen that displays a 3D image in an orientation corresponding to the map image in the second display mode. [Figure 8] This is a screen diagram showing an example of a surrounding display screen that displays an overhead view image oriented according to the map image in the second display mode. [Figure 9] A flowchart illustrating an example of processing performed by a vehicle surroundings image display device. [Modes for carrying out the invention]

[0008] Embodiments of the present invention will be described below with reference to the drawings. Figure 1 shows an example of the configuration of an in-vehicle device 3, including a vehicle surrounding image display device 100. The in-vehicle device 3 is mounted on a vehicle 1. The in-vehicle device 3 comprises a position detection unit 10, a detection unit 20, an operation unit 50, a display unit 60, and a vehicle surrounding image display device 100.

[0009] The position detection unit 10 detects the position of vehicle 1. The position detection unit 10 includes a GNSS receiver that receives GNSS (Global Navigation Satellite System) signals and a processor that calculates the position of vehicle 1 based on the GNSS signals received by the GNSS receiver. The GNSS receiver and processor are not shown in the diagram. The position detection unit 10 outputs position information indicating the position of vehicle 1 to the vehicle surrounding image display device 100.

[0010] The detection unit 20 captures images of the area around the vehicle 1 and detects the distance LD from the vehicle 1 to the toll booth FS. The toll booth FS corresponds to an example of a "specified facility". The detection unit 20 comprises an imaging unit 30 and a distance sensor 40.

[0011] In this embodiment, the case where the "specified facility" is a toll booth FS is described, but it is not limited to this. The specified facility can be any facility that may come into contact with vehicle 1 as it passes through. The specified facility may be, for example, a drive-through facility such as a fast-food restaurant. The specified facility may also include obstacles such as vehicles parked on the shoulder of the road on which vehicle 1 is traveling.

[0012] The imaging unit 30 captures images of the area around the vehicle 1. The imaging unit 30 includes a front camera 31 that captures the area in front of the vehicle 1, a rear camera 33 that captures the area behind the vehicle 1, a left side camera 35 that captures the area to the left of the vehicle 1, and a right side camera 37 that captures the area to the right of the vehicle 1. Each of these cameras includes an image sensor such as a CCD (Charge-Coupled Device) or CMOS (Complementary Metal-Oxide-Semiconductor), and a data processing circuit that generates captured images PC from the image sensor.

[0013] The shooting unit 30 adjusts the field of view of each of the four cameras (i.e., forward, rear, left, and right) so that it can capture a 360° range centered on the vehicle 1. The front camera 31, rear camera 33, left side camera 35, and right side camera 37 each capture the shooting range at a predetermined frame rate to generate captured image PC. The shooting unit 30 outputs the generated captured image PC to the vehicle surrounding image display device 100. The vehicle surrounding image display device 100 stores the input captured image PC in memory 140. Note that the front camera 31, rear camera 33, left side camera 35, and right side camera 37 may each consist of one camera or multiple cameras.

[0014] The distance sensor 40 detects the distance LD from the vehicle 1 to the toll booth FS. The distance sensor 40 is equipped with LiDAR (Light Detection and Ranging) at multiple locations on the vehicle body, such as the front, rear, left side, and right side, and acquires point cloud data using electromagnetic waves. Each point data that makes up the point cloud data indicates the distance LD between the vehicle 1 and an object (for example, the toll booth FS) that is within a predetermined distance range from the vehicle 1.

[0015] In this embodiment, a case where the distance sensor 40 is a LiDAR will be described, but the present invention is not limited thereto. The distance sensor 40 may be, for example, a radar or a sonar sensor. Further, in this embodiment, a case where LiDARs are arranged at a plurality of locations such as the front, rear, left side, and right side in the vehicle body constituting the vehicle 1 will be described, but the present invention is not limited thereto. The LiDAR may be arranged on the roof of the vehicle 1 to acquire point cloud data of the entire circumference (that is, the front, rear, left side, and right side).

[0016] In this embodiment, a case where the distance sensor 40 detects the distance LD from the vehicle 1 to the toll booth FS will be described, but the present invention is not limited thereto. The vehicle surrounding image display device 100 may detect the distance LD from the captured image PC captured by the imaging unit 30.

[0017] The operation unit 50 receives an operation of a user who has boarded the vehicle 1. The user is, for example, a driver. The operation unit 50 outputs an operation signal corresponding to the received operation to the vehicle surrounding image display device 100. The operations received by the operation unit 50 include, for example, an operation for instructing the start of image display processing and an operation for ending image display processing. The operation unit 50 includes, for example, an image display on switch (not shown) and an image display off switch (not shown). When the image display on switch is pressed, the vehicle surrounding image display device 100 receives an operation for instructing the start of image display processing. Further, when the image display off switch is pressed during the execution of the image display processing, the vehicle surrounding image display device 100 receives an operation for ending the image display processing.

[0018] Note that the "image display processing" is a process in which the vehicle surrounding image display device 100 displays, for example, a map image PG, a three-dimensional image PP, or an aerial view image PQ on the display 61. The "image display processing" will be described with reference to FIG. 1 for the functional configuration of the vehicle surrounding image display device 100, and will be further described with reference to FIGS. 2 to 8. Further, the map image PG, the three-dimensional image PP, and the aerial view image PQ will be described with reference to FIGS. 2 to 8.

[0019] The display unit 60 comprises a display 61 and a touch sensor 63. The display 61 can be a liquid crystal display, an organic EL display, or the like. Based on display data input from the vehicle surrounding image display device 100, the display unit 60 displays a map image PG, a 3D image PP, or an overhead image PQ on the display 61. The touch sensor 63 can be a resistive or capacitive type sensor. The touch sensor 63 is positioned on the display surface of the display 61. The display unit 60 detects the user's touch operation on the display 61 using the touch sensor 63 and generates a position signal indicating the position of the detected touch operation. In other words, the display 61 and the touch sensor 63 constitute a so-called "touch panel".

[0020] The vehicle surrounding image display device 100 is a computer equipped with a processor 130 such as a CPU (Central Processing Unit) and an MPU (Micro-Processing Unit), and memory 140 such as ROM (Read Only Memory) and RAM (Random Access Memory). In addition to these devices, the vehicle surrounding image display device 100 is equipped with storage devices such as an HDD (Hard Disk Drive) and an SSD (Solid State Drive), interface circuits for connecting sensors and peripheral devices, and an in-vehicle network communication circuit for communicating with other in-vehicle devices via an in-vehicle network. The vehicle surrounding image display device 100 realizes various functional configurations by having the processor 130 execute a control program PGM stored in the memory 140.

[0021] The vehicle surrounding image display device 100 includes, for example, a functional unit comprising a position detection unit 131, a form selection unit 132, a map image display unit 133, an image acquisition unit 134, a determination unit 135, an image selection unit 136, a surrounding image display unit 137, a map storage unit 141, and an image storage unit 142. Specifically, the processor 130 executes a control program PGM stored in the memory 140, thereby enabling the processor 130 to function as the position detection unit 131, the form selection unit 132, the map image display unit 133, the image acquisition unit 134, the determination unit 135, the image selection unit 136, and the surrounding image display unit 137. Furthermore, the processor 130 executes a control program PGM stored in the memory 140, enabling the memory 140 to function as the map storage unit 141 and the image storage unit 142.

[0022] The map storage unit 141 stores the map image PG. The map image PG is read out by the map image display unit 133.

[0023] The image storage unit 142 stores the captured image PC. The captured image PC is generated by the shooting unit 30. The image storage unit 142 also stores the 3D data of the vehicle 1 and the vehicle image QV. The 3D data of the vehicle 1 is read out by the surrounding image display unit 137 when generating the 3D image PP, and the vehicle image PV is generated from the 3D data of the vehicle 1. The vehicle image QV is read out by the surrounding image display unit 137 when generating the overhead image PQ. The vehicle image PV will be explained further with reference to Figure 4. The vehicle image QV will be explained further with reference to Figure 5.

[0024] The position detection unit 131 detects the position of the vehicle 1. For example, the position detection unit 131 detects the position of the vehicle 1 by acquiring position information indicating the position of the vehicle 1 from the position detection unit 10.

[0025] The mode selection unit 132 selects one display mode DFA from among multiple display modes DF as the display mode DF for the map image PG around the location of the vehicle 1. The mode selection unit 132 receives, for example, a user operation to the operation unit 50 and selects one display mode DFA from among multiple display modes DF according to the received operation. The multiple display modes DF include a first display mode DF1 and a second display mode DF2. The first display mode DF1 is a display mode DF that displays the map image PG on the display 61 such that the upward direction DU of the display 61 is the north direction of the map image PG. The second display mode DF2 is a display mode DF that displays the map image PG on the display 61 such that the upward direction DU of the display 61 is the direction of travel D1 of the vehicle 1 in the map image PG. The upward direction DU and the first display mode DF1 will be further explained with reference to Figure 3. The second display mode DF2 will be further explained with reference to Figure 6.

[0026] The map image display unit 133 displays the map image PG on the display 61 in one display mode DFA selected by the mode selection unit 132. The map image display unit 133 acquires location information indicating the position of vehicle 1 from the position detection unit 10, and reads the map image PG corresponding to the acquired location information from the map storage unit 141 to acquire the map image PG of the area around vehicle 1. The map image display unit 133 also displays the map image PG acquired from the map storage unit 141 on the display 61 in, for example, the first display mode DF1 or the second display mode DF2.

[0027] In this embodiment, the case in which the map image display unit 133 acquires a map image PG corresponding to location information from the map storage unit 141 to obtain a map image PG of the area around the vehicle 1 is described, but the embodiment is not limited to this. The vehicle surrounding image display device 100 may be connected to a server device in a communicative manner, and the map image display unit 133 may acquire a map image PG of the area around the vehicle 1 by receiving a map image PG corresponding to location information from the server device. In this case, the map storage unit 141 does not need to store the map image PG, so the configuration of the vehicle surrounding image display device 100 can be simplified.

[0028] The image acquisition unit 134 acquires images of the area around the vehicle 1. The image acquisition unit 134 acquires images of the area around the vehicle 1, for example, by acquiring captured images PC taken by each of the front camera 31, rear camera 33, left side camera 35, and right side camera 37 of the shooting unit 30.

[0029] The determination unit 135 determines whether the distance LD from vehicle 1 to toll booth FS is less than or equal to a first distance L1. The first distance L1 is, for example, 5m. The distance LD is detected, for example, by the detection unit 20. The determination unit 135 also determines whether the vehicle's speed is less than or equal to a threshold. The determination unit 135 obtains the vehicle's speed from the driving control ECU (not shown in the figure). The driving control ECU controls the vehicle's movement. The threshold is, for example, 10km / h. In the following description, the vehicle's speed may be referred to as vehicle speed.

[0030] The surrounding image display unit 137 displays an overhead image PQ on the display 61, oriented according to the map image PG of the display mode DFA selected by the mode selection unit 132. The surrounding image display unit 137 also displays an overhead image PP on the display 61, oriented according to the map image PG of the display mode DFA selected by the mode selection unit 132, with a virtual viewpoint set outside the vehicle 1. The virtual viewpoint corresponds to a virtual camera VC. The surrounding image display unit 137 generates the overhead image PQ and the 3D image PP by combining the captured images PC taken by the front camera 31, rear camera 33, left side camera 35, and right side camera 37 of the shooting unit 30. The virtual camera VC and the 3D image PP will be further explained with reference to Figure 2. For example, when the determination unit 135 determines that the vehicle 1's speed is below a threshold, the surrounding image display unit 137 displays the image selected by the image selection unit 136 on the display 61.

[0031] The image selection unit 136 selects either an overhead image PQ or a three-dimensional image PP in response to, for example, an operation from the user to the operation unit 50. The image selection unit 136 may also select either an overhead image PQ or a three-dimensional image PP depending on the positional relationship between the vehicle 1 and the toll booth FS. The image selection unit 136 may select the overhead image PQ if, for example, the distance LD is less than or equal to the second distance L2, which is shorter than the first distance L1. The image selection unit 136 may also select the three-dimensional image PP if, for example, the distance LD is longer than the second distance L2. In this configuration, if the distance LD is not less than or equal to the second distance L2, the image of the area around the vehicle 1 is displayed on the display 61 as a three-dimensional image PP with a virtual viewpoint set outside the vehicle 1. If the distance LD is less than or equal to the second distance L2, the image of the area around the vehicle 1 is displayed on the display 61 as an overhead image PQ. Therefore, by appropriately setting the second distance L2, it is possible to appropriately switch between displaying the image of the area around vehicle 1 as a 3D image PP or as an overhead image PQ.

[0032] Next, the method for generating a 3D image PP will be explained with reference to Figure 2. Figure 2 is a diagram showing an example of the method for generating a 3D image PP by the surrounding image display unit 137. As shown in Figure 2, the surrounding image display unit 137 places a hemispherical projection surface 7 in the virtual space. The projection surface 7 is set to have mutually orthogonal X, Y, and Z axes. A virtual vehicle VV is also placed on the bottom surface of the projection surface 7. The virtual vehicle VV is vehicle 1 in the virtual space. The virtual vehicle VV corresponds to the 3D data of vehicle 1 stored in the image storage unit 142. The Z axis is parallel to the vertical direction in the virtual space. The X and Y axes are parallel to the horizontal direction in the virtual space. The X axis is parallel to the long axis direction (longitudinal direction) of the virtual vehicle VV's body. The Y axis is parallel to the short axis direction (short direction) of the virtual vehicle VV's body. The positive direction of the Z axis is upward in the virtual space. The positive direction of the X-axis is the forward direction of the virtual vehicle VV's body, which is to the right in the virtual space. The positive direction of the Y-axis is the right direction of the virtual vehicle VV's body, which is downward in the virtual space.

[0033] The position of the virtual camera VC is set, for example, according to the positional relationship of vehicle 1 to the toll booth FS. The projection plane 7 is composed of a plurality of divided projection planes 72. The divided projection planes 72 include the first divided projection plane 721, the second divided projection plane 722, the third divided projection plane 723, the fourth divided projection plane 724, the fifth divided projection plane 725, and the sixth divided projection plane 726. The first to sixth divided projection planes 721 to 726 are positioned to the left of the virtual vehicle VV with respect to the central plane LC of the projection plane 7. The central plane LC contains the major axis of the virtual vehicle VV and is a plane parallel to the XZ plane.

[0034] Each of the first to sixth divided projection planes 721 to 726 is positioned at 30-degree intervals with respect to the central plane LC. For example, the first divided projection plane 721 corresponds to a range of 0 to 30 degrees with respect to the central plane LC. The second divided projection plane 722 corresponds to a range of 30 to 60 degrees with respect to the central plane LC. The third divided projection plane 723 corresponds to a range of 60 to 90 degrees with respect to the central plane LC. The fourth divided projection plane 724 corresponds to a range of 90 to 120 degrees with respect to the central plane LC. The fifth divided projection plane 725 corresponds to a range of 120 to 150 degrees with respect to the central plane LC. The sixth divided projection plane 726 corresponds to a range of 150 to 180 degrees with respect to the central plane LC.

[0035] On the first and second divided projection surfaces 721 and 722, for example, the left half of the image PC captured by the front camera 31 is placed. On the third and fourth divided projection surfaces 723 and 724, for example, the image PC captured by the left side camera 35 is placed. On the fifth and sixth divided projection surfaces 725 and 726, for example, the left half of the image PC captured by the rear camera 33 is placed. In this way, the captured image PC is placed on the first to sixth divided projection surfaces 721 to 726. When the captured image PC is placed on the divided projection surfaces 72, the captured image PC is deformed to match the shape of the divided projection surface 72. Similarly, on the six divided projection surfaces 72 located to the right of the virtual vehicle VV with respect to the central plane LC of the projection surface 7, the right half of the image PC captured by the front camera 31, the image PC captured by the right side camera 37, and the right half of the image PC captured by the rear camera 33 are placed.

[0036] In this way, with the captured images PC from the front camera 31, rear camera 33, left side camera 35, and right side camera 37 positioned on the projection surface 7, the surrounding image display unit 137 generates a 3D image PP by generating an image captured by the virtual camera VC.

[0037] Next, with reference to Figures 3 to 5, we will explain the map image PG in the first display mode DF1, the 3D image PP displayed in an orientation corresponding to the map image PG in the first display mode DF1, and the overhead image PQ displayed in an orientation corresponding to the map image PG in the first display mode DF1. Each of Figures 3 to 5 indicates a direction. In each of Figures 3 to 5, the north direction is the upward direction DU of the display 61, and the east direction is the rightward direction DR of the display 61.

[0038] First, with reference to Figure 3, the map image PG of the first display mode DF1 will be explained. Figure 3 is a screen diagram showing an example of a map display screen 800 that displays the map image PG of the first display mode DF1. The map display screen 800 is displayed on the display 61 by the map image display unit 133. The first display mode DF1 is a display mode DF in which the map image PG is displayed on the display 61 such that the upward direction DU of the display 61 is the north direction of the map image PG. The map display screen 800 displays highways RW, toll booths FS, and vehicle marks MV. On the map display screen 800, highways RW show an image of a highway, and toll booths FS show an image of a toll booth. On the map display screen 800, toll booths FS are displayed as rectangular marks.

[0039] A highway runway (RW) is a one-way road running from southwest to northeast. A highway runway consists of four lanes: RW1 (lane 1), RW2 (lane 2), RW3 (lane 3), and RW4 (lane 4). These lanes are arranged in this order, from northwest to southeast.

[0040] The vehicle mark MV indicates the position of vehicle 1. The vehicle mark MV is represented, for example, as an isosceles triangle, with the acute vertex indicating the front end of vehicle 1. The direction of travel D1 indicates the direction of travel of vehicle 1. The direction of travel D1 is northeast. That is, vehicle 1 is traveling in the northeast direction. The vehicle mark MV is displayed on the third lane RW3. That is, vehicle 1 is traveling on the third lane RW3.

[0041] The tollbooth FS extends in the width direction of the highway RW (from northwest to southeast). The tollbooth FS is positioned in the direction of travel of vehicle 1. Distance LD is the distance between vehicle 1 and the tollbooth FS.

[0042] Next, with reference to Figure 4, a three-dimensional image PP displayed in an orientation corresponding to the map image PG of the first display mode DF1 will be described. Figure 4 is a screen diagram showing an example of a surrounding display screen 810 that displays the three-dimensional image PP in an orientation corresponding to the map image PG of the first display mode DF1. The surrounding display screen 810 is displayed on the display 61 by the surrounding image display unit 137. The surrounding display screen 810 displays a vehicle image PV and a toll booth FS. The vehicle image PV is generated by the surrounding image display unit 137 from the three-dimensional data of vehicle 1 stored in the image storage unit 142. The three-dimensional image PP shown in Figure 4 is displayed in an orientation corresponding to the map image PG of the first display mode DF1. For example, the direction of travel D1 of vehicle 1 in the three-dimensional image PP coincides with the direction of travel D1 of vehicle 1 in the map image PG. The direction of travel D1 is northeast. Furthermore, the three-dimensional image PP shown in Figure 4 is an image captured by the virtual camera VC shown in Figure 2 when the virtual camera VC is positioned above and to the right of the virtual vehicle VV.

[0043] The tollbooth FS (Facility Stability) is equipped with four toll facilities: the first toll facility FS1, the second toll facility FS2, the third toll facility FS3, and the fourth toll facility FS4. The first toll facility FS1 is located between the first lane RW1 and the second lane RW2. The second toll facility FS2 is located between the second lane RW2 and the third lane RW3. The third toll facility FS3 is located between the third lane RW3 and the fourth lane RW4. The fourth toll facility FS4 is located on the southeast side of the fourth lane RW4. The first toll facility FS1, the second toll facility FS2, the third toll facility FS3, and the fourth toll facility FS4 are arranged in this order from northwest to southeast.

[0044] Vehicles traveling on the third lane (RW3) deposit their tolls into the third toll slot (CS3). The third toll slot (CS3) consists of the third toll slot (CS31) and the third toll slot (CS32). The third toll slot (CS32) is located on the third lane (RW3) side of the second toll facility (FS2). The third toll slot (CS31) is also located on the third lane (RW3) side of the third toll facility (FS3). For example, if a vehicle traveling on the third lane (RW3) is a left-hand drive vehicle, the toll is deposited into the third toll slot (CS31). For example, if a vehicle traveling on the third lane (RW3) is a right-hand drive vehicle, the toll is deposited into the third toll slot (CS32).

[0045] The second toll slot CS22 is located on the second lane RW2 side of the first toll facility FS1. The fourth toll slot CS42 is located on the fourth lane RW4 side of the third toll facility FS3. The fourth toll slot CS41 is located on the fourth lane RW4 side of the fourth toll facility FS4.

[0046] As shown in Figure 4, the third toll slot CS31 and the third toll slot CS32 are positioned to protrude toward the third lane RW3. Therefore, when vehicle 1 is traveling in the third lane RW3, the driver will be aware that they must take care not to come into contact with the third toll slot CS31 and the third toll slot CS32.

[0047] Furthermore, as shown in Figure 4, since the 3D image PP is displayed in an orientation corresponding to the map image PG of the first display mode DF1, the direction of travel D1 of the vehicle 1 in the map image PG and the direction of travel D1 of the vehicle 1 in the 3D image PP coincide. Therefore, by viewing the 3D image PP, the driver can easily recognize the situation around the vehicle 1.

[0048] Next, with reference to Figure 5, the overhead image PQ, which is displayed in an orientation corresponding to the map image PG of the first display mode DF1, will be described. Figure 5 is a screen diagram showing an example of a surrounding display screen 820 that displays the overhead image PQ in an orientation corresponding to the map image PG of the first display mode DF1. The surrounding display screen 820 is displayed on the display 61 by the surrounding image display unit 137. The surrounding display screen 820 displays the vehicle image QV, the second toll facility FS2, and the third toll facility FS3. The overhead image PQ shown in Figure 5 is displayed in an orientation corresponding to the map image PG of the first display mode DF1. For example, the direction of travel D1 of vehicle 1 in the overhead image PQ coincides with the direction of travel D1 of vehicle 1 in the map image PG.

[0049] As shown in Figure 5, the third toll slot CS32 is located on the third lane RW3 side of the second toll facility FS2. The third toll slot CS31 is also located on the third lane RW3 side of the third toll facility FS3.

[0050] As shown in Figure 5, the third toll slot CS31 and the third toll slot CS32 are positioned to protrude toward the third lane RW3. Therefore, when vehicle 1 is traveling in the third lane RW3, the driver will be aware that they must take care not to come into contact with the third toll slot CS31 and the third toll slot CS32.

[0051] Furthermore, as shown in Figure 5, since the overhead view image PQ is displayed in an orientation corresponding to the map image PG of the first display mode DF1, the direction of travel D1 of vehicle 1 in the map image PG and the direction of travel D1 of vehicle 1 in the overhead view image PQ coincide. Therefore, by viewing the overhead view image PQ, the driver can easily recognize the situation around vehicle 1.

[0052] Next, with reference to Figures 6 to 8, we will explain the map image PG in the second display mode DF2, the 3D image PP displayed in an orientation corresponding to the map image PG in the second display mode DF2, and the overhead image PQ displayed in an orientation corresponding to the map image PG in the second display mode DF2. Each of Figures 6 to 8 indicates a direction. North is the upper left direction of the display 61, and east is the upper right direction of the display 61.

[0053] First, the map image PG for the second display mode DF2 will be explained with reference to Figure 6. Figure 6 is a screen diagram showing an example of a map display screen 900 displaying the map image PG for the second display mode DF2. The second display mode DF2 is a display mode DF in which the map image PG is displayed on the display 61 so that the upward direction DU of the display 61 is the direction of travel D1 of the vehicle 1 in the map image PG. The map display screen 900 displays highways RW, toll booths FS, and vehicle marks MV. On the map display screen 900, highways RW show an image of a highway, and toll booths FS show an image of a toll booth. On the map display screen 900, toll booths FS are displayed as rectangular marks.

[0054] A highway runway (RW) is a one-way road running from southwest to northeast. A highway runway consists of four lanes: RW1 (lane 1), RW2 (lane 2), RW3 (lane 3), and RW4 (lane 4). These lanes are arranged in this order, from northwest to southeast.

[0055] The vehicle mark MV indicates the position of vehicle 1. The vehicle mark MV is represented, for example, as an isosceles triangle, with the acute vertex indicating the front end of vehicle 1. The direction of travel D1 indicates the direction of travel of vehicle 1. The direction of travel D1 is northeast. That is, vehicle 1 is traveling in the northeast direction. The vehicle mark MV is displayed on the third lane RW3. That is, vehicle 1 is traveling on the third lane RW3.

[0056] The tollbooth FS extends in the width direction of the highway RW (from northwest to southeast). The tollbooth FS is positioned in the direction of travel of vehicle 1. Distance LD is the distance between vehicle 1 and the tollbooth FS.

[0057] In the second display mode DF2 shown in Figure 6, the map image PG is displayed so that the upward direction DU of the display 61 is parallel to the direction of travel D1 of the vehicle 1.

[0058] Next, with reference to Figure 7, the three-dimensional image PP displayed in an orientation corresponding to the map image PG of the second display mode DF2 will be described. Figure 7 is a screen diagram showing an example of a surrounding display screen 910 that displays the three-dimensional image PP in an orientation corresponding to the map image PG of the second display mode DF2. The surrounding display screen 910 is displayed on the display 61 by the surrounding image display unit 137. The surrounding display screen 910 displays the vehicle image PV and the toll booth FS. The vehicle image PV is generated by the surrounding image display unit 137 from the three-dimensional data of the vehicle 1 stored in the image storage unit 142. The three-dimensional image PP shown in Figure 7 is displayed in an orientation corresponding to the map image PG of the second display mode DF2. For example, the direction of travel D1 of the vehicle 1 in the three-dimensional image PP coincides with the direction of travel D1 of the vehicle 1 in the map image PG. The direction of travel D1 is northeast. Furthermore, the three-dimensional image PP shown in Figure 7 is an image captured by the virtual camera VC shown in Figure 2 when the virtual camera VC is positioned above and to the right of the virtual vehicle VV.

[0059] The tollbooth FS consists of the first toll facility FS1, the second toll facility FS2, the third toll facility FS3, and the fourth toll facility FS4. The first toll facility FS1, the second toll facility FS2, the third toll facility FS3, and the fourth toll facility FS4 are arranged in this order from northwest to southeast. Vehicles traveling on the third lane RW3 deposit their tolls into the third toll slot CS3. The third toll slot CS3 consists of the third toll slot CS31 and the third toll slot CS32.

[0060] As shown in Figure 7, the third toll slot CS31 and the third toll slot CS32 are positioned to protrude toward the third lane RW3. Therefore, when vehicle 1 is traveling in the third lane RW3, the driver will be aware that they must take care not to come into contact with the third toll slot CS31 and the third toll slot CS32.

[0061] Furthermore, as shown in Figure 7, since the 3D image PP is displayed in an orientation corresponding to the map image PG of the first display mode DF1, the direction of travel D1 of the vehicle 1 in the map image PG and the direction of travel D1 of the vehicle 1 in the 3D image PP coincide. Therefore, by viewing the 3D image PP, the driver can easily recognize the situation around the vehicle 1.

[0062] Next, with reference to Figure 8, the overhead image PQ, which is displayed in an orientation corresponding to the map image PG of the second display mode DF2, will be described. Figure 8 is a screen diagram showing an example of a surrounding display screen 920 that displays the overhead image PQ in an orientation corresponding to the map image PG of the second display mode DF2. The surrounding display screen 920 is displayed on the display 61 by the surrounding image display unit 137. The surrounding display screen 920 displays the vehicle image QV, the second toll facility FS2, and the third toll facility FS3. The overhead image PQ shown in Figure 8 is displayed in an orientation corresponding to the map image PG of the second display mode DF2. For example, the direction of travel D1 of vehicle 1 in the overhead image PQ coincides with the direction of travel D1 of vehicle 1 in the map image PG. That is, the direction of travel D1 of vehicle 1 in the overhead image PQ is parallel to the upward direction DU of the display 61.

[0063] As shown in Figure 8, the third toll slot CS32 is located on the third lane RW3 side of the second toll facility FS2. The third toll slot CS31 is also located on the third lane RW3 side of the third toll facility FS3.

[0064] As shown in Figure 8, the third toll slot CS31 and the third toll slot CS32 are positioned to protrude toward the third lane RW3. Therefore, when vehicle 1 is traveling in the third lane RW3, the driver will be aware that they must take care not to come into contact with the third toll slot CS31 and the third toll slot CS32.

[0065] Furthermore, as shown in Figure 8, since the overhead image PQ is displayed in an orientation corresponding to the map image PG of the second display mode DF2, the direction of travel D1 of vehicle 1 in the map image PG and the direction of travel D1 of vehicle 1 in the overhead image PQ coincide. Therefore, by viewing the overhead image PQ, the driver can easily recognize the situation around vehicle 1.

[0066] Next, the processing of the vehicle surrounding image display device 100 will be described with reference to Figure 9. Figure 9 is a flowchart showing an example of the processing of the vehicle surrounding image display device 100. In Figure 8, the case in which the image selection unit 136 has already selected, for example, an overhead image PQ or a 3D image PP in response to an operation from the user to the operation unit 50 will be described. First, in step S101, the position detection unit 131 detects the position of the vehicle 1 by acquiring position information indicating the position of the vehicle 1 from the position detection unit 10. Next, in step S103, the form selection unit 132 receives an operation from the user to the operation unit 50 and selects one display form from among a plurality of display forms DF in response to the received operation. The plurality of display forms DF include a first display form DF1 and a second display form DF2.

[0067] Next, in step S105, the map image display unit 133 displays the map image PG on the display 61 in one display mode DFA selected in step S103. The map image display unit 133 displays the map image PG acquired from the map storage unit 141 on the display 61 in, for example, the first display mode DF1 or the second display mode DF2. Next, in step S107, the image acquisition unit 134 acquires an image of the area around the vehicle 1. The image acquisition unit 134 acquires an image of the area around the vehicle 1 by, for example, acquiring captured images PC taken by the front camera 31, rear camera 33, left side camera 35, and right side camera 37 of the shooting unit 30.

[0068] Next, in step S109, the determination unit 135 detects the distance LD from vehicle 1 to toll booth FS. The determination unit 135 obtains the distance LD from, for example, the distance sensor 40. Next, in step S111, the determination unit 135 determines whether the distance LD is less than or equal to the first distance L1. The first distance L1 is, for example, 5m. If the determination unit 135 determines that the distance LD is not less than or equal to the first distance L1 (step S109; NO), the process proceeds to step S121. If the determination unit 135 determines that the distance LD is less than or equal to the first distance L1 (step S109; YES), the process proceeds to step S113.

[0069] Then, in step S113, the determination unit 135 determines whether the vehicle speed 1 is below a threshold. The threshold is, for example, 10 km / h. If the determination unit 135 determines that the vehicle speed 1 is not below the threshold (step S113; NO), the process enters a waiting state. If the determination unit 135 determines that the vehicle speed 1 is below the threshold (step S113; YES), the process proceeds to step S115. Then, in step S115, the determination unit 135 determines whether or not to display the overhead image PQ.

[0070] If the determination unit 135 determines that the overhead image PQ should not be displayed (step S115; NO), the process proceeds to step S117. In step S117, the surrounding image display unit 137 displays the image of the area around the vehicle 1 as a 3D image PP on the display 61, oriented according to the map image PG of the display mode DFA selected in step S103. The process then proceeds to step S121. If the determination unit 135 determines that the overhead image PQ should be displayed (step S115; YES), the process proceeds to step S119. In step S119, the surrounding image display unit 137 displays the image of the area around the vehicle 1 as an overhead image PQ on the display 61, oriented according to the map image PG of the display mode selected in step S103. The process then proceeds to step S121.

[0071] Next, in step S121, the vehicle surrounding image display device 100 determines whether or not to terminate the image display process in response to the user operation received by the operation unit 50. If the vehicle surrounding image display device 100 determines that the image display process should not be terminated (step S121; NO), the process returns to step S101. If the vehicle surrounding image display device 100 determines that the image display process should be terminated (step S121; YES), the process is then terminated.

[0072] Step S101 corresponds to an example of a "position detection step". Step S103 corresponds to an example of a "form selection step". Step S105 corresponds to an example of a "map image display step". Step S107 corresponds to an example of an "image acquisition step". Step S111 corresponds to an example of a "determination step". Steps S117 and S119 correspond to an example of a "surrounding image display step".

[0073] As explained with reference to Figure 9, when the vehicle 1's speed is below a threshold, the system switches from the map image PG to a 3D image PP showing the area around vehicle 1, or to an overhead image PQ. Therefore, by appropriately setting the threshold, the system can appropriately switch from the map image PG to a 3D image PP showing the area around vehicle 1, or to an overhead image PQ.

[0074] As described above with reference to Figures 1 to 9, the vehicle surrounding image display device 100 according to this embodiment includes: a position detection unit 131 for detecting the position of the vehicle 1; a form selection unit 132 for selecting one display form DFA from among a plurality of display forms DF as the display form DF for the map image PG of the area around the vehicle 1's position; a map image display unit 133 for displaying the map image PG on the display 61 in one display form DFA; an image acquisition unit 134 for acquiring an image of the area around the vehicle 1; a determination unit 135 for determining whether the distance LD from the vehicle 1 to the toll booth FS is less than or equal to a first distance L1; and a surrounding image display unit 137 for displaying an image of the area around the vehicle 1 on the display 61 in an orientation corresponding to the map image PG of one display form DFA when the determination unit 135 determines that the distance LD is less than or equal to the first distance L1.

[0075] Therefore, when the determination unit 135 determines that the distance LD is less than or equal to the first distance L1, the surrounding image display unit 137 displays an image of the area around the vehicle 1 on the display 61 in an orientation corresponding to the map image PG of one display mode DFA. Thus, since the image of the area around the vehicle 1 is displayed in an orientation corresponding to the map image PG, the driver can easily recognize the situation around the vehicle 1. In addition, since the image of the area around the vehicle 1 is displayed when the distance LD is less than or equal to the first distance L1, by setting the first distance L1 appropriately, the image of the area around the vehicle 1 can be displayed at the appropriate timing.

[0076] Furthermore, the surrounding image display unit 137 displays an image of the area around the vehicle 1 as an overhead view image PQ on the display 61. Therefore, because the image of the area around the vehicle 1 is displayed as an overhead view image PQ, the driver can easily recognize the situation around the vehicle 1.

[0077] Furthermore, the surrounding image display unit 137 displays the image of the area around the vehicle 1 on the display 61 as a 3D image PP with a virtual viewpoint set outside the vehicle 1. Therefore, because the image of the area around the vehicle 1 is displayed as a 3D image PP, the driver can easily recognize the situation around the vehicle 1.

[0078] Furthermore, the system includes an image selection unit 136 that selects either an overhead image PQ or a three-dimensional image PP depending on the positional relationship of the vehicle 1 to the toll booth FS, and the surrounding image display unit 137 displays the image selected by the image selection unit 136 on the display 61. Therefore, depending on the positional relationship between vehicle 1 and the toll booth FS, either an overhead view image PQ or a 3D image PP is selected and the selected image is displayed. Thus, the overhead view image PQ or the 3D image PP can be appropriately selected and displayed. Consequently, the driver can easily recognize the surrounding conditions of vehicle 1.

[0079] Furthermore, the multiple display modes DF include a first display mode DF1 in which the map image PG is displayed on the display 61 such that the upward direction DU of the display 61 is the north direction of the map image PG, and a second display mode DF2 in which the map image PG is displayed on the display 61 such that the upward direction DU of the display 61 is the direction of travel D1 of the vehicle 1 in the map image PG. Therefore, by selecting the first display mode DF1, the driver can display the map image PG on the display 61 such that the upward direction DU of the display 61 is the north direction of the map image PG. Also, by selecting the second display mode DF2, the driver can display the map image PG on the display 61 such that the upward direction DU of the display 61 is the direction of travel D1 of the vehicle 1 in the map image PG. Therefore, driver convenience can be improved.

[0080] Furthermore, the vehicle surrounding image display method according to this embodiment includes a position detection step for detecting the position of the vehicle 1; a form selection step for selecting one display form DFA from among a plurality of display forms DF as the display form DF for the map image PG of the area around the position of the vehicle 1; a map image display step for displaying the map image PG on the display 61 in one display form DFA; an image acquisition step for acquiring an image of the area around the vehicle 1; a determination step for determining whether the distance LD from the vehicle 1 to the toll booth FS is less than or equal to a first distance L1; and, if the determination step determines that the distance LD is less than or equal to the first distance L1, a surrounding image display step for displaying an image of the area around the vehicle 1 on the display 61 in an orientation corresponding to the map image PG of one display form DFA. Therefore, the vehicle surrounding information display method according to this embodiment has the same effects as the vehicle surrounding image display device 100 according to this embodiment.

[0081] The embodiment described above is merely an example of one embodiment of the present invention, and can be modified and applied as desired without departing from the spirit of the present invention.

[0082] For example, Figure 1 is a diagram that classifies the components according to their main processing content in order to facilitate understanding of the present invention. The components can be further classified into many more components according to their processing content. Alternatively, a single component can be classified to perform even more processing. Furthermore, the processing of each component may be executed on one piece of hardware or on multiple pieces of hardware. Also, the processing of each component may be implemented by one program or by multiple programs.

[0083] Furthermore, in Figure 1, the vehicle surrounding image display device 100 may also include a display unit 50 as an integral part of the device.

[0084] Furthermore, in this embodiment, the vehicle surrounding image display device 100 includes a position detection unit 131, a shape selection unit 132, a map image display unit 133, an image acquisition unit 134, a determination unit 135, an image selection unit 136, and a surrounding image display unit 137, but is not limited to this. A server device that is communicably connected to the vehicle surrounding image display device 100 via a network such as the Internet may include at least one of the position detection unit 131, the shape selection unit 132, the map image display unit 133, the image acquisition unit 134, the determination unit 135, the image selection unit 136, and the surrounding image display unit 137. The server device may, for example, include a surrounding image display unit 137. In this case, the surrounding image display unit 137 may acquire an overhead image PQ captured by an artificial satellite.

[0085] Furthermore, when the vehicle surrounding information display method of the present invention is implemented using a computer, the control program PGM executed by the computer may be configured as a recording medium or a transmission medium for transmitting the control program PGM. Magnetic, optical, or semiconductor memory devices can be used as the recording medium. Specifically, portable or fixed recording media such as flexible disks, HDDs, CD-ROMs (Compact Disk Read Only Memory), DVDs, Blu-ray® Discs, magneto-optical disks, flash memory, and card-type recording media can be used. Alternatively, the recording medium may be a non-volatile storage device such as RAM, ROM, or HDD provided by the vehicle surrounding image display device 100. In addition, the control program PGM may be downloaded by the vehicle surrounding image display device 100 from a server device that is communicated with the vehicle surrounding image display device 100 via a network.

[0086] Furthermore, for example, the processing units in the flowchart shown in Figure 9 are divided according to the main processing content in order to facilitate understanding of the processing of the vehicle surrounding image display device 100, and the present invention is not limited by the way the processing units are divided or the names of the processing units. The processing of the vehicle surrounding image display device 100 may be further divided into more processing units depending on the processing content. Also, the processing of the vehicle surrounding image display device 100 may be divided so that one processing unit includes even more processing. [Explanation of symbols]

[0087] 100 Vehicle surrounding image display device 1 vehicle 10 Position detection unit 20 Detection unit 30 Photography Department 40 Distance Sensor 60 Display section 61 displays 130 processors 140 memory

Claims

1. A position detection unit that detects the position of the vehicle, A form selection unit selects one display form from among multiple display forms as the display form of the map image of the area around the vehicle's location. A map image display unit that displays the aforementioned map image on a display in the aforementioned single display mode, An image acquisition unit that acquires images of the area around the vehicle, A determination unit that determines whether the distance from the vehicle to the specific facility is less than or equal to a first distance, The system includes an surrounding image display unit that, when the determination unit determines that the distance is less than or equal to the first distance, displays an image of the area around the vehicle on the display in an orientation corresponding to the map image of one of the display modes, The aforementioned specific facility is a facility that may come into contact with the vehicle as the vehicle passes over it. The surrounding image display unit displays images of the vehicle's surroundings on the display as an overhead view or a three-dimensional image with a virtual viewpoint set outside the vehicle. The system includes an image selection unit that selects the overhead image when the distance from the vehicle to the specific facility is less than or equal to a second distance which is shorter than the first distance, and selects the three-dimensional image when the distance is longer than the second distance. The surrounding image display unit displays the image selected by the image selection unit on the display. Vehicle surroundings image display device.

2. The aforementioned multiple display formats are, A first display mode in which the map image is displayed on the display such that the top of the display is the north direction of the map image, A second display mode in which the map image is displayed on the display such that the upward direction of the display is the direction of travel of the vehicle in the map image, including, The vehicle surrounding image display device according to claim 1.

3. A position detection step for detecting the vehicle's position, A mode selection step involves selecting one display mode from among several display modes as the display mode for the map image of the area around the vehicle's location. A map image display step in which the map image is displayed on a display in the aforementioned single display mode, Image acquisition step of acquiring an image of the area around the vehicle, A determination step to determine whether the distance from the vehicle to the specific facility is less than or equal to a first distance, If the determination step determines that the distance is less than or equal to the first distance, the surrounding image display step includes displaying an image of the area around the vehicle on the display in an orientation corresponding to the map image of one display mode, The aforementioned specific facility is a facility that may come into contact with the vehicle as the vehicle passes over it. The surrounding image display step includes, when the distance from the vehicle to the specific facility is less than or equal to the second distance which is shorter than the first distance, displaying an overhead image of the area around the vehicle on the display; and when the distance from the vehicle to the specific facility is longer than the second distance, displaying an image of the area around the vehicle on the display as a three-dimensional image with a virtual viewpoint set outside the vehicle. Method for displaying images of the area around a vehicle.

Images