Control method and control device

Abstract

To generate an underfloor image with little deviation from an actual underfloor situation of a vehicle.SOLUTION: A vehicle includes: a first camera that transmits a first captured image obtained by capturing an image of a first region outside the vehicle; a second camera that transmits a second captured image obtained by capturing an image of a second region outside the vehicle; a control device that generates an underfloor image in which an underfloor of the vehicle is captured using the second captured image received at a predetermined timing; and a display device that displays the underfloor image. The control device sets an overlapping region overlapping both the first region and the second region, calculates a delay time of the second camera with respect to the first camera based on a temporal change in image characteristics in the overlapping region of the first captured image and a temporal change in image characteristics in the overlapping region of the second captured image, and generates the underfloor image based on the second captured image and the delay time when the vehicle is moving in the direction of the second region.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present disclosure relates to a control method and a control device. [Background technology]

[0002] BACKGROUND ART There is known a technique for assisting driving by displaying an image of a partial area under the floor of a vehicle such as an automobile (hereinafter referred to as an underfloor image) on a display device.

[0003] Patent Document 1 discloses that an underfloor image is generated using images captured by a plurality of cameras mounted on a vehicle. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-197785 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the images output by multiple cameras are not necessarily taken at the same time, and in such cases, if an underfloor image is generated using the captured images using conventional methods, there is a possibility that the generated underfloor image will differ from the actual underfloor condition of the vehicle.

[0006] An object of the present disclosure is to provide a technology for generating an underfloor image that is less likely to deviate from the actual underfloor condition of a vehicle. [Means for solving the problem]

[0007] One aspect of the present disclosure is a method for controlling a vehicle that has a plurality of wheels and can move in a predetermined direction using the wheels, the vehicle comprising: a first camera that transmits a first captured image of a first area outside the vehicle; a second camera that transmits a second captured image of a second area outside the vehicle; a control device that generates an underfloor image of an underfloor of the vehicle using the second captured image received at a predetermined timing; and a display device that displays the underfloor image, wherein the first captured image received at the predetermined timing was captured at a second time, and the second captured image received by the control device is one that was captured at a first time before the second time, and the control device sets an overlapping area that overlaps both the first area and the second area, calculates a delay time of the second camera relative to the first camera based on a change over time in image characteristics in the overlapping area of ​​the first captured image and a change over time in image characteristics in the overlapping area of ​​the second captured image, and generates the underfloor image based on the second captured image and the delay time when the vehicle is moving in the direction of the second area.

[0008] One aspect of the present disclosure is a control device provided in a vehicle having a plurality of wheels and capable of moving in a predetermined direction using the wheels, the vehicle including a first camera that transmits a first captured image of a first area outside the vehicle, a second camera that transmits a second captured image of a second area outside the vehicle, a control device that generates an underfloor image of an underfloor of the vehicle using the second captured image received at a predetermined time, and a display device that displays the underfloor image, wherein the first captured image received at the predetermined time was captured at a second time, and the second captured image is transmitted to a display device that displays the underfloor image. The second captured image received by the camera is captured at a first time before the second time, and the control device sets an overlapping area that overlaps both the first area and the second area, calculates a delay time of the second camera relative to the first camera based on a change in image characteristics over time in the overlapping area of ​​the first captured image and a change in image characteristics over time in the overlapping area of ​​the second captured image, and generates the underfloor image based on the second captured image and the delay time when the vehicle is moving in the direction of the second area.

[0009] These comprehensive or specific aspects may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a recording medium, or may be realized as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium. [Effects of the Invention]

[0010] According to the present disclosure, it is possible to generate an underfloor image that is less likely to deviate from the actual underfloor condition of a vehicle. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a schematic diagram illustrating a configuration example of a vehicle according to a first embodiment; [Figure 2] FIG. 1 is a block diagram showing an example of a device provided in a vehicle according to a first embodiment; [Figure 3]FIG. 1 is a diagram for explaining a method for generating an underfloor image when the vehicle according to the first embodiment is moving forward. [Figure 4] Diagram to explain the issues with underfloor images when the vehicle is reversing [Figure 5] FIG. 10 is a diagram for explaining a method for generating an underfloor image when the vehicle is moving backward according to the first embodiment; [Figure 6] 1 is a flowchart showing an example of processing performed by a control device according to the first embodiment; [Figure 7] FIG. 10 is a diagram for explaining an overlapping region according to the second embodiment. [Figure 8] Flowchart showing an example of delay time calculation processing according to the second embodiment [Figure 9] FIG. 10 is a diagram showing an example of luminance changes in two overlapping regions according to the second embodiment; DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments of the present disclosure will be described in detail with appropriate reference to the drawings. However, more detailed description than necessary may be omitted. For example, detailed descriptions of well-known matters and redundant descriptions of substantially identical configurations may be omitted. This is to avoid unnecessary redundancy in the following description and to facilitate understanding by those skilled in the art. Note that the accompanying drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims.

[0013] (Embodiment 1) <Vehicle configuration> Fig. 1 is a schematic diagram showing an example of the configuration of a vehicle 1 according to embodiment 1. Fig. 2 is a block diagram showing an example of a device provided in the vehicle 1 according to embodiment 1. Next, the configuration of the vehicle 1 will be described with reference to Figs. 1 and 2.

[0014] The vehicle 1 has front wheels 2A and rear wheels 2B and is capable of moving in a predetermined direction. In Fig. 1, the vehicle 1 has four wheels, but the vehicle 1 may have any number of wheels.

[0015] The vehicle 1 is equipped with a front camera 11, a rear camera 12, a left camera 13, a right camera 14, a plurality of sensors 15, a control device 16, a memory device 17, an input device 18, and a display device 19.

[0016] The front camera 11, rear camera 12, left camera 13, right camera 14, sensor 15, control device 16, input device 18, and display device 19 can transmit and receive information to and from each other via an in-vehicle network. Examples of the in-vehicle network connecting the sensor 15, control device 16, input device 18, and display device 19 include a Controller Area Network (CAN), a Local Interconnect Network (LIN), and FlexRay. Examples of the in-vehicle network connecting the front camera 11, rear camera 12, left camera 13, right camera 14, and control device 16 include a Gigabit Video Interface (GVIF), a PFD-LINK, and a Gigabit Multimedia Serial Link (GMSL). The storage device 17 may be included in the control device 16.

[0017] The front camera 11 is installed in front of the vehicle 1 (for example, on the front grill), and periodically captures an image of an area including the ground in front of the vehicle 1 (hereinafter referred to as the front area 21), generating a front captured image 31. The front camera 11 transmits the generated front captured image 31 to the control device 16.

[0018] The rear camera 12 is installed at the rear of the vehicle 1 (for example, below the rear trunk door), and periodically captures an image of an area including the ground behind the vehicle 1 (hereinafter referred to as the rear area 22) to generate a rear captured image 32. The rear camera 12 transmits the generated rear captured image 32 to the control device 16.

[0019] The left camera 13 is installed on the left side of the vehicle 1 (for example, on the left door mirror), and periodically captures an image of an area including the ground on the left side of the vehicle 1 (hereinafter referred to as the left area 23), generating a left side captured image 33. The left side camera 13 transmits the generated left side captured image 33 to the control device 16.

[0020] The right camera 14 is installed on the right side of the vehicle 1 (for example, on the right door mirror), and periodically captures an image of an area including the ground on the right side of the vehicle 1 (hereinafter referred to as the right area 24), generating a right side captured image 34. The right camera 14 transmits the generated left side captured image 33 to the control device 16.

[0021] The predetermined periods at which the front camera 11, rear camera 12, left camera 13, and right camera 14 take images may be synchronized. For example, the front camera 11, rear camera 12, left camera 13, and right camera 14 may take images at approximately the same time.

[0022] The sensor 15 senses the behavior of the vehicle 1 and transmits information indicating the sensing results to the control device 16. Examples of the sensor 15 include a shift sensor that detects the shift position, an accelerator sensor that detects the accelerator opening, a brake sensor that detects the amount of brake depression, a vehicle speed sensor that detects the vehicle speed, an acceleration sensor that detects acceleration, and a steering angle sensor that detects the angle of the front wheels 2A.

[0023] The input device 18 accepts input operations from the driver or passengers of the vehicle 1. The display device 19 displays images, information, etc. generated by the control device 16. The input device 18 and the display device 19 may be an integrated touch panel display. The input device 18 may include buttons, dials, etc. A plurality of display devices 19 may be installed in the vehicle. The input device 18 and the display device 19 may be included in, for example, a car navigation system and an In-Vehicle Infotainment (IVI) system.

[0024] The control device 16 controls various behaviors of the vehicle 1. The control device 16 may be interpreted as an Electronic Control Unit (ECU), a processor, a Central Processing Unit (CPU), a controller, or the like. The control device 16 has a storage device 17. The storage device 17 may be configured with a volatile storage medium (e.g., RAM) and / or a non-volatile storage medium (e.g., ROM, flash memory). The control device 16 may implement the functions described in this embodiment by executing a computer program in cooperation with the storage device 17.

[0025] In this embodiment, the control device 16 generates an underfloor image 50 using the captured images (31, 32, 33, 34) received from each camera (11, 12, 13, 14), and displays it on the display device 19. The underfloor image 50 is an image in which at least a portion of the ground surface under the floor of the vehicle 1 is captured. The underfloor image 50 may also capture at least a portion of the front wheels 2A and / or rear wheels 2B of the vehicle 1.

[0026] When the vehicle 1 is moving forward, the control device 16 stores the received front photographed image 31, left photographed image 33, and right photographed image 34 in the storage device 17, and generates an underfloor image 50 by transforming and combining these photographed images. A method for generating the underfloor image 50 when the vehicle 1 is moving forward will be described later (see FIG. 3).

[0027] When the vehicle 1 is reversing, the control device 16 stores the received rear photographed image 32, left photographed image 33, and right photographed image 34 in the storage device 17, and generates an underfloor image 50 by transforming and combining these photographed images. A method for generating the underfloor image 50 when reversing will be described later (see FIG. 5).

[0028] By displaying the underfloor image 50 on the display device 19, the control device 16 can inform the driver of the condition of the ground under the floor of the vehicle 1, and the positional relationship between the front wheels 2A (or rear wheels 2B) and obstacles (e.g., bollards).

[0029] In addition, the control device 16 can display the received front photographed image 31, rear photographed image 32, left photographed image 33, and right photographed image 34 on the display device 19, thereby conveying to the driver the condition of the ground around the vehicle 1 and the positional relationship between the vehicle 1 and surrounding obstacles.

[0030] 1, the control device 16 may display the generated underfloor image 50 and the received front photographed image 31, rear photographed image 32, left photographed image 33, and right photographed image 34 on the display device 19 in accordance with their respective positional relationships. This allows the driver to easily grasp the conditions of the ground under the floor of the vehicle 1, the conditions of the surrounding ground, and the positional relationships of obstacles.

[0031] <Method for generating underfloor images when moving forward> 3 is a diagram for explaining a method for generating an underfloor image 50 when the vehicle 1 according to the embodiment 1 is moving forward. Next, an example of a method for generating an underfloor image 50 when the vehicle 1 is moving forward will be described with reference to FIG.

[0032] Immediately after time t=1, the front camera 11, rear camera 12, left camera 13, and right camera 14 transmit to the control device 16 a front photographed image 31A, a rear photographed image 32A, a left photographed image 33A, and a right photographed image 34A, respectively, captured at time t=1. The front photographed image 31A includes at least an image of the ground area 5A. The control device 16 stores the received front photographed image 31A in the storage device 17.

[0033] Next, it is assumed that the vehicle 1 moves forward by a distance d1 between time t = 1 and time t = 2. Note that the distance may be interpreted as the amount of movement.

[0034] Immediately after time t=2, the front camera 11, rear camera 12, left camera 13, and right camera 14 transmit to the control device 16 the front photographed image 31B, rear photographed image 32B, left photographed image 33B, and right photographed image 34B, respectively, that were captured at time t=2. The front photographed image 31B includes at least an image of the ground area 5B. The control device 16 stores the received front photographed image 31B in the storage device 17.

[0035] The control device 16 generates an underfloor image 50B at time t=2 by shifting the forward captured image 31A received immediately after time t=1, which is stored in the storage device 17, by a distance d1 toward the rear of the vehicle 1, and stores the generated underfloor image 50B in the storage device 17. The underfloor image 50B includes an image of the ground area 5A located under the floor of the vehicle 1 at time t=2.

[0036] The control device 16 displays the front photographed image 31B, rear photographed image 32B, left photographed image 33B, and right photographed image 34B received immediately after time t=2, together with the underfloor image 50B generated above, on the display device 19. This allows the control device 16 to display an image showing the underfloor conditions of the vehicle 1 and the surrounding conditions at time t=2.

[0037] Next, it is assumed that the vehicle 1 moves forward by a distance d2 between time t=2 and time t=3.

[0038] Immediately after time t=3, the front camera 11, rear camera 12, left camera 13, and right camera 14 transmit to the control device 16 a front photographed image 31C, a rear photographed image 32C, a left photographed image 33C, and a right photographed image 34C, respectively, captured at time t=3. The front photographed image 31C includes at least an image of the ground area 5C. The control device 16 stores the received front photographed image 31C in the storage device 17.

[0039] The control device 16 shifts the underfloor image 50B at time t=2, which is stored in the storage device 17, by a distance d2 toward the rear of the vehicle 1. Furthermore, the control device 16 shifts the front captured image 31B, which is stored in the storage device 17 and received immediately after time t=2, by a distance d2 toward the rear of the vehicle 1 and combines it with the previously shifted underfloor image 50B to generate an underfloor image 50C at time t=3. The control device 16 stores the generated underfloor image 50C in the storage device 17. The underfloor image 50C includes images of the ground areas 5A and 5B located under the floor of the vehicle 1 at time t=3.

[0040] The control device 16 displays the front photographed image 31C, rear photographed image 32C, left photographed image 33C, and right photographed image 34C received immediately after time t=3, together with the underfloor image 50C generated above, on the display device 19. This allows the control device 16 to display an image showing the underfloor conditions of the vehicle 1 and the surrounding conditions at time t=3.

[0041] Next, it is assumed that the vehicle 1 moves forward by a distance d3 between time t=3 and time t=4.

[0042] Immediately after time t=4, the front camera 11, rear camera 12, left camera 13, and right camera 14 transmit to the control device 16 a front photographed image 31D, a rear photographed image 32D, a left photographed image 33D, and a right photographed image 34D, respectively, captured at time t=4. The front photographed image 31D includes at least an image of the ground area 5D. The control device 16 stores the received front photographed image 31D in the storage device 17.

[0043] The control device 16 shifts the underfloor image 50C at time t=3, which is stored in the storage device 17, by a distance d3 toward the rear of the vehicle 1. Furthermore, the control device 16 shifts the front captured image 31C received immediately after time t=3, which is stored in the storage device 17, by a distance d3 toward the rear of the vehicle 1 and combines it with the previously shifted underfloor image 50C to generate an underfloor image 50D at time t=4. The control device 16 stores the generated underfloor image 50D in the storage device 17. The underfloor image 50D includes images of ground areas 5A, 5B, and 5C located under the floor of the vehicle 1 at time t=4.

[0044] The control device 16 displays the front photographed image 31D, rear photographed image 32D, left photographed image 33D, and right photographed image 34D received immediately after time t=4, together with the underfloor image 50D generated above, on the display device 19. This allows the control device 16 to display an image showing the underfloor conditions of the vehicle 1 and the surrounding conditions at time t=4.

[0045] The control device 16 repeats the above process until the vehicle 1 stops. This allows the control device 16 to display on the display device 19 an underfloor image 50 showing the underfloor situation when the vehicle 1 stops, and an image showing the surrounding situation.

[0046] <Issues regarding underfloor images when reversing> FIG. 4 is a diagram for explaining a problem regarding the underfloor image when the vehicle 1 is moving backward.

[0047] In some cases, rear camera 12 performs a predetermined process that is more demanding than those of front camera 11, left camera 13, and right camera 14. For example, rear camera 12 performs a person detection process on rear-view captured image 32. This allows the presence of a person behind vehicle 1 to be detected when vehicle 1 is reversing, improving safety. In this case, at a predetermined timing when front camera 11, left camera 13, and right camera 14 transmit images captured at a second time, rear camera 12 transmits an image captured at the same second time for which person detection process has not been completed, but which was captured at a first time prior to the second time and for which person detection process has been completed. In such a case, a problem that arises when an underfloor image during reversing is generated using a method similar to that of FIG. 3 will be described with reference to FIG. 4.

[0048] Immediately after time t=2, front camera 11, left camera 13, and right camera 14 transmit to the control device, respectively, front photographed image 31B, left photographed image 33B, and right photographed image 34B captured at time t=2. However, rearward camera 12 has not yet completed the person detection process for rearward photographed image 32B captured at time t=2, and therefore transmits to the control device, rearward photographed image 32A captured at time t=1. The rearward photographed image 32A includes at least an image of ground area 5A. The control device stores the received rearward photographed image 32A in a storage device.

[0049] Next, it is assumed that the vehicle 1 moves backward by a distance d2 between time t=2 and time t=3.

[0050] Immediately after time t=3, the front camera 11, the left camera 13, and the right camera 14 transmit to the control device, respectively, the front captured image 31C, the left captured image 33C, and the right captured image 34C captured at time t=3. However, the rear camera 12 has not yet completed the person detection process for the rear captured image 32C captured at time t=3, and therefore transmits to the control device the rear captured image 32B captured at time t=2. The rear captured image 32B includes at least an image of the ground area 5B. The control device stores the received rear captured image 32B in a storage device.

[0051] When using the same method as in FIG. 3 , the control device generates an underfloor image 50C at time t=3 by shifting the rearward captured image 32A received immediately after time t=2, which is stored in the storage device, by a distance d2 toward the front of the vehicle 1. The control device stores the generated underfloor image 50C in the storage device. However, as shown in FIG. 4 , this method results in a discrepancy between the position of the ground area 5A included in the underfloor image 50C and the actual position of the ground area 5A under the floor of the vehicle 1 at time t=3. This may cause the driver to erroneously recognize the position of an obstacle under the floor. Furthermore, when the control device 16 displays the underfloor image 50C together with the forward captured image 31C, rearward captured image 32C, left-side captured image 33C, and right-side captured image 34C received immediately after time t=3 on the display device 19, the discrepancy (gaps) between the underfloor image 50C and the other captured images becomes noticeable.

[0052] Next, it is assumed that the vehicle 1 moves backward by a distance d3 between time t=3 and time t=4.

[0053] Immediately after time t=4, the front camera 11, the left camera 13, and the right camera 14 transmit to the control device, respectively, the front captured image 31D, the left captured image 33D, and the right captured image 34D captured at time t=4. However, the rear camera 12 has not yet completed the person detection process for the rear captured image 32D captured at time t=4, and therefore transmits to the control device the rear captured image 32C captured at t=3. The rear captured image 32C includes at least an image of the ground area 5C. The control device stores the received rear captured image 32C in a storage device.

[0054] When using the same method as in FIG. 3, the control device shifts the underfloor image 50C at time t=3 stored in the storage device by a distance d3 toward the front of the vehicle 1. Furthermore, the control device shifts the rearward captured image 32B received immediately after time t=3, which is stored in the storage device, by a distance d3 toward the rear of the vehicle 1 and combines it with the previously shifted underfloor image 50C to generate an underfloor image 50D at time t=4. The control device stores the generated underfloor image 50D in the storage device. However, as shown in FIG. 4, the positions of the ground areas 5A and 5B included in the underfloor image 50D are different from the actual positions of the ground areas 5A and 5B under the floor of the vehicle 1 at time t=4. Furthermore, when the control device displays the front photographed image 31D, rear photographed image 32D, left photographed image 33D, and right photographed image 34D received immediately after time t=4 together with the underfloor image 50D on the display device 19, a gap between the underfloor image 50D and the other photographed images becomes noticeable. This gap is not eliminated even after the vehicle 1 stops, as shown at time t=5.

[0055] Next, a method for generating an underfloor image during backing up according to this embodiment, which reduces the discrepancy between the displayed underfloor image and the actual condition of the ground under the vehicle floor, will be described.

[0056] <Method for generating underfloor images when reversing> FIG. 5 is a diagram for explaining a method for generating an underfloor image 50 when the vehicle 1 according to the first embodiment is moving backward.

[0057] Immediately after time t=2, the front camera 11, the left camera 13, and the right camera 14 transmit to the control device 16, respectively, the front photographed image 31B, the left photographed image 33B, and the right photographed image 34B captured at time t=2. However, the rearward photographed image 32A captured at time t=1 is transmitted to the control device 16 by the rearward photographed image 32B captured at time t=2 because the person detection process has not yet been completed for the rearward photographed image 32B. The rearward photographed image 32A includes at least an image of the ground areas 5A and 5B. The control device 16 stores the received rearward photographed image 32A in the storage device 17.

[0058] Next, it is assumed that the vehicle 1 moves backward by a distance d2 between time t=2 and time t=3.

[0059] Immediately after time t=3, the front camera 11, the left camera 13, and the right camera 14 transmit to the control device 16, respectively, the front photographed image 31C, the left photographed image 33C, and the right photographed image 34C captured at time t=3. However, the rearward photographed image 32C captured at time t=3 has not yet been subjected to person detection processing by the rearward photographed image 32B captured at time t=2. The rearward photographed image 32B includes at least an image of the ground areas 5B and 5C. The control device 16 stores the received rearward photographed image 32B in the storage device 17.

[0060] The control device 16 generates an underfloor image 50C at time t=3 by shifting the rear captured image 32A received immediately after time t=2, which is stored in the storage device 17, toward the front of the vehicle 1 by the sum of the distance d1 traveled between time t=1 and time t=2 and the distance d2 traveled between time t=2 and time t=3. The control device 16 stores the generated underfloor image 50C in the storage device 17. As a result, as shown in FIG. 5 , the positions of the ground areas 5A and 5B included in the underfloor image 50C match the positions of the ground areas 5A and 5B in the actual underfloor of the vehicle 1 at time t=3. In other words, there is no discrepancy between the underfloor image 50C and the actual underfloor situation of the vehicle 1 at time t=3.

[0061] The control device 16 also displays the front photographed image 31C, rear photographed image 32B, left photographed image 33C, and right photographed image 34C received immediately after time t=3, and the underfloor image 50C on the display device 19. In this case as well, no discrepancy (gap) occurs between the display of the underfloor image 50C and the display of the left photographed image 33C and right photographed image 34C.

[0062] Next, it is assumed that the vehicle 1 moves backward by a distance d3 between time t=3 and time t=4.

[0063] Immediately after time t=4, the front camera 11, the left camera 13, and the right camera 14 transmit to the control device 16, respectively, a front photographed image 31D, a left photographed image 33D, and a right photographed image 34D captured at time t=4. However, the rearward photographed image 32B, which was captured at time t=4, has not yet been subjected to person detection processing, so the rearward photographed image 32C, which was captured at time t=3, is transmitted to the control device 16. The rearward photographed image 32B includes at least an image of the ground areas 5C and 5D. The control device 16 stores the received rearward photographed image 32C in the storage device 17.

[0064] The control device 16 shifts the underfloor image 50C at time t=3, which is stored in the storage device 17, by a distance d3 toward the front of the vehicle 1. Furthermore, the control device 16 shifts the rearward captured image 32B, which is stored in the storage device 17 and received immediately after time t=3, by a distance d2+d3 toward the rear of the vehicle 1 and combines it with the previously shifted underfloor image 50C to generate an underfloor image 50D at time t=4. The control device 16 stores the generated underfloor image 50D in the storage device 17. As a result, as shown in FIG. 5 , the positions of the ground areas 5A, 5B, and 5C included in the underfloor image 50D match the positions of the ground areas 5A, 5B, and 5C under the actual floor of the vehicle 1 at time t=4. In other words, there is no discrepancy between the underfloor image 50D and the actual underfloor situation of the vehicle 1 at time t=4.

[0065] The control device 16 also displays the front photographed image 31D, rear photographed image 32C, left photographed image 33D, and right photographed image 34D received immediately after time t=4, and the underfloor image 50D on the display device 19. In this case, too, no discrepancy (gap) occurs between the display of the underfloor image 50D and the display of the left photographed image 33D and the right photographed image 34D.

[0066] Furthermore, as shown at time t=5, even when the vehicle 1 stops immediately after time t=4, there is no discrepancy (gap) between the display of the underfloor image 50D and the display of the left-side captured image 33E and the right-side captured image 34E received immediately after time t=5.

[0067] Furthermore, the rearward captured image 32D received immediately after time t=5 was captured at time t=4, but because the vehicle 1 is stopped, there is no change in the condition of the ground in the forward area 21 between time t=4 and time t=5. Therefore, when the vehicle 1 is stopped, the discrepancy between the rearward captured image 32D displayed at time t=5 and the actual condition of the forward area 21 of the vehicle 1 at time t=5 is eliminated.

[0068] 3, 4, and 5 may correspond to the frame interval of the camera. For example, if the time required for the person detection process by the rear camera 12 is within one frame interval, when the front camera 11, the left camera 13, and the right camera 14 transmit the captured image of the nth frame, the rear camera 12 may transmit the captured image of the (n-1)th frame, as shown in FIG. 2. For example, if the time required for the person detection process by the rear camera 12 is within two frame intervals, when the front camera 11, the left camera 13, and the right camera 14 transmit the captured image of the nth frame, the rear camera 12 may transmit the captured image of the (n-2)th frame.

[0069] <Processing flow> FIG. 6 is a flowchart showing an example of processing performed by the control device 16 according to the first embodiment.

[0070] The control device 16 detects the moving direction (forward or backward) of the vehicle 1 (S101). For example, the control device 16 detects from the shift sensor whether the shift position is forward (for example, D range) or backward (for example, R range).

[0071] Based on the detection result of step S101, the control device 16 determines whether the moving direction of the vehicle 1 is "forward" or "reverse" (S102). For example, if the control device 16 detects in step S101 that the shift position is forward, it determines that the moving direction of the vehicle is "forward," and if the control device 16 detects that the shift position is reverse, it determines that the moving direction of the vehicle is "reverse."

[0072] When the control device 16 determines in step S102 that the moving direction of the vehicle is "forward" (S102: forward), it executes the underfloor image generating method for forward movement described with reference to Fig. 3 (S103). Then, the process returns to step S101.

[0073] When the control device 16 determines in step S102 that the vehicle's movement direction is "reverse" (S102: reverse), it executes the underfloor image generating method for reverse described with reference to Fig. 5 (S104). That is, when the vehicle 1 is moving backward, the control device 16 generates the underfloor image 50 by a method different from that when the vehicle 1 is moving forward. Then, the process returns to step S101.

[0074] Through the above processing, the control device 16 can generate and display an underfloor image 50 that is almost identical to the actual underfloor situation of the vehicle 1 and that is also almost identical to the photographed image, whether the vehicle 1 is moving forward or backward.

[0075] Although the above description has been given of the case where the vehicle 1 moves forward or backward linearly, the above content also applies to the case where the vehicle 1 turns while moving forward or backward. For example, when the vehicle 1 turns while moving forward, the control device 16 may generate the underfloor image 50 by the underfloor image generation method for moving forward described in Fig. 3 using the front photographed image 31, the left side photographed image 33, and the right side photographed image 34. For example, when the vehicle 1 turns while moving backward, the control device 16 may generate the underfloor image 50 by the underfloor image generation method for moving backward described in Fig. 5 using the rear photographed image 32, the left side photographed image 33, and the right side photographed image 34.

[0076] Furthermore, in the above description, a case has been described in which the rear camera 12 performs a predetermined process that is more demanding than the front camera 11 (i.e., a case in which a delay occurs in the rear camera 12). However, this embodiment also applies to a case in which the front camera 11 performs a predetermined process that is more demanding than the rear camera 12 (i.e., a case in which a delay occurs in the front camera 11). In this case, the above-described front camera 11 and its related content may be interchanged with the rear camera 12 and its related content. Therefore, the camera in which no delay occurs may be referred to as the first camera, and the camera in which a delay occurs may be referred to as the second camera.

[0077] (Summary of the first embodiment) The above description of the first embodiment discloses the following techniques.

[0078] <Technology 1> A method for controlling a vehicle (1) having a plurality of wheels (e.g., front wheels 2A, rear wheels 2B) and capable of moving in a predetermined direction using the wheels, wherein the vehicle is equipped with a first camera that transmits a first captured image of a first area outside the vehicle, a second camera that transmits a second captured image of a second area outside the vehicle, a control device that generates an underfloor image of the underfloor of the vehicle using the first captured image or the second captured image received at a predetermined timing, and a display device that displays the underfloor image, wherein the first captured image received at the predetermined timing was captured at a second time, and the second captured image received at the predetermined timing was captured at a first time before the second time, and the control device generates the underfloor image using the first captured image when the vehicle is moving in the direction of the first area, and generates the underfloor image using the second captured image when the vehicle is moving in the direction of the second area using a method different from when the vehicle is moving in the direction of the first area. In this way, when the vehicle is moving in the direction of the second area, the underfloor image is generated using a method different from that used when using the first captured image, taking into consideration that the second captured image was captured at a second time different from that of the first captured image. This makes it possible to generate an underfloor image that is less likely to deviate from the actual underfloor situation.

[0079] <Technology 2> In the control method described in Technology 1, the first camera is a forward camera (11), the first area is an area (21) in front of the vehicle, the direction of the first area is the forward direction of the vehicle, and the first captured image is a forward captured image (31), the second camera is a rear camera (12), the second area is an area (21) behind the vehicle, the direction of the second area is the backward direction of the vehicle, and the second captured image is a rear captured image (32). In this way, when the vehicle is moving backward, the underfloor image is generated by a method different from that used when using the front-facing image, taking into consideration that the rear-facing image was captured at a second time different from the front-facing image. This makes it possible to generate an underfloor image that is less likely to deviate from the actual underfloor situation when the vehicle is backing up.

[0080] <Technology 3> In the control method described in Technology 2, when the vehicle is moving forward, the control device shifts the front-photographed image to the rear of the vehicle based on the amount of movement of the vehicle from the second time to a third time that is later than the second time, and generates the underfloor image at the third time; when the vehicle is moving backward, the control device shifts the rear-photographed image to the front of the vehicle based on the amount of movement of the vehicle from the first time to the third time, and generates the underfloor image at the third time. This makes it possible to generate an underfloor image that is less different from the actual underfloor situation when the vehicle is backing up.

[0081] <Technology 4> In the control method described in Technique 2 or 3, the vehicle further includes a left camera (13) that transmits a left-side image (33) of a left-side area (23) of the vehicle, and a right camera (14) that transmits a right-side image (34) of a right-side area (24) of the vehicle, and the control device displays the left-side image and the right-side image taken at the third time, as well as the underfloor image at the third time, on the display device. This allows the left and right captured images and the underfloor image to be displayed with almost no misalignment (gap).

[0082] <Technology 5> In the control method according to any one of Techniques 2 to 4, the intervals between the first time point, the second time point, and the third time point are based on the image capturing intervals between the front camera and the rear camera. This makes it possible to generate an underfloor image that is less different from the actual underfloor situation when the vehicle is backing up.

[0083] <Technology 6> In the control method according to any one of the first to fifth techniques, the second camera performs a predetermined process on the second captured image and then transmits the second captured image. As a result, by performing a predetermined process, the second camera can generate an underfloor image that is less different from the actual underfloor situation, even in a configuration in which a second captured image taken at a first time earlier than the second time is transmitted.

[0084] <Technology 7> In the control method described in Technique 6, the predetermined process is a process of detecting a person from the second captured image. As a result, by having the second camera perform person detection processing, it is possible to generate an underfloor image that is less different from the actual underfloor situation, even in a configuration in which a second captured image taken at a first time that is earlier than the second time is transmitted.

[0085] <Technology 8> A control device is provided in a vehicle (1) that has a plurality of wheels (e.g., front wheels 2A, rear wheels 2B) and is capable of moving in a predetermined direction using the wheels. The vehicle is provided with a first camera that transmits a first captured image of a first area outside the vehicle, a second camera that transmits a second captured image of a second area outside the vehicle, the control device that generates an underfloor image of the underfloor of the vehicle using the first captured image or the second captured image received at a predetermined timing, and a display device that displays the underfloor image, wherein the first captured image received at the predetermined timing was captured at a second time, and the second captured image received at the predetermined timing was captured at a first time before the second time, and the control device generates the underfloor image using the first captured image when the vehicle is moving in the direction of the first area, and generates the underfloor image using the second captured image when the vehicle is moving in the direction of the second area using a method different from when the vehicle is moving in the direction of the first area. In this way, when the vehicle is moving in the direction of the second area, the underfloor image is generated using a method different from that used when using the first captured image, taking into consideration that the second captured image was captured at a second time different from that of the first captured image. This makes it possible to generate an underfloor image that is less likely to deviate from the actual underfloor situation.

[0086] <Technology 9> In the control device described in Technology 8, the first camera is a forward camera (11), the first area is an area (21) in front of the vehicle, the direction of the first area is the forward direction of the vehicle, and the first captured image is a forward captured image (31), the second camera is a rear camera (12), the second area is an area (22) behind the vehicle, the direction of the second area is the backward direction of the vehicle, and the second captured image is a rear captured image (32). In this way, when the vehicle is moving backward, the underfloor image is generated by a method different from that used when using the front-facing image, taking into consideration that the rear-facing image was captured at a second time different from the front-facing image. This makes it possible to generate an underfloor image that is less likely to deviate from the actual underfloor situation when the vehicle is backing up.

[0087] <Technology 10> In the control device described in Technology 9, when the vehicle is moving forward, the control device shifts the front-photographed image toward the rear of the vehicle based on the amount of movement of the vehicle from the second time to a third time that is later than the second time, and generates the underfloor image at the third time; when the vehicle is moving backward, the control device shifts the rear-photographed image toward the front of the vehicle based on the amount of movement of the vehicle from the first time to the third time, and generates the underfloor image at the third time. This makes it possible to generate an underfloor image that is less different from the actual underfloor situation when the vehicle is backing up.

[0088] <Technology 11> In the control device described in Technology 9 or 10, the vehicle further includes a left camera (13) that transmits a left-side image (33) of an area (23) on the left side of the vehicle, and a right camera (14) that transmits a right-side image (34) of an area (24) on the right side of the vehicle, and the control device displays the left-side image and the right-side image taken at the third time, as well as the underfloor image at the third time, on the display device. This allows the left and right captured images and the underfloor image to be displayed with almost no misalignment (gap).

[0089] <Technology 12> In the control device according to any one of Techniques 9 to 11, the intervals between the first time point, the second time point, and the third time point are based on the intervals between images captured by the front camera and the rear camera. This makes it possible to generate an underfloor image that is less different from the actual underfloor situation when the vehicle is backing up.

[0090] <Technology 13> In the control device according to any one of techniques 8 to 12, the second camera performs a predetermined process on the second captured image and then transmits the second captured image. As a result, by performing a predetermined process, the second camera can generate an underfloor image that is less different from the actual underfloor situation, even in a configuration in which a second captured image taken at a first time earlier than the second time is transmitted.

[0091] <Technology 14> In the control device described in Technique 13, the predetermined process is a process of detecting a person from the second captured image. As a result, by having the second camera perform person detection processing, it is possible to generate an underfloor image that is less different from the actual underfloor situation, even in a configuration in which a second captured image taken at a first time that is earlier than the second time is transmitted.

[0092] (Embodiment 2) In the first embodiment, a case has been described in which a delay occurs in which rear camera 12 transmits an image captured in the (nk)th frame when front camera 11, left camera 13, and right camera 14 transmit an image captured in the nth frame. Here, k is an integer equal to or greater than 1, and in the first embodiment, a case has been described in which k is known in advance.

[0093] However, if the rear camera 12, the front camera 11, the left camera 13, and the right camera 14 operate asynchronously, the frame of the captured image transmitted from the rear camera 12 may be delayed by more than k frames. Hereinafter, the time including the time for the k frames and the additional delay time will be referred to as the rear camera delay time. The rear camera delay time is not always constant and may change, for example, every time the system including the front camera 11, the rear camera 12, the left camera 13, the right camera 14, and the control device 16 is restarted (for example, every time the engine of the vehicle 1 is turned on).

[0094] In the second embodiment, a method is described in which a rear camera delay time is calculated, and when implementing the underfloor image generating method for reversing the vehicle 1 described in the first embodiment, an underfloor image is generated taking the rear camera delay time into consideration, thereby preventing a discrepancy (gap) from occurring between the display of the left side photographed image 33 and the right side photographed image 34 and the display of the underfloor image 50, as in the first embodiment. Note that the configuration of the vehicle 1 in the second embodiment may be the same as that in the first embodiment, and therefore description thereof will be omitted.

[0095] FIG. 7 is a diagram illustrating an overlapping region according to the second embodiment.

[0096] 7, there is an overlapping area between the shooting area 63 of the left camera 13 and the shooting area 62 of the rear camera 12. The control device 16 sets an overlapping area 60 in at least a part of the overlapping area. In this embodiment, the overlapping area 60 is set for the left camera 13 and the rear camera 12, but the overlapping area 60 may also be set for the right camera 14 and the rear camera 12.

[0097] Fig. 8 is a flowchart showing an example of a delay time calculation process according to embodiment 2. Fig. 9 is a diagram showing an example of a luminance change in two overlapping regions 60 according to embodiment 2. Next, the delay time calculation process performed by the control device 16 will be described with reference to Figs. 8 and 9.

[0098] (S201) The control device 16 calculates the brightness in the overlapping area 60 of the left-side captured image 33 received from the left-side camera 13. By calculating the brightness in the overlapping area 60 of each left-side captured image 33 received at intervals of the frame rate, the control device 16 can obtain the change in brightness over time in the overlapping area 60 of the left-side captured image 33 captured by the left-side camera 13, as shown in FIG. 9(a). Hereinafter, the change in brightness over time in the overlapping area 60 of the left-side captured image 33 captured by the left-side camera 13 will be referred to as the brightness change of the left-side camera 13.

[0099] (S202) The control device 16 calculates the brightness in the overlapping area 60 of the rear-photographed images 32 received from the rear camera 12. The control device 16 calculates the brightness in the overlapping area 60 of each rear-photographed image 32 received at intervals of the frame rate, thereby obtaining the change in brightness over time in the overlapping area 60 of the rear-photographed images 32 taken by the rear camera 12, as shown in FIG. 9(b). Hereinafter, the change in brightness over time in the overlapping area 60 of the rear-photographed images 32 taken by the rear camera 12 will be referred to as the brightness change of the rear camera 12. Note that steps S201 and S202 may be executed in any order, or may be executed in parallel.

[0100] (S203) The control device 16 calculates the amount of deviation s in the time direction between the luminance change of the left camera 13 and the luminance change of the rear camera 12. As shown in Fig. 9, for example, the control device 16 calculates the amount of deviation s that can maximize the correlation between the luminance change of the left camera 13 and the luminance change of the rear camera 12. The control device 16 may calculate the amount of deviation s using known sub-pixel estimation.

[0101] (S204) The control device 16 calculates the rear camera delay time based on the calculated amount of deviation s, and stores it in the storage device 17.

[0102] When the control device 16 performs the underfloor image generating method for when the vehicle 1 is reversing, described in the first embodiment, it generates the underfloor image 50 based on the rear camera delay time stored in the storage device 17. For example, when generating the underfloor image 50 at time t=3 using the rear captured image 32 received at time t=2, the rear captured image 32 was captured the rear camera delay time before time t=1. Therefore, the control device 16 shifts the rear captured image 32 forward of the vehicle 1 by the distance x that the vehicle 1 moved backward from the time before the rear camera delay time before time t=1 to time t=3, and synthesizes it with the previously shifted underfloor image 50 to generate the underfloor image 50 at time t=3.

[0103] As a result, even if the rear camera delay time can change, for example, each time the system of vehicle 1 is restarted (for example, each time the engine of vehicle 1 is turned on), the control device 16 can display the underfloor image 50 with little discrepancy (gap) between the display of the left-side captured image 33 and the right-side captured image 34, as in embodiment 1.

[0104] In the above description, the rear camera delay time is calculated based on the amount of deviation between the time change in luminance in the overlapping region 60 of the left-side captured image 33 and the time change in luminance in the overlapping region 60 of the rearward captured image 32. However, the rear camera delay time may be calculated by other methods. For example, the rear camera delay time may be calculated based on the amount of deviation between the time change in the edge extraction result in the overlapping region 60 of the left-side captured image 33 and the time change in the edge extraction result in the overlapping region 60 of the rearward captured image 32. Alternatively, the rear camera delay time may be calculated based on the amount of deviation between the time change in RGB values ​​in the overlapping region 60 of the left-side captured image 33 and the time change in RGB values ​​in the overlapping region 60 of the rearward captured image 32. In other words, the control device 16 may calculate the rear camera delay time based on the amount of deviation between the time changes in image characteristics in the two overlapping regions 60.

[0105] Furthermore, the control device 16 performs the delay time calculation process shown in Fig. 8 while the vehicle 1 is traveling. As described above, calculation of the rear camera delay time requires the change in brightness over time in the overlap region 60 (i.e., the change in image characteristics over time), but no change in brightness occurs while the vehicle 1 is stopped. On the other hand, if the traveling speed of the vehicle 1 is too fast compared to the frame rate of the camera (i.e., the sampling rate of the brightness change), aliasing (folding) occurs in the data indicating the brightness change due to the sampling theorem, and there is a possibility that the amount of deviation between the brightness changes in the two overlap regions 60 cannot be calculated correctly.

[0106] Therefore, the control device 16 may perform the delay time calculation process shown in FIG. 8 when the traveling speed of the vehicle 1 is within a predetermined range. For example, after the engine of the vehicle 1 is turned on, the control device 16 may start the delay time calculation process shown in FIG. 8 when the traveling speed of the vehicle 1 becomes equal to or greater than the lower limit of the predetermined range, and may stop the delay time calculation process shown in FIG. 8 when the traveling speed of the vehicle 1 becomes equal to or greater than the upper limit of the predetermined range. This allows the control device 16 to accurately calculate the rear camera delay time. The predetermined range may have a lower limit of 2 km / h and an upper limit of 5.4 km / h, for example. However, these numerical values ​​are merely examples and may vary depending on the accuracy of the sensor that detects the traveling speed of the vehicle 1, the frame rate of the camera, the pattern of the ground captured by the camera, etc.

[0107] (Summary of the second embodiment) The above description of the second embodiment discloses the following techniques.

[0108] <Technology 1> A method for controlling a vehicle (1) having a plurality of wheels and capable of moving in a predetermined direction using the wheels includes the vehicle including a first camera (e.g., left side camera 13) that transmits a first captured image (e.g., left side captured image 33) of a first area (e.g., left side area 23) outside the vehicle, a second camera (e.g., rear camera 12) that transmits a second captured image (e.g., rear captured image 32) of a second area (e.g., rear area 22) outside the vehicle, a control device (16) that generates an underfloor image (50) of an underfloor of the vehicle using the second captured image received at a predetermined timing, and a display device (19) that displays the underfloor image, The received first photographed image was photographed at a second time, and the second photographed image received at the specified timing was photographed at a first time earlier than the second time. The control device sets an overlapping area (60) that overlaps both the first area and the second area, calculates a delay time of the second camera relative to the first camera based on the change over time in image characteristics in the overlapping area of ​​the first photographed image and the change over time in image characteristics in the overlapping area of ​​the second photographed image, and generates the underfloor image based on the second photographed image and the delay time when the vehicle is moving in the direction of the second area. In this way, when the vehicle is moving in the direction of the second area, the underfloor image is generated taking into consideration the fact that the second captured image was captured at a second time different from the first captured image and the delay time of the second camera, thereby making it possible to generate an underfloor image that is close to the actual underfloor situation.

[0109] <Technology 2> In the control method described in Technique 1, the control device calculates the delay time using a time change in image characteristics in the overlapping area obtained while the vehicle is traveling. This allows the delay time of the second camera to be calculated.

[0110] <Technology 3> In the control method described in Technology 2, the control device calculates the delay time based on the amount of deviation between the time change in image characteristics in the overlapping area of ​​the first captured image and the time change in image characteristics in the overlapping area of ​​the second captured image. This allows the delay time of the second camera to be calculated.

[0111] <Technology 4> In the control method according to Technique 2 or 3, the change over time in the image characteristic in the overlapping region is a change over time in the luminance of the image in the overlapping region. This makes it possible to calculate the delay time of the second camera based on the change over time in the brightness of the image in the overlapping area.

[0112] <Technology 5> In the control method described in any one of Techniques 1 to 4, the first camera is a left camera or a right camera, the first area is a left area or a right area of ​​the vehicle, the first captured image is a left image or a right image, the second camera is a rear camera, the second area is a rear area of ​​the vehicle, the direction of the second area is a backward direction of the vehicle, and the second captured image is a rear captured image. As a result, when the vehicle is moving backward, an underfloor image that is less different from the actual underfloor situation can be generated.

[0113] <Technology 6> In the control method described in Technology 5, when the vehicle is moving in a backward direction, the control device shifts the rear-photographed image to the front of the vehicle based on the moving distance of the vehicle and the delay time, and generates the underfloor image. This makes it possible to generate an underfloor image that is less likely to deviate from the actual underfloor situation.

[0114] <Technology 7> In the control method according to any one of the first to sixth techniques, the second camera performs a predetermined process on the second captured image and then transmits the second captured image. This makes it possible to generate an underfloor image that is less likely to deviate from the actual underfloor situation, even if a delay occurs due to the second camera performing a predetermined process on the second captured image.

[0115] <Technology 8> In the control method described in Technique 7, the predetermined process is a process of detecting a person from the second captured image. This makes it possible to generate an underfloor image that is less likely to deviate from the actual underfloor situation, even if a delay occurs due to the second camera performing processing to detect a person from the second captured image.

[0116] <Technology 9> A control device (16) provided in a vehicle (1) having a plurality of wheels and capable of moving in a predetermined direction using the wheels includes a first camera (e.g., left side camera 13) that transmits a first captured image (e.g., left side captured image 33) of a first area (e.g., left side area 23) outside the vehicle, a second camera (e.g., rear captured image 32) that transmits a second captured image of a second area (e.g., rear area 22) outside the vehicle, a control device that generates an underfloor image (50) of an underfloor of the vehicle using the second captured image received at a predetermined timing, and a display device (19) that displays the underfloor image, and the underfloor image received at the predetermined timing is displayed on the display device (19). The first captured image was captured at a second time, and the second captured image received at the specified timing was captured at a first time earlier than the second time. The control device sets an overlapping area (60) that overlaps both the first area and the second area, calculates a delay time of the second camera relative to the first camera based on the change over time in image characteristics in the overlapping area of ​​the first captured image and the change over time in image characteristics in the overlapping area of ​​the second captured image, and generates the underfloor image based on the second captured image and the delay time when the vehicle is moving in the direction of the second area. In this way, when the vehicle is moving in the direction of the second area, the underfloor image is generated taking into consideration the fact that the second captured image was captured at a second time different from the first captured image and the delay time of the second camera, thereby making it possible to generate an underfloor image that is close to the actual underfloor situation.

[0117] <Technology 10> In the control device described in Technique 9, the control device calculates the delay time using a change over time in image characteristics in the overlapping area obtained while the vehicle is traveling. This allows the delay time of the second camera to be calculated.

[0118] <Technology 11> In the control device described in Technology 10, the control device calculates the delay time based on the amount of deviation between the time change in image characteristics in the overlapping area of ​​the first captured image and the time change in image characteristics in the overlapping area of ​​the second captured image. This allows the delay time of the second camera to be calculated.

[0119] <Technology 12> In the control device according to Technique 10 or 11, the change over time in the image characteristic in the overlapping region is a change over time in the luminance of the image in the overlapping region.

[0120] <Technology 13> In the control device described in any one of Techniques 9 to 12, the first camera is a left camera or a right camera, the first area is a left area or a right area of ​​the vehicle, the first captured image is a left image or a right image, the second camera is a rear camera, the second area is a rear area of ​​the vehicle, the direction of the second area is a backward direction of the vehicle, and the second captured image is a rear captured image. As a result, when the vehicle is moving backward, an underfloor image that is less different from the actual underfloor situation can be generated.

[0121] <Technology 14> In the control device described in Technology 13, the control device shifts the rear-photographed image to the front of the vehicle based on the moving distance of the vehicle and the delay time, and generates the underfloor image. This makes it possible to generate an underfloor image that is less likely to deviate from the actual underfloor situation.

[0122] <Technology 15> In the control device according to any one of Techniques 9 to 14, the second camera performs a predetermined process on the second captured image and then transmits the second captured image. This makes it possible to generate an underfloor image that is less likely to deviate from the actual underfloor situation, even if a delay occurs due to the second camera performing a predetermined process on the second captured image.

[0123] <Technology 16> In the control device described in Technique 15, the predetermined process is a process of detecting a person from the second captured image. This makes it possible to generate an underfloor image that is less likely to deviate from the actual underfloor situation, even if a delay occurs due to the second camera performing processing to detect a person from the second captured image.

[0124] Although the embodiments have been described above with reference to the accompanying drawings, the present disclosure is not limited to such examples. It is clear that a person skilled in the art can conceive of various modifications, alterations, substitutions, additions, deletions, and equivalents within the scope of the claims, and it is understood that these also fall within the technical scope of the present disclosure. Furthermore, the components in the above-described embodiments may be combined in any manner without departing from the spirit of the invention. [Industrial Applicability]

[0125] The technology of the present disclosure is useful for vehicle driving assistance. [Explanation of symbols]

[0126] 1 vehicle 2A front wheel 2B rear wheel 5A,5B,5C,5D Ground area 11. Front camera 12 Rear camera 13 Left camera 14 Right camera 15 sensors 16 Control device 17 Storage device 18 Input Devices 19 Display device 21 Anterior area 22 Posterior area 23 Left area 24 Right area 31, 31A, 31B, 31C, 31D Frontal image 32, 32A, 32B, 32C, 32D Rear view images 33, 33A, 33B, 33C, 33D, 33E Left side image 34,34A,34B,34C,34D,34E Right side photographed image 50, 50B, 50C, 50D Underfloor image 60 Overlapping area 62,63 Shooting area

Claims

1. A method for controlling a vehicle having a plurality of wheels and capable of moving in a predetermined direction using the wheels, the method comprising: a first camera configured to transmit a first captured image of a first area outside the vehicle; a second camera configured to transmit a second captured image of a second area outside the vehicle; a control device that generates an underfloor image of an underfloor of the vehicle using the second captured image received at a predetermined timing; a display device that displays the underfloor image, the first captured image received at the predetermined timing is captured at a second time, the second captured image received at the predetermined timing was captured at a first time point that is earlier than the second time point, The control device setting an overlapping area that overlaps both the first area and the second area; calculating a delay time of the second camera relative to the first camera based on a time change in image characteristics in the overlapping region of the first captured image and a time change in image characteristics in the overlapping region of the second captured image; a control method for generating the underfloor image based on the second captured image and the delay time when the vehicle is moving in the direction of the second area;

2. 2. The control method according to claim 1, The control device calculates the delay time based on a change over time in image characteristics in the overlap region obtained while the vehicle is traveling.

3. 3. The control method according to claim 2, A control method in which the control device calculates the delay time based on the amount of deviation between the time change in image characteristics in the overlapping area of ​​the first captured image and the time change in image characteristics in the overlapping area of ​​the second captured image.

4. 3. The control method according to claim 2, A control method, wherein the change in image characteristics over time in the overlap region is a change in luminance of the image in the overlap region over time.

5. 2. The control method according to claim 1, the first camera is a left camera or a right camera, the first region is a left region or a right region of the vehicle, the first captured image is a left-side captured image or a right-side captured image, the second camera is a rear camera, the second area is a rear area of ​​the vehicle, the direction of the second area is a backward direction of the vehicle, A control method, wherein the second captured image is a rear captured image.

6. 6. The control method according to claim 5, The control device, when the vehicle is moving in a reverse direction, shifts the rear-view image to the front of the vehicle based on the moving distance of the vehicle and the delay time, and generates the underfloor image.

7. 2. The control method according to claim 1, a control method in which the second camera performs predetermined processing on the second captured image and then transmits the second captured image.

8. 8. The control method according to claim 7, The predetermined process is a process of detecting a person from the second captured image.

9. A control device provided in a vehicle that has a plurality of wheels and is capable of moving in a predetermined direction using the wheels, the control device comprising: a first camera configured to transmit a first captured image of a first area outside the vehicle; a second camera configured to transmit a second captured image of a second area outside the vehicle; a control device that generates an underfloor image of an underfloor of the vehicle using the second captured image received at a predetermined timing; a display device that displays the underfloor image, the first captured image received at the predetermined timing is captured at a second time, the second captured image received at the predetermined timing was captured at a first time point that is earlier than the second time point, The control device setting an overlapping area that overlaps both the first area and the second area; calculating a delay time of the second camera relative to the first camera based on a time change in image characteristics in the overlapping region of the first captured image and a time change in image characteristics in the overlapping region of the second captured image; a control device that generates the underfloor image based on the second captured image and the delay time when the vehicle is moving in the direction of the second area.

10. The control device according to claim 9, The control device calculates the delay time based on a change over time in image characteristics in the overlap region obtained while the vehicle is traveling.

11. The control device according to claim 10, The control device calculates the delay time based on the amount of deviation between the time change in image characteristics in the overlapping area of ​​the first captured image and the time change in image characteristics in the overlapping area of ​​the second captured image.

12. The control device according to claim 10, A control device, wherein the change in image characteristics over time in the overlap region is a change in luminance of the image in the overlap region over time.

13. The control device according to claim 9, the first camera is a left camera or a right camera, the first region is a left region or a right region of the vehicle, the first captured image is a left-side captured image or a right-side captured image, the second camera is a rear camera, the second area is a rear area of ​​the vehicle, the direction of the second area is a backward direction of the vehicle, The control device, wherein the second captured image is a rear captured image.

14. The control device according to claim 13, The control device shifts the rear-photographed image to a position forward of the vehicle based on the moving distance of the vehicle and the delay time, and generates the underfloor image.

15. The control device according to claim 9, The control device is configured to transmit the second captured image after the second camera performs a predetermined process on the second captured image.

16. 16. The control device according to claim 15, The control device, wherein the predetermined process is a process of detecting a person from the second captured image.

Images