Image Processing System

Abstract

To provide an image processing system configured to improve safety by allowing a user to satisfactorily confirm an obstacle surrounding a vehicle.SOLUTION: A head-mounted display unit 10 to be worn by a user on a vehicle 3 comprising around view cameras 2a-2d includes: a camera 14 which images a field of view of the user; a head-mounted display 13; a display-side communication unit 15 which wirelessly communicates with the vehicle 3; and a control unit 11 which generates a display image to be displayed on the head-mounted display 13. The control unit 11 includes an image synthesis unit 17 which synthesizes a vehicle surroundings image generated by the around view cameras 2a-2d input via the display-side communication unit 15 with a field-of-view image captured by the camera 14 to generate a mixed reality image, and causes the head-mounted display 13 to display the mixed reality image.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image processing system that improves visibility around a vehicle. [Background technology]

[0002] In recent years, various monitor devices such as around-view monitors have been developed to improve vehicle safety. For example, as described in Patent Document 1, the around-view monitor captures images of the surroundings of the vehicle using a camera mounted on the vehicle and displays the captured images on an image display device (monitor) provided in the vehicle cabin, thereby enabling vehicle occupants, particularly the driver, to visually recognize obstacles that are hidden behind the vehicle and cannot be seen.

[0003] Meanwhile, in recent years, MR technology has been developed to generate and provide a synthetic image (mixed reality image) by superimposing a virtual image on a real image captured by a camera, etc. Furthermore, a technology has been proposed to provide the synthetic image generated by this MR technology to a passenger (user) in a vehicle. For example, Patent Document 2 proposes a technology in which a landscape photographed by a camera mounted on a tourist vehicle is overlaid with a virtual image (e.g., tourist information, advertisements) related to the landscape, and the virtual image is displayed on a head-mounted display (head-mounted display device) worn by passengers such as tourists aboard the tourist vehicle, thereby providing the image to the passengers. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2007-41791 A [Patent Document 2] Restated WO2012 / 033095 publication Summary of the Invention [Problem to be solved by the invention]

[0005] Since the vehicle monitor device as described above can be checked only after the driver gets into the vehicle and turns on the vehicle power, if the driver is in a hurry, there is a possibility that the driver will start driving the vehicle without making a thorough check. One possible solution is to display an image of the vehicle's surroundings on a head-mounted display, as in Patent Document 2. However, since Patent Document 2 provides information to the vehicle occupants in the vehicle using images of the vehicle's surroundings and virtual images, there is a risk that the vehicle may start moving without fully checking for obstacles.

[0006] The present invention has been made in consideration of the above circumstances, and its object is to provide an image processing system that improves safety by enabling a user to fully check obstacles around the vehicle. [Means for solving the problem]

[0007] In order to achieve the above object, an image processing system of the present invention includes a portable terminal having an imaging unit for capturing images, an image display unit for displaying images, and a communication unit for communicating with a vehicle, and an image synthesis unit for generating a vehicle surroundings image of the vehicle viewed from above by combining images captured around the vehicle acquired via the communication unit, and generating a mixed reality image by superimposing the captured images captured by the imaging unit, the portable terminal being located within a predetermined range near the vehicle, cell phone When the line of sight of the user wearing the terminal is located near the door of the vehicle, The image synthesis unit On the outer wall surface of the vehicle or in the vicinity of the outer wall surface of the door of the vehicle The vehicle surroundings image is superimposed to generate the mixed reality image, and a control unit that displays the mixed reality image generated by the image synthesis unit on the image display unit.

[0008] This means: cell phoneWhen the user wearing the terminal is located within a predetermined range near the vehicle, the user can visually confirm both the image captured by the imaging unit and the image of the surroundings of the vehicle by visually confirming the vicinity of the outer wall surface of the vehicle or the outer wall surface of the vehicle door displayed on the image display unit. Therefore, the user can check from outside the vehicle for obstacles that are hidden behind the vehicle and cannot be directly seen with the naked eye.

[0010] good Preferably, the image synthesis unit generates the mixed reality image by making the vehicle in the captured image transparent and overlaying the vehicle surroundings image on the ground at the position of the vehicle.

[0011] As a result, the vehicle surroundings image is displayed on the ground of the captured image displayed on the image display unit outside the vehicle, so that if there is an obstacle around the vehicle, the position of the obstacle can be accurately visually confirmed. Also, a user carrying a mobile terminal can easily recognize the presence or absence of an obstacle around the vehicle before getting into the vehicle. Preferably, the mobile terminal is a head-mounted display device that is worn on the head.

[0012] This allows the user to easily wear a portable terminal having an imaging unit and an image display unit, and enables the user to check for obstacles around the vehicle from outside the vehicle. Preferably, when the head-mounted display device is worn and the vehicle is within the field of view and the vehicle is positioned near the vehicle, the mixed reality image is displayed on the image display unit. As a result, when a user wearing a head-mounted display device is located near a vehicle with the vehicle within their field of vision, a mixed reality image is displayed on the image display unit, allowing the user to check an image of the area around the vehicle. Effect of the Invention

[0013] According to the image processing system of the present invention, when a user carrying a mobile terminal having an imaging unit and an image display unit is located near a vehicle, a mixed reality image including an image of the surroundings of the vehicle is displayed, so that the user can visually confirm the surroundings of the vehicle. Therefore, even if the user carrying the mobile terminal gets into the vehicle and immediately starts moving, the user can increase the time to check the surroundings of the vehicle, thereby improving safety. [Brief description of the drawings]

[0014] [Figure 1] 1 is a schematic configuration diagram of a vehicle display system using a head mounted display unit according to an embodiment of the present invention. [Diagram 2] 1 is a first display example of an around-view image displayed on the head mounted display according to the present embodiment. [Diagram 3] 13 is a second display example of an around-view image displayed on the head mounted display according to the present embodiment. [Figure 4] 13 is a third display example of an around-view image displayed on the head mounted display according to the present embodiment. [Diagram 5] 4 is a fourth display example of an around-view image displayed on the head mounted display according to the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a configuration diagram of a vehicle display system 1 that uses a head mounted display unit 10 (head mounted display device, mobile terminal, image processing system) according to one embodiment of the present invention. The vehicular display system 1 includes around-view cameras 2 a , 2 b , 2 c , and 2 d , a head-mounted display unit 10 , and a vehicle-side communication device 12 .

[0016] In the vehicular display system 1, the around view cameras 2a, 2b, 2c, and 2d and the vehicle-side communication device 12 are mounted on a vehicle 3. The around view cameras 2a, 2b, 2c, and 2d are provided, for example, one each on the front, rear, left and right of the vehicle, and the four around view cameras 2a, 2b, 2c, and 2d capture images of the entire circumference of the vehicle 3 in the front, rear, left and right directions.

[0017] The vehicle-side communication device 12 transmits images of the surroundings of the vehicle input from the around-view cameras 2a, 2b, 2c, and 2d. Note that it is sufficient for the vehicle-side communication device 12 to be capable of transmitting at least within the vehicle cabin near the vehicle and the surroundings of the vehicle, for example, within a range of several meters. The head-mounted display unit 10 includes a head-mounted display 13 (image display unit), a camera 14 (imaging unit), a display side communication device 15 (communication unit), and a control unit 11 (control unit), and is worn on the head of a passenger (user) aboard the vehicle 3.

[0018] The head mounted display 13 is provided in front of the eyes of a user wearing the head mounted display unit 10, and has a function of displaying images and providing image information to the user. The camera 14 is provided, for example, near the head mounted display 13, and is arranged with its lens facing forward of the user. Therefore, the camera 14 captures (takes) an image in the direction in which the head of the user wearing the head mounted display unit 10 faces, that is, the forward direction which is the line of sight.

[0019] The display side communication device 15 and the vehicle side communication device 12 perform wireless communication with each other. The display side communication device 15 receives a captured image signal transmitted from the vehicle side communication device 12. The control unit 11 is a control device for controlling the display of the head mounted display 13, and includes an input / output device, a memory device (ROM, RAM, non-volatile RAM, etc.), a central processing unit (CPU), a timer, and the like.

[0020] The control unit 11 requests captured images of the surroundings of the vehicle input from the vehicle-side around view cameras 2a, 2b, 2c, 2d from the display-side communication device 15 via the vehicle-side communication device 12, and inputs the images captured by the around view cameras 2a, 2b, 2c, 2d received by the display-side communication device 15 and the captured image (image) by the camera 14, outputs a composite image (display image) signal to be displayed on the head-mounted display 13, and causes the head-mounted display 13 to display a composite image based on the composite image signal.

[0021] The control unit 11 includes an around view monitor image generating unit 16 and an image synthesizing unit 17 . The around-view monitor image generating unit 16 generates an around-view image 20 (surrounding image) from the images captured by the around-view cameras 2a, 2b, 2c, and 2d input via the vehicle-side communication device 12 and the display-side communication device 15. The around-view image 20 is an image obtained by combining an image 21 of the vehicle 3 and images of its surroundings (front, rear, left, right, and right) as viewed from above, as shown in FIG. 2, for example. The around-view image 20 can be generated in the around-view monitor image generating unit 16 using a known around-view monitor system (bird's-eye image display system) disclosed in Patent Document 1, for example, and therefore a detailed description thereof will be omitted. Note that, in the case where the around-view image 20 can be generated in the vehicle 3 and transmitted from the vehicle-side communication device 12, the around-view image 20 may be input via the display-side communication device 15 and used.

[0022] The image synthesis unit 17 inputs the captured image captured by the camera 14 of the head mounted display unit 10, as well as the around view image generated by the around view monitor image generation unit 16, aligns it with the position of the vehicle 3 in the captured image of the camera 14, and superimposes the around view image at a predetermined position of the vehicle 3 to generate a composite image (mixed reality image) of the field of view image and the around view image.

[0023] The composite image generated by the image synthesis unit 17 is displayed on the head mounted display 13. The head-mounted display unit 10 is equipped with a portable battery and a power switch (not shown), and by turning the power switch ON, the head-mounted display 13, camera 14, display-side communication device 15, and control unit 11 of the head-mounted display unit 10 become operable.

[0024] On the other hand, the around view cameras 2a to 2d and the vehicle side communication device 12 become operable when the power supply of the vehicle 3 is turned on. Furthermore, the head mounted display unit 10 is provided with a position detection section 18 for controlling the power supply of the around view cameras 2a to 2d. The position detection section 18 has a function of detecting the relative position of the head mounted display 13 with respect to the vehicle 3, and when the powered-on head mounted display unit 10 is located within a predetermined surrounding range (for example, within a range of several meters) near the vehicle 3 and within the vehicle cabin, the position detection section 18 causes the display side communication device 15 to transmit a power ON signal for turning on the around view cameras 2a to 2d. For example, the vehicle side communication device 12 is kept capable of receiving communication from the head mounted display unit 10 at all times, and when a power ON signal is received from the head mounted display unit 10, the vehicle side communication device 12 controls the around view cameras 2a to 2d to be turned on. Alternatively, the position detection unit 18 may be configured to have a function of outputting radio waves within a predetermined range when the head mounted display unit 10 is powered on, separate from the display side communication device 15, and to receive the radio waves on the vehicle 3 side and turn on the power of the around view cameras 2a to 2d. As a result, when a user wearing the head mounted display unit 10 is located within a predetermined range around the vehicle 3 or within the vehicle cabin, the around view cameras 2a to 2d are turned on, making it possible for the around view cameras 2a to 2d to capture images of the surroundings of the vehicle 3.

[0025] Next, a display example of the around-view image 20 in the head mounted display unit 10 of this embodiment will be described. Fig. 2 is a first display example of the around-view image 20. Fig. 3 is a second display example of the around-view image 20. Fig. 4 is a third display example of the around-view image 20. Note that Figs. 2 to 4 show a state in which a user is located outside the vehicle 3 and views the vehicle 3 from the right side of the vehicle, which is the driver's seat side (Figs. 2 and 3), or from the right front side (Fig. 4), in order for the driver to get into the vehicle 3, for example.

[0026] The vehicle 3 of this embodiment is a five-door automobile having five doors in total, two at the front and two at the rear of the vehicle, and a back door at the rear of the vehicle. As shown in FIG. 2, an around-view image 20 is displayed in the vicinity of the rear of a door handle 26 on the outer wall surface of a right front door 25 of a vehicle 3. Alternatively, as shown in FIG. 3, the around-view image 20 may be displayed in the vicinity of the upper front side of a right front door 25 of the vehicle 3.

[0027] The around view image 20 is displayed on the right side of the vehicle (driver's seat side), but may be displayed on the left side of the vehicle as well as on the right side of the vehicle. Furthermore, it may be displayed near the opening / closing switch of the rear gate of the vehicle 3. When the head mounted display unit 10 is worn by not only the driver but also other users, the around view image 20 may be displayed on the head mounted display units 10 of all users viewing the vehicle 3. When the driver is specified in the head mounted display unit 10, for example, when a head mounted display unit 10 for the driver is set, the around view image 20 may be displayed only on the head mounted display unit 10 for the driver.

[0028] The around view image 20 may be displayed at a predetermined display position (predetermined position), such as near the rear of the door handle 26 of the right front door 25 of the vehicle 3, constantly when the power of the vehicle 3 is turned on, or may be displayed for a predetermined period of time when the door lock device of the vehicle 3 is unlocked. The vehicle 3 is provided with a door locking / unlocking device that locks and unlocks the door lock devices of the vehicle 3 according to the user's line of sight detected by the head mounted display unit 10.

[0029] 2, the door locking / unlocking device displays a door locking / unlocking mark 30 superimposed on the side image of the vehicle 3 displayed by the head mounted display unit 10, in front of the door handle 26 of the right front door 25 located to the side of the driver's seat of the vehicle 3. The door locking / unlocking mark 30 has a shape, for example, of a rectangular mark combined with an arrow. The door locking / unlocking device regards the center of the image captured by the camera 14 of the head mounted display unit 10 as the gaze position, and when this gaze position is maintained within the rectangular shape of the door locking / unlocking mark 30 for a predetermined period of time, switches between locking and unlocking the door locking device of the vehicle 3. When switching between locking and unlocking, it is preferable to make the door locking / unlocking mark 30 flash.

[0030] When a door locking / unlocking device like this has a function for detecting the gaze position of a user wearing the head mounted display unit 10, the around view image 20 can be displayed for a predetermined period of time from the point when the user's gaze position is positioned on the door locking / unlocking mark 30. With the above-described configuration, a user wearing the head mounted display unit 10 can obtain a field of view similar to that seen directly with the naked eye, since an image of the area in front of the user captured by the camera 14 is displayed on the head mounted display 13.

[0031] Furthermore, the captured image displayed on the head mounted display 13 of the head mounted display unit 10 is combined with the captured images captured by the around view cameras 2a to 2d of the vehicle 3 and displayed as an around view image 20, so that the user wearing the head mounted display unit 10 can visually recognize both the field of view image and the vehicle surroundings image in the vicinity of the vehicle. Therefore, it is possible to confirm an obstacle that is hidden behind the vehicle 3 and cannot be directly seen with the naked eye.

[0032] In addition, the vehicle 3 and the head mounted display unit 10 are connected by wireless communication, and when the head mounted display unit 10 is located within a predetermined range around the vehicle 3, the around view cameras 2a to 2d can capture images of the surroundings of the vehicle 3, so that the user wearing the head mounted display unit 10 can visually recognize the vehicle surroundings image even outside the vehicle 3. Therefore, it is possible to check obstacles around the vehicle 3 before getting into the vehicle 3. This allows the user, particularly the driver of the vehicle, to increase the time to check the surroundings of the vehicle 3, even when, for example, the user gets into the vehicle 3 and immediately starts driving, thereby improving safety.

[0033] In addition, since the around view image 20 is displayed near the position of the door handle 26 of the right front door 25 on the driver's seat side, when the driver tries to open the right front door 25 from outside the vehicle and views the door handle 26, the around view image 20 is displayed near the line of sight. This allows the driver to easily view the around view image 20 when getting into the driver's seat of the vehicle 3.

[0034] In particular, in a vehicle 3 equipped with a visual door locking / unlocking device, the driver visually recognizes the door locking / unlocking mark 30 near the driver's side door handle 26 when locking or unlocking the vehicle 3, so that when unlocking the driver's side right front door 25, the driver can easily and for a relatively long period of time view the around view image 20 and adequately confirm obstacles. As shown in FIG. 4, the around view image 20 may be displayed on the ground at the parking position of the vehicle 3. The around view image 20 is displayed according to the position and size of the vehicle 3. When the around view image 20 is displayed on the ground, the vehicle 3 in the field of view image is displayed transparently. It is preferable that the vehicle 3 is not made completely transparent, but is made semi-transparent to the extent that the outer shape of the vehicle 3 can be identified. In the head mounted display unit 10 of this embodiment, the user's field of view image is captured by the camera 14 and displayed on the head mounted display 13, so that the vehicle 3 can be displayed transparently as shown in FIG. 4.

[0035] Then, the around-view image 20 is displayed on the ground at the position of the vehicle 3 through which the vehicle 3 is visible, so that an obstacle present in a position hidden by the vehicle 3 can be visually recognized from outside the vehicle 3. In addition, since the around-view image 20 is displayed according to the position and size of the vehicle 3, it is possible to visually confirm an obstacle in its actual position and actual size, and therefore the obstacle can be accurately recognized.

[0036] The present invention is not limited to the above-described embodiments. For example, the display position and size of the around-view image 20 may be set appropriately. In the above embodiment, the around-view image 20 is displayed on the vehicle 3 and its surroundings as viewed from outside the vehicle. However, the around-view image 20 may also be displayed inside the vehicle cabin, for example, as shown in FIG.

[0037] Figure 5 is a fifth display example of the around-view image 20, which is a display example when a driver wearing a head-mounted display unit 10 and seated in the driver's seat 31 of the vehicle 3 looks at the left front door 32 of the vehicle 3 from inside. In a vehicle 3 such as an automobile, the driver has difficulty seeing the left side of the vehicle on the passenger seat 33 side, as it is hidden behind the left front door 32.

[0038] 5, by displaying the around-view image 20 on the left front door 32, the side of the vehicle 3, including the left side of the vehicle, can be visually confirmed. Since the around-view image 20 is displayed by the head-mounted display unit 10, it becomes possible to arbitrarily display the around-view image 20 at any position of the vehicle 3, and therefore it becomes possible to display the around-view image 20 at an appropriate position as required, thereby improving the visibility of the around-view image 20.

[0039] 4, when a user wearing the head mounted display unit 10 is present inside the vehicle cabin, the vehicle 3 may be made semi-transparent and the around-view image 20 may be displayed on the ground around the vehicle 3 in accordance with the position and size of the vehicle 3. In the head mounted display unit 10 that displays the around-view image 20 in accordance with the size and position of the vehicle 3 while being transparent through the vehicle 3 in this manner, the around-view image 20 may be displayed only in a location that is difficult to see directly due to the left front door 32, for example, on the left side of the vehicle when viewed from inside the vehicle cabin.

[0040] Also, within the vehicle cabin, an icon showing an around-view image may be displayed, for example, on the upper part of the dashboard, and when a user wearing the head mounted display unit 10 looks at this icon for a predetermined period of time or longer, the around-view image 20 may be displayed at the position of this icon. When the user's line of sight moves away from the icon position, the display of the around-view image 20 may be stopped and the display may be restored to the icon.

[0041] In addition, in the above embodiment, when a specified position of vehicle 3 is within the user's field of view and the head-mounted display unit 10 is located near vehicle 3, a mixed reality image in which an image of the vehicle's surroundings is superimposed at a specified position is displayed on the head-mounted display 13. However, even if the specified position of vehicle 3 is not within the user's field of view, a mixed reality image may be displayed when the head-mounted display unit 10 is located near vehicle 3.

[0042] Furthermore, the configuration of the head mounted display unit 10 may also be changed as appropriate. The present invention can be widely applied to units including an image display unit that are mounted or carried by a user of a vehicle equipped with an around-view camera. [Explanation of symbols]

[0043] 1 Vehicle display system 3. Vehicle 10 Head-mounted display unit (head-mounted display device, mobile terminal, image processing system) 11 Control unit (control unit) 13 Head-mounted display (image display unit) 14 Camera (imaging unit) 15 Display side communication device (communication unit) 17 Image synthesis section 18 Position detection unit

Claims

1. a mobile terminal having an imaging unit for capturing an image, an image display unit for displaying the image, and a communication unit for communicating with a vehicle; a control unit which has an image synthesis unit which combines images of the surroundings of the vehicle acquired via the communication unit to generate a vehicle surroundings image of the vehicle viewed from above, and generates a mixed reality image by superimposing the imaged image captured by the imaging unit, and when the mobile terminal is located within a predetermined range near the vehicle and the line of sight of a user wearing the mobile terminal is located near a door of the vehicle, the image synthesis unit superimposes the vehicle surroundings image on an outer wall surface of the vehicle or an outer wall surface of the door of the vehicle in the captured image to generate the mixed reality image, and displays the mixed reality image generated by the image synthesis unit on the image display unit; An image processing system comprising:

2. 2. The image processing system according to claim 1, wherein the image synthesis unit generates the mixed reality image by making the vehicle in the captured image transparent and superimposing the vehicle surroundings image on the ground at the position of the vehicle.

3. 3. The image processing system according to claim 1, wherein the mobile terminal is a head-mounted display device that is mounted on a head.

4. The image processing system according to claim 3, characterized in that when the head-mounted display device is worn and the vehicle is within the field of view and the vehicle is located near the vehicle, the mixed reality image is displayed on the image display unit.

Images