Image processing device for vehicle

Abstract

Provided is a vehicle image processing device capable of confirming an object included in a range seen by a user. The vehicle image processing device (1) comprises: a camera (10) for photographing the surroundings of the vehicle and outputting image data; an image extraction unit (61) for extracting an image from the image data when a prescribed operation for instructing image acquisition is performed; an image processing unit (63) for processing the extracted image by matching it with the field of view of the user in the vehicle; and a display processing unit (64) and a display device (40) for displaying the processed image on a screen arranged at a position matching the field of view of the user.

Description

Technical Field

[0001] The present invention relates to a vehicle image processing device for capturing and displaying surrounding scenery viewed from a vehicle interior. Background Art

[0002] Conventionally, there is known a vehicle surrounding image providing device that performs coordinate transformation on an image captured by a photographing device based on information on a viewpoint position obtained and information on a setting state of a liquid crystal display so that an image identical to the situation outside the vehicle seen from the driver's viewpoint through the installation area of ​​the liquid crystal display is displayed on the liquid crystal display (for example, refer to Patent Document 1). By using this vehicle surrounding image providing device, an image equivalent to the scenery seen in the driver's line of sight can be displayed on the liquid crystal display.

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2007-96638 Summary of the invention

[0006] Problems to be solved by the invention

[0007] However, in the vehicle surrounding image providing device disclosed in the above-mentioned patent document 1, since only the image photographed by the photographing device is displayed on the liquid crystal display, the content of the image changes all the time when the vehicle is moving, and there is a problem that it is not easy to grasp the content. In particular, the closer the object is to the vehicle, the shorter the time it stays in the display range. Even if the driver wants to confirm the object, it immediately becomes invisible and the object in the display range cannot be confirmed.

[0008] The present invention has been made in view of the above point, and an object of the present invention is to provide an image processing device for a vehicle that can confirm an object included in a range viewed by a user.

[0009] Means for solving problems

[0010] In order to solve the above-mentioned problems, the vehicle image processing device of the present invention comprises: a camera unit that captures the surroundings of the vehicle and outputs image data; an image extraction unit that extracts an image from the image data when a prescribed operation for instructing image acquisition is performed; an image processing unit that processes the image extracted by the image extraction unit by matching it with the field of view of the user in the vehicle; and an image display unit that displays the image processed by the image processing unit on a screen arranged at a position that matches the user's line of sight. By performing a prescribed operation when an object that the user wants to confirm enters the field of view, an image containing the object can be displayed at a position that matches the user's line of sight, so it is easy to confirm an object contained in the range seen by the user.

[0011] In addition, it is desirable to further include: an image data storage unit that stores image data obtained by the above-mentioned camera unit over a predetermined period of time, and an image extraction unit that extracts images from the image data stored by the image data storage unit. By storing the image data and extracting the image, it is possible to avoid the situation where the image of the object to be confirmed is not captured in time.

[0012] Furthermore, it is desirable that the image extraction means can change the timing of extracting an image from the video data based on the vehicle's running speed, thereby preventing an object to be confirmed from being out of the imaging range due to a user's operation delay.

[0013] Furthermore, it is desirable that the image extraction means described above allows the timing of extracting an image from the video data to be changed based on a user's instruction, thereby enabling retroactive confirmation of an object that has entered the field of view.

[0014] In addition, it is desirable to further include: a sight line detection unit that detects the sight line of the user, and an image processing unit that generates an image corresponding to the user's field of view based on the sight line detected by the sight line detection unit. By extracting an image matching the direction the user is facing, processing and displaying it, it is possible to reproduce an image of a range actually seen by the user that is included in the user's field of view.

[0015] In addition, it is desirable to further include: an image recognition unit that performs image recognition on the image displayed by the above-mentioned image display unit to extract a partial image included in the image; and an information acquisition unit that acquires information related to the partial image extracted by the image recognition unit, and the image display unit displays the information acquired by the information acquisition unit on the screen in correspondence with the partial image. In this way, information about an object included in the user's field of view can be acquired without performing complicated operations.

[0016] In addition, it is desirable that the image display unit displays the partial image whose information is acquired by the information acquisition unit in a recognizable manner, and displays the information acquired by the information acquisition unit for the partial image whose display is instructed by the user. Thus, it is possible to prevent the screen from being flooded with information and deteriorating visual recognition, and to quickly display the information that the user wants to see. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a diagram showing a configuration of a vehicle image processing device according to an embodiment.

[0018] Figure 2 This is a flowchart showing an operation process of displaying an image identical to the scenery viewed by the user through the window of the vehicle on the window.

[0019] Figure 3 This is a flowchart showing the operation process of a modified example of performing image recognition on an image displayed on a window of a vehicle.

[0020] Description of symbols

[0021] 1 Image processing device for vehicle

[0022] 10.12 Camera

[0023] 20 Image data storage unit

[0024] 30 Vehicle speed detection unit

[0025] 32 Vehicle position and orientation detection unit

[0026] 40 Display device

[0027] 50 Map DB

[0028] 52 POI DB

[0029] 54 General DB

[0030] 60 Control Department

[0031] 61 Image Extraction Unit

[0032] 62 Eyesight Detection Unit

[0033] 63 Image Processing Department

[0034] 64 Display processing unit

[0035] 65 Image recognition unit

[0036] 66 Information Retrieval Department

[0037] 67 Operation detection unit DETAILED DESCRIPTION

[0038] Hereinafter, a vehicle image processing device according to an embodiment of the present invention will be described with reference to the drawings.

[0039] Figure 1 It is a diagram showing a configuration of a vehicle image processing device according to an embodiment. Figure 1 The vehicle image processing device 1 of the present embodiment shown in the figure displays, for example, an image similar to the scenery seen by passengers in the rear seats of the vehicle through the window on the window, stops the image (moving image) as needed, or provides information on objects (buildings, etc.) as display elements included in the image. Therefore, the vehicle image processing device 1 includes cameras 10 and 12, an image data storage unit 20, a vehicle speed detection unit 30, a vehicle position and orientation detection unit 32, a display device 40, a map DB (database) 50, a POI DB 52, a general DB 54, and a control unit 60.

[0040] The camera 10 captures the surroundings of the vehicle and outputs image data (moving images). For example, the camera 10 is installed on the right side of the vehicle and captures a wide range of scenery extending on the right side of the vehicle. The camera 12 is installed in the vehicle interior and captures the user's face.

[0041] The image data storage unit 20 stores the image data output from the camera 10 for a predetermined time. The latest image data corresponding to the predetermined time is stored in the image data storage unit 20, and the previous image data is deleted (overwritten).

[0042] The vehicle speed detection unit 30 detects the speed of the vehicle (vehicle speed).

[0043] The vehicle position and orientation detection unit 32 detects the position and orientation of the vehicle. The detection method may be a case where a GPS receiver is used, a case where an autonomous navigation sensor is used, or a case where a combination of these is used.

[0044] The display device 40 displays an image (moving image or still image) on the rear seat window (rear door window), for example, by projecting an image on a window glass with variable transparency called a smart window, or by using a transparent organic EL display or a transparent inorganic EL display.

[0045] The map DB 50 stores map data such as roads and their surrounding terrain and locations of facilities.

[0046] The POI DB 52 stores detailed information of facilities or natural objects (such as mountains) as POIs (Point of Interest). The POI DB 52 may be included in the map DB 50.

[0047] The general DB 54 stores detailed information on various objects included in the user's field of vision. For example, it contains information related to various objects other than facilities and natural objects stored in the POIDB 52 (which may be repeated). Among the various objects, tangible objects other than the POIs whose information is stored in the POI DB 52 are widely included. The general DB 54 may also be included in the POI DB 52.

[0048] The control unit 60 controls the entire vehicle image processing device 1, and performs an operation to display the image captured by the camera 10 on the display device 40, or to display information related to the POI or various objects contained in the image. Therefore, the control unit 60 includes an image extraction unit 61, a sight line detection unit 62, an image processing unit 63, a display processing unit 64, an image recognition unit 65, an information retrieval unit 66, and an operation detection unit 67. The control unit 60 can realize various functions by executing a predetermined action program using a CPU, a ROM, a RAM, etc.

[0049] The image extraction unit 61 extracts an image from the image data stored in the image data storage unit 20 when a predetermined operation (described later) is performed by the user.

[0050] The sight line detection unit 62 detects the position of the user's eyes and the direction in which they are looking (the position of the viewpoint and the direction of the sight line) based on the user's facial image captured by the camera 12 .

[0051] The image processing unit 63 processes the image extracted by the image extraction unit 61 to match the user's field of view. Specifically, the image processing unit 63 converts the viewpoint of the image extracted by the image extraction unit 61 to generate an image corresponding to the scenery viewed by the user through the screen of the display device 40 .

[0052] The display processing unit 64 displays the image generated by the image processing unit 63 on a screen (screen of the display device 40 ) arranged at a position matching the user's line of sight.

[0053] The image recognition unit 65 performs image recognition on the image generated by the image processing unit 63 and displayed on the screen of the display device 40 , and extracts a partial image included in the image.

[0054] The information search unit 66 searches and acquires detailed information of an object corresponding to the partial image extracted by the image recognition unit 65 using the map DB 50, POI DB 52, and general DB 54. The search may be delegated to an external search server connected via the Internet.

[0055] The operation detection unit 67 detects the operation performed by the user. For example, the screen of the display device 40 is included in the imaging range of the camera 12, and the pressing action of an operation button on the screen performed by the user is detected, or a transparent touch panel is arranged on the screen of the display device 40 to detect the pressing action of an operation button on the screen.

[0056] The above-mentioned camera 10 corresponds to the imaging component, the image extraction unit 61 corresponds to the image extraction component, the image processing unit 63 corresponds to the image processing component, the display processing unit 64 and the display device 40 correspond to the image display component, the image data storage unit 20 corresponds to the image data storage component, the camera 12 and the line of sight detection unit 62 correspond to the line of sight detection component, the image recognition unit 65 corresponds to the image recognition component, and the information retrieval unit 66 corresponds to the information acquisition component.

[0057] The vehicle image processing device 1 according to the present embodiment has such a configuration, and its operation will be described below.

[0058] Figure 2 The flowchart shows the operation process of displaying the same image as the scenery seen by the user through the window of the vehicle. In parallel with the operation process, the operation of storing the image data obtained by the camera 10 in the image data storage unit 20 is performed.

[0059] The image extraction unit 61 determines whether a user's finger has pressed (or touched) a "capture" button displayed on a portion of a screen using window glass in response to a predetermined operation for obtaining and displaying an image by the user (step 100). If the button has not been pressed, a negative determination is made, and this determination is repeated.

[0060] In addition, if the "Capture" button is pressed, a positive judgment is made in the determination of step 100. Next, the image extraction unit 61 extracts an image corresponding to the timing when the "Capture" button is pressed from the image data stored in the image data storage unit 20 (step 102). A delay occurs from the time when the user who sees the scenery outside through the car window recognizes the scenery of interest to the time when the "Capture" button is pressed. The image extraction unit 61 considers the delay and the vehicle speed at that time and determines one image to be extracted from the image data.

[0061] Next, the image processing unit 63 processes the extracted image by performing viewpoint conversion and the like to match the user's field of view (step 104). The display processing unit 64 displays the processed image on the screen (window glass) of the display device 40 (step 106). By viewing the displayed image in this way, the scenery seen at the place where the vehicle passes in an instant while driving can be reproduced, and the user can confirm various objects that have entered the field of view at that moment.

[0062] Thus, in the vehicle image processing device 1 of the present embodiment, by performing a predetermined operation (pressing the "Capture" button) when an object that the user wants to confirm enters the field of view, an image containing the object can be displayed at a position that matches the user's line of sight, so that it is easy to confirm the object contained in the range seen by the user. In particular, by extracting the image that stores the captured image data, it is possible to avoid a situation where the object to be confirmed is not captured in time. In addition, by making the timing of extracting the image from the image data variable based on the driving speed of the vehicle, it is possible to prevent the object to be confirmed from being out of the camera range due to a delay in the user's operation.

[0063] Furthermore, by detecting the viewpoint position or the direction of the line of sight by the line of sight detection unit 62 and extracting an image matching the direction the user is facing, processing and displaying the image, it is possible to reliably reproduce an image of a range actually seen by the user and included in the user's field of vision.

[0064] Figure 3 This is a flowchart showing the operation process of a modified example of performing image recognition on an image displayed on a window of a vehicle. Figure 3 The action process shown is relative to Figure 2 The operation process shown is different in that steps 110, 112, 114, 116, and 118 are added after step 106. Each of these added steps will be described below.

[0065] After the processed image is displayed (step 106 ) (or in parallel with or before the display operation), the image recognition unit 65 performs image recognition on the displayed image and extracts the partial image corresponding to the POI or other various objects contained in the image (step 110 ).

[0066] Furthermore, the information search unit 66 uses the map DB 50 , the POI DB 52 , and the general DB 54 to acquire detailed information of an object corresponding to the partial image extracted by image recognition (step 112 ).

[0067] Next, the information search unit 66 transmits an instruction to highlight the partial images for which detailed information has been obtained to the display processing unit 64, and highlights these partial images (step 114). For example, the corresponding partial images are highlighted or given special colors so that they can be displayed in a manner that can be identified from others.

[0068] Next, the information search unit 66 determines whether one of the highlighted partial images has been designated by the user (step 116). If no designation operation has been performed by the user, a negative determination is made, and this determination is repeated.

[0069] If one of the highlighted partial images is pointed out by the user, a positive judgment is made in the determination of step 116. The information search unit 66 displays detailed information of the partial image corresponding to the pointed out (step 118). For example, a window containing detailed information is displayed at a position adjacent to the partial image.

[0070] In this way, detailed information is displayed for a partial image extracted by image recognition, so that information about an object included in the user's field of view can be obtained without complicated operations. In addition, the partial image for which detailed information has been obtained is displayed in a recognizable manner, and detailed information is displayed when the user instructs to display information, so that detailed information is prevented from being overflowed in the screen and visual confirmation is deteriorated, and information that the user wants to see can be quickly displayed.

[0071] In addition, the present invention is not limited to the above-mentioned embodiments, and various modified embodiments can be performed within the scope of the gist of the present invention. For example, in the above-mentioned embodiments, the timing of extracting images from image data is determined based on the vehicle speed, etc., but it can also be made variable based on the user's instructions. The time from identifying the scene of interest to pressing the "Capture" button varies from user to user, so as long as the deviation of the image extraction timing based on the individual difference can be adjusted, the practicality can be further improved. For example, in Figure 2 In step 106 shown, the processed image is displayed, but along with this display, a button for shifting the timing of extracting the image from the video data by seconds is also displayed, and when the user presses this button, the timing of extracting the image is shifted by the number of times the user operates it, and the actions of steps 104 and 106 are repeated. In this way, the object that has entered the field of view can be reliably traced and confirmed.

[0072] In the above embodiment, the corresponding image is displayed by initially pressing the "Capture" button, but the pressing operation may be replaced by a voice instruction. For example, consider the case where the user's voice is determined to be "capture" through voice recognition processing and the image is extracted.

[0073] In addition, in the above-mentioned embodiment, the user's viewpoint position and the direction of the field of vision are detected based on the user's facial image captured by the camera 12, and image processing such as viewpoint transformation is performed by the image processing unit 63, but it can also be imagined that the user sits at a specified position and sees the outside scenery through the window glass, and image processing is performed using a fixed viewpoint position and line of sight direction.

[0074] Industrial Applicability

[0075] As described above, according to the present invention, when an object that the user wants to confirm enters the field of view, a predetermined operation is performed to display an image including the object at a position matching the user's line of sight, thereby facilitating confirmation of objects included in the user's visual range.

Claims

1. A vehicle image processing device, characterized in that: have: A camera component that captures the surroundings of the vehicle and outputs image data; an image extraction unit that extracts an image from the video data when a predetermined operation instructing image acquisition is performed; an image processing unit that processes the image captured by the image capturing unit by matching it with the field of view of the user in the vehicle cabin; an image display unit that displays the image processed by the image processing unit on a screen that is arranged at a position that matches the user's line of sight, an image recognition component that performs image recognition on the image displayed by the image display component and extracts a partial image contained in the image; and an information acquisition unit for acquiring information related to the partial image extracted by the image recognition unit, The image display unit displays the partial image whose information has been acquired by the information acquisition unit in a recognizable manner, and displays the information acquired by the information acquisition unit on the screen in correspondence with the partial image for which display of information has been instructed by the user.

2. The vehicle image processing device according to claim 1, wherein: Also available: an image data storage unit for storing the image data obtained by the imaging unit over a predetermined period of time, The image extracting means extracts an image from the image data stored by the image data storing means.

3. The vehicle image processing device according to claim 2, wherein: The image extraction means allows a timing of extracting an image from the video data to be variable based on a traveling speed of a vehicle.

4. The vehicle image processing device according to claim 2, wherein: The image extraction unit allows a timing of extracting an image from the video data to be changed based on a user's instruction.

5. The vehicle image processing device according to claim 1, wherein: Also available: The sight line detection component detects the user's sight line. The image processing unit generates an image corresponding to the visual field of the user based on the visual line detected by the visual line detection unit.

Images