Image processing device, mobile device, method and program

Abstract

An image processing device is provided that implements a configuration in which an output image on a display unit is switched and displayed according to a driver's behavior such as movement of the driver's head. Driver information indicating the behavior of the driver of a mobile device and captured images of a plurality of cameras capturing the surroundings of the mobile device from different viewpoints are input, and the output image on the display unit is switched according to the driver information. The plurality of cameras are, for example, a plurality of rear cameras mounted at the rear of the mobile device, and detect, for example, the driver's face or sight direction. An image in a direction corresponding to the detected driver's face or sight direction is selected as an output image to be displayed on the display unit, or an image in a direction indicated by a gesture of the driver is selected to be displayed on the display unit.

Description

Technical Field

[0001] The present disclosure relates to an image processing device, a mobile device, a method and a program. More specifically, the present disclosure relates to an image processing device, a mobile device, a method and a program for controlling the output of a display image to a display unit (monitor) at a driver's seat based on an image captured by a camera installed in a mobile device such as a car. Background Art

[0002] Nowadays, a camera monitoring system (CMS) can be installed and CMS has begun to be actually introduced into cars. In CMS, an image captured by a camera installed in a car is displayed on a monitor at the driver's seat, and the situation around the car, for example, the situation behind the car (in the rear direction) can be checked.

[0003] For example, Patent Document 1 (WO 2018 / 012299) discloses a camera monitoring system (CMS) in the present vehicle. In the CMS, a rear image of the vehicle captured by a rear camera installed at the rear of the vehicle, a right rear image and a left rear image captured by side cameras installed at rearview mirror positions on the right and left sides of the vehicle, or a composite image thereof, are displayed on a display unit.

[0004] Reference List

[0005] Patent Literature

[0006] Patent Document 1: WO 2018 / 012299 Summary of the invention

[0007] Problems to be solved by the present invention

[0008] A camera monitoring system (CMS) enables checking a surrounding state that has been checked with a backview mirror, such as a so-called rearview mirror and a sideview mirror of a conventional car, by using an image captured by a camera.

[0009] For example, three images of a rear captured image, a left rear captured image, and a right rear captured image are displayed separately on a display unit (monitor) that can be observed by the driver. The rear captured image is captured by a rear camera that images the rear of the vehicle. The left rear captured image and the right rear captured image are captured by a left camera and a right camera installed near the positions of the left and right mirrors of the vehicle, respectively. Alternatively, a single composite image generated by appropriately synthesizing these three images is displayed on the monitor.

[0010] However, in such an image display system, in the case of generating and displaying a continuous composite image from three images captured by the rear camera and the left and right side cameras, a subject captured in any of the three images sometimes disappears instead of being displayed. This is because the images captured by the three cameras of the rear camera, the L side camera, and the R side camera have different viewpoints, and this is caused by, for example, a process of converting an image from one viewpoint (e.g., the viewpoint of the rear camera) when generating a composite image.

[0011] If a synthetic image is generated with respect to the viewpoint of the rear camera when generating the synthetic image, the subjects captured by the left camera and the right camera (such as, for example, a motorcycle following closely behind a large vehicle and a vehicle approaching at high speed from a distance) disappear from the synthetic image. This is because, for example, the motorcycle is located in the area behind the large vehicle, that is, an occluded area from the viewpoint of the rear camera.

[0012] The present disclosure is made in view of problems such as the above, and an object of the present disclosure is to provide an image processing device, a mobile device, a method, and a program, wherein by sequentially switching and displaying images captured by multiple cameras with different imaging viewpoints according to the driver's behavior (such as movement of the head), the driver can almost continuously (seamlessly) check the rear side images observed from various viewpoints, similar to the case of checking conventional rearview mirrors and side mirrors.

[0013] Solution to the problem

[0014] A first aspect of the present disclosure is an image processing device including a data processing unit, the data processing unit performing image display control of the following contents:

[0015] enter

[0016] driver information showing behavior of a driver of the mobile device; and

[0017] images captured by a plurality of cameras that image conditions surrounding the mobile device from different viewpoints; and

[0018] According to the driver information, the image output to the display unit is switched to any image among the images captured by the plurality of cameras or a virtual viewpoint synthesized image.

[0019] In addition, a second aspect of the present disclosure is a mobile device, comprising:

[0020] a plurality of cameras that image conditions around the mobile device from different viewpoints;

[0021] a driver information detection unit that detects driver information showing behavior of a driver of the mobile device; and

[0022] A data processing unit inputs driver information and images captured by the plurality of cameras, and switches an image output to a display unit to any image among the images captured by the plurality of cameras or a virtual viewpoint synthesized image according to the driver information.

[0023] In addition, a third aspect of the present disclosure is an image processing method executed in an image processing device,

[0024] The data processing unit performs the following image display control:

[0025] enter

[0026] driver information showing behavior of a driver of the mobile device; and

[0027] images captured by a plurality of cameras that image conditions surrounding the mobile device from different viewpoints; and

[0028] According to the driver information, the image output to the display unit is switched to any image among the images captured by the plurality of cameras or a virtual viewpoint synthesized image.

[0029] In addition, a fourth aspect of the present disclosure is a display image control method executed in a mobile device, comprising:

[0030] an image capturing step, wherein a plurality of cameras images conditions surrounding the mobile device from different viewpoints;

[0031] a driver information detecting step in which a driver information detecting unit detects driver information showing a behavior of a driver of the mobile device; and

[0032] An image display control step, in which a data processing unit inputs driver information and images captured by the multiple cameras, and switches the image output to the display unit to any image among the images captured by the multiple cameras or a virtual viewpoint synthesized image according to the driver information.

[0033] Furthermore, a fifth aspect of the present disclosure resides in a program for executing image processing in an image processing apparatus, the program causing a data processing unit to execute image display control of:

[0034] enter

[0035] driver information showing behavior of a driver of the mobile device; and

[0036] images captured by a plurality of cameras that image conditions surrounding the mobile device from different viewpoints; and

[0037] According to the driver information, the image output to the display unit is switched to any image among the images captured by the plurality of cameras or a virtual viewpoint synthesized image.

[0038] Note that the program of the present disclosure can be provided to an information processing device and a computer system capable of executing various program codes by a storage medium or a communication medium capable of providing various program codes in a computer-readable format. By providing such a program in a computer-readable format, processing according to the program is performed on the information processing device and the computer system.

[0039] Based on the more detailed description of the embodiments and drawings described later in the present disclosure, other objects, features and advantages of the present disclosure will become apparent. Note that the system in the specification has a configuration of a logical set of multiple devices. The system is not limited to a system in which devices with various configurations are placed in the same housing.

[0040] Effects of the Invention

[0041] According to the configuration of one embodiment of the present disclosure, a configuration is achieved in which an image output to a display unit is switched and displayed according to a driver's behavior such as movement of the driver's head.

[0042] Specifically, for example, driver information showing the behavior of the driver of the mobile device and images captured by multiple cameras that image the conditions around the mobile device from different viewpoints are input. The image output to the display unit is switched according to the driver information. The multiple cameras are, for example, multiple rear cameras installed at the rear of the mobile device. For example, the driver's face or sight direction is detected. An image in a direction corresponding to the detected driver's face or sight direction is selected as an output image and displayed on the display unit. Alternatively, an image in a direction indicated by the driver's gesture is selected and displayed on the display unit.

[0043] This configuration realizes a configuration in which an image output to the display unit is switched and displayed according to the driver's behavior, such as movement of the driver's head.

[0044] Note that the effects described in the specification are merely exemplary and not restrictive, and additional effects may be exhibited. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A camera monitoring system (CMS) installed in a car is outlined.

[0046] Figure 2 The diagram shows an example of using CMS.

[0047] Figure 3 The diagram shows an example of using CMS.

[0048] Figure 4 The diagram shows an example of using CMS.

[0049] Figure 5 The configurations of the mobile device and the image processing device of the present disclosure and the processes therein are illustrated.

[0050] Figure 6 The configurations of the mobile device and the image processing device of the present disclosure and the processes therein are illustrated.

[0051] Figure 7 The configurations of the mobile device and the image processing device of the present disclosure and the processes therein are illustrated.

[0052] Figure 8 An example of image display on the display unit is illustrated.

[0053] Fig. 9 An example of image display on the display unit is illustrated.

[0054] Fig.10 An example of image display on the display unit is illustrated.

[0055] Fig.11 An example of image display on the display unit is illustrated.

[0056] Fig.12 The configurations of the mobile device and the image processing device and the processes therein are illustrated.

[0057] Fig.13 An example of image display on the display unit is illustrated.

[0058] Fig.14 An example of image display on the display unit is illustrated.

[0059] Fig.15 An example of a camera arrangement in a mobile device is illustrated.

[0060] Fig.16 An example of a camera arrangement in a mobile device is illustrated.

[0061] Fig.17 An example of a camera arrangement and displayed images in a mobile device is illustrated.

[0062] Fig.18 An example of a camera arrangement and displayed images in a mobile device is illustrated.

[0063] Fig.19 An example of a camera arrangement and displayed images in a mobile device is illustrated.

[0064] Fig. 20An example of a camera arrangement and displayed images in a mobile device is illustrated.

[0065] Fig.21 An example of a camera arrangement and displayed images in a mobile device is illustrated.

[0066] Fig. 22 An example of a camera arrangement in a mobile device is illustrated.

[0067] Fig.23 is a flowchart illustrating a processing sequence performed by the image processing apparatus.

[0068] Fig.24 is a flowchart illustrating a processing sequence performed by the image processing apparatus.

[0069] Fig.25 An example of the hardware configuration of the image processing apparatus is illustrated. DETAILED DESCRIPTION

[0070] The details of the image processing device, mobile device, method, and program of the present disclosure will be described below with reference to the drawings. Note that the description will be given according to the following items.

[0071] 1. Overview of Car Camera Monitoring System (CMS)

[0072] 2. Configuration Examples of the Image Processing Device and Mobile Device of the Present Disclosure

[0073] 3. Other embodiments

[0074] 4. Image processing device and processing sequence performed by mobile device

[0075] 5. Configuration example of image processing device

[0076] 6. Summary of the configuration of this disclosure

[0077] [1. Overview of Car Camera Monitoring System (CMS)]

[0078] First, refer to Figure 1 and subsequent figures outline a camera monitoring system (CMS) installed in a car.

[0079] Figure 1 An overview of a CMS installed in a car 10 is given. The CMS displays an image captured by a camera installed on the car on a display unit that a driver can observe.

[0080] Figure 1The car 10 in the figure includes a rear camera 11, a left camera 12, and a right camera 13. The rear camera 11 is installed at the rear of the car 10. The left camera 12 is installed on the left side of the car 10. The right camera 13 is installed on the right side of the car 10. Each of these cameras captures an image behind the car 10 as viewed from the installation position of each camera.

[0081] Images captured by these cameras or synthetic images generated based on these captured images are displayed on the display unit 20 inside the car 10. The driver of the car 10 can check the status behind the car 10 by viewing the displayed image on the display unit 20.

[0082] Will refer to Figure 2 20 and subsequent figures describe specific image examples displayed on the display unit 20. Figure 2 The figure shows the state behind the car 10. Figure 2 In the figure, the automobile 10 is traveling in the direction of the arrow (forward).

[0083] In this state, the three cameras of the rear camera 11 , the left camera 12 , and the right camera 13 capture images with each camera position as a viewpoint position.

[0084] Figure 3 An example of an image displayed on the display unit 20 that the driver of the car 10 can check is illustrated. Figure 3 An example is illustrated in which images captured by three cameras of the rear camera 11, the left camera 12, and the right camera 13 are individually displayed on the display unit 20. On the display unit 20, a rear camera image 21 captured by the rear camera 11 is displayed in the center of the display unit 20, a left camera image 22 captured by the left camera 12 is displayed on the left side of the display unit 20, and a right camera image 23 captured by the right camera 13 is displayed on the right side of the display unit 20.

[0085] The driver can check the surrounding conditions by viewing these three images in a similar way to viewing the rearview mirror and the left and right side mirrors.

[0086] However, when three images are displayed side by side in this way, the driver mainly pays attention to the central image corresponding to the rearview mirror. If the driver does not pay attention to the left and right images, the driver tends to ignore the information mainly reflected in the left and right images.

[0087] exist Figure 3 In the example of FIG. 2 , the bicycle reflected in the left camera image 22 is hidden behind the large vehicle in the central rear camera image 21 and is almost invisible. In this case, if the driver pays attention only to the rear camera image 21 displayed in the center of the display unit 20, the driver may be unaware of the danger of the bicycle.

[0088] Although Figure 3 An example is illustrated in which images captured by the cameras are individually displayed on the display unit 20 , but a configuration in which a composite image is generated by combining these three images into one image and displaying on the display unit 20 has been proposed and used. Figure 4 An example of display of a composite image is illustrated.

[0089] Figure 4 An example of a composite image 25 displayed on the display unit 20 is illustrated. The composite image 25 is generated with the viewpoint of the rear camera 11 as a reference. The rear camera 11, the left camera 12, and the right camera 13 capture images from different viewpoints, so that, for example, processing of converting the viewpoint position of each image, processing of converting the size of the object (subject), and processing of adjusting the position are necessary to generate one composite image. Specifically, image conversion processing such as, for example, affine transformation is performed.

[0090] A plurality of images are combined through such image conversion processing to generate one composite image. Figure 4 An example of a composite image generated by converting images captured by the left camera 12 and the right camera 13 into a captured image from the viewpoint of the rear camera 11 is illustrated. However, in the case of generating such a composite image, a subject captured by the left camera or the right camera may disappear.

[0091] Although in the above Figure 3 The bicycle as the subject can be clearly identified in the left camera image 22 in FIG. 1 , but the bicycle is hidden behind the vehicle and is Figure 4 The bicycle is barely visible in the composite image 25 in FIG. This is because the bicycle is in the area behind the large vehicle in front of the vehicle, i.e., it is an occluded area from the viewpoint of the rear camera.

[0092] The driver cannot confirm the bicycle approaching from behind even if viewing the synthetic image 25. If the driver drives in a manner such as to suddenly turn left, an accident such as a contact with a bicycle approaching from behind may occur.

[0093] [2. Configuration Examples of the Image Processing Device and Mobile Device of the Present Disclosure]

[0094] Next, we will refer to Figure 4 The following figures describe configuration examples of the image processing device and the mobile device of the present disclosure. Figure 4 Specifically, for example, images captured by a plurality of cameras having different imaging viewpoints are sequentially switched and displayed according to the driver's behavior such as, for example, movement of the head, face, and sight line.

[0095] Figure 5One configuration example of a car 10 (mobile device) of the present disclosure is illustrated. Figure 5 The car 10 in the embodiment includes a left camera 12 on the left side of the car 10 and a right camera 13 on the right side of the car 10 in the same manner as described above with reference to FIG. Figure 1 The above-described method is similar. Three rear cameras are installed at the rear of the car 10, namely, a rear center (C) camera 31, a rear left (L) camera 32, and a rear right (R) camera 33. The rear C camera 31 is installed at the center of the rear of the car 10. The rear L camera 32 is installed on the left side of the rear of the car 10. The rear R camera 33 is installed on the right side of the rear of the car 10.

[0096] Note that, although three cameras are installed at the rear of the car 10 in this example, a plurality of various cameras other than three may be used. An example of using three or more cameras will be described later.

[0097] Figure 6 An example of the imaging areas of three cameras installed at the rear of the car 10 is illustrated. Figure 6 The imaging areas of three cameras of the rear center (C) camera 31, the rear left (L) camera 32, and the rear right (R) camera 33 are shown. The rear C camera 31 is installed at the center of the rear of the car 10. The rear L camera 32 is installed on the left side of the rear of the car 10. The rear R camera 33 is installed on the right side of the rear of the car 10.

[0098] like Figure 6 As shown, the three cameras are arranged to image the rear of the car from different viewpoints, and parts of the imaging areas overlap each other. In the configuration of the present disclosure, images captured by multiple cameras with such different imaging viewpoints are sequentially switched and displayed according to the driver's behavior (e.g., movement of the head). The movement of the head is detected to detect movement comparable to the driver's actions and movements when viewing conventional rearview mirrors and side mirrors and use this as a trigger for switching the display.

[0099] Figure 7 The diagram shows the Figure 2 Similar imaging environment. The car 10 is traveling in the direction of the arrow in the figure (forward).

[0100] A large vehicle is traveling behind the car 10 , and a bicycle is traveling to the left rear of the large vehicle.

[0101] Figure 8 An example of an image displayed on the display unit 20 that can be checked by the driver of the car 10 is illustrated. A left camera image 22 captured by the left camera 12 is displayed on the left side of the display unit 20. A right camera image 23 captured by the right camera 13 is displayed on the right side of the display unit 20.

[0102] The image display area in the central portion of the display unit 20 is set as a rear camera image switching display area 41. According to the driver's behavior (e.g., movement of the head), images captured by three cameras installed at the rear of the car 10 (i.e., a rear center (C) camera 31, a rear left (L) camera 32, and a rear right (R) camera 33) are sequentially switched and displayed on the rear camera image switching display area 41. The rear C camera 31 is installed at the center of the rear of the car 10. The rear L camera 32 is installed on the left side of the rear of the car 10. The rear R camera 33 is installed on the right side of the rear of the car 10.

[0103] Will refer to Fig. 9 An example of image display on the rear camera image switching display area 41, that is, an image switching display example is described.

[0104] Fig. 9 Illustrated is correspondence data between the movement of the head of the driver 50 (the direction of the face or the sight line) and an example of image display on the rear camera image switching display area 41. Three examples of image display are given below.

[0105] (R) Display example of images captured by the rear R camera

[0106] (C) Display example of an image captured by the rear C camera

[0107] (L) Display example of an image captured by the rear L camera

[0108] Fig. 9 An example of (R) display of an image captured by the rear R camera in a case where the face or sight of the driver 50 is in the right direction is illustrated. In this way, in a case where the driver 50 of the car 10 looks to the right, an image captured by the rear right (R) camera 33 installed on the right side of the rear of the car 10 is displayed on the rear camera image switching display area 41 of the display unit 20.

[0109] also, Fig. 9 An example of (C) display of an image captured by the rear C camera is illustrated in a case where the face or sight line of the driver 50 is approximately in the front direction. In this way, in a case where the driver 50 of the car 10 looks forward, an image captured by the rear center (C) camera 31 installed at the rear center of the car 10 is displayed on the rear camera image switching display area 41 of the display unit 20.

[0110] also, Fig. 9An example of (L) display of an image captured by the rear L camera in a case where the face or sight line of the driver 50 is in the left direction is illustrated. In this way, in a case where the driver 50 of the car 10 looks to the left, an image captured by the rear left (L) camera 32 installed on the left side of the rear of the car 10 is displayed on the rear camera image switching display area 41 of the display unit 20.

[0111] Note that, in the case where the driver 50 exhibits behavior such as first looking to the right, then looking forward, and finally looking to the left, the displayed image on the rear camera image switching display area 41 of the display unit 20 is switched sequentially as follows: First, the image captured by the rear right (R) camera 33 is displayed, switched to the image captured by the rear center (C) camera 31, and then switched to displaying the image captured by the rear left (L) camera 32.

[0112] A driver information detection unit that detects the behavior of the driver 50 is installed in the automobile 10. The image processing device of the automobile 10 performs control of switching the display image on the display unit 20 based on the detection information from the driver information detection unit.

[0113] exist Fig. 9 In the (L) display example of the rear L camera captured image described on the left side of the three images, the image of the bicycle can be clearly confirmed. This is because the rear left (L) camera 33 captures the image from the left side of the rear of the car 10 and can capture the image of the bicycle behind the large vehicle that is not hidden in front of the bicycle.

[0114] The driver 50 can clearly confirm that the bicycle is behind the large vehicle by viewing the image.

[0115] Note that although Fig. 9 Each image in can be compared with the above reference Figure 8 The two images in the display example of the display unit 20 described above are displayed together on the central area between the left camera image 22 and the right camera image 23, but depending on the face or sight direction of the driver 50, only one of the images may be independently displayed on the display unit 20. Fig. 9 , while the left camera image 22 and the right camera image 23 are not displayed.

[0116] Note that, Fig. 9 The diagram shows that the display image on the display unit 20 is switched according to the face or sight direction of the driver 50 (the display image is the image on the rear camera image switching display area 41, the rear camera image switching display area 41 is Figure 8By setting the rear camera image switching display area 41 to the central area of ​​the display unit 20 in the rear view image, the position of the rear view image switching display area 41 can be changed according to, for example, the face or sight direction of the driver 50. That is, the position of the rear camera image switching display area 41 is changed to a position that matches the face or sight direction of the driver 50.

[0117] Will refer to Fig.10 Describes an example of displaying an image position change. Fig. 9 similar, Fig.10 The diagram shows correspondence data between the movement of the head of the driver 50 (the direction of the face or the sight line) and an image display example on the rear camera image switching display area 41. Three examples of image display are given below.

[0118] (R) Display example of images captured by the rear R camera

[0119] (C) Display example of an image captured by the rear C camera

[0120] (L) Display example of an image captured by the rear L camera

[0121] Fig.10 The display unit 20 in the embodiment is disposed at an upper portion of a front panel of the automobile, and has a display area which is long in the left-right direction.

[0122] Fig.10 An example of (L) display of an image captured by the rear L camera in a case where the face or sight line of the driver 50 is in the left direction is illustrated. In a case where the driver 50 looks to the left, an image captured by the rear left (L) camera 32 installed on the left side of the rear of the car 10 is displayed on the left end portion of the display unit.

[0123] also, Fig.10 An example of (C) display of an image captured by the rear C camera is illustrated in a case where the face or sight line of the driver 50 is approximately in the front direction. In a case where the driver 50 looks forward, an image captured by the rear center (C) camera 31 installed in the rear center of the car 10 is displayed in the center portion of the display unit.

[0124] also, Fig.10 An example of (R) display of an image captured by the rear R camera in a case where the face or sight of the driver 50 is in the right direction is illustrated. In a case where the driver 50 looks to the right, an image captured by the rear right (R) camera 33 installed on the right side of the rear of the car 10 is displayed on the right end portion of the display unit.

[0125] Fig.10An example is illustrated in which the position of the rear camera image switching display area 41 is changed to a position that matches the face or sight direction of the driver 50. Such display position control allows the driver 50 to display an image that the driver 50 wants to see in the direction of his / her own sight. Therefore, the driver 50 can positively confirm the image in the direction that the driver 50 is paying attention to while keeping the sight in that direction.

[0126] Note that control of the image display position is also performed based on detection information from a driver information detection unit installed in the car 10 , which detects the behavior of the driver 50 .

[0127] Furthermore, separately from the face or sight line direction of the driver 50 , the display image may be changed according to, for example, instructions using a head movement sequence, a hand, or a finger of the driver 50 .

[0128] Will refer to Fig.11 An example in which the configuration of a display image is changed according to an instruction with the hand or finger of the driver 50 is described.

[0129] Fig.11 An example of (L) display of an image captured by the rear L camera when the finger of the driver 50 indicates the left direction is illustrated. When the finger of the driver 50 indicates the left, an image captured by the rear left (L) camera 32 installed on the left side of the rear of the car 10 is displayed on the display unit 20.

[0130] also, Fig.11 The figure shows an example of a (C) display of an image captured by the rear C camera when the driver's finger indicates a substantially frontal direction. When the driver's finger 50 indicates the front, an image captured by the rear center (C) camera 31 installed at the rear center of the automobile 10 is displayed on the display unit 20.

[0131] also, Fig.11 An example of (R) display of an image captured by the rear R camera when the finger of the driver 50 indicates the right is illustrated. When the finger of the driver 50 indicates the right, an image captured by the rear right (R) camera 33 installed on the right side of the rear of the car 10 is displayed on the display unit 20.

[0132] Fig.11 An example is illustrated in which an image displayed on the rear camera image switching display area 41 is changed according to an instruction (gesture) using the hand or finger of the driver 50. Note that a sequence of movements of the head can be detected as a gesture. Such control enables the driver 50 to change the image without significantly changing the line of sight direction.

[0133] Fig.12FIG. 1 shows a configuration example for image switching display control provided in a car 10 as a mobile device. Fig.12 As shown, the automobile 10 includes a driver information detection unit 52 that detects the face or sight direction of the driver 50 or an indication (gesture) using a hand or finger. Specifically, the driver information detection unit 52 includes, for example, a camera or a motion sensor. The detection information from the driver information detection unit 52 is input to the data processing unit 60.

[0134] The data processing unit 60 performs a process of switching an image displayed on the display unit 20 , and changes a display position based on detection information from the driver information detection unit 52 .

[0135] Note that the user (driver 50) can perform an operation of switching the image displayed on the display unit 20. For example, Fig.13 As shown, for example, the user operation unit may be displayed on an operation unit or a touch panel type display unit provided near the display unit 20. The user can operate the operation unit to switch the display image.

[0136] Fig.13 The (L) display example of the rear L camera captured image in FIG. 1 is an example of a display image on the display unit 20 when the user moves the operation unit to the left. In this case, an image captured by the rear left (L) camera 32 installed on the left side of the rear of the car 10 is displayed on the display unit 20.

[0137] The (C) display example of the rear C camera captured image is an example of a display image on the display unit 20 when the user moves the operation unit to the center. In this case, an image captured by the rear center (C) camera 31 installed at the rear center of the car 10 is displayed on the display unit 20.

[0138] The (R) display example of the rear R camera captured image is an example of a display image on the display unit 20 when the user moves the operation unit to the right. In this case, an image captured by the rear right (R) camera 33 installed on the right side of the rear of the car 10 is displayed on the display unit 20.

[0139] Note that the same Fig. 9 In this setup, the same as in the above reference Figure 8 As with the display example of the display unit 20 described above, Figures 9 to 13 Each image in the display unit 20 is displayed together with the left camera image 22 and the right camera image 23. In addition, according to the direction of the face or line of sight of the driver 50 and the indication, only the left camera image 22 and the right camera image 23 can be displayed independently on the display unit 20. Figures 9 to 13, while the left camera image 22 and the right camera image 23 are not displayed.

[0140] refer to Figures 9 to 13 The image switching display example described is a configuration example in which images captured by three cameras installed at the rear of the car 10 (i.e., a rear left (L) camera 32, a rear center (C) camera 31, and a rear right (R) camera 33) are switched and displayed based on the position of the driver's head / face, line of sight direction, or indication.

[0141] Furthermore, an image captured by the left-side camera 12 and an image captured by the right-side camera 13 may be added to the images captured by the three cameras as a target for switching images.

[0142] Will refer to Fig.14 A specific example is described.

[0143] Fig.14 An example of a display image switching sequence of the display unit 20 based on, for example, the face or sight line direction of the driver 50 , an instruction, or an input to the operation unit is illustrated.

[0144] For example, as the face or sight line direction of the driver 50 changes from right to left, the display images on the display unit 20 are sequentially switched in the following order.

[0145] (1) Right camera image captured by the right camera 13

[0146] (2) Rear R camera captured image captured by the rear R camera 33

[0147] (3) Rear C camera captured image captured by the rear C camera 31

[0148] (4) Rear L camera captured image captured by the rear L camera 32

[0149] (5) Left camera captured image captured by left camera 12

[0150] Furthermore, as the face or sight line direction of the driver 50 changes from left to right, the display images on the display unit 20 are sequentially switched in the order of (5) to (1) above.

[0151] [3. Other embodiments]

[0152] refer to Figures 5 to 14 The described embodiment is a configuration example in which images captured by three cameras installed at the rear of the automobile 10 (i.e., a rear left (L) camera 32, a rear center (C) camera 31, and a rear right (R) camera 33) are switched and displayed based on the driver's face or line of sight direction, using, for example, a head gesture indication, or an input to an operating unit.

[0153] Any number of cameras other than three cameras may be installed at the rear of the car 10 .

[0154] Fig.15 An example is illustrated in which five cameras are installed at the rear of the car 10 . Fig.15 The diagram is similar to reference Figures 5 to 14 An example of an embodiment is described. In this example, a rear left (L) camera 32, a rear center (C) camera 31, and a rear right (R) camera 33 are installed. In addition, a rear center left (CL) camera 34 and a rear center right (CR) camera 35 are provided. The rear CL camera 34 captures an image from an intermediate viewpoint between the rear L camera 32 and the rear C camera 31. The rear CR camera 35 captures an image from an intermediate viewpoint between the rear C camera 31 and the rear R camera 33.

[0155] Fig.16 Examples of the imaging areas of the five cameras are illustrated. Fig.16 The diagram shows imaging areas of five cameras, namely, a rear left (L) camera 32, a rear center left (CL) camera 34, a rear center (C) camera 31, a rear center right (CR) camera 35, and a rear right (R) camera 33. These five cameras are installed from the left side to the right side of the rear of the car 10.

[0156] like Fig.16 As shown, five cameras image the rear of the car from different viewpoints. Parts of the imaging area overlap each other. By sequentially switching and displaying the images captured by the five cameras with these different imaging viewpoints according to the driver's behavior (such as the movement of the head), a smoother image switching display can be performed.

[0157] Fig.17 An example of switching images in a case where more cameras (c1 to cn=15) are deployed at the rear of the car 10 and the images are switched is illustrated.

[0158] By sequentially switching the manual images captured by the multiple cameras in this manner, the driver can observe images that change smoothly like a moving image.

[0159] Fig.18 The figure shows a case where the number of cameras installed at the rear of the car 10 is set to be less than Fig.17 . Although it is possible to switch images with a small number of cameras in this way, parallax between the cameras causes intermittent images. In this case, a process of calibrating each camera-captured image is performed to set and display the infinite point 101 of each image at a fixed position of the image. This process makes it possible to suppress the subject blur generated when switching images, and enables the image viewer (driver) to observe smoothly changing images without unnatural changes.

[0160] In addition, image correction of sweeping and shifting the image in the lateral direction at adjacent points in the image according to the movement of the driver's head can be performed. This process enables the driver to visually and intuitively recognize the switching between camera images.

[0161] Furthermore, in a case where a small number of cameras are installed at the rear of the car 10 and parallax between the cameras causes intermittent images, a virtual viewpoint image of an intermediate viewpoint between the two adjacent cameras may be synthesized and displayed based on images captured by the two adjacent cameras.

[0162] That is, image display control is performed to switch an image output to the display unit to any image among images captured by a plurality of cameras or a virtual viewpoint synthesized image.

[0163] Fig.19 An example of generating a virtual viewpoint image is shown. Here, Fig.19 The (1) image captured by camera C3 and the (4) image captured by camera C4 are images actually captured by the adjacent cameras C3 and C4.

[0164] Based on these two actually captured images, virtual viewpoint synthesis images of (2) and (3) are generated.

[0165] Here, the (2) virtual viewpoint synthesis image is a synthetic image generated at a ratio of 65% of the (1) camera C3 captured image and 35% of the (4) camera C4 captured image.

[0166] Here, the (3) virtual viewpoint synthesis image is a synthetic image generated at a ratio of 35% of the (1) camera C3 captured image and 65% of the (4) camera C4 captured image.

[0167] By sequentially switching and displaying Fig.19 The four images (1) to (4) in the figure can present a smoothly changing image to the driver.

[0168] However, it is important for the driver not to visually check the details of the image at the midway viewpoint between the cameras but to judge the rear situation. The driver sometimes cannot judge the switched situation only by instantly viewing the acquired image captured by the rear C camera 31 at the center of the rear of the car at the rear C camera position and the image taken from the camera deployed on the side. Switching and presenting multiple midway images to the driver enables the driver to recognize the image switching sensorily and naturally. Therefore, confusion in thinking when instant situation judgment is required can be avoided.

[0169] The images can be switched using switching based on, for example, the orientation of the face and the direction of the sight line, switching based on so-called gestures such as shaking the hand or neck, etc. Switching with gestures can bring the advantage of not having to maintain an inappropriate posture. However, it is noted that when the images are switched without depending on the orientation of the face or the direction of the sight line, the driver's viewing posture is the same. Therefore, it is preferred to display information indicating the switching and transition of the images, for example, auxiliary information such as a switching mark.

[0170] In the process of the present disclosure, by providing camera images of the rear of the vehicle obtained from a plurality of different viewpoints, a device for properly checking the rear without obstruction is provided. The camera images with different viewpoints are made to have continuity. An image in the middle of switching is generated by simulating the generation of images using human visual characteristics and presented on a monitor. In this way, the driver can sensorily grasp the viewpoint movement from the image. Information can be provided while suppressing a reduction or loss in grasping the situation. That is, ergonomically, the viewpoint can be well grasped by the driver who views the image wrapped between the cameras with different changes in viewpoints.

[0171] Fig. 20 An example is illustrated in which a camera position icon 102 is superimposed and displayed on an image displayed on the display unit 20. The camera position icon 102 indicates a camera imaging viewpoint. In this way, displaying the camera position icon 102 indicating the position of the camera that has captured the image together with the image enables the driver to immediately check the image capturing position and more quickly recognize the position of the subject in the image.

[0172] Fig.21 An example is illustrated in which a rear left end camera image and a left side camera image are displayed side by side on the display unit 20. The rear left end camera image is an image captured by a rear left end camera installed at the left end of the rear of the car 10. The left side camera image is an image captured by the left side camera 12. Such display enables the condition of the left side of the car 10 to be grasped with certainty.

[0173] Note that similarly, displaying the image captured by the rear right end camera installed at the right end of the rear of the car 10 and the image captured by the right side camera 13 side by side enables the situation of the right side of the car 10 to be grasped with certainty.

[0174] Although Fig.21An example is illustrated in which a pictogram representing a camera is displayed near the vehicle bumper, but a simple color mark, the entire screen, or the left and right frames may be displayed as bold and flashing. A flowing zebra crossing warning may be simply displayed. Another warning call display may be made as long as the driver can intuitively understand the arrangement of the camera when viewing the screen. In addition, a plurality of displays may be combined. In addition, a configuration that can be switched to a display method that does not impair intuitive understanding may be adopted according to the driver's preference.

[0175] Note that, for example, Fig. 22 As shown, the following two installation aspects are mainly adopted as aspects of installing the multiple cameras deployed at the rear of the car 10.

[0176] (a) Radial arrangement

[0177] (b) Linear arrangement

[0178] Here, (a) radial arrangement is an example of deploying a camera on a curve with a predetermined curvature, and is characterized by being able to capture a wider range of images. In contrast, the advantage of (b) linear arrangement is that, although the imaging range is narrowed, the amount of deformation of the object in each image is small, and the amount of correction processing of the displayed image is reduced. Note that, although it is possible to arrange in a direction converging to a rear adjacent point in a convergent arrangement, as opposed to the radial type of (a), the installation aspect has the disadvantage of not being able to obtain continuity in the direction toward the left camera 12 and the right camera 13. The image of the camera actually installed is used as the projection image of the central projection image, and the optical axis does not have to be deployed in each direction. The image can be converted from the fisheye projection image, and the image conversion can be used so that the optical axis of the virtual central projection image corresponds to the optical axis according to each installation aspect.

[0179] [4. Image processing device and processing sequence performed by mobile device]

[0180] Next, we will refer to Fig.23 The flowchart describes the image processing apparatus of the present disclosure and the processing sequence executed by the mobile device.

[0181] Fig.23 The processing of the flow chart in the following figures is performed in the mobile device or the image processing device installed in the mobile device. For example, the processing is performed under the control of a data processing unit including a processor such as a CPU. The processor has a function of executing a program according to a program stored in a storage unit in the device.

[0182] The following will describe Fig.23 Processing of each step in the process.

[0183] (Step S101)

[0184] First, in step S101 , the data processing unit detects the driver's face, sight direction, or gesture, or information about the driver's operation.

[0185] This is for example Fig.12 The data processing unit 60 in the embodiment inputs the detection information from the driver information detection unit 52 and the information about the driver's operation to the operation unit.

[0186] (Step S102)

[0187] Next, in step S102 , the data processing unit determines whether the driver's face, a change in the direction of sight, or a gesture, or an input of information about the driver's operation is detected.

[0188] If an input is detected, the process proceeds to step S103. If no input is detected, the detection process of step S101 is continued.

[0189] (Step S103)

[0190] Next, in step S103 , the data processing unit selects a camera-captured image in a direction according to the driver's face, sight direction, or gesture or operation information that has been input in step S102 .

[0191] Note that, for example, if there is no image that exactly matches the driver's line of sight direction, then the image in the closest direction is selected.

[0192] For example, if the driver's face, sight, or gesture, or an input to the operation unit is in the left direction, an image captured by a camera on the left side (rear L camera) at the rear of the car 10 is selected.

[0193] Furthermore, for example, if the driver's face, sight, or gesture, or input to the operation unit is in the center direction, an image captured by a camera (rear C camera) in the center of the rear of the car 10 is selected. Alternatively, for example, if the driver's face, sight, or gesture, or input to the operation unit is in the right direction, an image captured by a camera (rear R camera) on the right side of the rear of the car 10 is selected.

[0194] (Step S104)

[0195] Next, in step S104 , the data processing unit displays the image that has been selected in step S103 on the display unit.

[0196] Note that various settings can be adopted for the image displayed on the display unit, including, for example, a setting in which only the selected rear camera captured image is displayed, and a setting in which both side camera images captured by the left and right side cameras are displayed together.

[0197] exist Fig.23 In the process, when selecting a camera-captured image in the direction of the driver's face, gaze direction or gesture or operation information input in step S102, for example, and there is no image that completely matches the driver's gaze direction, processing of selecting an image in the nearest direction is performed.

[0198] Next, we will refer to Fig.24 The flowchart of FIG. 1 describes a processing sequence for synthesizing an image in the driver's line of sight direction in such a case (ie, in a case where there is no image that completely matches the driver's line of sight direction).

[0199] The following will describe Fig.24 Processing of each step in the process.

[0200] (Step S201)

[0201] First, in step S201 , the data processing unit detects the driver's face, sight direction, or gesture, or information about the driver's operation.

[0202] This is for example Fig.12 The data processing unit 60 in the embodiment inputs the detection information from the driver information detection unit 52 and the information about the driver's operation to the operation unit.

[0203] (Step S202)

[0204] Next, in step S202 , the data processing unit determines whether the driver's face, a change in the direction of sight, a gesture, or an input of information about the driver's operation is detected.

[0205] If an input is detected, the process proceeds to step S203. If no input is detected, the detection process of step S201 is continued.

[0206] (Step S203)

[0207] Next, in step S203 , the data processing unit determines the presence or absence of a camera captured image in a direction according to the driver's face, change in sight direction or gesture, or information on the driver's operation that has been input in step S202 .

[0208] If there is a captured image in the matching direction, the process proceeds to step S204.

[0209] In contrast, if no image is captured in the matching direction, the process proceeds to step S206.

[0210] (Step S204)

[0211] In the case where it is determined in step S203 that there is a camera capturing an image in the direction according to the driver's face, the change in the direction of the sight line, or the gesture or the information about the driver's operation that has been input in step S202, the processing of steps S204 and S205 is performed. This display is mainly applicable to the case where the cameras are disposed adjacently at the time of switching and the image is described without leaping. If the imaging directions of the switching images are away from each other, switching according to step S206 described later is performed.

[0212] In step S204 , a camera-captured image in a direction according to the driver's face, sight direction, or gesture or operation information input in step S202 is selected.

[0213] For example, if the driver's face, sight, or gesture, or an input to the operation unit is in the left direction, an image captured by a camera on the left side (rear L camera) at the rear of the car 10 is selected.

[0214] Furthermore, for example, if the driver's face, sight, or gesture, or input to the operation unit is located in the center direction, an image captured by a camera (rear C camera) in the center of the rear of the automobile 10 is selected. Alternatively, for example, if the driver's face, sight, or gesture, or input to the operation unit is in the right direction, an image captured by a camera (rear R camera) on the right side of the rear of the automobile 10 is selected.

[0215] (Step S205)

[0216] Next, in step S205 , the data processing unit displays the image that has been selected in step S204 on the display unit.

[0217] Note that various settings can be adopted for the image displayed on the display unit. The various settings include, for example, a setting in which only the selected rear camera captured image is displayed, and a setting in which both side camera images captured by the left and right side cameras are displayed together.

[0218] (Step S206)

[0219] In contrast, when it is determined in step S203 that no camera has captured an image in the direction of the driver's face, change in line of sight direction, gesture, or information about the driver's operation input in step S202, processing of steps S206 and S207 is performed.

[0220] First, in step S206, processing is performed to generate a camera-captured image in a direction according to the driver's face, change in sight direction, or gesture, or information about the driver's operation that has been input in step S202 as a composite image.

[0221] The synthetic image generation process corresponds to, for example, the above reference Fig.19 Describe the image synthesis process.

[0222] Image synthesis processing can be performed, for example, by simply weighting the images actually captured by adjacent cameras. In addition, an intermediate virtual image can be generated by detecting a moving object for each area and performing correction based on the detection information. It is noted that the driver's information loss is prevented by presenting a synthetic image of an intermediate viewpoint between the actual captured images to the driver. Specifically, for example, if the captured image is switched from the rear center (C) camera 31 to the side camera, the image changes greatly in an instant. The synthetic image effectively prevents the driver's information loss that may occur in this case.

[0223] Note that, for example, the driver's face, sight direction or gesture or the input of the image selection indication in step S206 generates various movements, such as medium movement and sharp movement. Steps 206 and 207 are performed as steps for generating one or two or more multiple intermediate images (i.e., composite images) according to the detected movement.

[0224] (Step S207)

[0225] Next, in step S207, the data processing unit displays the synthesized image generated in step S206 on the display unit. In addition, when the synthesized image in step S207 is output to the display unit, the driver can Fig.21 The camera position of the displayed image is grasped by superimposing and displaying a description of the camera position on the screen of the display unit 20 in the display unit. Note that various settings can be adopted for the image displayed on the display unit. The various settings include, for example, a setting in which only the image captured by the selected rear camera is displayed, and a setting in which two side camera images captured by the left and right side cameras are displayed together.

[0226] Furthermore, in the description of the sequence, the images are switched based on the driver's will, gesture or operation, for example, by movement of the face, sight and head. Figure 7 , Figure 8 and Fig. 9As described, an object behind the rear vehicle, i.e., an object in an occluded area, can be detected, and when the driver looks at the rear center (C) camera 31, a warning prompting the driver to visually check details can be displayed as a trigger for the driver to check the side camera image. The warning includes a zebra pattern and a flashing overlapping warning display at the boundary of the rear object that produces the occlusion.

[0227] That is, for example, if a large vehicle approaches from behind and only the image captured by the central camera is used, the approaching vehicle behind the large vehicle or the like is easily overlooked due to occlusion. In this case, a warning indicating that the object is in the occluded area is displayed. The warning prompts the driver to check the image captured by the camera with a different viewpoint. This processing enables the driver to change the camera viewpoint and check the rear situation with certainty even in the case of occlusion, thereby improving safety.

[0228] Note that the image can be switched in a configuration that provides different camera-captured images for a single flyby. Preferably, an image with a virtual viewpoint between cameras is output between different camera-captured images, and a display with image transition and continuity is performed. This process prevents the driver from losing control of the situation and enables a safe check of the rear side.

[0229] [5. Configuration Example of Image Processing Device]

[0230] Next, a hardware configuration example of an image processing apparatus that executes the above-described processing will be described.

[0231] Fig.25 A hardware configuration example of an image processing apparatus is illustrated.

[0232] A central processing unit (CPU) 501 functions as a data processing unit that executes various processes according to a program stored in a read only memory (ROM) 502 or a storage unit 508. For example, the processes are executed according to the sequence described in the above-described embodiments.

[0233] For example, programs and data to be executed by the CPU 501 are stored in a random access memory (RAM) 503. These CPU 501, ROM 502, and RAM 503 are connected to each other via a bus 504.

[0234] The CPU 501 is connected to an input / output interface 505 via a bus 504. An input unit 506 and an output unit 507 are connected to the input / output interface 505. The input unit 506 includes, for example, a status data acquisition unit such as various switches, a keyboard, a touch panel, a mouse, a microphone, a sensor, a camera, and a GPS. The output unit 507 includes, for example, a display and a speaker.

[0235] The CPU 501 inputs, for example, commands and status data input from the input unit 506 , executes various processes, and outputs the processing results to, for example, the output unit 507 .

[0236] The storage unit 508 connected to the input / output interface 505 includes, for example, a hard disk, and stores programs and various data to be executed by the CPU 501. The communication unit 509 functions as a transmission / reception unit for data communication via a network such as the Internet or a local area network, and communicates with external devices.

[0237] The drive 510 connected to the input / output interface 505 drives a removable medium 511 including a magnetic disk, an optical disk, a magneto-optical disk, and a semiconductor memory such as a memory card. The drive 510 records or reads data.

[0238] [6. Summary of the configuration of the present disclosure]

[0239] The embodiments of the present disclosure have been described in detail above with reference to specific embodiments. However, it is apparent that those skilled in the art may modify or replace these embodiments without departing from the gist of the present disclosure. That is, the present invention has been disclosed in an illustrative form and should not be construed restrictively. In order to determine the gist of the present disclosure, the claims should be considered.

[0240] Note that the technology disclosed in the specification can have the following configurations.

[0241] (1) An image processing device including a data processing unit, the data processing unit performing image display control of the following contents:

[0242] enter

[0243] driver information showing behavior of a driver of the mobile device; and

[0244] images captured by a plurality of cameras that image conditions surrounding the mobile device from different viewpoints; and

[0245] According to the driver information, the image output to the display unit is switched to any image among the images captured by the plurality of cameras or a virtual viewpoint synthesized image.

[0246] (2) The image processing device according to (1),

[0247] The plurality of cameras include a plurality of rear cameras mounted on the rear of the mobile device, and

[0248] The data processing unit switches to and displays any image or a virtual viewpoint synthesized image among the images captured by the plurality of rear cameras according to the driver information.

[0249] (3) The image processing device according to (1),

[0250] wherein the plurality of cameras are a plurality of rear cameras mounted on the rear of the mobile device and a side camera mounted on the side of the mobile device, and

[0251] The data processing unit switches to and displays any image or a virtual viewpoint synthesized image among the images captured by the plurality of rear cameras and side cameras according to the driver information.

[0252] (4) The image processing device according to any one of (1) to (3),

[0253] In which, the data processing unit displays a rear camera captured image and an image captured by a side camera side by side on a display unit, wherein the rear camera captured image is selected from images captured by multiple rear cameras installed at the rear of a mobile device based on driver information, and the side camera is installed on the side of the mobile device.

[0254] (5) The image processing device according to any one of (1) to (4),

[0255] The driver information is information obtained by detecting the driver's face or sight direction, and

[0256] The data processing unit displays an image in a direction corresponding to the driver's face or sight direction on the display unit.

[0257] (6) The image processing device according to any one of (1) to (4),

[0258] The driver information is obtained by detecting the driver's gestures.

[0259] The data processing unit displays an image in a direction corresponding to the driver's gesture on the display unit.

[0260] (7) The image processing device according to any one of (1) to (4),

[0261] The data processing unit displays an image in a direction according to the driver's input to the operating unit on the display unit.

[0262] (8) The image processing device according to any one of (1) to (7),

[0263] Among them, the virtual viewpoint synthesis image is a synthesized image based on images captured by multiple cameras, and is a synthesized image equivalent to an image captured from a virtual viewpoint between the imaging viewpoints of the multiple cameras.

[0264] (9) The image processing device according to any one of (1) to (8),

[0265] The driver information is information obtained by detecting the driver's face or sight direction, and

[0266] The data processing unit generates an image in a direction corresponding to the face or sight direction of the driver by synthesizing the images captured by the plurality of cameras, and displays the image on the display unit.

[0267] (10) The image processing device according to any one of (1) to (9),

[0268] The data processing unit performs an image calibration process to make the positions of the infinite points of each image consistent when switching the display images on the display unit.

[0269] (11) A mobile device comprising:

[0270] Multiple cameras that image the conditions surrounding the mobile device from different viewpoints.

[0271] a driver information detection unit that detects driver information showing behavior of a driver of the mobile device; and

[0272] A data processing unit that inputs driver information and images captured by the plurality of cameras and switches an image output to a display unit to any of the images captured by the plurality of cameras or a virtual viewpoint synthesized image according to the driver information.

[0273] (12) The mobile device according to (11),

[0274] The plurality of cameras include a plurality of rear cameras mounted on the rear of the mobile device, and

[0275] The data processing unit switches to and displays any image or a virtual viewpoint synthesized image among the images captured by the plurality of rear cameras according to the driver information.

[0276] (13) The mobile device according to (11) or (12),

[0277] The driver information detection unit detects the driver's face or sight direction, and

[0278] The data processing unit displays an image in a direction corresponding to the driver's face or sight direction on the display unit.

[0279] (14) An image processing method executed in an image processing device,

[0280] The data processing unit performs the following image display control:

[0281] enter

[0282] driver information showing behavior of a driver of the mobile device; and

[0283] images captured by a plurality of cameras that image conditions surrounding the mobile device from different viewpoints; and

[0284] According to the driver information, the image output to the display unit is switched to any image among images captured by a plurality of cameras or a virtual viewpoint synthesized image.

[0285] (15) A display image control method executed in a mobile device, comprising:

[0286] an image capturing step, wherein a plurality of cameras images conditions surrounding the mobile device from different viewpoints;

[0287] a driver information detecting step in which a driver information detecting unit detects driver information showing a behavior of a driver of the mobile device; and

[0288] An image display control step, in which a data processing unit inputs driver information and images captured by the multiple cameras, and switches the image output to the display unit to any image among the images captured by the multiple cameras or a virtual viewpoint synthesized image according to the driver information.

[0289] (16) A program for executing image processing in an image processing apparatus, the program causing a data processing unit to execute image display control of:

[0290] enter

[0291] driver information showing behavior of a driver of the mobile device; and

[0292] images captured by a plurality of cameras that image conditions surrounding the mobile device from different viewpoints; and

[0293] According to the driver information, the image output to the display unit is switched to any image among the images captured by the plurality of cameras or a virtual viewpoint synthesized image.

[0294] In addition, a series of processes described in the specification can be performed by hardware, software or a combination of the two. In the case of performing the process by software, the program recorded with the process sequence can be executed by being installed in a memory in a computer combined in special hardware or being installed in a general-purpose computer capable of performing various processes. For example, the program can be recorded in a recording medium in advance. In addition to being installed in a computer from a recording medium, the program can also be received via a network such as a local area network (LAN) and the Internet, and installed in a recording medium such as a built-in hard disk.

[0295] Note that the various processes described in the specification can be executed not only in chronological order according to the description, but also in parallel or individually according to the processing capacity or the needs of the device that executes the process. In addition, the system in the specification has a configuration of a logical collection of multiple devices. The system is not limited to a system in which the devices having each configuration are placed in the same housing.

[0296] Industrial Applicability

[0297] As described above, according to the configuration of one embodiment of the present disclosure, a configuration is achieved in which an image output to a display unit is switched and displayed according to the driver's behavior such as the movement of the driver's head.

[0298] Specifically, for example, driver information indicating the behavior of the driver of the mobile device and images captured by multiple cameras imaging the conditions around the mobile device from different viewpoints are input. The image output to the display unit is switched according to the driver information. The multiple cameras are, for example, multiple rear cameras installed at the rear of the mobile device. For example, the driver's face or sight direction is detected. An image in a direction corresponding to the detected driver's face or sight direction is selected as an output image and displayed on the display unit. Alternatively, an image in a direction indicated by the driver's gesture is selected and displayed on the display unit. In addition, when different cameras are switched to capture images, a composite image with an intermediate viewpoint is appropriately generated and displayed, thereby preventing the driver from losing control of the situation.

[0299] This configuration realizes a configuration in which an image output to the display unit is switched and displayed according to the driver's behavior such as movement of the driver's head.

[0300] Reference numerals list

[0301] 10. Cars

[0302] 11 Rear Camera

[0303] 12 Left camera

[0304] 13 Right camera

[0305] 20 Display unit

[0306] 21 Rear camera image

[0307] 22 Left camera image

[0308] 23 Right camera image

[0309] 25 Composite Image

[0310] 31 Rear center (C) camera

[0311] 32 Rear left (L) camera

[0312] 33 Rear right (R) camera

[0313] 34 Rear center left (CL) camera

[0314] 35 Rear Center Right (CR) Camera

[0315] 41 Rear camera image switching display area

[0316] 50 Driver

[0317] 52 Driver information detection unit

[0318] 60 Data Processing Unit

[0319] 101 Infinity Point

[0320] 102 Camera Location Icons

[0321] 501 CPU

[0322] 502 ROM

[0323] 503 RAM

[0324] 504 Bus

[0325] 505 Input / Output Interface

[0326] 506 Input Unit

[0327] 507 Output Unit

[0328] 508 storage unit

[0329] 509 Communication Unit

[0330] 510 Driver

[0331] 511 Removable Media

Claims

1. An image processing device comprising a data processing unit, wherein the data processing unit performs image display control of the following contents: enter driver information showing behavior of a driver of the mobile device; and images captured by a plurality of cameras that image conditions surrounding the mobile device from different viewpoints; and An image output to a display unit is switched to display a virtual view synthesis image having a viewpoint between two adjacent cameras among the plurality of cameras according to the driver information.

2. The image processing device according to claim 1, in, The plurality of cameras include a plurality of rear cameras mounted at a rear portion of the mobile device, and The data processing unit switches to and displays the virtual viewpoint synthetic image according to the driver information.

3. The image processing device according to claim 1, in, The plurality of cameras include: a plurality of rear cameras mounted at the rear of the mobile device and side cameras mounted at the side of the mobile device, and The data processing unit switches to and displays the virtual viewpoint synthetic image according to the driver information.

4. The image processing device according to claim 1, in, The data processing unit displays a rear camera captured image and an image captured by a side camera side by side on the display unit, the rear camera captured image being selected from images captured by a plurality of rear cameras installed at the rear of the mobile device according to the driver information, the side camera being installed at the side of the mobile device.

5. The image processing device according to claim 1, in, The driver information includes information obtained by detecting the direction of the driver's face or sight, and The data processing unit displays the virtual viewpoint synthesis image in a direction corresponding to a direction of a driver's face or sight line on the display unit.

6. The image processing device according to claim 1, in, The driver information includes information obtained by detecting a driver's gesture, and The data processing unit displays the virtual viewpoint synthesis image in a direction corresponding to the driver's gesture on the display unit.

7. The image processing device according to claim 1, in, The data processing unit displays the virtual viewpoint synthetic image in a direction according to the driver's input to the operation unit on the display unit.

8. The image processing device according to claim 1, in, The virtual viewpoint synthetic image is a synthetic image based on images captured by a plurality of cameras, and is a synthetic image comparable to an image captured from a virtual viewpoint between imaging viewpoints of the plurality of cameras.

9. The image processing device according to claim 1, in, The driver information includes information obtained by detecting the direction of the driver's face or sight, and The data processing unit generates the virtual viewpoint synthesis image in a direction corresponding to the driver's face or sight direction by synthesizing images captured by a plurality of cameras, and displays the virtual viewpoint synthesis image on the display unit.

10. The image processing device according to claim 1, in, The data processing unit performs an image calibration process to make positions of infinite points of respective images consistent when switching display images on the display unit.

11. A mobile device, comprising: a plurality of cameras that capture images of conditions surrounding the mobile device from different viewpoints; a driver information detection unit that detects driver information showing behavior of a driver of the mobile device; as well as A data processing unit that inputs the driver information and the images captured by the multiple cameras and switches the image output to a display unit to display a virtual viewpoint synthesis image according to the driver information, wherein the virtual viewpoint synthesis image has a viewpoint between two adjacent cameras among the multiple cameras.

12. The mobile device according to claim 11, in, The plurality of cameras include a plurality of rear cameras mounted at a rear portion of the mobile device, and The data processing unit performs switching to and displaying the virtual viewpoint synthesis image according to the driver information.

13. The mobile device according to claim 11, in, The driver information detection unit detects the direction of the driver's face or sight, and The data processing unit displays the virtual viewpoint synthesis image in a direction corresponding to a direction of a driver's face or sight line on the display unit.

14. An image processing method executed in an image processing device, in, The data processing unit performs image display control of the following contents: enter driver information showing behavior of a driver of the mobile device; and images captured by a plurality of cameras that image conditions surrounding the mobile device from different viewpoints; as well as An image output to a display unit is switched to display a virtual view synthesis image having a viewpoint between two adjacent cameras among the plurality of cameras according to the driver information.

15. A display image control method executed in a mobile device, comprising: an image capturing step, wherein a plurality of cameras capture images of conditions surrounding the mobile device from different viewpoints; a driver information detecting step, wherein a driver information detecting unit detects driver information showing a behavior of a driver of the mobile device; as well as An image display control step, wherein a data processing unit inputs the driver information and the images captured by the multiple cameras, and switches the image output to the display unit to display a virtual viewpoint synthesis image according to the driver information, wherein the virtual viewpoint synthesis image has a viewpoint between two adjacent cameras among the multiple cameras.

16. A non-transitory computer-readable storage medium storing instructions, wherein the instructions cause a data processing unit to perform image display control of the following contents: enter driver information showing behavior of a driver of the mobile device; and images captured by a plurality of cameras that image conditions surrounding the mobile device from different viewpoints; and An image output to a display unit is switched to display a virtual view synthesis image having a viewpoint between two adjacent cameras among the plurality of cameras according to the driver information.

Images