Display control device and display control method

Abstract

To provide a display control device capable of reducing a burden on an occupant when the occupant operates an image displayed on a display unit.SOLUTION: A display control device 30 includes: an occupant information acquisition unit 52 for acquiring angle information of a face of an occupant; an image acquisition unit 51 for acquiring a photographed image obtained by photographing a periphery of a vehicle; an image conversion unit 54 for converting the photographed image into a display image that is a virtual viewpoint image viewed from a virtual viewpoint; a virtual viewpoint setting unit 53 for setting a position of the virtual viewpoint based on an amount of change in an angle of the face of the occupant; and a display processing unit 55 for controlling to display a display image on a display unit 40.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a display control device and a display control method. [Background technology]

[0002] The vehicular display system disclosed in Patent Document 1 displays content on a plurality of displays arranged with non-display areas between reference display screens. An occupant status monitor is provided to detect the head position and angle of a vehicle occupant and the line of sight of the occupant. A display processing unit performs a process to prevent the important information from being hidden and displays it on one of the display screens of the plurality of displays on the condition that important information is included in the content to be displayed on the display based on the detection result of the occupant status monitor.

[0003] This conventional technology discloses that a control device detects the occupant's line of sight, head position, and head angle, and that when the occupant moves their head position left or right, the image displayed on the display unit moves left or right, the image is displayed in a reduced size, or the image is displayed tilted.

[0004] However, in the above-mentioned conventional technology, in order to greatly change the image displayed on the display unit, the occupant must move his / her head significantly, which places a heavy burden on the occupant. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] International Publication No. 2022 / 224754 Summary of the Invention [Problem to be solved by the invention]

[0006] Therefore, an object of the present disclosure is to provide a display control device and a display control method that can reduce the burden on an occupant when operating an image displayed on a display unit. [Means for solving the problem]

[0007] In order to achieve the above-mentioned objective, the display control device of the present disclosure includes an occupant information acquisition unit that acquires information regarding the angle of the occupant's face, an image acquisition unit that acquires an image taken of the surroundings of the vehicle, an image conversion unit that converts the image into a virtual viewpoint image viewed from a virtual viewpoint, a virtual viewpoint setting unit that sets the position of the virtual viewpoint based on the amount of change in the angle of the occupant's face, and a display processing unit that controls the virtual viewpoint image to be displayed on a display unit. Effect of the Invention

[0008] In the display control device of the present disclosure configured as above, when the occupant changes the angle of the face, the image displayed on the display unit changes according to the amount of change in the angle, which reduces the burden on the occupant when operating the image displayed on the display unit. [Brief description of the drawings]

[0009] [Figure 1] 1 is a functional block diagram showing a schematic configuration of a display control system including a display control device according to a first embodiment. [Diagram 2] 11A and 11B are diagrams for explaining the relationship between an image displayed on a display unit and the angle of an occupant's face. [Diagram 3] 5A to 5C are diagrams for explaining a procedure for calculating the amount of change in yaw angle and pitch angle, which is executed by an occupant information acquisition unit. [Figure 4] 10A to 10C are diagrams for explaining a procedure for calculating a virtual viewpoint position executed by a virtual viewpoint setting unit; [Diagram 5] 10A to 10C are diagrams for explaining a first conversion process executed by a ground model projection unit. [Figure 6] 11A and 11B are diagrams for explaining a projection area setting process executed by a viewpoint reflecting unit; [Figure 7] 4 is a diagram for explaining a first conversion process executed in a ground model projection unit and a second conversion process executed in a viewpoint reflection unit. FIG. [Figure 8]4 is a flowchart for explaining an example of an operation of the display control device according to the first embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] (First embodiment) A display control device according to a first embodiment of the present disclosure will be described below with reference to the drawings. FIG. 1 is a functional block diagram showing a schematic configuration of a display control system 100 including a display control device 30 according to a first embodiment. The display control device 30 shown in FIG. 1 is an embodiment of a display control device according to the present disclosure, and the present disclosure is not limited to this embodiment. The display control device 30 is mounted on a moving body such as an automobile. Hereinafter, the moving body on which the display control device 30 of this embodiment is mounted will be referred to as the host vehicle and described.

[0011] The display control system 100 includes an imaging device 10, an occupant monitoring unit 20, a display control device 30, and a display unit 40. As shown in FIG. 2, the display control system 100 generates a display image 42 to be displayed in a display area 41 of the display unit 40 according to the amount of change in the angle of the face of the driver, who is the occupant 1, and displays the display image 42 on the display unit 40. The diagram in the center of the page of FIG. 2 shows the display image 42 displayed on the display unit 40 when the angle of the face of the occupant 1 is a reference angle described later, and the diagrams on the right and left of the page show the display image 42 displayed on the display unit 40 when the angle of the face of the occupant 1 changes from the reference angle. For example, FIG. 2 is an example of the display image 42 displayed on the display unit 40 when the angle φ of the face of the occupant 1 changes in the Yaw direction from the reference angle θ. The angle φ of the face and the reference angle θ will be described later. When converting the amount of change, which is the amount by which the angle φ of the face of the occupant 1 has changed from the reference angle θ, into a virtual viewpoint position, the positive and negative of the amount of change may be reversed as shown in the lower part of FIG. 2 under "with left-right inversion". For example, the display area of ​​the display unit 40 is set to a projection surface, and the projection center when an image projected from the projection center to the projection surface is displayed corresponds to the virtual viewpoint. As will be described in detail later, the virtual viewpoint in this disclosure does not necessarily correspond to the position of the driver's eyes. The display control system 100 can display an image as if the outside of the vehicle was seen through the display unit 40 from a virtual viewpoint.

[0012] The imaging device 10 is installed outside the vehicle and captures images of the surroundings of the vehicle. The imaging device 10 outputs the captured images to the display control device 30 as captured image data according to a predetermined protocol. In this embodiment, the imaging device 10 includes a front camera installed in front of the vehicle. Note that the imaging device 10 is not limited to a front camera, and may include a rear camera installed behind the vehicle, and side cameras installed on the left and right front and left and rear sides of the vehicle.

[0013] The occupant monitoring unit 20 is mounted on the vehicle. The occupant monitoring unit 20 monitors the state of the driver, who is the occupant 1, based on an image captured by the in-vehicle camera 21. The occupant monitoring unit 20 may be a known one. The in-vehicle camera 21 is a camera that captures the interior of the vehicle, including the driver, and is provided, for example, near the display unit 40 and facing the interior of the vehicle. The occupant monitoring unit 20 detects the angle of the face of the occupant 1 using a known method based on the image captured by the in-vehicle camera 21, and outputs information on the acquired face angle (hereinafter referred to as "angle information") to the display control device 30. In this embodiment, the occupant monitoring unit 20 outputs a yaw angle and a pitch angle that indicate the direction in which the face of the occupant 1 is facing in the horizontal and vertical directions as face angle information.

[0014] The display control device 30 is an information processing device that executes processing related to the creation and display of images, and includes a processing unit (control unit) 50 and a storage unit 60, as shown in FIG. 1. The display control device 30 is configured, for example, by an ECU having storage devices such as a CPU, a GPU, a RAM, and a ROM. The processing unit 50 is mainly configured by a CPU and the like of the ECU, and controls the overall operation of the display control device 30 by expanding a predetermined program stored in the ROM into the RAM and executing it. The storage unit 60 is mainly configured by a storage device of the ECU, but may also include an external server or database. The display control device 30 may be configured by one ECU, or may be configured by multiple ECUs, and each function of the processing unit 50 described later may be distributed, or data to be stored may be distributed. In addition, the display control device 30 may be configured to realize some or all of its functions using hardware such as an FPGA or an ASIC. Furthermore, one ECU may be configured to have not only the function of the display control device 30, but also the function of a camera ECU that controls the imaging device 10 and the function of the occupant monitoring unit 20.

[0015] The processing unit 50 controls the entire display control device 30, generates a display image based on the captured image data input from the imaging device 10 and the angle information input from the occupant monitoring unit 20, and displays the image on the display unit 40. To enable this, the processing unit 50 functions as an image acquisition unit 51, an occupant information acquisition unit 52, a virtual viewpoint setting unit 53, an image conversion unit 54, and a display processing unit 55 as shown in Fig. 1, but is not limited to this configuration.

[0016] The image acquisition unit 51 acquires captured image data of the surroundings of the vehicle from the imaging device 10, and outputs the data to the image conversion unit .

[0017] The occupant information acquisition unit 52 acquires angle information from the occupant monitoring unit 20, and calculates the amount of change in the face angle, i.e., the amount of change in the yaw angle and the pitch angle, based on the acquired angle information and information on a predetermined reference face angle (hereinafter referred to as “reference angle information”). The occupant information acquisition unit 52 outputs the calculated amount of change in the yaw angle and the pitch angle to the virtual viewpoint setting unit 53.

[0018] The reference angle information is stored in advance in the reference angle storage unit 61 of the storage unit 60. For example, the reference angle may be information on the face angle when the occupant 1 faces forward (front), with the face direction being set as the reference angle. For example, the face angle when the occupant 1 faces the display unit 40 may be the reference angle. The angle information acquired from the occupant monitoring unit 20 is information on the face angle after the occupant 1 changes the face direction with respect to the reference angle.

[0019] The calculation procedure of the yaw angle and pitch angle change amount by the occupant information acquisition unit 52 will be described below with reference to FIG. 3. The yaw angle (Yaw) is the angle of the face of the occupant 1 in the horizontal direction (left-right direction), and the pitch angle (Pitch) is the angle of the face of the occupant 1 in the vertical direction (up-down direction). If the reference angle is θ (θ yaw ,θ pitch ), and the face angle obtained from the DMS20 (i.e., the face angle after the face orientation is changed) is φ(φ yaw ,φ pitch ), the occupant information acquisition unit 52 calculates the change in the face angle A (A yaw ,A pitch ) is calculated by the following formulas (1) and (2). In the following formulas (1) and (2), θ yaw and θ pitch are the yaw angle and pitch angle as the reference angle, and φ yaw and φ pitch are the yaw angle and pitch angle as the face angle after movement, and A yaw and A pitch are the amounts of change in the yaw angle and the pitch angle.

[0020] A yaw = φ yaw - θ yaw (1) A pitch = φ pitch - θ pitch (2)

[0021] The virtual viewpoint setting unit 53 determines the amount of change in the yaw angle and the pitch angle (A yaw ,A pitch ), the position of a virtual viewpoint of the occupant 1 in the horizontal and vertical directions (hereinafter referred to as a "virtual viewpoint position T") is calculated (set). The virtual viewpoint setting unit 53 outputs the calculated virtual viewpoint position to the image conversion unit 54.

[0022] The calculation procedure of the virtual viewpoint position T by the virtual viewpoint setting unit 53 will be described below with reference to Fig. 4. P shown in Fig. 4 is the virtual viewpoint position when the angle of the face of the occupant 1 is a reference angle θ, and T is the virtual viewpoint position when the angle of the face of the occupant changes from the reference angle θ to an angle φ. If the coordinates of the virtual viewpoint position P are P(Px, Py, Pz) and the coordinates of the virtual viewpoint position T are T(Tx, Ty, Tz), the virtual viewpoint setting unit 53 calculates the coordinates of the virtual viewpoint position T by the following formulas (3), (4), and (5). In the following formulas (3) and (4), f yaw and f pitch is a predetermined increasing function. A publicly known increasing function can be used as this increasing function. The increasing function may be one that adds a predetermined additional value according to the amount of change in the face angle to constantly increase the value of the virtual viewpoint position T. For example, the increasing function may be one that increases the additional value according to the degree of the amount of change, such as increasing the additional value as the amount of change in the face angle increases, thereby increasing the value of the virtual viewpoint position T. Note that the virtual viewpoint setting unit 53 may set the virtual viewpoint position T by inverting the amount of change in the face angle of the occupant 1 in the up-down direction (pitch direction) and left-right direction (yaw direction).

[0023] Tx = Px + f yaw (A yaw ) (3) Ty = Py + f pitch (A pitch ) (4) Tz = Pz (5)

[0024] The image conversion unit 54 generates a display image 42 (an image of the captured image 70 viewed from a virtual viewpoint) to be displayed in the display area 41 of the display unit 40, based on a captured image 70 captured by the imaging device 10 and the virtual viewpoint position T calculated by the virtual viewpoint setting unit 53. For this purpose, the image conversion unit 54 executes a first conversion process for converting the coordinates of the captured image 70 from the image coordinate system of the captured image 70 to coordinates of a vehicle coordinate system which is the coordinate system of the host vehicle, and a second conversion process for converting the coordinates of the vehicle coordinate system to a display coordinate system which is the coordinate system of the display unit 40.

[0025] 1, the image conversion unit 54 has a ground model projection unit 541 and a viewpoint reflection unit 542. The ground model projection unit 541 performs correction processing and ground model image generation processing which is a first conversion processing. The ground model projection unit 541 also performs calculation processing of a homography matrix M used in the first conversion processing. The ground model projection unit 541 stores the calculated homography matrix M in the conversion information storage unit 62 of the storage unit 60.

[0026] As a correction process, the ground model projection unit 541 corrects the lens distortion (for example, lens distortion aberration, chromatic aberration, etc.) of the imaging device 10 using a known method for the captured image 70 input from the image acquisition unit 51. Note that this correction process may be performed by the image acquisition unit 51 instead of the ground model projection unit 541, and the image acquisition unit 51 may output the captured image 70 after the lens distortion correction process to the ground model projection unit 541.

[0027] As a ground model image generation process, the ground model projection unit 541 converts the image coordinate system into the vehicle coordinate system so as to map and project the corrected captured image 70 onto a surface set in the vehicle coordinate system and display it, thereby generating a ground model image 72. This ground model image generation process is a first conversion process that converts the image coordinate system into the vehicle coordinate system using the projective transformation matrix M stored in the conversion information storage unit 62.

[0028] The ground model image generation process will be described below with reference to FIG. 5. The upper diagram on the paper of FIG. 5 is an image coordinate system based on the captured image 70 after the correction process, and the lower diagram on the paper of FIG. 5 is a vehicle coordinate system based on the vehicle itself. The image coordinate system is a two-dimensional coordinate system with the upper left corner of the captured image 70 after the correction process as the origin O (0,0). Xa and Ya are mutually orthogonal axes, and the unit of each axis is pixels (px). The vehicle coordinate system is a three-dimensional coordinate system with the origin O (0,0,0) at a predetermined position of the vehicle itself, where Xb is an axis extending in the vehicle width direction (horizontal direction), Yb is an axis perpendicular to Xb and extending in the up-down direction (vertical direction) of the vehicle, and Zb is an axis perpendicular to Xb and Yb and extending forward of the vehicle. The unit of each axis of Xb, Yb, and Zb is mm.

[0029] The surface set in the vehicle coordinate system is set to a surface corresponding to the ground (driving surface) on which the vehicle runs. In this embodiment, this surface is referred to as a ground model 71. The area indicated by reference numeral 72 in FIG. 5 indicates an image (hereinafter referred to as a "ground model image 72") in which the captured image 70 is mapped onto the ground model 71 by a ground model image generation process. The area indicated by reference numeral 73 in FIG. 5 is an area in which the captured image 70 is not mapped, that is, an area not captured by the imaging device 10 (hereinafter referred to as a "non-mapping area 73").

[0030] The ground model projection unit 541 converts the coordinates of each pixel of the captured image 70 after the correction process into the coordinates of the ground model 71 by substituting the following formula (6). a and y a are the x and y coordinates in the image coordinate system, and x b and z b are the x and z coordinates of the vehicle coordinate system. Here, the coordinates (x a ,y a ) is defined as a, and the coordinate (x b , z b The relationship between the homogeneous coordinates a and b is expressed by the following formula (6). Note that the value λ b indicates the magnification in homogeneous coordinate b. This value λ bRegardless of the value of (except for 0), the homogeneous coordinate b represents the same coordinate in the vehicle coordinate system.

[0031]

number

[0032] The projective transformation matrix M is calculated in advance, and the calculation procedure can be performed as follows, with reference to the left and center diagrams of FIG. 7. a1, a2, a3, and a4 shown in FIG. 7 are reference points referenced in the captured image 70 in the image coordinate system. Also, b1, b2, b3, and b4 shown in FIG. 7 are coordinates (x,y) of the image coordinate system, which are reference points in the ground model 71 in the vehicle coordinate system that correspond to the reference points a1, a2, a3, and a4 in the image coordinate system. a ,y a ) and the coordinate (x b , z b The relationship with the homogeneous coordinate b representing θ can be expressed by the above equation (6).

[0033] The ground model projection unit 541 sets four reference points b1, b2, b3, and b4 in the vehicle coordinate system, which are points captured on the corrected captured image 70. The coordinates of each of the reference points b1, b2, b3, and b4 are determined by actual measurement and input to the ground model projection unit 541. Next, the ground model projection unit 541 determines the coordinates of the four reference points a1, a2, a3, and a4 in the image coordinate system of the corrected captured image 70.

[0034] The ground model projection unit 541 calculates the homography transformation matrix M by substituting the coordinates of each reference point identified as described above into the above equation (6) and solving the simultaneous equations including each element of the homography transformation matrix M. The ground model projection unit 541 stores the calculated homography transformation matrix M in the transformation information storage unit 62 of the storage unit 60.

[0035] The viewpoint reflecting unit 542 performs a projection area setting process, a calculation process of a projective transformation matrix N, and a display image generation process which is a second conversion process. The projection area setting process will be described below with reference to FIG. 6. As shown in FIG. 6, when the projection surface is the display unit 40 with the virtual viewpoint position T input from the virtual viewpoint setting unit 53 as a reference, the viewpoint reflecting unit 542 calculates an area (projection area 74) on the ground model 71 projected from the virtual viewpoint position T onto the display area 41 of the display unit 40. The area surrounded by points c1, c2, c3, and c4 shown in FIG. 6 is the display area 41 of the display unit 40, and the area surrounded by points b5, b6, b7, and b8 on the ground model 71 is the projection area 74 projected onto the display area 41. The viewpoint reflecting unit 542 calculates the coordinates of points b5 to b8 based on the coordinates of the virtual viewpoint position T and the coordinates of points c1 to c4.

[0036] More specifically, the viewpoint reflecting unit 542 sets straight lines connecting the virtual viewpoint position T and each of the points c1 to c4, using the coordinates T (Tx, Ty, Tz) of the virtual viewpoint position T and the coordinates of the points c1 to c4 at the four corners of the display area 41. Next, the viewpoint reflecting unit 542 detects intersections b5 to b8 between these straight lines and the ground model 71, recognizes the area surrounded by the intersections b5 to b8 as the projection area 74 corresponding to the display area 41, and calculates the coordinates of each of the intersections b5 to b8.

[0037] The calculation process of the projective transformation matrix N will be described as follows with reference to the central and right diagrams of Fig. 7. The right diagram of Fig. 7 is a display coordinate system based on the display area 41 of the display unit 40. The display coordinate system is a two-dimensional coordinate system with the upper left corner of the display area 41 as the origin O(0,0). Xc and Yc are axes perpendicular to each other, and the unit of each axis is pixels (px).

[0038] On the ground model 71 in the vehicle coordinate system, an area surrounded by intersections b5, b6, b7, and b8 is defined as a projection area 74 calculated by the viewpoint reflection unit 542, and the coordinates (x b ,z bIn the display coordinate system of the display unit 40, the reference points corresponding to the intersections b5, b6, b7, and b8 are c1, c2, c3, and c4, respectively, and the coordinates (x c ,y c The homogeneous coordinates b and c are expressed by the following equation (7). c indicates the magnification in the homogeneous coordinate c. This value λ c Regardless of the value of (except 0), the homogeneous coordinate c represents the same coordinate in the display coordinate system.

[0039]

number

[0040] The viewpoint reflecting unit 542 substitutes the coordinates of the reference points c1 to c4 in the display coordinate system of the display unit 40 calculated in the projection area setting process and the coordinates of the intersection points b5 to b8 in the vehicle coordinate system into the above equation (7), and calculates the projection transformation matrix N by solving simultaneous equations including each element of the projection transformation matrix N. The viewpoint reflecting unit 542 stores the calculated projection transformation matrix N in the transformation information storage unit 62 of the storage unit 60.

[0041] Furthermore, as a display image generation process, the viewpoint reflecting unit 542 substitutes the coordinates of each point of the projection area 74 of the ground model image 72 corresponding to each pixel of the display area 41 of the display unit 40 into the above formula (7) using the projective transformation matrix N calculated in the above calculation process to convert the coordinates of the projection area 74 into the display coordinate system. In this manner, the viewpoint reflecting unit 542 generates image data of the display image 42 corresponding to the image of the projection area 74 on the ground model image 72. The viewpoint reflecting unit 542 outputs the generated image data to the display processing unit 55.

[0042] Based on the image data input from the viewpoint reflecting section 542, the display processing section 55 causes the display area 41 of the display section 40 to display the display image 42 corresponding to this image data.

[0043] The storage unit 60 temporarily or non-temporarily stores a control program for operating the display control device 30, and various data and parameters used during various operations in the processing unit 50. As described above, the reference angle storage unit 61 of the storage unit 60 temporarily or non-temporarily stores reference angle information of the face when the angle of the face is the reference angle. The conversion information storage unit 62 of the storage unit 60 temporarily or non-temporarily stores the projective transformation matrix M and the projective transformation matrix N used in the ground model image generation process (first conversion process) and the display image generation process (second conversion process), respectively.

[0044] An example of the operation of the display control system 100 according to the first embodiment having the above-mentioned configuration will be described below with reference to the flowchart in Fig. 8. Fig. 8 shows an example of the operation of the display control device 30, but the operation of the display control device 30 is not limited to the operation in Fig. 8.

[0045] First, in step S1, the image acquisition unit 51 acquires the captured image 70 captured by the imaging device 10, and outputs it to the image conversion unit 54. Then, in step S2, the occupant information acquisition unit 52 acquires angle information on the face angle of the occupant 1 from the occupant monitoring unit 20. In the next step S3, the occupant information acquisition unit 52 calculates the amount of change in the face angle using the above-mentioned formulas (1) and (2) based on the acquired angle information and the reference angle information acquired from the reference angle storage unit 61, and outputs the amount of change in the face angle to the virtual viewpoint setting unit 53.

[0046] In the next step S4, the virtual viewpoint setting unit 53 calculates the virtual viewpoint position T of the occupant 1 based on the amount of change input from the occupant information acquisition unit 52 using the above-mentioned equations (3) and (4), and outputs it to the image conversion unit 54.

[0047] In the next step S5, the ground model projection unit 541 performs a correction process to correct lens distortion on the captured image 70. Next, in step S6, the ground model projection unit 541 converts the coordinates of the captured image 70 after the correction process into coordinates in the vehicle coordinate system using the projective transformation matrix M acquired from the transformation information storage unit 62 and the above-mentioned equation (6), thereby generating a ground model image 72.

[0048] In the next step S7, the viewpoint reflecting unit 542 calculates an area (projection area 74) on the ground model image 72 to be projected onto the display area 41 of the display unit 40, based on the virtual viewpoint position T input from the virtual viewpoint setting unit 53. That is, the viewpoint reflecting unit 542 calculates the coordinates of intersection points b5 to b8 surrounding the projection area 74, based on the coordinates of the virtual viewpoint position T and the coordinates of points c1 to c4 at the four corners of the display area. Next, in step S8, the viewpoint reflecting unit 542 substitutes the coordinates of points c1 to c4 in the display coordinate system and the coordinates of intersection points b5 to b8 in the vehicle coordinate system into the above-mentioned equation (7) to calculate the projective transformation matrix N.

[0049] In the next step S9, the viewpoint reflection unit 542 substitutes each coordinate of the projection area 74 into the aforementioned equation (7) and converts it into coordinates of the display coordinate system, thereby generating image data of the display image 42 to be displayed in the display area 41, and outputs it to the display processing unit 55.

[0050] Then, in step S10, the display processing unit 55 displays the display image 42 corresponding to the image data input from the viewpoint reflection unit 542 in the display area 41 of the display unit 40 based on the image data. As shown in Fig. 2, the display image 42 in the direction corresponding to the angle of the face of the occupant 1 is displayed in the display area 41.

[0051] As described above, the display control device 30 of this embodiment converts the captured image 70 of the periphery of the vehicle into a ground model image 72 based on the virtual viewpoint position T set based on the change in the angle of the face of the occupant 1, and converts the ground model image 72 into a display image 42 to be displayed on the display unit 40. Then, the display image 42 is displayed on the display unit 40, so that the occupant 1 can view the display image 42 according to the angle of the face. Such a display image 42 is appropriately connected to the scenery viewed through the front window, and the occupant 1 can view the display image 42 without feeling strange. Moreover, when the occupant 1 wants to change the image displayed on the display unit 40, he or she only needs to change the angle of the face up, down, left, and right without moving his or her head significantly. Therefore, the display control device 30 of this embodiment can reduce the burden on the occupant 1 when the occupant 1 changes the image displayed on the display unit 40.

[0052] The display control device 30 of this embodiment also includes a storage unit 60 (reference angle storage unit 61) that stores a reference angle at which the angle of the face of the occupant 1 is a predetermined angle. The virtual viewpoint setting unit 53 sets the position T of the virtual viewpoint based on the amount of change in the angle of the face of the occupant 1 relative to the reference angle. This allows the virtual viewpoint setting unit 53 to obtain the amount of change in the face angle with higher accuracy. As a result, the display control device 30 can present the occupant 1 with a more appropriate display image 42 that corresponds to the angle of the face.

[0053] In addition, in the display control device 30 of this embodiment, the occupant information acquisition unit 52 acquires a yaw angle and a pitch angle as the angle of the face of the occupant 1. Then, the virtual viewpoint setting unit 53 moves the position T of the virtual viewpoint in the horizontal direction based on the amount of change in the yaw angle, and moves the position T of the virtual viewpoint in the vertical direction based on the amount of change in the pitch angle. With this configuration, the virtual viewpoint setting unit 53 can calculate the amount of change in the angle of the face more accurately and more quickly, and the display control device 30 can perform the display control process more efficiently and more accurately.

[0054] Although an embodiment of the present disclosure has been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and design changes that do not deviate from the gist of the present disclosure are included in the present disclosure.

[0055] For example, the display control device 30 may be configured to include a viewpoint position acquisition unit that acquires information regarding the eye position of the occupant 1. Based on the information regarding the eye position acquired by the viewpoint position acquisition unit, the virtual viewpoint setting unit 53 sets a virtual viewpoint at a position corresponding to the eye position of the occupant 1 when the angle of the face of the occupant 1 is a reference angle, and moves the position of the virtual viewpoint from the eye position of the occupant 1 to a position based on the amount of change when the angle of the face of the occupant 1 is not the reference angle. With this configuration, the virtual viewpoint setting unit 53 can set the position of the virtual viewpoint according to the eye position of the occupant 1, and can set the position of the virtual viewpoint more appropriately and more easily. For example, when the angle of the face of the occupant 1 when the face of the occupant 1 faces the display unit 40 is the reference angle, the virtual viewpoint can be set at a position corresponding to the eye position of the occupant 1 while the face of the occupant 1 faces the display unit 40. Furthermore, when the face of the occupant 1 is not facing the display unit 40, the virtual viewpoint can be set at a position based on the amount of change in the angle of the face of the occupant 1.

[0056] In addition, the display control device 30 in the above embodiment sets the face angle when the face of the occupant 1 is at the reference angle as the reference angle (reference angle information), but is not limited to this. For example, the reference angle can be the face angle information when the face angle was acquired last time. In this case, the occupant information acquisition unit 52 updates the reference angle information in the reference angle storage unit 61 every time the face angle is acquired. With this configuration, the display control device 30 can calculate the amount of change when the occupant 1 changes the face angle this time, using the face angle before the occupant 1 changes the face angle as a reference. [Explanation of symbols]

[0057] 1: Crew member 30: Display control device 40: Display unit 42: Display image (virtual viewpoint image) 50: Processing section 51: Image acquisition section 52: Occupant information acquisition unit 53: Virtual viewpoint setting unit 54: Image conversion unit 55: Display processing unit 60: Storage unit 70: Captured image (image) A: Amount of change θ: Reference angle λ : Face angle

Claims

1. an occupant information acquisition unit that acquires information regarding the angle of the occupant's face; an image acquisition unit that acquires an image of the surroundings of the vehicle; an image conversion unit that converts the image into a virtual viewpoint image viewed from a virtual viewpoint; a virtual viewpoint setting unit that sets the position of the virtual viewpoint based on an amount of change in the angle of the face of the occupant; a display processing unit that controls the virtual viewpoint image to be displayed on a display unit; A display control device comprising:

2. a storage unit that stores a reference angle that is a predetermined angle of the face of the occupant; The virtual viewpoint setting unit sets the position of the virtual viewpoint based on an amount of change that is an amount by which an angle of the face of the occupant has changed with respect to the reference angle. The display control device according to claim 1 .

3. A viewpoint position acquisition unit that acquires information regarding the eye position of the occupant, the virtual viewpoint setting unit sets the virtual viewpoint at a position corresponding to an eye position of the occupant when the angle of the face of the occupant is the reference angle; When the angle of the face of the passenger is not the reference angle, the position of the virtual viewpoint is moved from the position of the eyes of the passenger to a position based on the amount of change. The display control device according to claim 2 .

4. the occupant information acquisition unit acquires a yaw angle and a pitch angle as angles of the face of the occupant, The virtual viewpoint setting unit moves the position of the virtual viewpoint in a horizontal direction based on the amount of change in the yaw angle, and moves the position of the virtual viewpoint in a vertical direction based on the amount of change in the pitch angle. The display control device according to any one of claims 1 to 3.

5. A display control method executed by a control unit of a display control device mounted on a vehicle, comprising: an occupant information acquisition step of acquiring information regarding the angle of the occupant's face; an image acquisition step of acquiring an image of the surroundings of the vehicle; an image conversion step of converting the image into a virtual viewpoint image viewed from a virtual viewpoint; a virtual viewpoint setting step of setting the position of the virtual viewpoint based on an amount of change in the angle of the face of the occupant; a display processing step of controlling the virtual viewpoint image to be displayed on a display unit; A display control method comprising:

6. a storage step of storing a reference angle, which is a predetermined angle of the face of the occupant, in a storage unit; The virtual viewpoint setting step sets a position of the virtual viewpoint based on an amount of change that is an amount of change in an angle of the face of the occupant with respect to the reference angle. The display control method according to claim 5 .

7. A viewpoint position acquiring step of acquiring information regarding the eye position of the occupant, the virtual viewpoint setting step sets the virtual viewpoint at a position corresponding to an eye position of the occupant when the angle of the face of the occupant is the reference angle, When the angle of the face of the passenger is not the reference angle, the position of the virtual viewpoint is moved from the position of the eyes of the passenger to a position based on the amount of change. The display control method according to claim 6.

8. The occupant information acquisition step acquires a yaw angle and a pitch angle as angles of the face of the occupant, The virtual viewpoint setting step moves a position of the virtual viewpoint in a horizontal direction based on the amount of change in the yaw angle, and moves a position of the virtual viewpoint in a vertical direction based on the amount of change in the pitch angle. The display control method according to any one of claims 5 to 7.

Images