Information processing apparatus and control program

Abstract

To display a transmission image capable of suppressing discomfort felt by a crew member.SOLUTION: An image generation device 40 functioning as an information processing device acquires a projective transformation matrix M which becomes first transformation information making the coordinates of a captured image GC transformable from the coordinate system of an image to the coordinate system of a vehicle, calculates a projective transformation matrix N which becomes second transformation information making the coordinates of a reference range to be displayed on a display part 50 transformable from the coordinate system of a vehicle 2 to the coordinate system of the display part 50, and generates a display target image GD by using the projective transformation matrixes M and N, by an image generation part 46C.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an information processing apparatus and a control program.

Background Art

[0002] There has been proposed an information processing apparatus mounted on a vehicle and capable of displaying an image that expands the driver's field of view. For example, Patent Document 1 describes a vehicle surrounding video providing apparatus that performs coordinate conversion on a video captured by a photographing device so that the same video as when the outside of the vehicle can be seen through the installation area of the liquid crystal display from the driver's viewpoint position is displayed on the liquid crystal display.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in Patent Document 1, the specific method of how to set the display area among the videos captured by the photographing device is not clear, and the specific method of how to perform coordinate conversion on the videos captured by the photographing device and display them is also not clear. Unless the generation of an image within the range visible to the passenger and the association of the image with a display unit such as a liquid crystal display are accurately performed, the displayed image and the scenery directly visible to the passenger will not be continuous, and the passenger may feel a sense of discomfort.

[0005] The present invention has been made in view of the above circumstances, and an object thereof is to enable the display of a transmissive image capable of suppressing the discomfort of the passenger.

Means for Solving the Problems

[0006] To achieve the above object, the information processing apparatus of the present invention includes an information acquisition unit that acquires a captured image obtained by imaging the surroundings of a moving body, a display processing unit that causes a display unit visible to an occupant of the moving body to display an image to be displayed, a first conversion information acquisition unit that acquires first conversion information that enables conversion of the coordinates of the captured image from the coordinate system of the image to the coordinate system of the moving body, a reference range calculation unit that specifies a projection center corresponding to the viewpoint of the occupant of the moving body and calculates a reference range on a set surface set in the coordinate system of the moving body to be projected onto the display range of the display unit with reference to the projection center, a second conversion information acquisition unit that calculates second conversion information that enables conversion of the coordinates of the reference range from the coordinate system of the moving body to the coordinate system of the display unit, and an image generation unit that calculates each pixel value of the display range of the display unit from each pixel value of the captured image corresponding to the reference range by using the first conversion information and the second conversion information, and generates the image to be displayed by using the calculated pixel values. When the straight line drawn from the projection center to the upper edge of the display range of the display unit does not intersect the setting surface, the reference range calculation unit changes the position of the upper edge of the display range of the display unit to a lowered position so that the straight line intersects the setting surface, and calculates a reference range corresponding to the changed display range.

Advantages of the Invention

[0007] According to the present invention, it is possible to display a transparent image that can suppress the discomfort of the occupant.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Embodiments for Carrying Out the Invention

[0009] Hereinafter, embodiments will be described with reference to the accompanying drawings. FIG. 1 is a diagram showing the configuration of a display control system 1 according to an embodiment of the present invention. The display control system 1 includes a photographing device 10, a user recognition device 20, a user interface 30 (hereinafter referred to as "UI30"), an image creation device 40, and a display unit 50. The display control system 1 is a system mounted on a vehicle 2 (FIG. 2), and each member constituting the display control system 1 can also be referred to as an in-vehicle device. The vehicle 2 is a moving body that moves together with a user US who is an occupant, and in the present embodiment, it is a four-wheeled vehicle called an automobile. However, the vehicle 2 may be a vehicle other than a four-wheeled vehicle or a moving body other than a vehicle.

[0010] FIG. 2 is a diagram showing the surroundings of the driver's seat of the vehicle 2 on which the display control system 1 is mounted. FIG. 2 shows a part of the interior of the vehicle as seen from the user US (driver) seated in the driver's seat. As shown in FIG. 2, the vehicle 2 includes a front window 2W, a steering wheel 2H operated by the user US, and a display unit 50 provided on the dashboard in front of the steering wheel 2H. An in-vehicle camera 12 included in the photographing device 10 is arranged near the display unit 50. The size, position of the display unit 50, and the position and size of the in-vehicle camera 12 may be changed as appropriate.

[0011] The display unit 50 is arranged in front of the user US and below the front window 2W, is visible to the user US, and has a wide display screen that extends in the vehicle width direction. The display unit 50 displays the image created by the image creation device 40, in other words, displays the output image of the image creation device 40. A widely circulated display device such as a liquid crystal display device can be widely applied to the display unit 50. In the present embodiment, the display unit 50 is configured by one display device, but the display unit 50 may be configured by a plurality of display devices.

[0012] When the user US drives the vehicle 2, the user US visually recognizes the scenery around the vehicle including the front of the vehicle through the front window 2W. The display control system 1 is a system that captures the scenery around the vehicle by the external camera 11 of the imaging device 10, converts the captured imaging image GC into a transparent image GD that can be seen when viewed through the display unit 50 from the user US, and displays the transparent image GD on the display unit 50. That is, the transparent image GD is an image to be displayed and is also the output image output by the image creation device 40. Also, the transparent image GD can be called a viewpoint conversion image adjusted to the viewpoint of the user US, and can also be called a vision expansion image useful for driving support that expands the vision of the user US. Hereinafter, for convenience of explanation, the image displayed on the display unit 50 is referred to as "display target image GD".

[0013] FIG. 2 shows a case where there are two lines LN extending linearly on the ground in front of the vehicle. In the present embodiment, since a highly accurate display target image GD adjusted to the viewpoint of the user US is generated, as shown in FIG. 2, each line LN visually recognized through the front window 2W and each line LN' displayed on the display unit 50 can be arranged in a straight line. Therefore, the area that the driver cannot directly view can be accurately displayed, which is suitable for driving support. In FIG. 2, the center lines of each line LN are shown with the symbol LC attached. As shown in FIG. 2, since the center line LC of each line LN coincides with the center line of each line LN' to be displayed, each line LN and LN' are continuous and do not give the user US a sense of discomfort.

[0014] FIG. 3 is a diagram showing the surroundings of the driver's seat of the vehicle 2 when a highly accurate display target image GD cannot be generated. As shown in FIG. 3, when viewed from the user US, each line LN' displayed on the display unit 50 is displayed shifted from each line LN visually recognized through the front window 2W. That is, the scenery visually recognized by the user US through the front window 2W and the display target image GD displayed on the display unit 50 are not continuous, and there is a possibility that the user US may feel a sense of discomfort. Also, as an image for expanding the driver's field of view, since it is not an accurate image, there may be a case where it is not preferable as an image for driving support.

[0015] As shown in FIG. 1, the imaging device 10 includes an external camera 11 that images the surroundings of the vehicle 2 and an in-vehicle camera 12 that images the user US inside the vehicle. The external camera 11 images at least the ground located in front of the user US inside the vehicle and behind the area visually recognized by the user US through the front window 2W. The data of the captured image GC captured by the external camera 11 is output to the image creation device 40.

[0016] As shown in FIG. 2, the in-vehicle camera 12 is arranged in front of the user US, so that it can image an image including both eyes and the face of the user US. The data of the captured image GF captured by the in-vehicle camera 12 is output to the user recognition device 20. Widely circulated imaging devices can be applied to the external camera 11 and the in-vehicle camera 12.

[0017] The user recognition device 20 includes a face recognition unit 21 that recognizes the face of the user US using the captured image GF, and a viewpoint detection unit 22 that detects the viewpoint of the user US using the captured image GF. The face recognition unit 21 performs processing to recognize the position of the face of the user US, the orientation of the face, each part of the face (including the eyes), and user actions related to the face using known image recognition techniques. User actions related to the face are, for example, actions such as winking and nodding.

[0018] The viewpoint detection unit 22 detects the viewpoint coordinates PE, which are the position of the viewpoint of the user US, using the recognition result of the face recognition unit 21. Note that the viewpoint detection unit 22 may directly detect the viewpoint coordinates PE of the user US from the captured image GF without using the recognition result of the face recognition unit 21. As shown in FIG. 4, the viewpoint detection unit 22 detects the left-eye viewpoint position TL and the right-eye viewpoint position TR as the viewpoint coordinates PE. The viewpoint positions TL and TR can also be referred to as the positions of the left and right eyes, and are information used to specify the range to be displayed on the display unit 50. Strictly speaking, the viewpoint positions TL and TR are preferably the positions where the user US recognizes the video, for example, the positions on the left and right retinas where light from the outside is converted into video. As shown in FIG. 4, since the viewpoint positions TL and TR are separated from each other left and right, the regions SL and SR visible from each of the viewpoint positions TL and TR are slightly different. The detection results of the face recognition unit 21 and the viewpoint detection unit 22 are output to the image creation device 40 as user information DU.

[0019] In this way, the user recognition device 20 uses the captured image GF of the in-vehicle camera 12 to identify at least the viewpoint positions TL and TR of the user US, and outputs user information DU that can identify at least the viewpoint positions TL and TR to the image creation device 40. In FIG. 1, the case where the user recognition device 20 is separate from the image creation device 40 is shown, but the user recognition device 20 may be integrated with the image creation device 40.

[0020] UI30 is an interface for inputting instructions from user US, and includes known user interfaces such as operation buttons, touch panels, voice input devices, etc. The user instructions input via UI30 are input into the image creation device 40. FIG. 1 shows a display control system 1 in which the photographing device 10, the user recognition device 20, UI30, the image creation device 40, and the display unit 50 are separate, but the configuration is not limited thereto. For example, at least any one of the photographing device 10, the user recognition device 20, UI30, and the display unit 50 may be configured integrally with the image creation device 40.

[0021] As shown in FIG. 1, the image creation device 40 includes an input interface 41 (hereinafter referred to as "input I / F 41"), a processor 42, and a storage unit 43, and functions as an information processing device that performs processing related to image creation and display. The input I / F 41 includes a hardware environment for connecting to predetermined devices such as the user recognition device 20, the photographing device 10, and UI30, and has a function of transmitting and receiving data and signals according to a protocol defined in advance with each device. The data and signals input by the input I / F 41 are output to the processor 42 of the image creation device 40. This input I / F 41 corresponds to the "information acquisition unit" of the present invention.

[0022] The processor 42 is an arithmetic processing device including a CPU (Central Processing Unit) and an MPU (Micro Processing Unit), etc. The processor 42 functions as a projection center specifying unit 44, a dominant eye setting unit 45, an image processing unit 46, a display processing unit 47, etc. by operating according to the control program DP stored in the storage unit 43.

[0023] The processor 42 can also be said to be a computer that controls the operation of the image creation device 40. By the processor 42 controlling the operation of the image creation device 40, the operation of the entire display control system 1 is controlled. That is, the processor 42 can also be said to be a computer that controls the operation of the display control system 1.

[0024] The processor 42 may be composed of a single processor or a plurality of processors. Further, the processor 42 may be composed of a SoC (System-on-a-chip) integrated with a part or all of the storage unit 43 and other circuits. Also, the processor 42 can be configured by a combination of a CPU that executes the control program DP and a DSP that executes predetermined arithmetic processing. Furthermore, the processor 42 may be configured such that all functions of the processor 42 are implemented in hardware, or may be configured using a programmable device.

[0025] In addition to the control program DP, the storage unit 43 stores data such as the previous viewpoint coordinates PE detected by the viewpoint detection unit 22, the previous projection center T specified by the projection center specifying unit 44, the line of sight X set by the line of sight setting unit 45, and the conversion information M, N, etc. A well-known storage device can be widely applied to this storage unit 43.

[0026] Here, the conversion information M is the first conversion information that enables the coordinates of the captured image GF to be converted from the coordinate system of the image data to the coordinate system of the vehicle 2. More specifically, it is the projection transformation matrix M described later. Also, the conversion information N is the second conversion information that enables the image to be displayed on the display unit 50 among the captured images GF to be converted from the coordinate system of the vehicle 2 to the coordinate system of the display unit 50. More specifically, it is the projection transformation matrix N described later.

[0027] The projection center specifying unit 44 sets a projection center T corresponding to the viewpoint of the user US based on the viewpoint coordinates PE detected by the viewpoint detection unit 22 and the like. In the present embodiment, the projection center T is determined as a single point, the display area of the display unit 50 is set as the projection plane, and an image projected from the projection center T onto the projection plane is created as the display target image GD.

[0028] By the way, since humans have two eyes, that is, two viewpoints, when the projection center T is fixed at a single point, in principle, it is impossible to perfectly reproduce the image when a human sees an object. As a compromise, a method of setting the midpoint between the two eyes as the projection center T can be considered, but a sense of discomfort remains. On the other hand, which eye's information is processed more preferentially in vision varies from person to person. Therefore, in the present embodiment, for the purpose of reducing the above-mentioned sense of discomfort, the dominant eye setting unit 45 sets the dominant eye X of the user US, and sets the projection center T that reflects the dominant eye X. Note that the dominant eye is the eye that is recognized more dominantly between the left and right eyes.

[0029] In the present embodiment, the dominant eye setting unit 45 performs a dominant eye setting process for setting the dominant eye X of the user US. The dominant eye setting process will be described later. The image processing unit 46 includes a lens correction unit 46A, a projection conversion unit 46B, and an image generation unit 46C. The lens correction unit 46A performs a correction process for correcting the distortion of the lens of the external camera 11 on the captured image GC acquired via the input I / F 41. The distortion of the lens is, for example, the distortion aberration or chromatic aberration of the lens. A known correction process may be applied to the process of correcting the distortion of the lens.

[0030] FIG. 5 is a diagram for supplementing to facilitate the understanding of the first conversion process described later. In FIG. 5, the image coordinate system of the corrected captured image GC is shown by two axes consisting of the horizontal axis Xc and the vertical axis Yc, and the vehicle coordinate system is shown by three axes consisting of Xv, Yv, and Zv. The image coordinate system is a coordinate system based on the captured image GC, and is a two-dimensional coordinate system with the upper left corner of the captured image GC as the origin O, and the unit of each axis is a pixel (px). The vehicle coordinate system is a coordinate system based on vehicle 2, which is a three-dimensional coordinate system with a certain position of vehicle 2 as the origin O. Xv is an axis extending in the vehicle width direction, Yv is an axis extending in the vehicle up-and-down direction, Zv is an axis extending in the vehicle forward direction, and the unit of each axis is mm. In this embodiment, the setting surface SM is set on a surface corresponding to the ground (which can also be referred to as the driving surface) on which vehicle 2 travels. This setting surface SM can also be referred to as the surface of the object whose visual field is to be expanded or the surface to be visually recognized by the user US.

[0031] As shown in FIG. 5, the projection conversion unit 46B calculates a projection conversion matrix M for mapping the corrected captured image GC onto the setting surface SM (a surface such that the coordinate of the Yv axis corresponds to the height of the ground). The region indicated by the reference sign GN in FIG. 5 is a region where the image in the captured image GC is not mapped, in other words, it is a region outside the captured image GC.

[0032] The projection conversion unit 46B performs a process of calculating a reference range (corresponding to the range of the driving surface) to be displayed on the display unit 50. FIG. 6 is a diagram showing the processing content of the projection conversion unit 46B. As shown in FIG. 6, the projection conversion unit 46B calculates which range on the setting surface SM set in the vehicle coordinate system is projected onto the display range (the range surrounded by the coordinates PD1 to PD4) of the display unit 50, which is the projection surface, based on the projection center T. More specifically, the projection conversion unit 46B calculates the coordinates Pv1 to Pv4 using the coordinates of the projection center T and the coordinates PD1 to PD4.

[0033] In this embodiment, the range HC surrounded by the coordinates Pv1 to Pv4 is referred to as the reference range, and the image within the reference range (range HC) corresponds to the image to be displayed on the display unit 50. In this embodiment, the projection center specifying unit 44 and the projection conversion unit 46B correspond to the "reference range calculation unit" of the present invention.

[0034] The image generation unit 46C generates the image data of the display target image GD by generating the image data corresponding to the image of the reference range (range HC) using the captured image GC. Further, the display processing unit 47 performs processing to cause the display unit 50 to display an image corresponding to the image data generated by the image generation unit 46C based on the image data.

[0035] In order to create the display target image GD to be displayed on the display unit 50 from the captured image GC captured by the external camera 11, a first conversion process for accurately converting the coordinates of the captured image GC from the coordinate system of this image GC to the coordinate system of the vehicle 2 and a second conversion process for accurately converting the coordinates of the reference range HC to the coordinate system of the display unit 50 are required. The information required for the first conversion process is as follows.

[0036] As shown in FIG. 7, the reference points in the captured image GC in the image coordinate system are set as points Pc1, Pc2, Pc3, Pc4, and the homogeneous coordinates representing the coordinates (xc, yc) of the image coordinate system are set as Pc. On the setting surface SM (the surface corresponding to the ground) in the vehicle coordinate system, the reference points corresponding to the respective reference points Pc1 to Pc4 are set as points Pv1, Pv2, Pv3, Pv4, and the homogeneous coordinates representing the coordinates (xv, zv) of the vehicle coordinate system are set as Pv. These reference points Pc1 to Pc4 and the reference points Pv1 to Pv4 are used when calculating the projective transformation matrix M described later. The relationship between the homogeneous coordinates Pc and Pv can be expressed by Equation (1).

[0037]

Equation

[0038] M is a projective transformation matrix for converting the coordinates of the image coordinate system to the coordinates of the vehicle coordinate system, and corresponds to the "first conversion information" of the present invention. The projective transformation matrix M can be calculated by the following procedure. Points captured in the captured image GC with the distortion of the lens of the external camera 11 corrected, four points Pv1, Pv2, Pv3, and Pv4 in the coordinate system of the vehicle 2 are defined, and the coordinates of each of the points Pv1 to Pv4 are specified by actual measurement. Next, the coordinates of the four points Pc1, Pc2, Pc3, and Pc4 in the image coordinate system are specified. In this case, on the captured image GC with the distortion of the lens of the external camera 11 corrected, the coordinates of each of the points Pc1 to Pc4 are to be specified. By substituting these specified coordinates into Equation (1) and solving the system of simultaneous equations including each element of the projective transformation matrix M, the projective transformation matrix M is calculated. The calculated projective transformation matrix M is stored in the storage unit 43 as the transformation information M.

[0039] Note that the process of calculating the projective transformation matrix M may be performed using the arithmetic processing function of the processor 42. When the processor 42 calculates the projective transformation matrix M, the processor 42 functions as the "first transformation information acquisition unit" of the present invention. The projective transformation matrix M may be calculated by a device having an arithmetic processing function other than the image creation device 40. In this case, the calculated projective transformation matrix M may be acquired by the image creation device 40 via the input I / F 41 and stored in the storage unit 43. In this case, the input I / F 41 functions as the "first transformation information acquisition unit" of the present invention. Further, the input I / F 41 may have a communication function or the like for connecting to a predetermined communication network.

[0040] The information required for the second conversion process is as follows. As shown in FIG. 8, the reference range calculated by the projection conversion unit 46B on the setting surface SM in the vehicle coordinate system is the range of points Pv1, Pv2, Pv3, and Pv4, and the homogeneous coordinates representing the coordinates (xv, zv) of the vehicle coordinate system are defined as Pv. In the coordinate system of the display unit 50, the reference points corresponding to the respective reference points Pv1 to Pv4 are points PD1, PD2, PD3, and PD4, and the homogeneous coordinates representing the coordinates (xD, yD) of the coordinate system of the display unit 50 are defined as PD. The coordinate system of the display unit 50 is a coordinate system based on the display unit 50, a two-dimensional coordinate system with the upper left corner of the display unit 50 as the origin O, the horizontal axis as XD, and the vertical axis as YD, and the unit is pixels (px). These reference points Pv1 to Pv4 and the reference points PD1 to PD4 are used when calculating the projection conversion matrix N described later. The relationship between the homogeneous coordinates Pv and PD can be expressed by Equation (2).

[0041]

Equation

[0042] N is a projection conversion matrix that converts the coordinates of the vehicle coordinate system into the coordinates of the display unit 50, and corresponds to the "second conversion information" of the present invention. The values λv and λD included in Equations (1) and (2) indicate the magnification factors in the respective homogeneous coordinates Pv and PD. No matter what values the values λv and λD take other than the value 0, each homogeneous coordinate represents the same coordinate on each coordinate system.

[0043] The product H (= NM) of the projection conversion matrix N and the projection conversion matrix M is a projection conversion matrix that converts the coordinates (xc, yc) of the image coordinate system into the coordinates (xD, yD) of the display unit 50. The relational expression using this projection conversion matrix H is Equation (3). By obtaining the projection conversion matrix H, it becomes possible to easily perform coordinate conversion between the coordinates of the image coordinate system and the coordinates of the display unit 50.

[0044]

Equation

[0045] FIG. 9 is a flowchart showing the creation process of the display target image GD in the image creation device 40. In step S1, the image creation device 40 acquires the captured image GC captured by the external camera 11 via the input I / F 41. In step S2, the image creation device 40 performs a correction process of correcting the lens distortion on the captured image GC by the lens correction unit 46A. In step S3, the image creation device 40 performs a setting process of setting the projection center T based on the viewpoint coordinates PE of the user US and the dominant eye X set by the dominant eye setting unit 45 by the projection center specifying unit 44.

[0046] In step S4, the image creation device 40 calculates the coordinates PD1 to PD4 at the four corners of the display unit 50 and the coordinates Pv1 to Pv4 at the four corners of the reference range shown in FIGS. 6 and 8 by calculating the reference range to be displayed on the display unit 50 by the projection conversion unit 46B. In step S5, the image creation device 40 substitutes the coordinates PD1 to PD4 in the coordinate system of the display unit 50 and the coordinates Pv1 to Pv4 in the vehicle coordinate system into Equation (2) by the projection conversion unit 46B, and solves the simultaneous equations including each element of the projective transformation matrix N to calculate the projective transformation matrix N. The calculated projective transformation matrix N is stored in the storage unit 43 as the transformation information N.

[0047] In step S6, the image creation device 40 calculates the product H of the projective transformation matrix N and the projective transformation matrix M (hereinafter referred to as the projective transformation matrix H) by the projection conversion unit 46B. This projection conversion unit 46B corresponds to the "second conversion information acquisition unit" of the present invention. Although the case where the projection conversion unit 46B calculates the projective transformation matrices N and M has been exemplified, it is not limited to this configuration. The process of calculating the projective transformation matrices N and M may be performed using the arithmetic processing function of the processor 42. In addition to the processor 42, a device for calculating the projective transformation matrices N and M may be provided separately. Further, the projective transformation matrix H may be stored in the storage unit 43.

[0048] In step S7, the image creation device 40 uses the projection conversion matrix H by the image generation unit 46C to calculate the pixel value of each pixel of the display unit 50 from the pixel values of the pixels of the captured image GC corresponding to this pixel value, generates data of the output image based on the calculated pixel value, and outputs it as the image data of the display target image GD. Note that for the process of calculating the pixel value of each pixel of the display unit 50 from the pixel values of the pixels of the captured image GC, a known technique may be applied. Also, the pixel value of each pixel of the display unit 50 may be calculated from the pixel values of a plurality of pixels combined with the periphery of the pixel of the captured image GC corresponding to this pixel value. In step S8, the image creation device 40 performs a process of causing the display unit 50 to display an image corresponding to the display target image GD by the display processing unit 47.

[0049] The process of step S3 will be further described with reference to FIG. 10. In step S1A, the projection center specifying unit 44 acquires the binocular viewpoint coordinates PE detected by the viewpoint detection unit 22 via the input I / F 41. In step S2A, the projection center specifying unit 44 determines whether the dominant eye X has been set by determining whether information for specifying the dominant eye X is stored in the storage unit 43. If the dominant eye X has not been set (step S2A; NO), the projection center specifying unit 44 performs a dominant eye setting process for setting the dominant eye (step S10A).

[0050] If the dominant eye X has been set (step S2A; YES), the projection center specifying unit 44 sets the position of the dominant eye X to the viewpoint coordinates PE1 (step S3A). In the next step S4A, the projection center specifying unit 44 determines whether the previous projection center T is stored in the storage unit 43. If the previous projection center T is stored (step S3A; YES), the projection center specifying unit 44 calculates the distance between the previous projection center T and the viewpoint coordinates PE1, and determines whether this distance is equal to or greater than a predetermined threshold value (step S5A).

[0051] Here, when the distance is equal to or greater than the threshold value, it means that the viewpoint of the user US has moved relatively greatly compared to the previous time. In other words, when the reference range is set using the previous projection center T and the image corresponding to the reference range (display target image GD) is displayed on the display unit 50, it corresponds to the case where the deviation between the image (display target image GD) and the scenery directly visible to the occupant exceeds the allowable range. That is, the threshold value is set to a threshold value within which the above deviation is within the allowable range. Also, if the image displayed on the display unit 50 is changed at any time due to a slight deviation in the viewpoint, there is a risk that the image displayed on the display unit 50 may appear to vibrate. However, setting this threshold value also serves to suppress this.

[0052] When the above distance is equal to or greater than the threshold value (step S5A; YES), or when the previous projection center T is not stored (step S3A; NO), the projection center specifying unit 44 sets the viewpoint coordinates PE1 as the current projection center T and stores the viewpoint coordinates PE1 in the storage unit 43 as the previous projection center T (step S6A). As a result, the projection center T stored in the storage unit 43 is updated.

[0053] On the other hand, when the distance is less than the threshold value (step S5A; NO), the previous projection center T stored in the storage unit 43 is set as the current projection center T (step S7A). Therefore, when the viewpoint of the user US hardly changes from the previous time, the previous projection center T is used as the current projection center T1, and the rewriting process of the storage unit 43 becomes unnecessary. The above is the process of step S3.

[0054] The advantageous direction setting process in step S10A will be described with reference to FIG. 11. In step S1B, the advantageous direction setting unit 45 detects the position of the feature point S that can identify the position of the face of the user US. The feature point S is a position that can be specified from the captured image GF of the in-vehicle camera 12 acquired via the input I / F 41, and any position that can identify the position of the face of the user US can be adopted. In this embodiment, the eye point setting unit 45 detects the position of the "midpoint between both eyes" as the feature point S as shown in FIG. 11 by using the detection result of the user recognition device 20.

[0055] In step S2B, the eye point setting unit 45 sets a variable α that changes according to the position of the feature point S. More specifically, when the feature point S is at the reference position as the general driving position of the user US, the variable α has a value of 0.5. As the feature point S deviates to either the left or right from the reference position, the variable α approaches the value of 0, and as the feature point S deviates to the other side (left or right) from the reference position, the variable α approaches the value of 1. In this configuration, as shown in FIG. 11, when the feature point S deviates to the right by a specified amount from the reference position, the variable α becomes 0, and when the feature point S deviates to the left by the specified amount from the reference position, the variable α becomes 1. The specified amount is, for example, 10 cm, but it may be set as appropriate.

[0056] In step S3B, the eye point setting unit 45 calculates a projection center T' including the variable α as a parameter, generates a display target image GD using the projection center T' by the image processing unit 46, and causes the display unit 50 to display an image corresponding to the display target image GD.

[0057] Here, T’ = αL + (1 - α)R ··· Equation (4) T' is the projection center for the eye point setting process, L is the coordinate of the left eye, R is the coordinate of the right eye, and T', L, and R are three-dimensional coordinates based on the origin of the vehicle coordinate system.

[0058] By the process of step S3B described above, a projection center T' corresponding to the position of the user US is set, and a display target image GD adjusted to the set projection center T' is displayed on the display unit 50.

[0059] In step S4B, the image creation device 40 performs a predetermined notification process on the user US and determines whether or not a predetermined response of the user US has been detected. Here, the predetermined notification process is a process of notifying the user US, by means of display, voice, or the like, that a predetermined response is requested when the image displayed on the display unit 50 and the image visually recognized through the front window 2W are at consecutive positions. For example, the predetermined response is a user action such as a wink or a nod, or a user action that can be input via the UI 30 (uttering a voice corresponding to "OK" or operating a predetermined operator).

[0060] When the predetermined response of the user US is not detected (step S4B: NO), the image creation device 40 proceeds to the process of step S1B. Therefore, the processes of steps S1B to S3B are repeatedly executed. For this reason, when the user US moves at least the face left and right, the projection center T' changes within the range between the right eye and the left eye, and the display target image GD corresponding to the changed projection center T' is displayed in real time.

[0061] When the predetermined response of the user US is detected (step S4B: YES), the image creation device 40 causes the dominant eye setting unit 45 to store information on the dominant eye X based on the value of the variable α in the storage unit 43. Generally, since the dominant eye is either the left or the right, the variable α is predicted to be the value 0 or the value 1. If the value is 0, information indicating that the dominant eye X is the right eye may be stored, and if the value is 1, information indicating that the dominant eye X is the left eye may be recorded. Also, if the variable α is 0 or more and less than 0.5, the dominant eye X may be determined to be the right eye, and if the variable α is 0.5 or more and less than 1.0, the dominant eye X may be determined to be the left eye. Further, if the variable α is 0 or more and less than 0.3, the dominant eye X may be determined to be the right eye, if the variable α is 0.7 or more and less than 1.0, the dominant eye X may be determined to be the left eye, and if the variable α is 0.3 or more and less than 0.7, the dominant eye setting process may be performed again assuming that the dominant eye X cannot be determined.

[0062] Also, as information on the dominant eye X, the value of the variable α itself, that is, the value of the variable α at the time when a predetermined response of the user US is detected, may be stored in the storage unit 43. In this case, the projection center T may be calculated by substituting the stored variable α into Equation (4). Since the value of the variable α to be stored is not limited to the value 0 or the value 1, in other words, since the dominant eye is not limited to either the left or the right, it becomes easier to set the projection center T so that the user US does not actually feel discomfort. If there is a user US who can be regarded as having no dominant eye, for example, the value of the variable α is stored as 0.5, and it becomes possible to set an appropriate projection center T for that user US.

[0063] Note that Equation (4) is an example of a mathematical formula and may be changed as appropriate. For example, when Equation (5) is adopted instead of Equation (4), if the variable α is less than 0 to 0.5, the dominant eye X may be determined to be the left eye, and if the variable α is 0.5 or more and less than 1.0, the dominant eye X may be determined to be the right eye. T’=(1-α)L+αR ··· Equation (5)

[0064] The reference range calculation process in step S4 will be described with reference to FIGS. 12 and 13. As shown in FIGS. 12 and 13, in step S1C, the projection conversion unit 46B determines whether a straight line LX drawn from the projection center T to the upper edge D1 of the display range of the display unit 50 intersects the setting surface SM. As shown by reference numeral A in FIG. 13, when the straight line LX intersects the setting surface SM, based on the projection center T, the coordinates on the setting surface SM to be projected onto the upper edge D1 of the display unit 50 (corresponding to reference numeral V1 in FIG. 13) can be specified. Note that the projection center T and the coordinates on the setting surface SM to be projected onto the lower edge D2 of the display range of the display unit 50 are indicated by reference numeral V2 in FIG. 13. Therefore, when the straight line LX intersects the setting surface SM (step S1C; YES), the projection conversion unit 46B can specify the range of coordinates V1 to V2 corresponding to the reference range (step S2C in FIG. 12).

[0065] On the other hand, as shown by reference sign B in FIG. 13, when the position of the projection center T is lower than the upper edge D1 of the display range of the display unit 50, since the straight line LX does not intersect the setting surface SM, the coordinates on the setting surface SM to be projected onto the upper edge D1 of the display unit 50 cannot be specified, and thus the reference range cannot be specified. For example, when the position of the eyes of the user US is lower than the display unit 50, there is a possibility that the situation as shown by reference sign B occurs. If the reference range cannot be specified, the display target image GD corresponding to the transmissive image visible when the display unit 50 is seen through by the user US cannot be displayed.

[0066] Therefore, in the present embodiment, as shown by reference sign C in FIG. 13, the projection conversion unit 46B performs a process (the process of step S3C in FIG. 12) of changing the position of the upper edge D1 of the display range of the display unit 50 to a position D1' where the straight line LX intersects the setting surface SM.

[0067] By performing the process of changing to the position D1', it becomes possible to specify the reference range (step S2C in FIG. 12). Therefore, based on the specified reference range, the display target image GD can be generated, and the display unit 50 can display a transmissive image visible when the display unit 50 is seen through by the user US. In this case, the display target image GD corresponding to the specified reference range corresponds to the image in the range between the position D1' and the lower edge D2 among the upper and lower display ranges of the display unit 50. For this reason, it is preferable to display the display target image GD in the range between the position D1' and the lower edge D2, and for the remaining range (the upper and lower ranges between the position D1' and the upper edge D1), it is preferable to display an appropriate image (for example, a single-color image).

[0068] Various methods can be considered for setting the position D1'. For example, a process of setting a position that is lower than the projection center T by a predetermined height as the position D1' may be adopted. According to this process, the position D1' can be calculated easily and in a short time. Also, a process of setting a value that is lowered by a predetermined distance from the upper edge D1 until the straight line LX intersects the setting surface SM, and setting the above value at the time when the straight line LX intersects the setting surface SM as the position D1' may be adopted.

[0069] As described above, the image creation device 40 of the present embodiment acquires the captured image GC by the input I / F 41 that functions as an information acquisition unit, and causes the display processing unit 47 to display the display target image GD on the display unit 50. Further, the image creation device 40 acquires a projection transformation matrix M that enables the coordinates of the captured image GC to be transformed from the coordinate system of the image to the coordinate system of the vehicle 2 by the processor 42 or the input I / F 41. Further, the image creation device 40 identifies the projection center T by the projection center identification unit 44 and the projection transformation unit 46B that function as a reference range calculation unit, and calculates a reference range on the setting surface SM to be projected onto the display range of the display unit 50 with the projection center T as a reference. Further, the image creation device 40 calculates a projection transformation matrix N that enables the coordinates of the reference range to be transformed from the coordinate system of the vehicle 2 to the coordinate system of the display unit 50 by the processor 42 including the projection transformation unit 46B. Then, the image creation device 40 generates the display target image GD using the projection transformation matrices M and N by the image generation unit 46C.

[0070] Since the projection transformation matrices M and N are used, highly accurate coordinate transformation is possible among the coordinate system of the image, the coordinate system of the vehicle 2, and the coordinate system of the display unit 50. Therefore, a highly accurate display target image GD matching the viewpoint of the user US who is an occupant can be generated from the captured image GC. As a result, as shown in FIG. 2, the scenery visually recognized by the user US through the front window 2W and the display target image GD displayed on the display unit 50 can be continuously displayed. As a result, it becomes possible to display a transparent image that can suppress the sense of discomfort of the user US who is an occupant.

[0071] Further, the image creation device 40 acquires user information DU that can identify at least the position of the viewpoint of the user US through the input I / F 41. Also, the image creation device 40 identifies the projection center T based on the position of the viewpoint identified from the user information DU by the projection center identification unit 44 and the projection conversion unit 46B that function as a reference range calculation unit. As a result, it becomes easier to accurately identify the projection center T corresponding to the user US, and it becomes easier to generate a high-precision display target image GD that matches the viewpoint of the user US. In this embodiment, since user information DU that can identify at least the position of the viewpoint of the user US is acquired in real time and the image seen by the user US is displayed in real time, an image suitable for driving support can be displayed.

[0072] Note that although the case where the user information DU input to the image creation device 40 is information indicating the recognition result of the user recognition device 20 has been exemplified, the user information DU input to the image creation device 40 may be the data of the captured image GF of the in-vehicle camera 12. That is, when the image creation device 40 includes a configuration corresponding to the user recognition device 20, it may be configured to input the data of the captured image GF from the outside. In this case, the configuration corresponding to the user recognition device 20 functions as an information acquisition unit that acquires user information DU that can identify the position of the viewpoint.

[0073] Also, as shown in FIG. 13, when the straight line LX drawn from the projection center T1 to the upper edge D1 of the display range of the display unit 50 does not intersect the setting surface SM, the image creation device 40 changes the position of the upper edge D1 of the display range of the display unit 50 to a lowered position D1' so that the straight line LX intersects the setting surface SM, and calculates the reference range corresponding to the changed display range. Here, the setting surface SM is set to a traveling surface that exists at a position lower than the display range of the display unit 50. According to this configuration, since the position of the projection center T is lower than the upper edge D1 of the display range of the display unit 50, it is possible to avoid a situation where the reference range cannot be calculated. Therefore, it becomes easier to display a transparent image that can suppress the discomfort of the user US for users US of various physiques, and even when the same user US assumes various postures, it becomes easier to continuously display a transparent image that can suppress the discomfort of the user US.

[0074] Further, the image creation device 40 sets the dominant eye X of the user US by the dominant eye setting unit 45, and sets the projection center T at a position reflecting the dominant eye X by the projection center specifying unit 44. According to this configuration, it is possible to display an image with reduced discomfort compared to the case where the dominant eye X is not reflected.

[0075] Further, since the image creation device 40 uses the projective transformation matrix N as the second transformation information that can convert the coordinates of the reference range from the coordinate system of the vehicle 2 to the coordinate system of the display unit 50, it becomes easier to easily obtain the image of the display range of the display unit 50 from the image of the reference range by using projective transformation. Further, since the image creation device 40 uses the projective transformation matrix M as the first transformation information that can convert the coordinates of the captured image GF from the coordinate system of the image to the coordinate system of the vehicle 2, it becomes easier to easily obtain the image of the range defined in the coordinate system of the vehicle from the captured image GF by using projective transformation. In the present embodiment, the projective transformation matrix H, which is the product of the projective transformation matrix N and the projective transformation matrix M, is obtained, and the coordinates of the captured image GF are converted from the coordinate system of the vehicle 2 to the coordinate system of the display unit 50 by using this projective transformation matrix H. Therefore, it is not necessary to separately perform the arithmetic processing using the projective transformation matrix N and the arithmetic processing using the projective transformation matrix M, which is advantageous for reducing the amount of calculation and shortening the arithmetic processing time.

[0076] The above-described present embodiment is merely an example of an embodiment of the present invention, and can be arbitrarily modified and applied without departing from the gist of the present invention.

[0077] For example, the location where the display unit 50 is placed is not limited to the dashboard and may be, for example, the pillar part. The content displayed on the display unit 50 is not limited to an image of the front of the vehicle and depends on the positional relationship between the display unit 50 and the user US. Therefore, when the display unit 50 is located on the side of the user US, an image including the side of the vehicle is displayed on the display unit 50, and when the display unit 50 is located behind the user US, an image including the rear of the vehicle is displayed on the display unit 50.

[0078] The external camera 11 to be used is not limited to one unit and may be a plurality of units. For example, the captured images of the front camera that captures the front of the vehicle and the left and right side cameras that capture the sides of the vehicle may be combined and mapped. Thereby, a display target image GD mapped with an image of the side that could not be captured by the front camera can be obtained.

[0079] The image of the external camera 11 to be used is not limited to the image at that moment, and past images may be combined. Thereby, it becomes possible to obtain a display target image GD mapped with an image below the front hood or bonnet of the vehicle or an image below the tire.

[0080] In addition, in the above embodiment, the case where the present invention is applied to the image creation device 40 shown in FIG. 1 and the control program DP of the image creation device 40 has been described, but it is not limited thereto. The information processing device of the present invention only needs to include at least a configuration related to the display of the image creation device 40, and within that range, it may include all the configurations of the display control system 1, or may not include some of the configurations of the image creation device 40. In addition, each component shown in FIG. 1 is a schematic diagram classified according to the main processing content for easy understanding of the present invention of the application, and these components can be further classified into more components according to the processing content. Also, one component can be classified so as to execute more processes.

[0081] In addition, the processing of each component may be executed by one piece of hardware or by a plurality of pieces of hardware. Also, the processing of each component may be realized by one program or by a plurality of programs.

[0082] In addition, the processing units of the flowcharts shown in each figure are divided according to the main processing contents. The embodiments are not limited by the ways and names of dividing the processing units of each flowchart. Also, the processing order of the above flowcharts is not limited to the illustrated examples.

[0083] Also, when realizing the control program DP of the present invention by a computer, it is also possible to configure it in the form of a recording medium for recording the control program DP or a transmission medium for transmitting the program. As the recording medium, a magnetic, optical recording medium or a semiconductor memory device can be used. Specifically, examples of the recording medium include a flexible disk, HDD (Hard Disk Drive), CD-ROM (Compact Disk Read Only Memory), DVD, Blu-ray (registered trademark) Disc, and magneto-optical disk. Also, as the recording medium, a portable or fixed recording medium such as a flash memory or a card-type recording medium can be mentioned. Further, the above recording medium may be a non-volatile storage device such as a RAM, ROM, or HDD which is an internal storage device provided in the display device.

Explanation of Reference Numerals

[0084] 1 Display control system 2 Vehicle (mobile object) 2W Front window 10 Photographing device 11 External camera (information acquisition unit) 12 In-vehicle camera 20 User recognition device (information acquisition unit) 21 Face recognition unit 22 Viewpoint detection unit 30 User interface 40 Image creation device (information processing device) 41 Input Interface (Information Acquisition Unit, First Conversion Information Acquisition Unit) 42 Processor (First Conversion Information Acquisition Unit) 43 Memory Unit 44 Projection Center Identification Unit (Reference Range Calculation Unit) 45 Edge Setting Unit (Reference Range Calculation Unit) 46 Image Processing Unit 46A Lens Correction Unit 46B Projection Conversion Unit (Reference Range Calculation Unit, Second Conversion Information Acquisition Unit) 46C Image Generation Unit 47 Display Processing Unit 50 Display Unit T Projection Center N Projection Transformation Matrix (First Conversion Information) M Projection Transformation Matrix (Second Conversion Information) SM Setting Surface (Ground, Driving Surface) US User (Occupant, Driver)

Claims

1. An information acquisition unit that acquires a captured image obtained by imaging the surroundings of a moving body; A display processing unit that causes a display unit visible to an occupant of the moving body to display an image to be displayed; A first conversion information acquisition unit that acquires first conversion information that enables conversion of the coordinates of the captured image from the coordinate system of the image to the coordinate system of the moving body; A reference range calculation unit that specifies a projection center corresponding to the viewpoint of an occupant of the moving body, and calculates a reference range on a set surface set in the coordinate system of the moving body to be projected onto the display range of the display unit with reference to the projection center; A second conversion information acquisition unit that calculates second conversion information that enables conversion of the coordinates of the reference range from the coordinate system of the moving body to the coordinate system of the display unit; An image generation unit that calculates each pixel value of the display range of the display unit from each pixel value of the captured image corresponding to the reference range using the first conversion information and the second conversion information, and generates the image to be displayed using the calculated pixel values; comprising When a straight line drawn from the projection center to the upper edge of the display range of the display unit does not intersect the set surface, the reference range calculation unit changes the position of the upper edge of the display range of the display unit to a lowered position so that the straight line intersects the set surface, and calculates a reference range corresponding to the changed display range. An information processing apparatus.

2. The information acquisition unit acquires user information capable of specifying at least the position of the viewpoint of the occupant, The information processing apparatus according to claim 1, wherein the reference range calculation unit specifies the projection center based on the position of the viewpoint specified from the user information.

3. The information processing apparatus according to claim 1 or 2, wherein the set surface is set on a traveling surface existing at a position lower than the display range of the display unit.

4. having an eye preference setting unit that sets the eye preference of the occupant, The information processing apparatus according to any one of claims 1 to 3, wherein the reference range calculation unit sets the projection center at a position reflecting the dominant eye of the occupant.

5. The information processing apparatus according to any one of claims 1 to 4, wherein the first conversion information and the second conversion information are projective transformation matrices.

6. A computer, An information acquisition unit that acquires a captured image obtained by imaging the surroundings of a moving body, A display processing unit that displays an image to be displayed on a display unit visible to an occupant of the moving body, A first conversion information acquisition unit that acquires first conversion information that enables conversion of the coordinates of the captured image from the coordinate system of the image to the coordinate system of the moving body, A reference range calculation unit that specifies a projection center corresponding to the viewpoint of an occupant of the moving body, and calculates a reference range on a set surface set in the coordinate system of the moving body to be projected onto the display range of the display unit with reference to the projection center, A second conversion information acquisition unit that calculates second conversion information that enables conversion of the coordinates of the reference range from the coordinate system of the moving body to the coordinate system of the display unit, An image generation unit that calculates each pixel value of the display range of the display unit from each pixel value of the captured image corresponding to the reference range by using the first conversion information and the second conversion information, and generates the image to be displayed by using the calculated pixel values, to function as, When a straight line drawn from the projection center to the upper edge of the display range of the display unit does not intersect the set surface, the reference range calculation unit changes the position of the upper edge of the display range of the display unit to a lowered position so that the straight line intersects the set surface, and calculates a reference range corresponding to the changed display range. Control program.

Images