Circuit device, display system, and electronic device

Abstract

The present application provides a circuit device, a display system, and an electronic device. The circuit device includes a storage section that stores a rendered image and a warp processing section. The warp processing section includes a coordinate conversion section, a coordinate address conversion section, and an output section. The coordinate conversion section converts an output coordinate that is a coordinate of a display image into an input coordinate that is a coordinate on the rendered image, by coordinate conversion based on a warp parameter and a rotation correction parameter. The coordinate conversion section converts the input coordinate into a read address of the storage section. The output section reads out rendered image pixel data from the read address of the storage section, and outputs pixel data at the output coordinate of the display image based on the read-out pixel data.

Description

TECHNICAL FIELD

[0001] The present application relates to a circuit device, a display system, an electronic apparatus, and the like. BACKGROUND

[0002] A head-up display that displays information superimposed on a user's field of view by displaying an image on a transparent screen or the like is known. By combining such a head-up display and AR technology, it is possible to cause the head-up display to display a virtual object in such a manner that the virtual object follows an actual object such as a preceding vehicle or a road. AR is an abbreviation for Augmented Reality.

[0003] In Patent Literature 1 and Patent Literature 2, a technology of performing vibration correction for causing a virtual object of AR to follow an actual object is disclosed. In Patent Literature 1, a low-frequency band of a posture change is corrected by image processing, and a high-frequency band of a posture change is corrected by a projection optical unit. By correcting the high-frequency band of the posture change with the projection optical unit, in the case where a posture change of a vehicle occurs, a position of a projection image is corrected so that a virtual image is superimposed on a correct position with respect to a real image. In Patent Literature 2, an image that needs vibration correction and an image that does not need vibration correction are synthesized on a frame memory, and distortion correction is performed on the synthesized image.

[0004] Patent Literature 1: Japanese Patent Application Publication No. 2019-98756

[0005] Patent Literature 2: Japanese Patent Application Publication No. 2020-050328

[0006] In the above-described Patent Literature 1 and Patent Literature 2, a specific structure of warp processing in the case where correction such as vibration correction is not performed in the warp processing is not disclosed. That is, in Patent Literature 1, a high-frequency band of a posture change is corrected with a projection optical unit, and a posture change is not corrected in warp processing. In addition, in Patent Literature 2, Figure 6 correction circuit described in Patent Literature 1 and Patent Literature 2 performs distortion correction, but a specific structure of the correction circuit is not described. SUMMARY

[0007] One embodiment of the present disclosure relates to a circuit device that performs display control of a head-up display that displays a virtual object corresponding to an actual object in an actual space in a display region, the circuit device including: a storage section that stores a rendering image including the virtual object; and a warp processing section that performs warp processing on the rendering image to generate a display image displayed in the display region, the warp processing section including: a coordinate conversion section that converts an output coordinate that is a coordinate on the display image into an input coordinate that is a coordinate on the rendering image by coordinate conversion based on a warp parameter corresponding to a distortion of an image due to an optical system and a rotation correction parameter, thereby obtaining the input coordinate to which rotation processing that corrects a rotation indicated by the rotation correction parameter is added; a coordinate address conversion section that converts the input coordinate into a read address of the storage section; and an output section that reads out pixel data of the rendering image from the read address of the storage section, and outputs pixel data at the output coordinate of the display image based on the read pixel data.

[0008] In addition, another embodiment of the present disclosure relates to a display system that is a head-up display that displays a virtual object corresponding to an actual object in an actual space in a display region, the display system including: a rendering image generation section that generates a rendering image including the virtual object; a storage section that stores the rendering image; and a warp processing section that performs warp processing on the rendering image to generate a display image displayed in the display region, the warp processing section including: a coordinate conversion section that converts an output coordinate that is a coordinate on the display image into an input coordinate that is a coordinate on the rendering image by coordinate conversion based on a warp parameter corresponding to a distortion of an image due to an optical system and a rotation correction parameter, thereby obtaining the input coordinate to which rotation processing that corrects a rotation indicated by the rotation correction parameter is added; a coordinate address conversion section that converts the input coordinate into a read address of the storage section; and an output section that reads out pixel data of the rendering image from the read address of the storage section, and outputs pixel data at the output coordinate of the display image based on the read pixel data.

[0009] Further, another embodiment of the present disclosure relates to an electronic device including the above-described circuit device. BRIEF DESCRIPTION OF DRAWINGS

[0010] Figure 1 is an example of HUD-based AR display.

[0011] Figure 2 is a diagram illustrating delay compensation of the present embodiment.

[0012] Figure 3is a configuration example of a circuit device.

[0013] Figure 4 is a schematic diagram of coordinate conversion.

[0014] Figure 5 is a first detailed configuration example of a circuit device.

[0015] Figure 6 is a first configuration example of a display system.

[0016] Figure 7 is a timing chart that explains an operation of the display system in the first configuration example.

[0017] Figure 8 is a second detailed configuration example of a circuit device.

[0018] Figure 9 is a second configuration example of a display system.

[0019] Figure 10 is a timing chart that explains an operation of the display system in the second configuration example.

[0020] Figure 11 is an explanatory diagram of processing in a third detailed configuration example and a fourth detailed configuration example.

[0021] Figure 12 is a third detailed configuration example of a circuit device.

[0022] Figure 13 is a fourth detailed configuration example of a circuit device.

[0023] Figure 14 is a configuration example of an electronic device.

[0024] Label Explanation

[0025] 10: preceding vehicle; 20: virtual object; 24: first display object; 25: second display object; 30: automobile; 40: display region; 100: display system; 110: tracking processing section; 120: rendered image generating section; 130: warp processing section; 131: coordinate conversion section; 132: coordinate address conversion section; 133: output section; 134: distortion correction section; 135: rotation correction section; 136: interpolation processing section; 137: output buffer; 140: rotation correction parameter operation section; 145: warp parameter storage section; 146: warp parameter correction section; 150: processing device; 160: circuit device; 161: storage section; 162: warp parameter selection section; 400: HUD; 450: sensor; 500: electronic device; 532: projection device; AR1: first region; AR2: second region; CWPM: corrected warp parameter; DSIM: display image; IM24: image of first display object; IM25: image of second display object; RC: rotation center; RENIM: rendered image; RTPM: rotation correction parameter; WPM: warp parameter. DETAILED DESCRIPTION

[0026] Hereinafter, a preferred embodiment of the present disclosure will be described in detail. In addition, the present embodiment described below is not intended to unduly limit the content recited in the claims, and the structures described in the present embodiment are not necessarily all essential structural elements.

[0027] 1. Structure example of circuit device

[0028] First, an AR display based on a HUD will be described. Figure 1 is an example of an AR display based on a HUD. HUD is an abbreviation of head-up display, and AR is an abbreviation of augmented reality. Hereinafter, an example in which an automobile 30 is equipped with a HUD will be mainly described, but the HUD of the present embodiment can be mounted on various moving bodies such as an airplane, a ship, or a two-wheeled vehicle.

[0029] In Figure 1 , an AR display viewed by a driver via a HUD is shown. The driver views a preceding vehicle 10, a road on which the preceding vehicle 10 travels, and scenery around them through a front windshield. In addition, the driver views a virtual image projected by the HUD into a display region 40 of the windshield, and the virtual image looks like a virtual object 20 overlapping with the actual space from the driver's viewpoint. The display region 40 indicates a range in which the HUD can project the virtual image, and the virtual object 20 is displayed in the display region 40. Figure 1The virtual object 20 in the HUD is an AR display object, displayed in a way that its position follows the preceding vehicle 10 or the road. That is, the HUD wants to display the virtual object 20 in a way that maintains the relative positional relationship between the preceding vehicle 10 or the road and the virtual object 20 as seen from the driver's perspective.

[0030] The HUD display system uses sensors such as LiDAR to track a vehicle 10, renders a virtual object 20 based on the tracking results, and then distorts the rendered image for display on the HUD. During the period from the timing of sensor sampling to the timing of displaying the virtual object 20 on the HUD, there is a delay caused by tracking processing, rendering processing, distortion processing, or data communication. This delay caused by such processing or communication is called latency.

[0031] The delay causes positional and rotational offsets between the virtual object 20 and the preceding vehicle 10, etc. That is, the position and angle of the preceding vehicle 10, etc., when the display system renders the virtual object 20 deviates from the actual position and angle of the preceding vehicle 10, etc., when the virtual object 20 is displayed on the HUD, resulting in the virtual object 20 and the preceding vehicle 10 being displayed at an offset. For example, in the rendering process, even if the position of the preceding vehicle 10, etc., at a future display timing is predicted, since there is time between the prediction point and the display, the predicted position and angle of the preceding vehicle 10, etc., may deviate from the actual position and angle of the preceding vehicle 10, etc., when the virtual object 20 is displayed on the HUD. Furthermore, in the following embodiments, the correction of rotational offset is mainly described. However, the circuit device or display system of this disclosure can further correct for positional offsets.

[0032] Figure 2 This is a diagram illustrating the delay compensation in this embodiment. Figure 2 The left image shows an example of a HUD display without delay compensation. For example... Figure 2 As shown in the left figure, when a car equipped with a HUD rotates counterclockwise by θ, the display area 40 of the HUD rotates counterclockwise by θ. RC represents the center of rotation. Thus, the virtual object 20 is displayed with the virtual object 20 rotated by θ relative to the preceding vehicle 10, which is the actual object. Here, we assume a rotation speed that cannot be followed during rendering, that is, a rotation speed at which the delay from rendering to HUD display becomes a problem.

[0033] exist Figure 2 The right-hand image shows an example of a HUD display with delay compensation applied. For example... Figure 2In the right view of FIG. 1, in the present embodiment, the virtual object 20 is displayed by being rotated clockwise by θ with RC as the center, so that a rotation shift caused by the delay is compensated for, and the preceding vehicle 10 as the actual object is made to coincide with the display of the virtual object 20. By performing such delay compensation, even in the case where the vehicle equipped with the HUD is rotated, it is possible to avoid a rotation shift in the display. In the present embodiment, by performing the delay compensation in the warping process as close to the display timing as possible, it is possible to reduce the position shift caused by the delay as much as possible, and the details will be described later.

[0034] In addition, the circuit device and the display system of the present disclosure can also perform a rotation shift correction that corrects a mounting error of the HUD in addition to the delay compensation. When the correction angle of the former is θllt and the correction angle of the latter is θerr, the total correction angle is θ = θllt + θerr. θllt is a parameter that varies depending on the rotation of the HUD, and θerr is a fixed parameter corresponding to the mounting error. In the following embodiments, the case where the correction angle is θ = θllt will be mainly described, but by adding θerr to the correction angle θ, it is possible to additionally correct the mounting error as described above.

[0035] Figure 3 is a configuration example of the circuit device 160 in the present embodiment. The circuit device 160 performs display control of the HUD 400 that displays the virtual object 20 corresponding to the actual object of the actual space in the display region 40. The circuit device 160 includes a storage section 161 and a warping processing section 130. The circuit device 160 is also referred to as a HUD controller, and is configured by, for example, an integrated circuit device in which a circuit is integrated on a semiconductor substrate.

[0036] The storage section 161 temporarily stores a rendered image RENIM including the virtual object 20. The rendered image RENIM is input to the circuit device 160 from an external MPU or the like, but can also be configured to be rendered within the circuit device 160. The storage section 161 is a line buffer that stores image data of the number of lines required for warping processing. The number of lines is only required to be the maximum moving line number in the vertical direction in the warping processing plus a margin. The storage section 161 is, for example, a semiconductor memory such as a RAM. RAM is an abbreviation for Random Access Memory.

[0037] The warp processing section 130 generates a display image DSIM displayed on the display region 40 by performing warp processing on the rendered image RENIM, and outputs the display image DSIM to the HUD 400. The warp processing is a coordinate transformation between the rendered image RENIM and the display image DSIM, and in the present embodiment includes distortion correction and rotation offset correction. In the present embodiment, the warp processing section 130 is an inverse warp engine. Inverse warp refers to finding a transformation of each pixel of an output image of a warp engine from a pixel at an arbitrary position in an input image. An inverse warp engine refers to a warp engine having a function of inverse warp.

[0038] The warp processing section 130 includes a coordinate transformation section 131, a coordinate address transformation section 132, and an output section 133. The warp processing section 130 is constituted by a logic circuit, for example, a gate array with automatic wiring or a standard cell array with automatic wiring.

[0039] The coordinate transformation section 131 transforms a coordinate on the display image DSIM, that is, an output coordinate (Xtrg, Ytrg), into a coordinate on the rendered image RENIM, that is, an input coordinate (Xsrc, Ysrc), by coordinate transformation based on the warp parameter and the rotation correction parameter. Thus, the coordinate transformation section 131 finds the input coordinate (Xsrc, Ysrc) to which a rotation process of correcting rotation indicated by the rotation correction parameter is added.

[0040] The warp parameter is a parameter corresponding to distortion of an image caused by an optical system of the HUD 400. The optical system is, for example, a lens for projecting an image onto a screen, a screen on which an image is projected, or both. The distortion of the image is, for example, distortion caused by a distortion aberration of the lens, distortion caused by a curvature of the screen, or both. The distortion correction is a correction that eliminates the above-mentioned distortion of the image, and the image after the distortion correction is displayed as a virtual image without distortion in a visual field of a viewer by being projected onto the screen.

[0041] The rotation correction parameter is a parameter corresponding to a rotation offset between an actual object and a virtual object displayed following the actual object. The rotation offset correction is a process of correcting the rotation offset. The rotation offset is generated due to a change in posture of a mobile body on which the HUD is mounted, and the rotation offset correction is performed in accordance with the change in posture of the mobile body.

[0042] Figure 4 A schematic diagram showing coordinate transformation. In Figure 4In the above, f() indicates distortion correction, and r() indicates rotation shift correction. The coordinate transformation of the distortion correction is represented by the following equation (1). In the following equation (1), if a transformation matrix is assumed to be Adis, each element within Adis corresponds to a distortion parameter. Further, the coordinate transformation of the rotation shift correction is represented by the following equation (2). In the following equation (2), if a transformation matrix is assumed to be Arot, cos θ and sin θ within Arot correspond to rotation correction parameters. (cx, cy) is a coordinate of the rotation center RC. Further, (cx, cy) can also be used as the rotation correction parameters. y ) is a coordinate of the rotation center RC. Further, (cx, cy) can also be used as the rotation correction parameters. θ

[0043]

[0044]

[0045] The above equation (1) is a polynomial of the following equations (3), (4) represented in a matrix form. That is, the distortion parameters correspond to coefficients of each term of the polynomial. Further, an example of a 4th degree polynomial is shown in the above equation (1) and the following equations (3), (4), but the degree of the polynomial is not limited to 4th degree.

[0046] Xrot = a 14 Xtrg 4 +a 13 Ytrg 4 +a 12 Xtrg 3 Ytrg+a 11 Xtrg 2 Ytrg 2 +a 10 XtrgYtrg 3

[0047] +a9Xtrg 3 +a8Ytrg 3 +a7Xtrg 2 Ytrg+a6XtrgYtrg 2

[0048] +a5Xtrg 2 +a4Ytrg 2 +a3XtrgYtrg

[0049] +a2Xtrg+a1Ytrg+a0

[0050] …(3)

[0051] Yrot = b 14 Xtrg 4 +b 13 Ytrg 4 +b 12 Xtrg​3 Ytrg+b 11 Xtrg 2 Ytrg 2 +b 10 XtrgYtrg 3

[0052] +b9Xtrg 3 +b8Ytrg 3 +b7Xtrg 2 Ytrg+b6XtrgYtrg 2

[0053] +b5Xtrg 2 +b4Ytrg 2 +b3XtrgYtrg

[0054] +b2Xtrg+b1Ytrg+b0

[0055] ... (4)

[0056] The coordinate conversion section 131 performs the distortion correction and the rotation shift correction described above in combination or in two stages. In the case of performing in combination, this is described later in the first detailed configuration example, and in the case of performing in two stages, this is described later in the second detailed configuration example. Here, a summary is explained using mathematical expressions.

[0057] In the case of performing in combination, the distortion parameters are corrected by rotation correction parameters as shown in the following expressions (5) to (7), and the corrected distortion parameters are input to the coordinate conversion section 131. In the following expressions (5) to (7), each element of the transformation matrix Atot corresponds to the corrected distortion parameters. The coordinate conversion section 131 performs the distortion correction and the rotation shift correction simultaneously by performing coordinate conversion using the transformation matrix Atot as shown in the following expression (8).

[0058]

[0059] a' 14 = a 14 cos θ - b 14 sin θ

[0060] a' 13 = a 13 cos θ - b 13 sin θ

[0061] a' 12 = a 12 cos θ - b 12 sin θ

[0062]

[0063] a'0= a0cos θ - b0sin θ + cx(l - cos θ) + cy sin θ

[0064] (6)

[0065] b' 14 = a 14 sin θ + b 14 cos θ

[0066] b' 13 = a 13 sin θ + b 13 cos θ

[0067] b' 12 = a 12 sin θ + b 12 cos θ

[0068]

[0069] b'0= a0sin θ + b0cos θ + cy(l - cos θ) - cx sin θ

[0070] (7)

[0071]

[0072] In the case of performing in 2 stages, the coordinate conversion section 131 performs the coordinate conversion of the rotation correction shown in the above formula (2) on the converted coordinate (xrot, yrot) after the coordinate conversion of the distortion correction shown in the above formula (1) is performed, thereby performing the distortion correction and the rotation offset correction in 2 stages.

[0073] Figure 3 The coordinate address conversion section 132 shown in the figure converts the input coordinate (Xsrc, Ysrc) found by the coordinate conversion section 131 into the read address of the storage section 161. That is, when the address of the pixel data of the input coordinate (Xsrc, Ysrc) stored is set as ADDRxy, the coordinate address conversion section 132 converts the input coordinate (Xsrc, Ysrc) into the read address ADDRxy.

[0074] The output section 133 reads out the pixel data of the rendering image RENIM from the read address ADDRxy of the storage section 161, and outputs the pixel data at the output coordinate (Xtrg, Ytrg) of the display image DSIM on the basis of the read pixel data. In addition, as described later, the output section 133 can also read out a plurality of pixel data around the input coordinate (Xsrc, Ysrc), perform an interpolation process on the plurality of pixel data, and thereby output the pixel data at the output coordinate (Xtrg, Ytrg).

[0075] According to the above embodiment, the coordinate transformation based on the distortion parameters and the rotation correction parameters is executed immediately before the pixel data is read out from the storage section 161, and the display image DSIM is output based on the read-out pixel data. Thereby, the delay from the rotation offset correction to the output of the display image DSIM can be minimized. In the case where the present embodiment is applied to the above-described AR display, the virtual object 20 can be caused to follow the preceding vehicle 10 as the actual object with high precision.

[0076] Further, in the above-described Patent Document 1, the measurement time of the posture data is made shorter than the communication time of the image data or the like, the projection optical unit performs correction of a high frequency band using the posture data, and thereby the positional offset of the actual image and the projection image due to the communication time is suppressed. However, since the data processing time for controlling the projection optical unit and the operation time of the constituent components are required, the positional offset due to the time remains. Further, in the above-described Patent Document 2, a distortion parameter table of the amount of one screen on which the vibration correction is implemented is prepared on the memory, and the distortion processing is performed using the distortion parameter table of the amount of one screen, and thereby the positional offset is suppressed. However, since the time for preparing the distortion parameter table of the amount of one screen on the memory is required, the positional offset due to the time remains. Further, since the specific structure of the distortion processing is not described, the contents related to the delay in the distortion processing are unclear.

[0077] Further, in the present embodiment, as shown in the above equations (3), (4), the distortion parameters are the coefficients of the polynomial that corrects the distortion of the image.

[0078] In the table method distortion correction, a plurality of parameters corresponding to a plurality of coordinates are used. For example, in the above-described Patent Document 2, since the vibration correction is implemented for each of the plurality of parameters, the operation takes time. In this regard, according to the present embodiment, since the distortion correction is performed by the polynomial, as shown in the above equations (3), (4), the distortion processing can be implemented by only one set of distortion parameters. In the case where the distortion correction and the rotation offset correction are implemented together, the distortion parameters are multiplied by the rotation matrix, but the distortion parameters are one set, and thus the matrix operation is performed once, and the operation time is shortened compared to the table method.

[0079] Further, in the present embodiment, the table method distortion correction can also be employed. For example, in the case where the distortion correction and the rotation offset correction are implemented in two stages, since the distortion correction and the rotation offset correction are operated respectively, it is considered that the operation time of the rotation offset correction is the same regardless of whether the distortion correction is the table method or the polynomial method.

[0080] 2. First detailed structure example

[0081] Figure 5is a first detailed configuration example of the circuit device 160. In Figure 5 The circuit device 160 includes a warp parameter storage section 145 and a warp parameter correction section 146 in the above embodiment. Further, the output section 133 includes an interpolation processing section 136 and an output buffer 137. Further, the same reference numerals are attached to the constituent elements already explained, and the explanation about the constituent elements is appropriately omitted.

[0082] The warp parameter storage section 145 stores the warp parameter WPM used for the distortion correction. The warp parameter storage section 145 is a semiconductor memory such as a RAM or a nonvolatile memory. As described above, the warp parameter WPM is each element of the transformation matrix Adis constituting the distortion correction, and specifically, is a coefficient indicating a polynomial of the distortion correction.

[0083] The rotation correction parameter RTPM is input to the warp parameter correction section 146. The rotation correction parameter RTPM is input to the circuit device 160 from an external MPU or the like, for example, but can also be configured to calculate the rotation correction parameter RTPM within the circuit device 160 from the tracking information. As described above, the rotation correction parameter RTPM is cosθ and sinθ included in the transformation matrix Arot of the rotation offset correction. θ is a parameter that varies according to the tracking information of a moving body or the like, and is updated, for example, every frame. However, the update interval of θ can be arbitrary.

[0084] The warp parameter correction section 146 calculates the corrected warp parameter CWPM by correcting the warp parameter WPM using the rotation correction parameter RTPM. This calculation is the calculation shown in the above formula (5), and the corrected warp parameter CWPM is each element constituting the transformation matrix Atot. The coordinate conversion section 131 performs coordinate conversion using the corrected warp parameter CWPM. This coordinate conversion is the calculation shown in the above formula (8).

[0085] According to the above embodiment, the warp parameter correction section 146 calculates the corrected warp parameter CWPM before the coordinate conversion by the coordinate conversion section 131, and the coordinate conversion section 131 performs coordinate conversion of the output coordinates (Xtrg, Ytrg) into the input coordinates (Xsrc, Ysrc) using the corrected warp parameter CWPM. Thereby, compared to the structure in which a warp parameter table of the amount of one screen is prepared temporarily in a memory as in the above Patent Literature 2, it is possible to shorten the delay time from the acquisition of the rotation correction parameter RTPM to the output of the display image DSIM. More specifically, by performing the distortion correction using a polynomial as above, the calculation of the corrected warp parameter CWPM requires only one matrix calculation, and thus the calculation time is shortened compared to the table method.

[0086] The interpolation processing section 136 generates pixel data of the display image DSIM by interpolation processing. Specifically, the coordinate address conversion section 132 generates a read address group based on a reference read address that is a read address. The read address group is a plurality of read addresses for reading out a pixel data group of a periphery of the input coordinate (Xsrc, Ysrc) output by the coordinate conversion section 131 from the storage section 161. The reference read address is a read address corresponding to one pixel data in the pixel data group, for example, a read address of a pixel data closest to the input coordinate (Xsrc, Ysrc). The interpolation processing section 136 reads out the pixel data group corresponding to the read address group, performs interpolation processing of the pixel data group, thereby generating pixel data of the output coordinate (Xtrg, Ytrg).

[0087] The output buffer 137 buffers the pixel data output by the interpolation processing section 136 and outputs to the HUD 400. The output buffer 137 is constituted by a FIFO memory or a line buffer, and the like, as long as it can temporarily store pixel data corresponding to a period shorter than one frame. For example, the output buffer 137 temporarily stores several pixels to several tens of lines or so of pixel data.

[0088] Figure 6 is a first configuration example of the display system 100. The display system 100 includes the processing device 150 and the circuit device 160.

[0089] The processing device 150 includes the tracking processing section 110, the rendered image generation section 120, and the rotation correction parameter operation section 140. The processing device 150 is, for example, a processor such as a CPU, a GPU, or a microcomputer.

[0090] The output signal of the sensor 450 is input to the tracking processing section 110. The sensor 450 is a sensor that detects the position, posture, or motion of a mobile body, a viewer, or an actual object. The sensor 450 is provided to the mobile body, and includes, for example, a Lidar, an IMU, a camera, an eye tracking sensor, or a head tracking sensor. The Lidar is an abbreviation of Light Detection and Ranging, and is a sensor that acquires three-dimensional information such as z mapping. The IMU is an abbreviation of Inertial Measurement Unit, and is a sensor that detects the motion of a single axis or multiple axes. The IMU is constituted by, for example, an acceleration sensor, a gyro sensor, or a combination thereof. The camera is a sensor that captures an image as two-dimensional information. The eye tracking sensor is a sensor that detects the position, line-of-sight direction, or both of the eyes of a viewer. The head tracking sensor is a sensor that detects the position, posture, or both of the head of a viewer.

[0091] The mobile body is an object that moves in the actual space while the HUD 400, the viewer, and the sensor 450 are placed thereon, such as a car, a two-wheeled vehicle, an airplane, or a ship. The viewer is a user who views the virtual image projected by the HUD 400, and is a driver or a passenger of the mobile body. The actual object is an object that exists in the actual space. The actual object is an object whose position or posture in the HUD display region viewed by the viewer changes when the position or posture of the mobile body, the viewer, or the actual object changes.

[0092] The tracking processing section 110 tracks the position, posture, or motion of the mobile body, the viewer, or the actual object based on the output signal of the sensor 450, and outputs the result thereof as tracking information. For example, the tracking processing section 110 tracks the actual object based on two-dimensional ranging information from a Lidar or two-dimensional images from a camera. In addition, the tracking processing section 110 tracks a car based on information of acceleration or angular velocity from an IMU. In addition, the tracking processing section 110 tracks the eyes of the driver based on information of the position or line-of-sight direction of the eyes from an eye tracking sensor.

[0093] The tracking information can be any form of information as long as it is information indicating the position, posture, or motion of the mobile body, the viewer, or the actual object. For example, the tracking information is a coordinate indicating the position in the actual space, an angle indicating the posture, a vector indicating parallel movement, or an angular velocity indicating rotation, or the like. Alternatively, the tracking information can be information obtained by converting the above-mentioned coordinate in the actual space or the like into a coordinate, an angle, a vector, or an angular velocity on an image, or the like. The tracking information includes first tracking information of the mobile object, second tracking information of the viewer, and third tracking information of the actual object. However, the tracking information can omit the second tracking information of the viewer as long as it includes at least one of the first to third tracking information.

[0094] The tracking processing section 110 tracks the position, posture, or motion of the mobile body, the viewer, or the actual object based on the output signal of the sensor 450, and outputs the result thereof as tracking information. For example, the tracking processing section 110 tracks the actual object based on two-dimensional ranging information from a Lidar or two-dimensional images from a camera. In addition, the tracking processing section 110 tracks a car based on information of acceleration or angular velocity from an IMU. In addition, the tracking processing section 110 tracks the eyes of the driver based on information of the position or line-of-sight direction of the eyes from an eye tracking sensor.

[0095] Tracking information can be any form of information representing the position, posture, or motion of a moving object, viewer, or actual object. For example, tracking information can be coordinates representing a position in actual space, angles representing posture, vectors representing parallel movement, or angular velocities representing rotation. Alternatively, tracking information can be obtained by converting the aforementioned coordinates in actual space into coordinates, angles, vectors, or angular velocities on an image. Tracking information includes first tracking information of the moving object, second tracking information of the viewer, and third tracking information of the actual object. However, it is sufficient for the tracking information to include at least one of the first to third tracking information; for example, the second tracking information of the viewer can be omitted.

[0096] The rendering image generation unit 120 renders virtual objects based on tracking information of moving objects, viewers, or actual objects, and outputs a rendered image (RENIM) containing the virtual objects. Specifically, the rendering image generation unit 120 determines the position of the actual object that can be seen in the display area of ​​the HUD 400, and renders the virtual object at the position corresponding to the position of the actual object.

[0097] The rotation correction parameter calculation unit 140 calculates the rotation correction parameter RTPM, which serves as a delay compensation parameter, based on the tracking information. The tracking information used here is information sampled after the tracking information used in the rendering process. From the viewpoint of minimizing delay, it is preferable to use tracking information obtained immediately before or as close as possible to the parameter calculation. Furthermore, the delay compensation parameter is a parameter used to compensate for the display offset between the virtual object in the rendered image and the virtual object at the display timing. In this embodiment, the delay compensation parameter is a rotation angle on the image data, or sinθ and cosθ when that rotation angle is set to θ.

[0098] The circuit device 160 includes a storage unit 161, a distortion parameter storage unit 145, a distortion parameter correction unit 146, and a distortion processing unit 130. The operation of each unit is as follows: Figure 5 As explained in the text.

[0099] in addition, Figure 6 The structure of the processing device 150 and circuit device 160 is just one example. The display system 100 can simply include a tracking processing unit 110, a rendering image generation unit 120, a rotation correction parameter calculation unit 140, a storage unit 161, a distortion processing unit 130, a distortion parameter storage unit 145, and a distortion parameter correction unit 146. For example, in Figure 6 In the circuit device 160, the distortion parameter storage unit 145 and the distortion parameter correction unit 146 may also be included in the processing device 150.

[0100] Figure 7This is a timing diagram illustrating the operation of the display system 100 in the first structural example. Furthermore, Figure 7 It is a schematic diagram representing the timing of an action, not a strict timing representation.

[0101] The rendering image generation unit 120 generates a rendering image RENIM based on the tracking information obtained at time tr1. In this embodiment, a delay, i.e., a rotational offset of the virtual object, is generated based on the time from time tr1 to the HUD display.

[0102] The rotation correction parameter calculation unit 140 calculates the rotation correction parameter RTPM based on the tracking information obtained at time tr1 and the tracking information obtained at time tp1 after time tr1. Thus, the rotation correction parameter RTPM is calculated to compensate for the rotation between time difference tp1 and tr1.

[0103] The distortion parameter correction unit 146 uses the rotation correction parameter RTPM to calculate the corrected distortion parameter CWPM. The distortion processing unit 130 uses the corrected distortion parameter CWPM to perform distortion processing and outputs the display image DSIM of the first frame to the HUD 400.

[0104] Similarly, the rendering image generation unit 120 generates a rendering image RENIM based on the tracking information obtained at time tr2. The rotation correction parameter calculation unit 140 calculates the rotation correction parameter RTPM based on the tracking information obtained at time tr2 and the tracking information obtained at time tp2 after time tr2. The distortion parameter correction unit 146 calculates the corrected distortion parameter CWPM using the rotation correction parameter RTPM. The distortion processing unit 130 performs distortion processing using the corrected distortion parameter CWPM and outputs the display image DSIM of the second frame to the HUD 400.

[0105] In this embodiment, after the corrected warp parameter CWPM is calculated, a coordinate transformation using the corrected warp parameter CWPM is performed immediately. Therefore, the time from the time tp1, tp2 used in the calculation of the rotation correction parameter RTPM to the display of the DSIM image can be shortened. Specifically, the rotation correction parameter RTPM used for the warp processing of the second frame after the first frame is updated in the first frame. That is, the delay from time tp1 to the HUD display is shorter than one frame. The timing of updating the rotation correction parameter RTPM only needs to be within the first frame, but it is preferably timed as close as possible to the beginning of the second frame. For example, the update timing is preferably within the latter half of the period after dividing the first frame into two equal parts, and more preferably within the last period after dividing the first frame into four equal parts.

[0106] 3. Second detailed structural example

[0107] Figure 8This is a second detailed structural example of the circuit device 160. Figure 8 In the circuit device 160, a distortion parameter storage unit 145 is included. Additionally, the coordinate transformation unit 131 includes a distortion correction unit 134 and a rotation correction unit 135, and the output unit 133 includes an interpolation processing unit 136 and an output buffer 137. Furthermore, the same reference numerals are used to denote the components already described, and descriptions of these components are omitted where appropriate.

[0108] The distortion correction unit 134 performs a first coordinate transformation based on the distortion parameter WPM to transform the output coordinates (Xtrg, Ytrg) and obtains the coordinates (Xrot, Yrot). The first coordinate transformation is the operation shown in equation (1) above.

[0109] The rotation correction unit 135 performs a second coordinate transformation by rotating the coordinates (Xrot, Yrot) after the first coordinate transformation using the rotation correction parameter RTPM, and calculates the input coordinates (Xsrc, Ysrc). The second coordinate transformation is the operation shown in equation (2) above.

[0110] According to the above embodiment, since rotational offset correction is performed separately from distortion correction, it is not necessary to calculate the distortion parameter WPM. Furthermore, the rotational correction unit 135 performs rotational offset correction just before the coordinate address transformation. Therefore, compared to the structure in Patent Document 2 described above, which temporarily prepares a distortion parameter table for one frame in memory, the delay from obtaining the rotational correction parameter RTPM to outputting the displayed image DSIM can be shortened.

[0111] Figure 9 This is a second structural example of the display system 100. The display system 100 includes a processing unit 150 and a circuit unit 160. The processing unit 150 includes a tracking processing unit 110, a rendering image generation unit 120, and a rotation correction parameter calculation unit 140. Each part is as follows... Figure 6 As described in the diagram. The circuit device 160 includes a storage unit 161, a distortion parameter storage unit 145, and a distortion processing unit 130. Each part is as described in... Figure 8 As explained in the text.

[0112] in addition, Figure 9 The structure of the processing device 150 and circuit device 160 is just one example; any display system 100 that includes a tracking processing unit 110, a rendering image generation unit 120, a rotation correction parameter calculation unit 140, a storage unit 161, a distortion processing unit 130, and a distortion parameter storage unit 145 is acceptable. For example, in Figure 9 In the circuit device 160, the rotation correction parameter calculation unit 140 included in the processing device 150 may also be included in the circuit device 160.

[0113] Figure 10 This is a timing diagram illustrating the operation of the display system 100 in the second structural example. Furthermore, Figure 10 It is a schematic diagram representing the timing of an action, not a strict timing representation.

[0114] The rendering image generation unit 120 generates a rendering image RENIM based on the tracking information obtained at time tr1. In this embodiment, a delay, i.e., a rotational offset of the virtual object, is generated based on the time from time tr1 to the HUD display.

[0115] The rotation correction parameter calculation unit 140 calculates the rotation correction parameter RTPM based on the tracking information obtained at time tr1 and the tracking information obtained at time tp1 after time tr1. Thus, the rotation correction parameter RTPM is calculated to compensate for the rotation between time difference tp1 and tr1.

[0116] The distortion correction unit 134 performs distortion correction using the distortion parameter WPM. The rotation correction unit 135 performs rotation offset correction on the distortion-corrected coordinates using the rotation correction parameter RTPM. The coordinate address transformation unit 132 transforms the rotation offset corrected coordinates into a read address, and the output unit 133 reads the pixel data from the read address and outputs the first frame display image DSIM to the HUD 400 based on the pixel data.

[0117] Similarly, the rendering image generation unit 120 generates a rendering image RENIM based on the tracking information obtained at time tr2. The rotation correction parameter calculation unit 140 calculates the rotation correction parameter RTPM based on the tracking information obtained at time tr2 and the tracking information obtained at time tp2 after time tr2. The distortion correction unit 134 performs distortion correction using the warp parameter WPM. The rotation correction unit 135 performs rotation offset correction on the distortion-corrected coordinates using the rotation correction parameter RTPM. The coordinate address transformation unit 132 transforms the rotation offset-corrected coordinates into a read address, and the output unit 133 reads pixel data from the read address and outputs the display image DSIM of the second frame to the HUD 400 based on the pixel data.

[0118] In this embodiment, after the rotation correction parameter RTPM is calculated, a rotation coordinate transformation using the RTPM is performed immediately. Therefore, the time from the time tp1, tp2 used in the calculation of the rotation correction parameter RTPM to the display of the DSIM image can be shortened. Specifically, the rotation correction parameter RTPM used for the distortion processing of the second frame after the first frame is updated in the first frame. That is, the delay from time tp1 to the HUD display is shorter than one frame. The timing of updating the rotation correction parameter RTPM only needs to be within the first frame, but it is preferably timed as close as possible to the beginning of the second frame. For example, the update timing is preferably within the latter half of the period after dividing the first frame into two equal parts, and more preferably within the last period after dividing the first frame into four equal parts.

[0119] 4. Third and Fourth Detailed Structural Examples

[0120] Figure 11 Explanatory diagrams showing the processes in the third and fourth detailed structural examples are provided. Here, the common parts of the third and fourth detailed structural examples are explained; regarding the differences, [further details are provided]. Figure 12 and Figure 13 This will be discussed later. Furthermore, in Figure 11 This example mainly illustrates a HUD installed in a car 30, but the HUD in this embodiment can also be installed in various mobile bodies such as airplanes, ships, or two-wheeled vehicles.

[0121] like Figure 11 As shown in the upper left figure, the rendered image includes image IM24 as the first display object and image IM25 as the second display object. Images IM24 and IM25 refer to a portion of the rendered image, not to images distinct from the rendered image.

[0122] Here, the first display object is the display object in the AR display, that is, the virtual object displayed in the HUD that follows the actual object in the actual space. Figure 11 In the lower segment of the image, the preceding vehicle 10 is the actual object, and the first display object 24 is displayed following the preceding vehicle 10. The image IM24 of the first display object is used to make the HUD display the image of the first display object 24. Figure 11 In the upper left image, the part marked with a shaded line is image IM24.

[0123] The second display object is a display object in the HUD that does not follow the actual object; rather, it is a display object whose display position is fixed in the display image DSIM output from the circuit device 160 to the HUD 400. When the positional relationship between the viewer and the HUD remains unchanged, the display position of the second display object is fixed in the viewer's field of vision. Figure 11In the lower section of the image, the fixed display of the "100km / h" characters is the second display object 25. The image IM25 of the second display object is used to make the HUD display the image of the second display object 25. Figure 11 In the left image, the part with the "100km / h" character is image IM25.

[0124] like Figure 11 As shown in the upper left and upper right images, the distortion processing unit 130 performs distortion correction and rotation offset correction on the first region AR1 in the rendered image. That is, the coordinate transformation unit 131 calculates the input coordinates with added rotation processing for the first region AR1 by performing a coordinate transformation based on the distortion parameter WPM and the rotation correction parameter RTPM. The distortion processing unit 130 performs distortion correction on the second region AR2 in the rendered image, but does not perform rotation offset correction. That is, the coordinate transformation unit 131 calculates the input coordinates without added rotation processing for the second region AR2 by performing a coordinate transformation based on the distortion parameter WPM. Through the above, the display image is given distortion correction that eliminates the distortion generated in the HUD 400. In addition, the display image includes the image IM24' of the first display object after distortion correction and rotation offset correction, and the image IM25' of the second display object after distortion correction.

[0125] Here, the first region AR1 refers to the region in the rendered image that includes the image IM24 of the first display object but does not include the image IM25 of the second display object. Specifically, the first region AR1 is the region in the rendered image other than the second region AR2. Furthermore, if the rendered image includes an image of a third display object and that third display object is an AR display, the first region AR1 is set to include both the image IM24 of the first display object and the image of the third display object.

[0126] The second region AR2 refers to the region in the rendered image that contains the image IM25 of the second display object. Specifically, the second region AR2 is the entire region containing the image IM25 of the second display object. Furthermore, when the rendered image contains an image of a third display object and that third display object is displayed fixedly, the second region AR2 is set in a manner that includes both the image IM25 of the second display object and the image of the third display object.

[0127] like Figure 11 As shown in the upper right and lower diagrams, the distortion processing unit 130 outputs the display image to the HUD 400. The HUD 400 projects the display image onto a screen, which the viewer then views. Thus, the displayed image, as a virtual image, overlaps with the actual space as seen by the viewer. Figure 11The lower figure shows the AR display viewed by the driver as a viewer via HUD 400. The driver views the preceding vehicle 10, the road on which the preceding vehicle 10 is traveling, and the surrounding scenery through the windshield. Additionally, the driver views a virtual image projected by HUD 400 onto a display area 40 on the screen. Display area 40 represents the range within which HUD 400 can project virtual images.

[0128] The virtual image includes a first display object 24 and a second display object 25. Specifically, in Figure 12 In the upper right image, the portion of the displayed image other than the image IM24' of the first display object and the image IM25' of the second display object is displayed as transparent black in the HUD 400. The first display object 24 is projected using the image IM24' of the first display object in the displayed image projected by the HUD 400, and the second display object 25 is projected using the image IM25' of the second display object. Furthermore, in the virtual image, the portion other than the aforementioned display objects is transparent; from the viewer's perspective, only the display objects overlap with the actual space and are thus visible. Here, the entire virtual image generated by the HUD 400 is referred to as the "virtual image," and the portion that is not transparent but is recognized by the viewer is referred to as the "display object." Moreover, the background of the display object does not necessarily have to be transparent; in this case, a specific portion of the virtual image can also be referred to as the display object.

[0129] According to the above implementation, rotational offset correction is performed on the first display object 24 that follows the actual object, while no rotational offset correction is performed on the second display object 25 that does not follow the actual object. Therefore, for the first display object 24 in the virtual image projected by the HUD 400, an AR display with applied rotational offset correction can be achieved, enabling it to accurately follow the moving vehicle 10 compared to the case without rotational offset correction. On the other hand, for the second display object 25, which is not an AR display object, it can be fixedly displayed in the viewer's field of vision of the HUD, becoming an easily viewable display unaffected by changes in the position or posture of the car 30.

[0130] Figure 12 This is the third detailed structural example of circuit device 160. In Figure 13 In this example, compared to the first detailed structural example, the circuit device 160 includes a twist parameter selection unit 162. Furthermore, the same reference numerals are used to denote the components already described, and descriptions of these components are appropriately omitted.

[0131] The distortion parameter selection unit 162 selects the corrected distortion parameter CWPM when the first region AR1 corresponding to the AR display is distorted, and selects the distortion parameter WPM when the second region AR2 corresponding to the fixed display is distorted. The selected parameter is output to the distortion processing unit 130 as the distortion parameter PRMQ. The coordinate transformation unit 131 performs coordinate transformation using the distortion parameter PRMQ. Thus, the coordinate transformation unit 131 performs coordinate transformation based on the distortion parameter WPM and the rotation correction parameter RTPM for the first region AR1 corresponding to the AR display, and performs coordinate transformation based on the distortion parameter WPM for the second region AR2 corresponding to the fixed display.

[0132] The distortion parameter selection unit 162 can determine the first region AR1 or the second region AR2 based on the coordinates of the pixels distorted by the distortion processing unit 130. In inverse distortion, it determines which of the first region AR1 or the second region AR2 the pixels of the displayed image on the output side correspond to. A correspondence is established between the rendered image and the displayed image through distortion correction, and the region can be determined based on this correspondence.

[0133] According to this embodiment, distortion correction and rotation offset correction are performed on the image IM24 of the first display object by performing a coordinate transformation based on the distortion parameter WPM and the rotation correction parameter RTPM on the first region AR1. Furthermore, distortion correction is performed on the image IM25 of the second display object by performing a coordinate transformation based on the distortion parameter WPM on the second region AR2, but rotation offset correction is not performed.

[0134] Figure 13 This is the fourth detailed structural example of circuit device 160. In Figure 14 In this example, compared to the second detailed structural example, the rotation correction unit 135 performs rotation processing corresponding to the region. Furthermore, the same reference numerals are used to label the already described components, and descriptions of these components are appropriately omitted.

[0135] For the first region AR1 corresponding to the AR display, the distortion correction unit 134 performs a first coordinate transformation, transforming the output coordinates (Xtrg, Ytrg) according to the distortion parameter WPM. The rotation correction unit 135 performs a second coordinate transformation, rotating the coordinates (Xrot, Yrot) after the first coordinate transformation according to the rotation correction parameter RTPM. The rotation correction unit 135 outputs the input coordinates (Xsrc, Ysrc) obtained through the second coordinate transformation to the coordinate address transformation unit 132. On the other hand, for the second region AR2 corresponding to the fixed display, the distortion correction unit 134 performs the first coordinate transformation, but the rotation correction unit 135 does not perform the second coordinate transformation. The rotation correction unit 135 outputs the coordinates (Xrot, Yrot) after the first coordinate transformation as the input coordinates (Xsrc, Ysrc) to the coordinate address transformation unit 132. The rotation correction unit 135 determines, for example, whether the input coordinates (Xsrc, Ysrc) belong to the first region AR1 or the second region AR2. Alternatively, the rotation correction unit 135 can calculate the input coordinates (Xsrc, Ysrc) based on the output coordinates (Xtrg, Ytrg) and determine whether the input coordinates (Xsrc, Ysrc) belong to the first region AR1 or the second region AR2. The rotation correction unit 135 performs a second coordinate transformation if the input coordinates (Xsrc, Ysrc) belong to the first region AR1, and does not perform a second coordinate transformation if the input coordinates (Xsrc, Ysrc) belong to the second region AR2.

[0136] According to this embodiment, distortion correction and rotation offset correction are performed on the image IM24 of the first display object by performing a first coordinate transformation and a second coordinate transformation on the first region AR1. Conversely, distortion correction is performed on the image IM25 of the second display object by performing a first coordinate transformation on the second region AR2 but not a second coordinate transformation, without performing rotation offset correction.

[0137] 5. Electronic equipment

[0138] ​ This is an example of the structure of an electronic device 500 that uses circuitry device 160. The electronic device 500 includes a processing unit 150 and a HUD 400.

[0139] The HUD 400 includes a circuit device 160 as a HUD controller and a projection device 532. A processing device 150 sends a rendered image to the circuit device 160. The circuit device 160 distorts the rendered image and outputs a display control signal along with the resulting display image to the projection device 532. The projection device 532 includes, for example, a display driver, a liquid crystal display panel, a light source, and optical devices. The display driver causes the liquid crystal display panel to display an image based on image data and display control signals received from the circuit device 160. The light source emits projection light onto the liquid crystal display panel, and the projection light passing through the liquid crystal display panel is incident on the optical devices. The optical devices project the projection light passing through the liquid crystal display panel onto a screen. The screen is, for example, the windshield of a moving vehicle, but a dedicated screen can also be provided. The moving vehicle is a car, airplane, or ship, etc.

[0140] The circuit apparatus of this embodiment described above performs display control on a head-up display that displays a virtual object corresponding to an actual object in the actual space on the display area. The circuit apparatus includes: a storage unit that stores a rendered image containing the virtual object; and a distortion processing unit that distorts the rendered image to generate a display image displayed on the display area. The distortion processing unit includes a coordinate transformation unit, a coordinate address transformation unit, and an output unit. The coordinate transformation unit transforms the output coordinates, which are coordinates on the display image, into input coordinates, which are coordinates on the rendered image, based on a distortion parameter and a rotation correction parameter corresponding to the image distortion caused by the optical system. This process yields the input coordinates, which are then corrected for rotation as indicated by the rotation correction parameter. The coordinate address transformation unit transforms the input coordinates into a read address of the storage unit. The output unit reads pixel data of the rendered image from the read address of the storage unit and, based on the read pixel data, outputs pixel data at the output coordinates of the displayed image.

[0141] According to this embodiment, immediately before reading pixel data from the storage unit, a coordinate transformation based on distortion parameters and rotation correction parameters is performed, and a display image is output based on the read pixel data. This minimizes the delay from rotation offset correction to the output of the display image. For example, when this embodiment is applied to AR displays, virtual objects can follow real objects with high precision.

[0142] Furthermore, in this embodiment, the circuit device may also include a torsion parameter correction unit that calculates the corrected torsion parameter by correcting the torsion parameter using rotation correction parameters. The coordinate transformation unit may also perform coordinate transformation using the corrected torsion parameter.

[0143] According to this embodiment, immediately before the coordinate transformation unit performs coordinate transformation, the distortion parameter correction unit calculates the corrected distortion parameters, and the coordinate transformation unit uses the corrected distortion parameters to perform coordinate transformation on the output coordinates to transform them into input coordinates. Therefore, compared to the structure in Patent Document 2 above, which temporarily prepares a table of distortion parameters for one frame in memory, the delay from obtaining the rotation correction parameters to outputting the displayed image can be shortened.

[0144] Alternatively, in this embodiment, the coordinate transformation unit may perform a first coordinate transformation that transforms the output coordinates by a distortion parameter and a second coordinate transformation that rotates the coordinates after the first coordinate transformation by a rotation correction parameter as coordinate transformation.

[0145] According to this embodiment, since rotational offset correction is performed separately from distortion correction, calculation of the distortion parameters is unnecessary. Furthermore, the rotational correction unit performs rotational offset correction immediately before coordinate address transformation. Therefore, compared to a structure like Patent Document 2, which temporarily stores a table of distortion parameters for one frame in memory, the delay from obtaining the rotational correction parameters to outputting the displayed image can be shortened.

[0146] In addition, in this embodiment, the rotation correction parameter may also be a delay compensation parameter that compensates for the delay of rendering processing delay, including the rendering image, based on at least one of the following tracking information: the first tracking information of the moving body equipped with the head-up display, the second tracking information of the viewer of the head-up display, and the third tracking information of the actual object.

[0147] According to this embodiment, after calculating the corrected distortion parameters, a coordinate transformation using these parameters is performed immediately. Therefore, the time from obtaining the tracking information used for calculating the rotation correction parameters to displaying the image can be shortened. Thus, by performing rotation offset correction using rotation correction parameters obtained at a timing as close as possible to the display timing of the image, the delay from rendering the rendered image to displaying the displayed image can be compensated for with high precision.

[0148] Furthermore, in this embodiment, the distortion parameter can also be the coefficient of a polynomial that corrects for image distortion.

[0149] In tabular distortion correction, multiple parameters corresponding to multiple coordinates are used. For example, in Patent Document 2 mentioned above, the calculation takes time because vibration correction is performed on each of these multiple parameters. In this respect, according to this embodiment, since distortion correction is performed using a polynomial, distortion processing can be achieved using only one set of distortion parameters. Therefore, the calculation time can be shortened compared to the tabular method.

[0150] Furthermore, in this embodiment, the coordinate transformation unit may also use the rotation correction parameters updated in the first frame to calculate the input coordinates with added rotation processing during the distortion processing performed on the second frame after the first frame.

[0151] According to this embodiment, the delay from the time the tracking information used for calculating rotation correction parameters is obtained to the time the HUD is displayed is shorter than one frame. Therefore, rotation offset correction is performed using rotation correction parameters obtained at a timing as close as possible to the display timing of the displayed image.

[0152] In this embodiment, the coordinate address transformation unit may also generate a read address group based on a reference read address. The output unit may also include an interpolation processing unit that reads the pixel data group corresponding to the read address group, performs interpolation processing on the pixel data group, and thereby generates pixel data for the output coordinates.

[0153] In inverse warp, the input coordinates obtained by transforming the output coordinates may not necessarily correspond to the pixel positions of the rendered image. According to this embodiment, pixel data at the input coordinates can be interpolated based on a pixel data set, and this pixel data can be used as the pixel data for the output coordinates.

[0154] In this embodiment, the storage unit may also store a rendered image containing images of a first display object and a second display object, which are virtual objects. The coordinate transformation unit may also calculate the input coordinates with added rotation processing for the image of the first display object in the rendered image by performing a coordinate transformation based on distortion parameters and rotation correction parameters. The coordinate transformation unit may also calculate the input coordinates without added rotation processing for the image of the second display object in the rendered image by performing a coordinate transformation based on distortion parameters.

[0155] According to this embodiment, rotational offset correction is performed on a first display object that follows the actual object, but not on a second display object that does not follow the actual object. Therefore, for the first display object in the virtual image projected by the HUD, an AR display with applied rotational offset correction can be achieved. On the other hand, for the second display object, which is not an AR display object, it can be fixed in the viewer's field of vision on the HUD, becoming an easy-to-view display unaffected by changes in the position or posture of the moving object.

[0156] Furthermore, in this embodiment, the circuit device may also include: a distortion parameter correction unit that corrects the distortion parameter using a rotation correction parameter, thereby calculating the corrected distortion parameter; and a parameter selection unit. The storage unit may store a rendered image containing an image of a first display object and an image of a second display object. Alternatively, for the image of the first display object in the rendered image, the parameter selection unit may select the corrected distortion parameter, and the coordinate transformation unit may perform coordinate transformation using the corrected distortion parameter. Similarly, for the image of the second display object in the rendered image, the parameter selection unit may select the distortion parameter, and the coordinate transformation unit may perform coordinate transformation using the distortion parameter.

[0157] According to this embodiment, distortion correction and rotational offset correction are performed on the image of the first display object by performing a coordinate transformation based on distortion parameters and rotational correction parameters. Furthermore, distortion correction is performed on the image of the second display object by performing a coordinate transformation based on distortion parameters, but rotational offset correction is not performed.

[0158] In this embodiment, the storage unit may also store a rendered image containing both an image of the first display object and an image of the second display object. The coordinate transformation unit may also perform a first coordinate transformation on the image of the first display object in the rendered image, outputting coordinates through a distortion parameter transformation, and a second coordinate transformation, rotating the coordinates after the first coordinate transformation using a rotation correction parameter. The coordinate transformation unit may also perform a first coordinate transformation on the image of the second display object in the rendered image.

[0159] According to this embodiment, distortion correction and rotation offset correction are performed on the image of the first display object by performing a first coordinate transformation and a second coordinate transformation. Conversely, distortion correction is performed on the image of the second display object by performing a first coordinate transformation but not a second coordinate transformation, without performing rotation offset correction.

[0160] Furthermore, the display system of this embodiment displays virtual objects corresponding to actual objects in the actual space within the display area. The display system includes: a rendering image generation unit that generates a rendered image containing the virtual object; a storage unit that stores the rendered image; and a distortion processing unit that performs distortion processing on the rendered image to generate a display image displayed in the display area. The distortion processing unit includes a coordinate transformation unit, a coordinate address transformation unit, and an output unit. The coordinate transformation unit transforms the output coordinates, which are coordinates on the display image, into input coordinates, which are coordinates on the rendered image, based on a distortion parameter and a rotation correction parameter corresponding to the image distortion caused by the optical system. This yields the input coordinates, which are then corrected for the rotation indicated by the rotation correction parameter. The coordinate address transformation unit transforms the input coordinates into a read address for the storage unit. The output unit reads pixel data of the rendered image from the read address of the storage unit and, based on the read pixel data, outputs pixel data at the output coordinates of the displayed image.

[0161] Furthermore, in this embodiment, the display system may also include a distortion parameter correction unit that calculates the corrected distortion parameter by correcting the distortion parameter using rotation correction parameters. The coordinate transformation unit may also perform coordinate transformation using the corrected distortion parameter.

[0162] Alternatively, in this embodiment, the coordinate transformation unit may perform a first coordinate transformation by transforming the output coordinates through the distortion parameter and a second coordinate transformation by rotating the coordinates after the first coordinate transformation through the rotation correction parameter as coordinate transformation.

[0163] In addition, in this embodiment, the display system may also include a rotation correction parameter calculation unit. The rotation correction parameter calculation unit may calculate the rotation correction parameters based on at least one of the following tracking information: first tracking information of the moving body mounted on the head-up display, second tracking information of the viewer on the head-up display, and third tracking information of the actual object. The rotation correction parameters may also be delay compensation parameters that compensate for delays in rendering processing, including the rendering image.

[0164] Furthermore, the electronic device of this embodiment includes the circuit device described in any one of the above embodiments.

[0165] Furthermore, while this embodiment has been described in detail above, those skilled in the art will readily understand that various modifications can be made without substantially departing from the novel aspects and effects of this disclosure. Therefore, all such modifications are included within the scope of this disclosure. For example, in the specification or drawings, a term described at least once with a broader or synonymous term can be replaced with that different term anywhere in the specification or drawings. Additionally, all combinations of this embodiment and its modifications are also included within the scope of this disclosure. Furthermore, the structure and operation of circuit devices, display systems, HUDs, and electronic devices are not limited to those described in this embodiment, and various modifications can be implemented.

Claims

1. A circuit device, characterized in that, This circuit device controls the display of a head-up display, which displays virtual objects corresponding to actual objects in the real space in a display area. The circuit device includes: Storage unit, which stores rendered images containing the virtual objects; and The distortion processing unit distorts the rendered image to generate a display image displayed in the display area. The distortion processing unit includes: The coordinate transformation unit transforms the output coordinates, which are coordinates on the displayed image, into input coordinates, which are coordinates on the rendered image, by using a coordinate transformation based on a distortion parameter corresponding to the distortion of the image caused by the optical system and a rotation correction parameter, which is a delay compensation parameter that compensates for the delay including the rendering processing delay of the rendered image. The input coordinates are then obtained after rotation processing, which corrects for the rotation represented by the rotation correction parameter. The coordinate address transformation unit transforms the input coordinates into a read address of the storage unit; as well as The output unit reads the pixel data of the rendered image from the read address of the storage unit, and outputs the pixel data at the output coordinates of the displayed image based on the read pixel data.

2. The circuit device according to claim 1, characterized in that, The circuit device includes a torsion parameter correction unit, which calculates the corrected torsion parameter by correcting the torsion parameter using the rotation correction parameter. The coordinate transformation unit performs the coordinate transformation using the corrected distortion parameters.

3. The circuit device according to claim 1, characterized in that, The coordinate transformation unit performs a first coordinate transformation by transforming the output coordinates using the distortion parameter and a second coordinate transformation by rotating the coordinates after the first coordinate transformation using the rotation correction parameter, as the coordinate transformation.

4. The circuit device according to any one of claims 1 to 3, characterized in that, The rotation correction parameter is a parameter used to compensate for the delay, including the rendering processing delay of the rendered image, based on at least one of the following: first tracking information of the moving body equipped with the head-up display, second tracking information of the viewer of the head-up display, and third tracking information of the actual object.

5. The circuit device according to any one of claims 1 to 3, characterized in that, The distortion parameter is the coefficient of a polynomial that corrects the distortion of the image.

6. The circuit device according to any one of claims 1 to 3, characterized in that, In the distortion processing performed on the second frame after the first frame, the coordinate transformation unit uses the rotation correction parameters updated in the first frame to calculate the input coordinates after the additional rotation processing.

7. The circuit device according to any one of claims 1 to 3, characterized in that, The coordinate address transformation unit generates a read address group based on the reference read address used as the read address. The output unit includes an interpolation processing unit, which reads out the pixel data group corresponding to the read address group, performs interpolation processing on the pixel data group, and thereby generates the pixel data of the output coordinates.

8. The circuit device according to claim 1, characterized in that, The storage unit stores the rendered image, which includes an image of a first display object and an image of a second display object, which are the virtual objects. The coordinate transformation unit, for the image of the first display object in the rendered image, calculates the input coordinates after the rotation processing by performing a coordinate transformation based on the distortion parameters and the rotation correction parameters. The coordinate transformation unit calculates the input coordinates without the added rotation processing for the image of the second display object in the rendered image by performing the coordinate transformation based on the distortion parameters.

9. The circuit device according to claim 8, characterized in that, The circuit device includes: A torsion parameter correction unit, which corrects the torsion parameter using the rotation correction parameter and calculates the corrected torsion parameter; and Parameter selection section The storage unit stores the rendered image containing the image of the first display object and the image of the second display object. For the image of the first display object in the rendered image, the parameter selection unit selects the corrected distortion parameter, and the coordinate transformation unit performs the coordinate transformation using the corrected distortion parameter. For the image of the second display object in the rendered image, the parameter selection unit selects the distortion parameter, and the coordinate transformation unit performs the coordinate transformation using the distortion parameter.

10. The circuit device according to claim 8, characterized in that, The storage unit stores the rendered image containing the image of the first display object and the image of the second display object. The coordinate transformation unit performs a first coordinate transformation on the image of the first display object in the rendered image, transforming the output coordinates using the distortion parameters, and a second coordinate transformation, rotating the coordinates after the first coordinate transformation using the rotation correction parameters, as the coordinate transformation. The coordinate transformation unit performs the first coordinate transformation on the image of the second display object in the rendered image as the coordinate transformation.

11. A display system, characterized in that, This display system is a head-up display system that displays virtual objects corresponding to actual objects in the display area. The display system includes: A rendering image generation unit generates a rendering image containing the virtual object; Storage unit, storing the rendered image; and The distortion processing unit distorts the rendered image to generate a display image displayed in the display area. The distortion processing unit includes: The coordinate transformation unit transforms the output coordinates, which are coordinates on the displayed image, into input coordinates, which are coordinates on the rendered image, by using a coordinate transformation based on a distortion parameter corresponding to the distortion of the image caused by the optical system and a rotation correction parameter, which is a delay compensation parameter that compensates for the delay including the rendering processing delay of the rendered image. The input coordinates are then obtained after rotation processing, which corrects for the rotation represented by the rotation correction parameter. The coordinate address transformation unit transforms the input coordinates into a read address of the storage unit; as well as The output unit reads the pixel data of the rendered image from the read address of the storage unit, and outputs the pixel data at the output coordinates of the displayed image based on the read pixel data.

12. The display system according to claim 11, characterized in that, The display system includes a distortion parameter correction unit, which calculates the corrected distortion parameter by correcting the distortion parameter using the rotation correction parameter. The coordinate transformation unit performs the coordinate transformation using the corrected distortion parameters.

13. The display system according to claim 11, characterized in that, The coordinate transformation unit performs a first coordinate transformation by transforming the output coordinates using the distortion parameter and a second coordinate transformation by rotating the coordinates after the first coordinate transformation using the rotation correction parameter, as the coordinate transformation.

14. The display system according to any one of claims 11 to 13, characterized in that, The display system includes a rotation correction parameter calculation unit, which calculates the rotation correction parameter based on at least one of the following: first tracking information of a moving body equipped with the head-up display, second tracking information of a viewer of the head-up display, and third tracking information of the actual object.

15. An electronic device, characterized in that, The electronic device includes the circuit arrangement as described in any one of claims 1 to 10.

Images