Circuit arrangement, display system and image processing method

Abstract

Circuit device, display system, and image processing method. Display position shift of AR display can be accurately corrected in a HUD in which AR display and fixed display are mixed. A circuit device (160) includes a storage section (161) and a warp processing section (130). The storage section (161) stores a warp target image (WTIM). The warp processing section (130) performs warp processing on the warp target image WTIM to generate a display image (DSIM) displayed in a display region (40). The warp processing section (130) performs warp processing on an image (IM20) of a first display object in the warp target image (WTIM) in accordance with a first warp parameter (PRM1) that performs vibration correction and distortion correction. In addition, the warp processing section (130) performs warp processing on an image (IM25) of a second display object in the warp target image (WTIM) in accordance with a second warp parameter (PRM2) that performs distortion correction.

Description

TECHNICAL FIELD

[0001] The present application relates to a circuit device, a display system, and an image processing method. BACKGROUND

[0002] A head-up display that displays information superimposed in a user's field of view by displaying an image on a transparent screen or the like is known. By combining such a head-up display and AR technology, it is possible to cause the head-up display to display a virtual object in such a manner that the virtual object of the AR follows a real object such as a preceding vehicle or a road. AR is an abbreviation for Augmented Reality.

[0003] Sometimes it is desirable to perform vibration correction for causing a virtual object of the AR to follow a real object, but it is desirable to make an instrument display or the like that does not need to follow a real object a fixed display. Furthermore, in a head-up display, an image is distorted due to a curved surface of a transparent screen or the like, so distortion correction for correcting the distortion is performed. Patent Document 1 discloses a technology in which an image that needs to be subjected to vibration correction and an image that does not need to be subjected to vibration correction are synthesized on a frame memory, and the synthesized image is subjected to distortion correction. Furthermore, Patent Document 2 discloses a technology in which position correction and distortion correction are performed separately, and in the distortion correction, the parameter of the distortion correction is changed for each image region. Furthermore, Patent Document 3 discloses a technology in which a plurality of images including a first image that is subjected to position correction and a second image that is not subjected to position correction are synthesized, but the first image and the second image become independent images.

[0004] Patent Document 1: Japanese Patent Application Publication No. 2020-050328

[0005] Patent Document 2: Japanese Patent Application Publication No. 2009-179214

[0006] Patent Document 3: Japanese Patent Application Publication No. 2017-94882

[0007] From the viewpoint of accurately correcting a positional deviation of a virtual object of the AR from a real object, it is preferable that the timing of the vibration correction be close to the timing at which display is performed in the head-up display. However, in Patent Documents 1 and 2, the distortion correction is performed after the vibration correction or the position correction, so the processing time from the vibration correction or the position correction to the output to the head-up display becomes long. Furthermore, in Patent Document 3, the synthesis is performed after the correction of a plurality of different images, so the processing time from the position correction to the output to the head-up display becomes long. As described above, in the above-described related art, there is a problem that in a HUD in which AR display and fixed display are mixed, the processing time from the vibration correction or the position correction to the output to the head-up display becomes long, and the correction of the positional deviation is insufficient. SUMMARY

[0008] One embodiment of the present application relates to a circuit device used in a head-up display that displays a first display object corresponding to a real object of a real space and a second display object in a display region, the circuit device including: a storage section that stores a warped object image including an image of the first display object and an image of the second display object; and a warp processing section that performs warp processing on the warped object image to generate a display image displayed in the display region, the warp processing section performing the warp processing on the image of the first display object in the warped object image in accordance with a first warp parameter that performs vibration correction that shifts a display position of the first display object with respect to the real object and distortion correction corresponding to at least one of distortion of an optical system and bending of the display region, the warp processing section performing the warp processing on the image of the second display object in the warped object image in accordance with a second warp parameter that performs the distortion correction.

[0009] Further, another embodiment of the present application relates to a display system that is a display system of a head-up display that displays a first display object corresponding to a real object of a real space and a second display object in a display region, the display system including: a storage section that stores a warped object image including an image of the first display object and an image of the second display object; and a warp processing section that performs warp processing on the warped object image to generate a display image displayed in the display region, the warp processing section performing the warp processing on the image of the first display object in the warped object image in accordance with a first warp parameter that performs vibration correction that shifts a display position of the first display object with respect to the real object and distortion correction corresponding to at least one of distortion of an optical system and bending of the display region, the warp processing section performing the warp processing on the image of the second display object in the warped object image in accordance with a second warp parameter that performs the distortion correction.

[0010] Further, another aspect of the present application relates to an image processing method, which is an image processing method in a head-up display that displays a first display object corresponding to a real object of a real space and a second display object in a display region, performs a warp processing on a warped object image that includes an image of the first display object and an image of the second display object, and generates a display image to be displayed in the display region, performs the warp processing on the image of the first display object in the warped object image in accordance with a first warp parameter that performs a vibration correction that shifts a display position of the first display object with respect to the real object and a distortion correction that corresponds to at least one of a curvature of the display region and a distortion of an optical system, and performs the warp processing on the image of the second display object in the warped object image in accordance with a second warp parameter that performs the distortion correction. BRIEF DESCRIPTION OF DRAWINGS

[0011] Figure 1 is a first configuration example of a circuit device.

[0012] Figure 2 is an explanatory diagram of a process performed by the first configuration example of a circuit device.

[0013] Figure 3 is a second configuration example of a circuit device.

[0014] Figure 4 is an explanatory diagram of a process performed by the second configuration example of a circuit device.

[0015] Figure 5 is an explanatory diagram of display position shift based on tilt displacement or vertical displacement.

[0016] Figure 6 is a diagram that explains delay time compensation.

[0017] Figure 7 is a configuration example of a display system in the related art.

[0018] Figure 8 is a process flow of a display system in the related art.

[0019] Figure 9 is a configuration example of a display system in the present embodiment.

[0020] Figure 10 is a detailed process flow of a tracking processing section.

[0021] Figure 11 is a process flow of a display system.

[0022] Figure 12 is a detailed configuration example of a display system.

[0023] Figure 13 is an explanatory diagram of the warping process.

[0024] Figure 14 is a diagram showing the correspondence relationship between the tracking information of a mobile body or the like and the delay time compensation based on the tracking information.

[0025] Figure 15 is an example of a delay time compensation parameter.

[0026] Figure 16 is a diagram explaining a method of calculating a delay time compensation parameter from tracking information.

[0027] Explanation of Reference Signs

[0028] 10: preceding vehicle; 11, 12: real object; 20: 1st display object; 21: virtual object; 25: 2nd display object; 30: car; 32: mobile body; 34: screen; 35: windshield; 36: projection unit; 40: display region; 50: eye; 52: observer; 60: processing device; 61: sensor interface; 62: tracking processing section; 63: rendered image generation section; 65: sensor; 70: HUD controller; 71: warping processing section; 81: display; 90: display system; 100: display system; 110: tracking processing section; 111: visual-inertial odometry processing; 112: inertial odometry processing; 113: prediction processing; 120: rendered image generation section; 130: warping processing section; 131: 1st correction section; 132: 2nd correction section; 133: composition section; 140: parameter calculation section; 145: storage section; 150: processing device; 151: interface; 160: circuit device; 161: storage section; 162: warping parameter selection section; 163: layer selection section; 270: processing device; 410: display; 500: sensor; AR1: 1st region; AR2: 2nd region; CIM1: 1st correction image; CIM2: 2nd correction image; DSIM: display image; IM20: image of 1st display object; IM25: image of 2nd display object; PRM1: 1st warping parameter; PRM2: 2nd warping parameter; WTIM: warped object image. DETAILED DESCRIPTION

[0029] Hereinafter, a preferred embodiment of the present application will be described in detail. Note that the present embodiment described below is not intended to unduly limit the content recited in the claims, and all the structures described in the present embodiment are not necessarily essential structural elements.

[0030] 1. 1st Structural Example

[0031] Figure 1A first configuration example of the circuit device 160 used for the HUD is shown. The HUD is an abbreviation for head-up display. The circuit device 160 includes a storage section 161, a warp processing section 130, and a warp parameter selection section 162. The circuit device 160 is also referred to as a HUD controller, and is constituted by, for example, an integrated circuit device in which a circuit is integrated on a semiconductor substrate.

[0032] The storage section 161 temporarily stores the warp target image WTIM input to the circuit device 160. Specifically, the storage section 161 is a line buffer that stores image data of a number of lines required for warp processing. The number of lines is a number of lines added to a maximum moving line number in the vertical direction in warp processing by a margin. The storage section 161 is, for example, a semiconductor memory such as a RAM. The RAM is an abbreviation for Random Access Memory.

[0033] The warp processing section 130 performs warp processing on the warp target image WTIM, thereby generating a display image DSIM, and outputs the display image DSIM to the HUD 400. The warp processing is coordinate conversion between the warp target image WTIM and the display image DSIM, and in the present embodiment, includes distortion correction and vibration correction. As described later, the warp processing section 130 performs distortion correction and vibration correction on an image region corresponding to AR display, and performs only distortion correction on an image region corresponding to fixed display. The AR is an abbreviation for augmented reality. The distortion correction and the vibration correction are performed together by coordinate conversion of the warp processing. That is, a warp parameter representing the coordinate conversion is generated from a parameter for the distortion correction and a parameter for the vibration correction, and the coordinate conversion is performed using the warp parameter, thereby performing the distortion correction and the vibration correction at the same time.

[0034] The warp processing section 130 can be either of a forward warp engine or a reverse warp engine. The forward warp is a conversion that moves each pixel of an input image to the warp engine to an arbitrary position in an output image. The forward warp engine is a warp engine having a function of the forward warp. The reverse warp is a conversion that finds each pixel of an output image of the warp engine from a pixel at an arbitrary position in an input image. The reverse warp engine is a warp engine having a function of the reverse warp.

[0035] The distortion parameter selection section 162 selects the first distortion parameter PRM1 when performing the distortion process on the image region corresponding to the AR display, and selects the second distortion parameter PRM2 when performing the distortion process on the image region corresponding to the fixed display. The selected parameter is output as the distortion parameter PRMQ to the distortion process section 130. The first distortion parameter PRM1 is a distortion parameter that performs both the distortion correction and the vibration correction. The second distortion parameter PRM2 is a distortion parameter that performs the distortion correction and does not perform the vibration correction. The distortion process section 130 performs the distortion process using the distortion parameter PRMQ output from the distortion parameter selection section 162, whereby the image region corresponding to the AR display is subjected to the distortion correction and the vibration correction, and the image region corresponding to the fixed display is subjected to only the distortion correction. In addition, the first distortion parameter PRM1 and the second distortion parameter PRM2 are input from the outside of the circuit device 160. Alternatively, the circuit device 160 can include a parameter operation section that generates the first distortion parameter PRM1 and the second distortion parameter PRM2.

[0036] Figure 2 An explanatory diagram of the process performed by the first structure example of the circuit device 160 is shown. In Figure 2 In the present embodiment, the HUD is mounted on the automobile 30, but the HUD of the present embodiment can be mounted on various moving bodies such as an airplane, a ship, or a two-wheeled vehicle.

[0037] As shown in the left drawing of FIG. 20, the image IM20 of the first display object and the image IM25 of the second display object are included in the distortion target image. The images IM20 and IM25 mean a part of the distortion target image, and do not mean an image different from the distortion target image. Figure 2

[0038] Here, the first display object is a display object in the AR display, that is, a virtual object that is displayed in the HUD while following a real object in the real space. In the lower drawing of FIG. 20, the preceding vehicle 10 is a real object, and the first display object 20 is displayed while following the preceding vehicle 10. The image IM20 of the first display object is an image for causing the HUD to display the first display object 20. In the left drawing of FIG. 20, the portion of the figure to which the hatching is applied is the image IM20. Figure 2 Figure 2 The second display object is a display object that is not displayed while following a real object in the HUD, and is a display object whose display position is fixed in the display image DSIM output from the circuit device 160 to the HUD 400. In the case where the positional relationship between the observer who views the HUD and the HUD does not change, the display position of the second display object is fixed in the field of view of the observer. In the left drawing of FIG. 20, the portion of the figure to which the hatching is applied is the image IM25 of the second display object.

[0039] The second display object is a display object that is not displayed while following a real object in the HUD, and is a display object whose display position is fixed in the display image DSIM output from the circuit device 160 to the HUD 400. In the case where the positional relationship between the observer who views the HUD and the HUD does not change, the display position of the second display object is fixed in the field of view of the observer. In the left drawing of FIG. 20, the portion of the figure to which the hatching is applied is the image IM25 of the second display object. Figure 2 ​​In the left image, the fixed display of the "100km / h" characters is the second display object 25. The image IM25 of the second display object is used to make the HUD display the image of the second display object 25. Figure 2 In the left image, the part with the "100km / h" text is image IM25.

[0040] The distorted object image corresponds to the input image in the distortion processing that performs coordinate transformation between the input and output images. Specifically, the distorted object image is the image input from the storage unit 161 to the distortion processing unit 130, and is an image containing the image IM20 of the first display object and the image IM25 of the second display object. For example, the rendering processing for AR display is performed in a processing device outside the circuit device 160, and the rendered image is input to the circuit device 160. At least the image IM20 of the first display object is rendered in the rendered image. For example, the image IM25 of the second display object is also rendered in the rendering process, and this rendered image is stored in the storage unit 161 as a distorted object image. Alternatively, the circuit device 160 may also include an overlay processing unit (not shown), which may overlay the image IM25 of the second display object onto the rendered image and store it in the storage unit 161 as a distorted object image.

[0041] Furthermore, the distorted object image is not limited to a rendered image. For example, multiple images of the first display object IM20 can be prepared in advance, with the image IM20 positioned at various display locations. The image selected from these multiple images is then input into the circuit device 160 as the distorted object image. In this case, the image corresponding to the location where the first display object should be displayed in the AR display is selected from the multiple images.

[0042] like Figure 2 As shown in the left and middle images, the distortion processing unit 130 performs distortion correction and vibration correction on the first region AR1 in the distorted object image, and distortion correction on the second region AR2 in the distorted object image, but does not perform vibration correction. Thus, the displayed image is given distortion to eliminate the distortion generated in the HUD 400. Furthermore, the displayed image includes the first display object image IM20', which has undergone distortion correction and vibration correction, and the second display object image IM25', which has undergone distortion correction.

[0043] Specifically, the distortion parameter selection section 162 selects the first distortion parameter PRM1 when performing the distortion process of the first region AR1, and selects the second distortion parameter PRM2 when performing the distortion process of the second region AR2. Thus, the correction of each region described above is achieved. The distortion parameter selection section 162 can determine the first region AR1 or the second region AR2 according to the coordinates of the pixel for which the distortion process section 130 is performing the distortion process. In the forward distortion, it is determined which of the first region AR1 or the second region AR2 the pixel of the distortion target image on the input side belongs to. In the reverse distortion, it is determined which of the first region AR1 or the second region AR2 the pixel of the display image on the output side corresponds to. The distortion target image and the display image correspond to each other through the distortion correction, and the regions can be determined according to the correspondence relationship thereof.

[0044] Here, the first region AR1 is a region that includes the image IM20 of the first display object and does not include the image IM25 of the second display object in the distortion target image. Specifically, the first region AR1 is a region other than the second region AR2 in the distortion target image. In addition, in a case where the distortion target image includes an image of a third display object and the third display object is an AR display, the first region AR1 is set in a manner that includes the image IM20 of the first display object and the image of the third display object.

[0045] The second region AR2 is a region that includes the image IM25 of the second display object in the distortion target image. Specifically, the second region AR2 is a region that includes the entire image IM25 of the second display object. In addition, in a case where the distortion target image includes an image of a third display object and the third display object is a fixed display, the second region AR2 is set in a manner that includes the image IM25 of the second display object and the image of the third display object.

[0046] In addition, the first region AR1 and the second region AR2 can also be defined in the display image. In this case, the first region AR1 includes the image IM20' of the first display object that has been subjected to the distortion correction and the vibration correction, and the second region AR2 includes the image IM25' of the second display object that has been subjected to the distortion correction. As described above, the distortion target image and the display image correspond to each other through the distortion correction, and thus the second region AR2 can be set in a manner that includes the image IM25' of the second display object, and the region other than the second region AR2 can be set as the first region AR1.

[0047] The distortion correction is a process of coordinate conversion of the distorted object image according to the curvature of the display area and / or the distortion of the optical system. The coordinate conversion is a coordinate conversion to eliminate the image distortion caused by the screen curvature. The display image is an image displayed in the display panel of the HUD 400, and the display image displayed in the display panel is projected onto the screen by the projection unit, whereby a virtual image without distortion is displayed in the visual field of the observer.

[0048] The vibration correction is a process of correcting the display position shift of the first display object displayed in the AR display following the real object. The display position shift is caused by the change in the position or posture of the mobile body on which the HUD is mounted, and the vibration correction is performed according to the change in the position or posture of the mobile body. In the following description, the vibration correction is described as a process of correcting the display position shift of the first display object displayed in the AR display following the real object. Figure 2 In the following drawing, the automobile 30 is a mobile body, and the position shift of the preceding vehicle 10 and the first display object 20 is caused by the change in the position or posture of the automobile 30.

[0049] The vibration correction includes the shift correction. In addition, the vibration correction can also include the rotation correction, the scale correction, or both. The shift is the position shift of the first display object 20 with respect to the preceding vehicle 10 in the horizontal direction or the vertical direction. The shift correction is a correction to move the first area AR1 of the distorted object image in the horizontal direction or the vertical direction in parallel in order to eliminate the shift. The rotation is the rotation of the first display object 20 with respect to the preceding vehicle 10 clockwise or counterclockwise. The rotation correction is a correction to rotate the first area AR1 of the distorted object image clockwise or counterclockwise in order to eliminate the rotation. The scale is the enlargement or reduction of the size of the first display object 20 with respect to the preceding vehicle 10. The scale correction is a correction to enlarge or reduce the first area AR1 of the distorted object image in order to correct the scale.

[0050] The vibration correction is, for example, the delay time compensation described later. The delay time compensation is a compensation for the display position shift occurring from the time when the rendered image is rendered to the time when the distortion process is performed. Specifically, a difference between the display position of the first display object decided or predicted from the tracking information at the time of rendering and the display position of the first display object decided from the tracking information at the time of the distortion process is compensated. The tracking information is obtained from a sensor, and the details thereof are described later. In addition, the vibration correction is not limited to the delay time compensation, but is a process of correcting the change in the position or posture of the mobile body that is not completely followed in the rendering. That is, the above difference can not be used, and, for example, the low-frequency component of the tracking information can be used in the rendering, and the high-frequency component of the tracking information can be used in the vibration correction of the distortion process.

[0051] As described above, the HUD 400 is a device that displays the first display object 20 in the AR display following the real object 10. The HUD 400 includes the display panel 400A, the projection unit 400B, and the vibration correction unit 400C. Figure 2As shown in the middle and right figures, the distortion processing unit 130 outputs the display image to the HUD400. The HUD400 projects the display image onto a screen, and an observer views the screen, thus seeing the display image as a virtual image superimposed on the real space. Figure 2 The diagram below illustrates the AR display viewed by the observer, i.e., the driver, via the HUD400. The driver views the preceding vehicle 10, the road on which the preceding vehicle 10 is traveling, and the surrounding scenery through the windshield. Additionally, the driver views a virtual image projected by the HUD400 onto a display area 40 within the windshield. The display area 40 represents the range within which the HUD400 can project virtual images.

[0052] The virtual image contains a first display object 20 and a second display object 25. Specifically, in Figure 2 In the right-hand image, the portion outside the images IM20' of the first display object and IM25' of the second display object is displayed as transparent black in the HUD400. The first display object 20 is projected from the image IM20' of the first display object in the image projected by the HUD400, and the second display object 25 is projected from the image IM25' of the second display object. Furthermore, in the virtual image, the portion outside the aforementioned display objects is transparent, and the observer sees only the display objects superimposed on the real space. Here, the entire virtual image generated by the HUD400 is referred to as the "virtual image," and the non-transparent portion that the observer can perceive is referred to as the "display object." Additionally, the background of the display object does not necessarily have to be transparent; in this case, a specific portion of the virtual image can also be referred to as the display object.

[0053] The first display object 20 in the virtual image projected by the HUD400 has undergone vibration correction, thus accurately following the preceding vehicle 10 compared to the case without vibration correction. Furthermore, the second display object 25 has not undergone vibration correction, therefore becoming a fixed display in the observer's field of vision, making it an easy-to-view display unaffected by changes in the position or attitude of the vehicle 30.

[0054] In the above-described embodiment, the circuitry 160 is configured to display the HUD 400 of the first display object 20 and the second display object 25 corresponding to the real object of the real space in the display region 40. The circuitry 160 includes the storage section 161 and the warp processing section 130. The storage section 161 stores the warped object image WTIM including the image IM20 of the first display object and the image IM25 of the second display object. The warp processing section 130 performs warp processing on the warped object image WTIM to generate a display image DSIM displayed in the display region 40. The warp processing section 130 performs warp processing on the image IM20 of the first display object in the warped object image WTIM according to the first warp parameter PRM1. The first warp parameter PRM1 is a warp parameter that performs the vibration correction of the display position offset of the first display object 20 with respect to the real object and the distortion correction corresponding to the curvature of the display region 40. Further, the warp processing section 130 performs warp processing on the image IM25 of the second display object in the warped object image WTIM according to the second warp parameter PRM2. The second warp parameter PRM2 is a warp parameter that performs the distortion correction.

[0055] Thus, the vibration correction is performed with respect to the first display object 20 that follows the real object, and the vibration correction is not performed with respect to the second display object 25 that does not follow the real object. Thereby, the AR display is realized, and with respect to the display object that is not the AR display object, fixed display in the field of view of the observer of the HUD is possible. Further, according to the present embodiment, the vibration correction is performed together with the distortion correction in the warp processing, and thus, the vibration correction is performed at the timing of the warp processing close to the timing of the HUD display. Thereby, compared with the prior art in which the processing time from the vibration correction to the display is long as described in the above-described Patent Documents 1 to 3, the display position offset of the real object with respect to the first display object 20 is accurately corrected.

[0056] Further, in the present embodiment, the circuitry 160 includes the warp parameter selection section 162. The warp parameter selection section 162 outputs the first warp parameter PRM1 to the warp processing section 130 when the warp processing section 130 performs warp processing on the first region AR1 in which the first display object 20 is displayed. Further, the warp parameter selection section 162 outputs the second warp parameter PRM2 to the warp processing section 130 when the warp processing section 130 performs warp processing on the second region AR2 in which the second display object 25 is displayed.

[0057] Thus, the first area AR1 is subjected to the warping process based on the first warping parameter PRM1, whereby the image IM20 of the first display object is subjected to the distortion correction and the vibration correction. Further, the second area AR2 is subjected to the warping process based on the second warping parameter PRM2, whereby the image IM25 of the second display object is subjected to the distortion correction without being subjected to the vibration correction.

[0058] Further, in the present embodiment, the storage section 161 is a line buffer.

[0059] The warping process is performed, for example, in units of one line or a plurality of lines. Thus, the storage section 161 can store the image data of one line or a plurality of lines required for the warping process in the warped object image. Thus, the storage section 161 can not be a frame memory but a line buffer.

[0060] Further, the operation of the above-described circuit device 160 can also be performed as an image processing method of the HUD as follows. In this case, the subject of each step is not limited to the circuit device 160. For example, the display system of the HUD described later can be the subject of each step.

[0061] The image processing method is an image processing method of a head-up display that displays a first display object 20 and a second display object 25 corresponding to a real object of a real space in a display region 40. In the image processing method, a warped object image WTIM including an image IM20 of the first display object and an image IM25 of the second display object is subjected to a warping process, and a display image DSIM displayed in the display region 40 is generated. In the image processing method, the image IM20 of the first display object in the warped object image WTIM is subjected to the warping process in accordance with a first warping parameter PRM1 that performs a vibration correction that shifts the display position of the first display object 20 with respect to the real object and a distortion correction that corresponds to the curvature of the display region 40. In the image processing method, the image IM25 of the second display object in the warped object image WTIM is subjected to the warping process in accordance with a second warping parameter PRM2 that performs the distortion correction.

[0062] 2. Second Structural Example

[0063] Figure 3 A second structural example of the circuit device 160 is shown. The circuit device 160 includes a storage section 161, a warping processing section 130, and a layer selection section 163. The warping processing section 130 includes a first correction section 131, a second correction section 132, and a synthesis section 133. Further, regarding the structures and operations that are the same as those of the first structural example, appropriate description is omitted.

[0064] Figure 4An explanatory diagram of the processing performed by the 2nd configuration example of the circuit device 160 is shown. In addition, the action after the display image is output to the HUD is the same as the 1st configuration example.

[0065] As shown in the left and center diagrams of FIG. 13, the 1st correction section 131 performs a warping process using the 1st warping parameter PRM1 on the warping target image WTIM, and thereby generates the 1st corrected image CIM1. This warping process is performed on the entire warping target image WTIM. That is, the 1st corrected image CIM1 is an image in which the entire warping target image WTIM has been subjected to distortion correction and vibration correction. The 1st corrected image CIM1 includes the image IM20a of the 1st display object that has been subjected to distortion correction and vibration correction, and the image IM25a of the 2nd display object that has been subjected to distortion correction and vibration correction. Figure 4

[0066] The 2nd correction section 132 performs a warping process using the 2nd warping parameter PRM2 on the warping target image WTIM, and thereby generates the 2nd corrected image CIM2. This warping process is performed on the entire warping target image WTIM. That is, the 2nd corrected image CIM2 is an image in which the entire warping target image WTIM has been subjected to distortion correction. The 2nd corrected image CIM2 includes the image IM20b of the 1st display object that has been subjected to distortion correction, and the image IM25b of the 2nd display object that has been subjected to distortion correction.

[0067] The 1st correction section 131 and the 2nd correction section 132 are different warping engines, respectively. That is, the warping process of the 1st correction section 131 and the warping process of the 2nd correction section 132 are performed in parallel by the respective warping engines. The warping engine can be either a forward warping engine or a reverse warping engine.

[0068] As shown in the center and right diagrams of FIG. 13, the synthesis section 133 synthesizes the 1st corrected image CIM1 and the 2nd corrected image CIM2, and thereby generates the display image DSIM. Here, the 1st region AR1 and the 2nd region AR2 are defined in the display image DSIM. The synthesis section 133 selects the 1st corrected image CIM1 in the 1st region AR1 and selects the 2nd corrected image CIM2 in the 2nd region AR2. Thereby, as shown in the right diagram of FIG. 13, the display image DSIM is output that includes the image IM20a of the 1st display object that has been subjected to distortion correction and vibration correction, and the image IM25b of the 2nd display object that has been subjected to distortion correction but not vibration correction. Information that specifies the 1st region AR1 and the 2nd region AR2 is input from the layer selection section 163 to the synthesis section 133, and the synthesis section 133 performs the above selection in accordance with this information. Figure 4 Figure 4

[0069] ​​​Alternatively, the synthesizing section 133 can also perform synthesis based on α-blending. That is, the synthesizing section 133 sets the blending ratio of the first corrected image CIM1 to a1 and the blending ratio of the second corrected image CIM2 to 1-a1 in the first area AR1, and blends the pixel values of the respective pixels within the first area AR1. Further, the synthesizing section 133 sets the blending ratio of the first corrected image CIM1 to 1-a2 and the blending ratio of the second corrected image CIM2 to a2 in the second area AR2, and blends the pixel values of the respective pixels within the second area AR2. For example, if a1=a2=1 is set, the same as the synthesis based on the above-described selection is performed. Information designating the first area AR1 and the second area AR2 is input from the layer selecting section 163 to the synthesizing section 133, and the synthesizing section 133 performs the above-described α-blending in accordance with the information.

[0070] Further, the "synthesis" herein is not a large-delayed synthesis of synthesizing the entire 1 frame at once on the frame memory. In the above-described embodiment, selection and α-blending are cited as examples of the synthesis. More specifically, the first correcting section 131 and the second correcting section 132 perform the distortion processing on a line-by-line basis, and output the results thereof on a line-by-line basis. At this time, the synthesizing section 133 performs selection or α-blending on a line-by-line basis, and thereby outputs the display image on a line-by-line basis. Further, the processing can also be performed in units of a plurality of lines, but the number of lines is less than the number of lines of 1 frame.

[0071] In the above-described embodiment, the distortion processing section 130 includes the first correcting section 131, the second correcting section 132, and the synthesizing section 133. The first correcting section 131 performs distortion processing on the warped target image WTIM in accordance with the first distortion parameter PRM1, and outputs the first corrected image CIM1. The second correcting section 132 performs distortion processing on the warped target image WTIM in accordance with the second distortion parameter PRM2, and outputs the second corrected image CIM2. The synthesizing section 133 synthesizes the first corrected image CIM1 and the second corrected image CIM2, and outputs the display image DSIM.

[0072] Thus, the distortion correction and the vibration correction based on the first correcting section 131 and the distortion correction based on the second correcting section 132 can be performed in parallel. Further, the synthesizing section 133 synthesizes the first corrected image CIM1 obtained through the distortion correction and the vibration correction and the second corrected image CIM2 obtained through the distortion correction, and thereby the display image DSIM in which the area subjected to the vibration correction and the area not subjected to the vibration correction are mixed can be generated.

[0073] Further, in the present embodiment, the synthesis section 133 outputs the first corrected image CIM1 as the display image DSIM with respect to the first region AR1 in which the first display object 20 is displayed. Further, the synthesis section 133 outputs the second corrected image CIM2 as the display image DSIM with respect to the second region AR2 in which the second display object 25 is displayed.

[0074] Thus, the first region AR1 subjected to the distortion correction and the vibration correction and the second region AR2 subjected to the distortion correction and not subjected to the vibration correction are synthesized and output as the display image DSIM. Thereby, the vibration correction is performed with respect to the first display object 20 displayed in AR, and the vibration correction is not performed with respect to the second display object 25 displayed fixedly. Further, the synthesis is performed by the selection of the regions, and thus, the processing time from the vibration correction to the HUD display is shorter than the related art in which the synthesis is performed in units of frames on the frame memory. Thereby, the display position shift can be reduced.

[0075] Further, in the present embodiment, the synthesis section 133 performs the alpha blending of the first corrected image CIM1 and the second corrected image CIM2 and outputs the display image DSIM.

[0076] Thus, the first corrected image CIM1 subjected to the distortion correction and the vibration correction and the second corrected image CIM2 subjected to the distortion correction and not subjected to the vibration correction are alpha-blended. Thereby, the display image DSIM including the first display object 20 subjected to the distortion correction and the vibration correction and the second display object 25 subjected to the distortion correction and not subjected to the vibration correction is generated. Further, the synthesis is performed by the alpha blending, and thus, the processing time from the vibration correction to the HUD display is shorter than the related art in which the synthesis is performed in units of frames on the frame memory. Thereby, the display position shift can be reduced.

[0077] 3. Display system

[0078] Next, the display system including the circuit device 160 will be described. Here, the case where the vibration correction is the delay time compensation will be described as an example, but, as described above, the vibration correction is not limited to the delay time compensation. Further, the first display object which is the object of the vibration correction will be mainly described, but, as described above, the second display object which is displayed fixedly in the HUD is also displayed. Hereinafter, the first display object will be referred to as a virtual object. As described above, the virtual object is an object which is displayed in the HUD in following the real object in the real space.

[0079] As described above, the display system including the circuit device 160 is configured to perform the vibration correction with respect to the virtual object displayed in AR and not to perform the vibration correction with respect to the second display object displayed fixedly in the HUD. Thus, the display position shift can be reduced. Figure 2The display system of the HUD mounted on the automobile 30 tracks the preceding vehicle 10 or the like using a sensor such as a Lidar, and renders a virtual object 20 based on the tracking result, and performs a warp process on the rendered image to cause the HUD to display the virtual object 20. At this time, there is a delay due to the tracking process, the rendering process, the warp process, or data communication, or the like, from the time when the sensor is sampled to the time when the virtual object 20 is displayed on the HUD. This delay due to the process or the communication or the like is referred to as a delay time.

[0080] The delay time is a cause of a positional deviation of the virtual object 20 from the preceding vehicle 10 or the like. That is, the position of the preceding vehicle 10 or the like at the time when the display system renders the virtual object 20 deviates from the position of the preceding vehicle 10 or the like at the time when the virtual object 20 is actually displayed on the HUD, and thus the virtual object 20 and the preceding vehicle 10 or the like are deviated. For example, in the rendering process, even if the position of the preceding vehicle 10 or the like at the future display time is predicted, there is a time from the predicted point of time to the display, and thus the predicted position of the preceding vehicle 10 or the like can deviate from the position of the preceding vehicle 10 or the like at the time when the virtual object 20 is actually displayed on the HUD.

[0081] In Figure 5 In the following, as an example, a diagram illustrating a display position deviation based on a tilt displacement or a vertical displacement is shown. xyz is a coordinate system fixed in an automobile on which a HUD is mounted. The z direction is a traveling direction of the automobile, and the x direction and the y direction are orthogonal to each other and orthogonal to the z direction. When the y direction coincides with the plumb direction, the x direction is a horizontal direction, and the y direction is a vertical direction or an up-down direction. When the posture of the automobile becomes such that the y direction does not coincide with the plumb direction, the x direction and the y direction do not coincide with the horizontal and the vertical, but in the following, for the sake of convenience, the x direction is referred to as the horizontal direction, and the y direction is referred to as the vertical direction or the up-down direction. The direction of the arrow shown in the diagram is set as a positive direction, and in the case where it is desired to indicate the positive and negative directions, it is described as +x direction, -x direction, or the like.

[0082] Figure 5 A diagram of the first layer of FIG. 1 illustrates a state in which there is no positional deviation of the virtual object 21 from the real object 11. The real object 11 is an object existing in a real space, and is an object that becomes a tracking target of AR display. For example, Figure 2 The preceding vehicle 10 or the like of the lower diagram of FIG. 1 is a real object. The projection unit 36 of the HUD projects a virtual image onto the windshield 35, and the virtual object 21 in the virtual image is seen by the eyes 50 of the driver.

[0083] Figure 5The figure of the 2nd layer shows the positional shift of the virtual object 21 with respect to the real object 11 when the windshield 35 and the projection unit 36 are shifted in the +y direction by WSM. The vertical shift WSM is caused by the tilt rotation or the vertical shift of the automobile 30. The tilt rotation is the rotation around the axis parallel to the x direction, and the vertical shift is the parallel movement in the y direction. When the display system of the HUD renders the virtual object 21, the state of the figure of the 1st layer is assumed, but there is a time during which the display to the HUD is not performed, and thus, when the vertical shift WSM is generated in this time, the vertical shift WSM is not reflected in the rendered image. Therefore, the display position of the virtual object 21 is shifted in the +y direction due to the vertical shift WSM of the windshield 35 or the like, and the virtual object 21 is seen to be shifted in the +y direction with respect to the real object 11. For example, in the case where the automobile 30 repeatedly performs the tilt motion due to the unevenness of the road, the virtual object 21 is seen to be shifted in the up-and-down manner with respect to the real object 11.

[0084] Figure 5 The figure of the 3rd layer shows the positional shift of the virtual object 21 with respect to the real object 11 when the driver's eye 50 is shifted in the +y direction by DEM. The windshield 35 is not moved, but the eye 50 is shifted in the +y direction with respect to the windshield 35, and thus, the display position of the virtual object 21 is shifted in the -y direction. Therefore, the virtual object 21 is seen to be shifted in the -y direction with respect to the real object 11.

[0085] Figure 5 The figure of the 4th layer shows the positional shift of the virtual object 21 with respect to the real object 11 when the real object 11 is shifted in the -y direction by RJM. The windshield 35 and the eye 50 are not moved, and thus, the display position of the virtual object 21 is not shifted, but the real object 11 is shifted in the -y direction, and thus, the virtual object 21 is seen to be shifted in the +y direction with respect to the real object 11.

[0086] Figure 6 is a figure that explains the delay time compensation of the present embodiment. Figure 6 shows the real object 11 and the virtual object 21 as viewed from the driver. Figure 6 The left figure of is a state in which the display positions of the real object 11 and the virtual object 21 coincide when viewed from the driver. As Figure 6 As shown in the middle figure of, in the case where the driver moves the head in the upper left direction, the real object 11 is shifted in the upper left direction by MV. The display position of the virtual object 21 is not changed, and thus, the real object 11 and the virtual object 21 are shifted by MV. As Figure 6In the right view shown in FIG. 1, in the present embodiment, the shift amount MV based on the delay time is compensated, whereby the display positions of the real object 11 and the virtual object 21 are made coincident. By performing such delay time compensation, in the state of the transition from the left view to the right view when the driver moves the head to the upper left, the positional deviation state of the middle view can be avoided. Details will be described later, but in the present embodiment, the delay time compensation is performed in the warping processing as close to the display timing as possible, whereby the positional deviation based on the delay time is reduced as much as possible.

[0087] In order to consider what kind of delay time exists in the display system of the HUD, a typical display system in the related art and the delay time generated in the display system will be described.

[0088] Figure 7 FIG. 1 is a configuration example of a display system 90 in the related art. The display system 90 includes a sensor 65, a processing device 60, a HUD controller 70, and a HUD 80.

[0089] The sensor 65 is a sensor that detects the position, posture, or movement of the automobile, the driver, or the real object.

[0090] The processing device 60 includes a sensor interface 61, a tracking processing section 62, and a rendered image generation section 63. The sensor interface 61 is an interface circuit that receives the output signal of the sensor 65. The tracking processing section 62 is constituted by a CPU and tracks the position, posture, or movement of the automobile, the driver, or the real object based on the output signal of the sensor 65. The rendered image generation section 63 is constituted by a GPU and renders the virtual object in such a manner that the virtual object is displayed in a manner of following the real object based on the tracking information of the automobile, the driver, or the real object.

[0091] The HUD controller 70 includes a warping processing section 71 that performs warping processing on the rendered image. The HUD 80 includes a display 81 that displays the image subjected to the warping processing. The display 81 is a display device such as a liquid crystal display device. The image displayed in the display 81 is projected onto the windshield glass through an optical system, whereby the virtual object of the rendered image is displayed.

[0092] Figure 8 A processing flow of the display system 90 is shown. Here, an example in which an IMU and a camera are used as the sensor 65 is shown. The IMU detects at a much higher sampling rate than the frame rate of the camera. tck, tck+1, tck+2 show the exposure timing of the camera. That is, the image obtained by the camera shows the position of the real object at the timing tck or the like. The image signal processing is processing performed after the exposure of the camera ends, and includes signal readout and development processing from the image sensor, and the like.

[0093] The tracking processing section 62 tracks the position of the real object based on the image after the image signal processing according to the visual-inertial odometry. In this stage, the position of the real object at the exposure time tck of the camera is detected. The tracking processing section 62 tracks the position of the real object by the inertial odometry according to the output signal of the IMU at the time tod after the visual-inertial odometry ends and the result of the visual-inertial odometry. By this processing, the position of the real object at the time tod is detected while compensating for the delay time from the exposure time tck of the camera to the time tod of the inertial odometry.

[0094] The rendering image generation section 63 renders the virtual object at the position corresponding to the real object detected by the inertial odometry, and outputs the rendered image to the warp processing section 71. At this time, the data communication time from the GPU constituting the rendering image generation section 63 to the HUD controller 70 is generated. The warp processing section 71 performs the warp processing on the rendered image, and the image after the warp processing is displayed in the HUD 80. The 1st line to the final line are the horizontal scanning lines of the display 81, and are scanned line by line. Therefore, in the image display of 1 frame, the time for scanning the 1st line to the final line is required, but here the time when the display of the 1st line is performed is set as representative and is set as the display time tdp.

[0095] The virtual object displayed in the HUD 80 corresponds to the position of the real object at the time tod. Therefore, there is a delay time from the time tod to the display time tdp, and the real object and the virtual object can be shifted due to the movement of the car, the driver, or the real object during this period.

[0096] As described above, by displaying the virtual object rendered using the tracking information detected before the display time tdp, there is a problem that the display position is shifted due to the delay time from the time when the tracking information is detected to the display time tdp. It is preferable to correct the display position shift as close to the time of the HUD display as possible. As described above, in Patent Documents 1 to 3, there is a problem that the processing time from the correction of the position shift to the output to the head-up display becomes long, and the correction of the position shift is insufficient.

[0097] Figure 9 is a configuration example of the display system 100 in the present embodiment. The display system 100 causes the HUD 400 to display the virtual object corresponding to the real object according to the output signal of the sensor 500. The display system 100 includes a tracking processing section 110, a rendering image generation section 120, a warp processing section 130, a parameter calculation section 140, a storage section 161, and a warp parameter selection section 162.

[0098] In addition, Figure 9The circuit device 160 is not shown, but the circuit blocks included in the circuit device 160 are included in the display system 100. The hardware configuration example of the display system 100 including the circuit device 160 is described later. Furthermore, Figure 9 The first configuration example of the circuit device 160 shown in Figure 1 is incorporated in the display system 100, but in the case where the second configuration example of the circuit device 160 shown in Figure 3 is incorporated in the display system 100, the layer selection section 163 can be provided instead of the distortion parameter selection section 162.

[0099] The sensor 500 is a sensor that detects the position, posture, or movement of a mobile body, an observer, or a real object. The sensor 500 is provided to the mobile body, and includes, for example, a Lidar, an IMU, a camera, an eye tracking sensor, or a head tracking sensor. The Lidar is an abbreviation for Light Detection and Ranging, and is a sensor that acquires three-dimensional information such as a z map. The IMU is an abbreviation for Inertial Measurement Unit, and is a sensor that detects the movement of one axis or multiple axes. The IMU is constituted by, for example, an acceleration sensor, a gyro sensor, or a combination thereof. The camera is a sensor that captures an image, which is two-dimensional information. The eye tracking sensor is a sensor that detects the position, line-of-sight direction, or both of the observer's eyes. The head tracking sensor is a sensor that detects the position, posture, or both of the observer's head.

[0100] The mobile body is an object that carries the HUD 400, the observer, and the sensor 500 and moves in a real space, and is, for example, an automobile, a two-wheeled vehicle, an airplane, or a ship. The observer is a user who views the virtual image projected to the HUD 400, and is the operator or the passenger of the mobile body. The real object is an object that exists in the real space. The real object can be an object whose position or posture in the HUD display region viewed by the observer changes when the position or posture of the mobile body, the observer, or the real object changes.

[0101] The tracking processing section 110 tracks the position, posture, or movement of the mobile body, the observer, or the real object based on the output signal of the sensor 500, and outputs the result as tracking information. For example, the tracking processing section 110 tracks the real object based on two-dimensional ranging information from the Lidar or a two-dimensional image from the camera. Furthermore, the tracking processing section 110 tracks the automobile based on information of acceleration or angular velocity from the IMU. Furthermore, the tracking processing section 110 tracks the driver's eyes based on information of the position or line-of-sight direction of the eyes from the eye tracking sensor.

[0102] The tracking information is information indicating the position, posture, or movement of the mobile body, the observer, or the real object, and can be information of any form. For example, the tracking information is a coordinate indicating the position in the real space, an angle indicating the posture, a vector indicating the parallel movement, or an angular velocity indicating the rotation, or the like. Alternatively, the tracking information can be information in which the coordinate in the real space or the like is converted into a coordinate, an angle, a vector, or an angular velocity on the image, or the like. The tracking information includes first tracking information of the mobile body, second tracking information of the observer, and third tracking information of the real object. However, the tracking information can include at least one of the first to third tracking information, and for example, the second tracking information of the observer can be omitted.

[0103] The rendering image generation section 120 renders the virtual object based on the tracking information of the mobile body, the observer, or the real object, and outputs a rendering image including the virtual object. Specifically, the rendering image generation section 120 finds a position at which the real object is seen in the display region of the HUD 400, and renders the virtual object at a position corresponding to the position of the real object. In addition, the rendering image generation section 120 renders a second display object that is fixedly displayed, in addition to the first display object, that is, the virtual object.

[0104] The parameter operation section 140 finds a delay time compensation parameter based on the tracking information, corrects a warping parameter used in the distortion correction based on the delay time compensation parameter, and outputs the corrected warping parameter. The corrected warping parameter is a first warping parameter. In addition, the parameter operation section 140 outputs a warping parameter used in the distortion correction as a second warping parameter. The tracking information used here is tracking information that is sampled after the tracking information used in the rendering process. From the viewpoint of minimizing the delay time, it is preferable to use tracking information that is acquired immediately before the parameter operation or as close to the time of the parameter operation as possible.

[0105] The delay time compensation parameter is a parameter that compensates for a difference between the position of the virtual object in the rendering image and the position of the virtual object at the display time. The delay time compensation parameter indicates a shift amount or a rotation angle on the image data. More specifically, the delay time compensation parameter indicates a shift amount or a rotation angle in an image that has no distortion before the warping process.

[0106] The first warping parameter and the second warping parameter are parameters of coordinate conversion in the warping process, and are parameters that correspond the coordinates of the input image in the warping process to the coordinates of the output image. The warping parameter is, for example, a matrix or a table indicating the coordinate conversion between the input image and the output image.

[0107] The warping process section 130 warps the rendering image based on the first warping parameter and the second warping parameter, and outputs a warping image. Figures 1-4The display image is an image displayed in a display panel of the HUD 400, and the displayed image in the display panel is projected onto a screen by a projection unit, whereby an observer who views the screen sees a real space and a virtual image superimposed thereon.

[0108] The HUD 400 includes a projection unit for projecting a virtual image onto a screen. The projection unit includes a display panel that displays a display image, and an optical system that projects the display image displayed in the display panel as a virtual image onto a screen. The screen is a windshield or the like of a moving body. Alternatively, the HUD 400 can include a dedicated screen onto which a virtual image is projected. The optical system described above includes a mirror and a lens.

[0109] Figure 10 The detailed processing flow of the tracking processing section 110 is shown in Figure 11 The processing flow of the display system 100 is shown in FIG. 6. In addition, Figure 10 The tracking processing shown is an example, and the method of tracking a real object or the like is not limited thereto. Furthermore, the sensor used in the tracking processing can be selected as appropriate in accordance with the method to be employed.

[0110] Figure 11 The operation of the sensor shown is as described in Figure 8 The tracking processing section 110 performs visual-inertial odometry processing 111 based on the image IMG(tck) obtained by imaging by the camera at time tck and the motion information MTI(tck) of the moving body sampled by the IMU at time tck, and estimates the position and attitude Pr(tck) of the real object and the moving body at time tck.

[0111] The tracking processing section 110 performs inertial odometry processing 112 based on the position and attitude Pr(tck) and the motion information MTI(tnowl) of the moving body sampled by the IMU at time tnowl, and estimates the position and attitude Pc(tck, tnowl) of the real object and the moving body at time tnowl. tnowl is after tck and before the start of the inertial odometry processing 112. More preferably, tnowl is after the end of the visual-inertial odometry processing 111 and before the start of the inertial odometry processing 112.

[0112] The tracking processing section 110 performs prediction processing 113 based on the position and attitude Pc(tck, tnowl), and predicts the position and attitude Pp(tck, tnowl, tfuture) of the real object and the moving body at time tfuture after tnowl. In Figure 11In this case, an example in which tfuture = tdp is set to predict the position and attitude Pp(tck, tnow1, tdp) is shown. tdp is a display time of the HUD. The prediction processing 113 predicts the future position and attitude Pp, for example, from a differential value of the position and attitude Pc obtained in time series.

[0113] The rendering image generation section 120 renders the virtual object based on the position and attitude Pp(tck, tnow1, tdp), and outputs a rendering image. If the HUD 400 is caused to display this rendering image directly, the virtual object is displayed in correspondence with the position of the real object indicated by the position and attitude Pp.

[0114] Here, the predicted position and attitude Pp(tck, tnow1, tdp) is predicted using information of tck and tnow1, and is not the actual sensor output position based on tdp. That is, in a case where actual motion occurring between tnow1 and tdp is different from the motion predicted by the prediction processing 113, a position offset based on a delay time can occur between the virtual object and the real object. That is, in the example of FIG. 6, the position and attitude Pp(tck, tnow1, tdp) is predicted using information of tck and tnow1, and is not the actual sensor output position based on tdp. That is, in a case where actual motion occurring between tnow1 and tdp is different from the motion predicted by the prediction processing 113, a position offset based on a delay time can occur between the virtual object and the real object. Figure 11 In the example of FIG. 6, the delay time that is the compensation target is a delay time based on a processing time between tnow1 and tdp. In addition, the delay time that is the compensation target is not limited thereto. For example, in a case where the rendering processing is performed using Pr(tck) with omission of the inertial odometry and the prediction processing, a delay time based on a processing time between tck and tdp becomes the compensation target.

[0115] In the present embodiment, the above-described position offset based on the delay time is compensated for as described below. The tracking processing section 110 performs the visual inertial odometry processing 111 based on the image IMG(tck+1) obtained by the camera imaging at the time tck+1 and the motion information MTI(tck+1) of the mobile body sampled by the IMU at the time tck+1, and estimates the position and attitude Pr(tck+1) of the real object and the mobile body at the time tck+1. tck+1 is the next camera exposure time of tck.

[0116] The tracking processing section 110 performs the inertial odometry processing 112 based on the position and attitude Pr(tck+1) and the motion information MTI(tnow2) of the mobile body sampled by the IMU at the time tnow2, and estimates the position and attitude Pc(tck+1, tnow2) of the real object and the mobile body at the time tnow2.

[0117] The tracking processing section 110 performs the prediction processing 113 based on the position and attitude Pc(tck+1, tnow2), and predicts the position and attitude Pp(tck+1, tnow2, tdp) of the real object and the mobile body at tdp.

[0118] The parameter calculation section 140 calculates the delay time compensation parameter from the position and attitude Pp(tck, tnowl, tdp) and the position and attitude Pp(tck+1, tnow2, tdp). Specifically, the parameter calculation section 140 calculates the delay time compensation parameter that eliminates the positional shift of the virtual object due to the difference between the position and attitude Pp(tck, tnowl, tdp) and the position and attitude Pp(tck+1, tnow2, tdp). The parameter calculation section 140 corrects the distortion parameter using the delay time compensation parameter, and outputs the corrected distortion parameter.

[0119] The distortion processing section 130 performs distortion processing on the rendering image using the corrected distortion parameter, and generates a display image. In addition, the corrected distortion parameter corresponds to the first distortion parameter described in the first embodiment, and the distortion processing using the corrected distortion parameter corresponds to distortion processing including distortion correction and vibration correction. This distortion processing is performed on a second region including the virtual object in the rendering image, which is the distortion target image. The distortion processing section 130 performs distortion processing including only distortion correction on a first region in the rendering image. Figures 1-4

[0120] The distortion processing section 130 sequentially performs distortion processing on the first row to the final row in the display panel of the HUD 400 according to the scanning timing from the first row to the final row, and sequentially outputs the data of each row after the distortion processing corresponding to the scanning timing to the HUD 400. In the first embodiment, the display timing of the first row is set to tdp, but tdp can also be any timing within the frame period during which the first row to the final row are displayed. Figure 8

[0121] ​​In the above embodiment, the display system 100 is a display system of the HUD 400 that displays the first display object 20 and the second display object 25 corresponding to the real object of the real space in the display region 40. The display system 100 includes the storage section 161 and the warp processing section 130. The storage section 161 stores the warped object image WTIM including the image IM20 of the first display object and the image IM25 of the second display object. The warp processing section 130 performs the warp processing on the warped object image WTIM to generate the display image DSIM displayed in the display region 40. The warp processing section 130 performs the warp processing on the image IM20 of the first display object in the warped object image WTIM according to the first warp parameter PRM1. The first warp parameter PRM1 is a warp parameter that performs the vibration correction of the display position of the first display object 20 with respect to the real object and the distortion correction corresponding to the curvature of the display region 40 and / or the distortion of the optical system. Further, the warp processing section 130 performs the warp processing on the image IM25 of the second display object in the warped object image WTIM according to the second warp parameter PRM2. The second warp parameter PRM2 is a warp parameter that performs the distortion correction.

[0122] Further, in the present embodiment, the display system 100 includes the parameter operation section 140 that operates the first warp parameter PRM1 and the second warp parameter PRM2. The parameter operation section 140 operates the first warp parameter PRM1 according to the tracking information. The tracking information is at least one of the first tracking information of the mobile body on which the HUD 400 is mounted, the second tracking information of the observer of the HUD 400, and the third tracking information of the real object.

[0123] Thus, the parameter operation section 140 can operate the parameter of the vibration correction according to the tracking information. Thereby, the parameter operation section 140 can operate the first warp parameter PRM1 according to the parameter of the vibration correction, and can output the first warp parameter PRM1 that performs the vibration correction and the distortion correction to the warp processing section 130.

[0124] Further, in the present embodiment, the display system 100 includes the rendering image generation section 120 that renders the warped object image WTIM. The parameter operation section 140 operates the delay time compensation parameter as the parameter of the vibration correction to obtain the first warp parameter PRM1 according to the tracking information. The delay time compensation parameter is a parameter that compensates for the delay time including the rendering processing delay time of the warped object image WTIM.

[0125] Thus, the delay time compensation can be performed by the vibration correction. That is, the position shift of the virtual object due to the delay time based on the rendering process between tnow1 and tdp is compensated according to the delay time compensation parameter. The delay time compensation is performed in the distortion process immediately before the display, whereby the residual delay time can be minimized, and thus, the AR display with high followability of the virtual object to the real object can be realized.

[0126] Further, in the present embodiment, the rendering image generation section 120 generates the rendering image according to the tracking information at the first time. The parameter operation section 140 operates the delay time compensation parameter according to the tracking information at the second time after the first time and before the display time tdp of the image and the tracking information at the first time. In the present embodiment, the parameter operation section 140 operates the delay time compensation parameter according to the tracking information at the second time after the first time and before the display time tdp of the image and the tracking information at the first time. Figure 8 In the example of FIG. 10, tnow1 corresponds to the first time, and tnow2 corresponds to the second time.

[0127] Thus, the delay time between the first time and the display time is shortened to the delay time between the second time and the display time, and thus, the position shift based on the delay time is reduced. The use of the second distortion parameter based on the delay time compensation parameter is described with reference to the example of FIG. 11. Figure 8 In the example of FIG. 10, tnow1 corresponds to the first time, and tnow2 corresponds to the second time.

[0127] Thus, the delay time between the first time and the display time is shortened to the delay time between the second time and the display time, and thus, the position shift based on the delay time is reduced. The use of the second distortion parameter based on the delay time compensation parameter is described with reference to the example of FIG. 11.

[0128] Further, in the present embodiment, the parameter operation section 140 corrects the distortion parameter of the coordinate conversion corresponding to the bending of the display region according to the delay time compensation parameter, and thus, outputs the first distortion parameter. The distortion processing section 130 performs the distortion process using the first distortion parameter with respect to the first region AR1 of the distortion target image WTIM, and thus, performs the delay time compensation process.

[0129] Thus, the distortion process is performed using the first distortion parameter based on the delay time compensation parameter, and thus, the delay time compensation can be performed in the distortion process. For example, a configuration of the image processing in which the delay time compensation is separately performed from the distortion process is also considered, but the processing load is increased in correspondence with the image processing of the delay time compensation, and further, the delay time of the distortion process remains after the delay time compensation. In the present embodiment, the delay time compensation and the distortion process are integrated, and thus, the increase of the processing load can be suppressed, and the delay time can be minimized.

[0130] 4. Detailed configuration example of display system

[0131] Hereinafter, a hardware configuration example of the display system 100 will be described, but the hardware configuration of the display system 100 is not limited to this. For example, hereinafter, the parameter calculation section 140 is provided to the processing device 150, but the parameter calculation section 140 can also be provided to the circuit device 160. Further, hereinafter, the circuit device 160 includes the distortion parameter selection section 162, but in the case of adopting the second configuration example of the display system 100, a layer selection section 163 can be included instead of the distortion parameter selection section 162. Figure 3

[0132] Figure 12 is a detailed configuration example of the display system 100. The display system 100 includes the processing device 150 and the circuit device 160. The processing device 150 is constituted by, for example, an SoC, and includes the interface 151, the tracking processing section 110, the rendered image generation section 120, the parameter calculation section 140, and the storage section 145. The SoC is an abbreviation of System on Chip. The circuit device 160 includes the storage section 161, the distortion processing section 130, and the distortion parameter selection section 162. In addition, the same reference numerals are attached to the structural elements that have already been described, and the description of the structural elements is appropriately omitted.

[0133] The interface 151 is an interface for inputting an output signal of the sensor 500 to the processing device 150. The interface 151 is, for example, a reception circuit that receives the output signal of the sensor 500. Alternatively, the interface 151 can also be a terminal to which the output signal of the sensor 500 is input. In this case, the output signal of the sensor 500 input to the terminal is input to a processor that constitutes the tracking processing section 110, and the processor performs reception processing on the output signal of the sensor 500.

[0134] The tracking processing section 110 performs tracking processing in accordance with the output signal of the sensor 500. The rendered image generation section 120 renders a virtual object in accordance with the tracking information output from the tracking processing section 110. The tracking processing section 110 and the rendered image generation section 120 are constituted by a processor such as a CPU or a GPU. The CPU is an abbreviation of Central Processing Unit. The GPU is an abbreviation of Graphical Processing Unit. The tracking processing section 110 and the rendered image generation section 120 can be realized by one processor, or can be realized by separate processors, respectively.

[0135] ​The storage section 145 is a semiconductor memory such as a RAM or a nonvolatile memory, and stores the distortion parameter before the delay time compensation. The RAM is an abbreviation of Random Access Memory. As described above, the distortion parameter is a matrix or a table. The distortion parameter can be either of a forward mapping parameter and a reverse mapping parameter. The forward mapping parameter is a parameter that corresponds each pixel of the input image with a movement destination coordinate corresponding to the each pixel, or a parameter that corresponds each pixel of the input image with a relative movement amount toward a movement destination coordinate corresponding to the each pixel. The reverse mapping parameter is a parameter that corresponds each pixel of the output image with a reference source coordinate corresponding to the each pixel, or a parameter that corresponds each pixel of the output image with a relative movement amount with respect to a reference source coordinate corresponding to the each pixel.

[0136] The parameter operation section 140 calculates the delay time compensation parameter based on the tracking information, corrects the distortion parameter read out from the storage section 145 based on the delay time compensation parameter, and outputs the corrected distortion parameter as a first distortion parameter. In addition, the parameter operation section 140 outputs the distortion parameter read out from the storage section 145 as a second distortion parameter. The parameter operation section 140 can be implemented by the same processor as the processors of the tracking processing section 110 and the rendered image generation section 120, or can be implemented by a different processor from them.

[0137] The distortion processing section 130 performs a distortion process on the rendered image using the first distortion parameter and the second distortion parameter, thereby generating a display image. Here, the rendered image corresponds to the distortion target image. The distortion processing section 130 is constituted by a logic circuit, for example, a gate array in which a wiring is automatically arranged or a standard cell array in which a wiring is automatically arranged. The HUD 400 includes a display 410, and causes the display 410 to display the display image. The display 410 is a display device such as a liquid crystal display device. The HUD 400 includes a projection optical unit not shown. The projection optical unit projects the image displayed in the display 410 onto a screen, thereby superimposing a virtual object on a real space.

[0138] 5. Distortion process

[0139] Hereinafter, a detailed description will be given of the distortion process with the reverse distortion as an example. However, the distortion process can be either of the forward distortion and the reverse distortion, and the distortion processing section 130 can be either of a forward distortion engine and a reverse distortion engine.

[0140] Figure 13This is an explanatory diagram of the distortion processing in this embodiment. The input image is a rendered image. The output image is a display image shown on the LCD display panel of a HUD, etc. (xsrc, ysrc) represent the horizontal and vertical positions in the input image. (xdst, ydst) represent the horizontal and vertical positions in the output image.

[0141] When g() is used to represent the distortion parameter correction using the time delay compensation parameter, the position in the time delay compensation image is represented as shown in equation (1).

[0142] (xlat,ylat)=g(xsrc,ysrc)…(1)

[0143] When using f() to represent distortion correction based on the torsion parameter corresponding to the curvature of the screen, the position in the output image is represented as shown in equation (2).

[0144] (xdst, ydst)=f(xlat, ylat)…(2)

[0145] According to equations (1) and (2) above, the coordinate transformation between the input image and the output image is expressed as shown in equation (3).

[0146] (xdst, ydst)=f·g(xsrc, ysrc)…(3)

[0147] The reverse twist uses the inverse transformation of equation (3) above, and thus, as in equation (4) below.

[0148] (xsrc, ysrc) = g -1 ·f -1 (xdst,ydst)…(4)

[0149] In this embodiment, the parameter calculation unit 140 calculates the delay time compensation parameter g. -1 operation g -1 ·f -1 Therefore, the distortion parameter f -1 Perform correction and output the corrected torsion parameter g. -1 ·f -1 The distortion processing unit 130 uses the corrected distortion parameter g. -1 ·f -1 The distortion process of equation (4) above is performed to generate the output image.

[0150] Additionally, based on the corrected torsion parameter g -1 ·f -1 The object of the distortion processing is Figures 1-4 The first region AR1 is described in the diagram, and the second region AR2 is subjected to a process based on the distortion parameter f. -1twist processing. That is, the corrected twist parameter g -1 ·f -1 is a first twist parameter, the twist parameter f -1 is a second twist parameter.

[0151] Figure 14 is a graph showing the correspondence relation of the tracking information of a moving body or the like and the delay time compensation based on the tracking information. Here, the 6-axis motion information is obtained as the tracking information.

[0152] The yaw displacement Δα is a rotational displacement in which an axis parallel to the vertical direction, that is, the y direction is set as a rotational axis. The yaw displacement Δα causes the real object seen by the observer through the screen to be horizontally displaced. The horizontal displacement is a displacement in the horizontal direction, that is, the x direction. Therefore, the method of the delay time compensation is the horizontal displacement.

[0153] The pitch displacement Δβ is a rotational displacement in which an axis parallel to the horizontal direction, that is, the x direction is set as a rotational axis. The pitch displacement Δβ causes the real object seen by the observer through the screen to be vertically displaced. The vertical displacement is a displacement in the vertical direction, that is, the y direction. Therefore, the method of the delay time compensation is the vertical displacement.

[0154] The roll displacement Δγ is a rotational displacement in which an axis parallel to the front-rear direction of the moving body, that is, the z direction is set as a rotational axis. The roll displacement Δγ causes the real object seen by the observer through the screen to be rotated. Therefore, the method of the delay time compensation is the rotation.

[0155] The horizontal displacement Δx is a displacement in the x direction, which causes the real object seen by the observer through the screen to be horizontally displaced. Therefore, the method of the delay time compensation is the horizontal displacement.

[0156] The vertical displacement Δy is a displacement in the y direction, which causes the real object seen by the observer through the screen to be vertically displaced. Therefore, the method of the delay time compensation is the vertical displacement.

[0157] The front-rear displacement Δz is a displacement in the z direction, which causes the real object seen by the observer through the screen to be reduced or enlarged. Therefore, the method of the delay time compensation is the reduction or zooming.

[0158] Figure 15 Examples of the delay time compensation parameter implementing the above compensation method are shown. Figure 15 The matrix on the right side of the equation shown corresponds to Figure 13 g -1 described in the above equation. m13 indicates the horizontal displacement, and m23 indicates the vertical displacement. m12 indicates the horizontal skew. The horizontal skew is a conversion that distorts a rectangle into a parallelogram, but here it is used as an approximation of the rotation. m11 indicates the horizontal scale, and m22 indicates the vertical scale. m11 and m22 correspond to the reduction or zooming.

[0159] For example, in the case where only a vertical shift is generated, m11 = m22 = 1, m12 = m13 = 0. In this case, the equation of the distortion correction becomes the following equation (5). Figure 15

[0160] (xsrc, ysrc) = (xlat, ylat + m23)... (5)

[0161] In the case where only a vertical shift is generated, m11 = m22 = 1, m12 = m13 = 0. In this case, the equation of the distortion correction becomes the following equation (5). Figure 13 In the case where only a vertical shift is generated, m11 = m22 = 1, m12 = m13 = 0. In this case, the equation of the distortion correction becomes the following equation (5). -1 The inverse conversion f

[0162] Figure 16 is a diagram illustrating a method of calculating a delay time compensation parameter from tracking information.

[0163] According to the geometric positional relationship of the rotation center of yaw, roll, and pitch, the head of the observer, the HUD screen, and the real object, a displacement generated in the mobile body, the observer, or the real object can be converted into a shift amount on the screen or a rotation angle. Then, a matrix on the right side of the equation of the distortion correction can be determined from the shift amount or the rotation angle. Figure 15

[0164] In the case where only a vertical shift is generated, m11 = m22 = 1, m12 = m13 = 0. In this case, the equation of the distortion correction becomes the following equation (5). Figure 16 In the case where only a vertical shift is generated, m11 = m22 = 1, m12 = m13 = 0. In this case, the equation of the distortion correction becomes the following equation (5). Figure 15 In the case where only a vertical shift is generated, m11 = m22 = 1, m12 = m13 = 0. In this case, the equation of the distortion correction becomes the following equation (5).

[0165] In the case where only a vertical shift is generated, m11 = m22 = 1, m12 = m13 = 0. In this case, the equation of the distortion correction becomes the following equation (5).

[0166] ​​Let the distance between the observer 52's head and the real object 12 be DPT, and the distance between the screen 34 and the real object 12 be DFT. Assuming that the screen 34 and the real object 12 are stationary, for example, when the observer's head is raised by +Δyp, the position of the real object 12 on the screen 34 as seen from the observer 52 appears to be raised relative to the screen 34 by +(DFT / DPT)×Δyp. If we assume the real object 12 is far enough, we can approximate it as DFT / DPT = 1; therefore, the vertical displacement of the real object 12 is Δyp. Thus, the delay time compensation parameter in this case is m23 = Δyp.

[0167] In the embodiment described above, the display system 100 includes a storage unit 145, which stores a table as distortion parameters that corresponds to coordinates in the display image and the amount of movement representing the destination of the movement at those coordinates. The parameter calculation unit 140 shifts the amount of movement in the table according to the delay time compensation parameter and outputs the table with the shifted amount of movement as the first distortion parameter PRM1.

[0168] exist Figure 13 In the example, f -1 () is stored as a table in storage unit 145. Figure 15 In the example described in equation (5) above, the amount of movement of the table that causes (xdst, ydst) to move toward (xlat, ylat) is, for example, a vertical shift of m23. That is, the table is corrected to cause (xdst, ydst) to move toward (xlat, ylat+m23).

[0169] Alternatively, the storage unit 145 may store parameters of the transformation formula that converts the coordinates in the display image to the coordinates of the moving destination as distortion parameters. The parameter calculation unit 140 shifts the coordinates of the moving destination according to the delay time compensation parameter and outputs the parameters of the transformation formula after the coordinates of the moving destination have been shifted as the first distortion parameter PRM1.

[0170] The transformation expression is, for example, a polynomial, whose coefficients are stored as parameters in storage unit 145. Figure 13 In the example, f represents -1 The parameters of the conversion formula () are stored in the storage unit 145. Figure 15 In the example described in equation (5) above, the shift of the transformation formula that moves (xdst, ydst) toward (xlat, ylat) is, for example, a shift of m23 in the vertical direction. That is, the transformation formula is corrected to move (xdst, ydst) toward (xlat, ylat+m23).

[0171] In addition, although the present embodiment has been described in detail as described above, a person skilled in the art can easily understand that various modifications can be made without departing from the new matters and effects of the present application. Therefore, such modified examples are all included in the scope of the present application. For example, in the specification or the drawings, a term recorded at least once together with a different term that is more general or synonymous can be replaced with the different term in any part of the specification or the drawings. Furthermore, all combinations of the present embodiment and the modified examples are also included in the scope of the present application. Furthermore, the structure and the operation of the sensor, the head-up display, the processing device, the circuit device, the display system, the electronic device, the mobile body, and the like are not limited to those described in the present embodiment, and various modified embodiments can be made.

Claims

1. A circuit device used in a head-up display, the head-up display displaying a first display object and a second display object corresponding to real objects in real space in a display area, characterized in that, The circuit device includes: a storage section that stores a warp target image including an image of the first display object and an image of the second display object; and a warp processing section that performs warp processing on the warp target image using a first warp parameter and a second warp parameter to generate a display image displayed in the display region, the first warp parameter is a parameter of coordinate conversion including both a vibration correction of a display position shift of the first display object with respect to the real object and a distortion correction corresponding to a curvature of the display region, the second warp parameter is a parameter of coordinate conversion of the distortion correction corresponding to the curvature of the display region, the warp processing section includes: a first correction section that outputs a first corrected image by performing the coordinate conversion of the entire warp target image using the first warp parameter while performing both the vibration correction and the distortion correction; a second correction section that outputs a second corrected image by performing the coordinate conversion of the entire warp target image using the second warp parameter while performing the distortion correction; and a combining section that combines the first corrected image and the second corrected image to output the display image.

2. The circuit device according to claim 1, wherein the first warp parameter is a parameter obtained by correcting a warp parameter for the distortion correction according to a delay time compensation parameter that compensates for a delay time including a rendering processing delay time of the warp target image, the second warp parameter is the warp parameter for the distortion correction.

3. The circuit device according to claim 1, wherein the combining section outputs the first corrected image as the display image for a first region in which the first display object is displayed, the combining section outputs the second corrected image as the display image for a second region in which the second display object is displayed.

4. The circuit device according to claim 1, wherein the combining section performs alpha blending of the first corrected image and the second corrected image to output the display image.

5. The circuit device according to any one of claims 1 to 4, wherein the storage section is a line buffer. The display system includes:

6. A display system that is a display system of a head-up display that displays a first display object corresponding to a real object of a real space and a second display object in a display region, characterized by a storage section that stores a warp target image including an image of the first display object and an image of the second display object; and a warp processing section that performs warp processing on the warp target image using a first warp parameter and a second warp parameter to generate a display image displayed in the display region, the first warp parameter is a parameter of coordinate conversion including both a vibration correction of a display position shift of the first display object with respect to the real object and a distortion correction corresponding to a curvature of the display region, the second warp parameter is a parameter of coordinate conversion of the distortion correction corresponding to the curvature of the display region, the warp processing section includes: a first correction section that outputs a first corrected image by performing the coordinate conversion of the entire warp target image using the first warp parameter while performing both the vibration correction and the distortion correction; a second correction section that outputs a second corrected image by performing the coordinate conversion of the entire warp target image using the second warp parameter while performing the distortion correction; and a combining section that combines the first corrected image and the second corrected image to output the display image. a first correction section that outputs a first corrected image by performing the coordinate conversion on the entire of the warping target image using the first warping parameter while performing the vibration correction and the distortion correction; a second correction section that outputs a second corrected image by performing the coordinate conversion on the entire of the warping target image using the second warping parameter while performing the distortion correction; and a synthesis section that synthesizes the first corrected image and the second corrected image to output the display image.

7. The display system according to claim 6, wherein the display system includes a parameter calculation section that calculates the first warping parameter and the second warping parameter, the parameter calculation section calculates the first warping parameter based on tracking information that is at least one of first tracking information of a moving body on which the head-up display is mounted, second tracking information of an observer of the head-up display, and third tracking information of the real object.

8. The display system according to claim 7, wherein the display system includes a rendering image generation section that renders the warping target image, the parameter calculation section calculates a delay time compensation parameter that compensates for a delay time including a rendering processing delay time of the warping target image based on the tracking information, the first warping parameter is calculated by correcting a warping parameter for the distortion correction based on the delay time compensation parameter, the warping parameter for the distortion correction is output as the second warping parameter.

9. An image processing method that is an image processing method in a head-up display that displays a first display object corresponding to a real object of a real space and a second display object in a display region, characterized by performing warping processing on a warping target image that includes an image of the first display object and an image of the second display object using a first warping parameter and a second warping parameter to generate a display image to be displayed in the display region, the first warping parameter being a parameter of coordinate conversion that includes both a vibration correction of a display position shift of the first display object with respect to the real object and a distortion correction corresponding to a curvature of the display region, and the second warping parameter being a parameter of coordinate conversion of the distortion correction corresponding to the curvature of the display region, in the warping processing, outputting a first corrected image by performing the coordinate conversion on the entire of the warping target image using the first warping parameter while performing the vibration correction and the distortion correction, outputting a second corrected image by performing the coordinate conversion on the entire of the warping target image using the second warping parameter while performing the distortion correction, synthesizing the first corrected image and the second corrected image to output the display image.

10. The image processing method according to claim 9, wherein ​ The first warping parameter is a parameter obtained by correcting a warping parameter for the distortion correction according to a delay time compensation parameter that compensates for a delay time including a rendering processing delay time of the warped object image, The second warping parameter is the warping parameter for the distortion correction.

Images