CIRCUIT DEVICE, DISPLAY SYSTEM, AND IMAGE PROCESSING METHOD

Abstract

To provide a circuit arrangement that, in a HUD in which AR display and fixed display are mixed, can accurately correct deviation of the display position of the AR display.SOLUTION: A circuit arrangement 160 includes a storage unit 161 and a warp processing unit 130. The storage unit 161 stores a warp target image WTIM. The warp processing unit 130 performs warp processing on the warp target image WTIM and creates a display image DSIM to be displayed in a display area 40. The warp processing unit 130 performs the warp processing on an image IM20 of a first display object in the warp target image WTIM based on a first warp parameter PRM1 for performing shake correction and distortion correction. The warp processing unit 130 performs the warp processing on an image IM25 of a second display object in the warp target image WTIM based on a second warp parameter PRM2 for performing the distortion correction.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a circuit device, a display system, an image processing method, and the like. [Background technology]

[0002] A head-up display is known that displays information superimposed on the user's field of vision by displaying an image on a transparent screen or the like. By combining such a head-up display with AR technology, it is possible to display a virtual object on the head-up display so that the virtual object follows a real object such as a car ahead or a road. AR is an abbreviation for Augmented Reality.

[0003] While vibration correction is performed to make the virtual object of AR follow the real object, there are cases where it is desired to display a meter display, etc., which does not need to follow the real object, fixedly. In addition, in a head-up display, since an image is distorted due to a curved surface such as a transparent screen, distortion correction is performed to correct the distortion. Patent Document 1 discloses a technology in which an image requiring vibration correction and an image not requiring vibration correction are synthesized on a frame memory, and the synthesized image is subjected to distortion correction. Patent Document 2 discloses a technology in which position correction and distortion correction are performed separately, and in the distortion correction, the parameters of the distortion correction are changed for each image area. Patent Document 3 discloses a technology in which a plurality of images including a first image with position correction and a second image without position correction are synthesized, but the first image and the second image are separate images. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2020-050328 A [Patent Document 2] JP 2009-179214 A [Patent Document 3] JP 2017-94882 A Summary of the Invention [Problem to be solved by the invention]

[0005] From the viewpoint of accurately correcting the positional deviation between the virtual object and the real object in AR, it is desirable that the timing of vibration correction is close to the timing of display on the head-up display. However, in Patent Documents 1 and 2, since distortion correction is performed after vibration correction or position correction, the processing time from vibration correction or position correction to output to the head-up display is long. In Patent Document 3, since a plurality of different images are corrected and then synthesized, the processing time from position correction to output to the head-up display is long. As described above, in the above conventional technology, in a HUD in which AR display and fixed display are mixed, the processing time from vibration correction or position correction to output to the head-up display is long, and there is a problem that the correction of position deviation is insufficient. [Means for solving the problem]

[0006] One aspect of the present disclosure relates to a circuit device used in a head-up display that displays a first display object corresponding to a real object in a real space and a second display object in a display area, the circuit device including: a memory unit that stores a warp target image including an image of the first display object and an image of the second display object; and a warp processing unit that performs warp processing on the warp target image to generate a display image to be displayed in the display area, wherein the warp processing unit performs the warp processing on an image of the first display object among the warp target images, based on first warp parameters that perform vibration correction for a display position deviation of the first display object relative to the real object and distortion correction in accordance with a curvature of the display area, and performs the warp processing on an image of the second display object among the warp target images, based on second warp parameters that perform the distortion correction.

[0007] Another aspect of the present disclosure relates to a display system for a head-up display that displays a first display object corresponding to a real object in a real space and a second display object in a display area, the display system including: a storage unit that stores a warp target image including an image of the first display object and an image of the second display object; and a warp processing unit that performs warp processing on the warp target image to generate a display image to be displayed in the display area, wherein the warp processing unit performs the warp processing on an image of the first display object among the warp target images, based on first warp parameters that perform vibration correction for a display position deviation of the first display object relative to the real object and distortion correction in accordance with a curvature of the display area, and performs the warp processing on an image of the second display object among the warp target images, based on second warp parameters that perform the distortion correction.

[0008] Yet another aspect of the present disclosure relates to an image processing method for a head-up display that displays a first display object corresponding to a real object in a real space and a second display object in a display area, the image processing method comprising: performing warp processing on a warp target image including an image of the first display object and an image of the second display object to generate a display image to be displayed in the display area; performing the warp processing on an image of the first display object among the warp target images based on first warp parameters that perform vibration correction for a display position deviation of the first display object relative to the real object and distortion correction in accordance with a curvature of the display area; and performing the warp processing on an image of the second display object among the warp target images based on second warp parameters that perform the distortion correction. [Brief description of the drawings]

[0009] [Figure 1] 1 shows a first configuration example of a circuit device. [Diagram 2] FIG. 4 is an explanatory diagram of a process performed by a first configuration example of a circuit device. [Diagram 3] 2 shows a second configuration example of a circuit device. [Figure 4] FIG. 11 is an explanatory diagram of a process performed by a second configuration example of a circuit device. [Diagram 5] 5A and 5B are explanatory diagrams for explaining a display position shift caused by a pitch displacement or a vertical displacement. [Figure 6] FIG. 1 is a diagram illustrating latency compensation. [Figure 7] 1 shows an example of the configuration of a display system according to the prior art. [Figure 8] 3 is a process flow of a display system according to the prior art. [Figure 9] 1 shows an example of the configuration of a display system according to the present embodiment. [Figure 10] 4 is a detailed processing flow of a tracking processing unit. [Figure 11] 1 shows the processing flow of the display system. [Figure 12] 1 shows an example of a detailed configuration of a display system. [Figure 13] FIG. [Figure 14] 1 is a diagram showing the correspondence between tracking information of a moving object or the like and latency compensation based on the tracking information; [Figure 15] Example of latency compensation parameters. [Figure 16] 1A and 1B are diagrams illustrating a method for calculating latency compensation parameters from tracking information. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] A preferred embodiment of the present disclosure will be described in detail below. Note that the embodiment described below does not unduly limit the contents described in the claims, and all of the configurations described in the embodiment are not necessarily essential configurations.

[0011] 1. First configuration example 1 shows a first configuration example of a circuit device 160 used in a HUD. HUD is an abbreviation for head-up display. The circuit device 160 includes a storage unit 161, a warp processing unit 130, and a warp parameter selection unit 162. The circuit device 160 is also called a HUD controller, and is formed, for example, by an integrated circuit device in which circuits are integrated on a semiconductor substrate.

[0012] The storage unit 161 temporarily stores the warp target image WTIM input to the circuit device 160. Specifically, the storage unit 161 is a line buffer that stores image data of the number of lines required for warp processing. The number of lines may be approximately the maximum number of lines of vertical movement in the warp processing plus a margin. The storage unit 161 is, for example, a semiconductor memory such as a RAM. RAM is an abbreviation for Random Access Memory.

[0013] The warp processing unit 130 performs warp processing on the warp target image WTIM to generate a display image DSIM, and outputs the display image DSIM to the HUD 400. The warp processing is a coordinate conversion between the warp target image WTIM and the display image DSIM, and includes distortion correction and vibration correction in this embodiment. As described later, the warp processing unit 130 performs distortion correction and vibration correction on an image area corresponding to AR display, and performs only distortion correction on an image area corresponding to fixed display. AR is an abbreviation for augmented reality. The distortion correction and vibration correction are performed collectively by the coordinate conversion of the warp processing. That is, the warp parameters indicating the coordinate conversion are generated from the distortion correction parameters and the vibration correction parameters, and the coordinate conversion is performed using the warp parameters, so that the distortion correction and the vibration correction are performed simultaneously.

[0014] The warp processing unit 130 may be either a forward warp engine or an inverse warp engine. Forward warp is a transformation that moves each pixel of an input image to a warp engine to an arbitrary position in an output image. A forward warp engine is a warp engine that has a forward warp function. An inverse warp is a transformation that determines each pixel of an output image of a warp engine from a pixel at an arbitrary position in an input image. An inverse warp engine is a warp engine that has an inverse warp function.

[0015] The warp parameter selection unit 162 selects the first warp parameter PRM1 when an image region corresponding to AR display is warped, and selects the second warp parameter PRM2 when an image region corresponding to fixed display is warped. The selected parameter is output to the warp processing unit 130 as the warp parameter PRMQ. The first warp parameter PRM1 is a warp parameter that performs both distortion correction and vibration correction. The second warp parameter PRM2 is a warp parameter that performs distortion correction but does not perform vibration correction. The warp processing unit 130 performs warp processing using the warp parameter PRMQ output by the warp parameter selection unit 162, thereby performing distortion correction and vibration correction for the image region corresponding to AR display, and performing only distortion correction for the image region corresponding to fixed display. The first warp parameter PRM1 and the second warp parameter PRM2 are input from outside the circuit device 160. Alternatively, the circuit device 160 may include a parameter calculation unit that generates the first warp parameter PRM1 and the second warp parameter PRM2.

[0016] Fig. 2 shows an explanatory diagram of the process performed by the first configuration example of the circuit device 160. Fig. 2 mainly describes an example in which a HUD is mounted on an automobile 30, but the HUD of this embodiment may be mounted on various moving objects such as an airplane, a ship, or a motorcycle.

[0017] As shown in the left diagram of Fig. 2, the warp target image includes an image IM20 of a first display object and an image IM25 of a second display object. Images IM20 and IM25 refer to parts of the warp target image, and do not refer to images separate from the warp target image.

[0018] Here, the first display object refers to a display object in AR display, that is, a virtual object displayed on the HUD following a real object in the real space. In the right diagram of Fig. 2, a vehicle ahead 10 is a real object, and a first display object 20 is displayed following the vehicle ahead 10. An image IM20 of the first display object is an image for displaying the first display object 20 on the HUD. In the left diagram of Fig. 2, the hatched portion of the figure is the image IM20.

[0019] The second display object is a display object that does not follow a real object in the HUD, and is a display object whose display position is fixed in the display image DSIM output from the circuit device 160 to the HUD 400. When the positional relationship between the viewer looking at the HUD and the HUD does not change, the display position of the second display object is fixed in the field of view of the viewer. In the right diagram of FIG. 2, the characters "100km / h" that are fixedly displayed are the second display object 25. The image IM25 of the second display object is an image for displaying the second display object 25 on the HUD. In the left diagram of FIG. 2, the part of the characters "100km / h" is the image IM25.

[0020] The warp target image corresponds to an input image in a warp process that performs coordinate conversion between an input image and an output image. Specifically, the warp target image is an image input from the storage unit 161 to the warp processing unit 130, and is an image including the image IM20 of the first display object and the image IM25 of the second display object described above. For example, a rendering process for AR display is performed in a processing device outside the circuit device 160, and the rendering image is input to the circuit device 160. At least the image IM20 of the first display object is rendered in the rendering image. For example, the image IM25 of the second display object is also rendered in the rendering process, and the rendering image is stored in the storage unit 161 as the warp target image. Alternatively, the circuit device 160 may include an overlay processing unit (not shown), and the overlay processing unit may overlay the image IM25 of the second display object on the rendering image and store it in the storage unit 161 as the warp target image.

[0021] Note that the warp target image is not limited to a rendering image. For example, a plurality of images in which the image IM20 of the first display object is arranged at various display positions may be prepared in advance, and an image selected from the plurality of images may be input to the circuit device 160 as the warp target image. In this case, an image corresponding to the position where the first display object should be displayed in the AR display is selected from the plurality of images.

[0022] 2, the warp processing unit 130 performs distortion correction and vibration correction on a first region AR1 of the warp target image, and performs distortion correction but not vibration correction on a second region AR2 of the warp target image. As a result, the display image is given a distortion that cancels the distortion occurring in the HUD 400. The display image also includes an image IM20' of the first display object that has been corrected for distortion and vibration, and an image IM25' of the second display object that has been corrected for distortion.

[0023] Specifically, when warping the first region AR1, the warp parameter selection unit 162 selects the first warp parameter PRM1, and when warping the second region AR2, the warp parameter selection unit 162 selects the second warp parameter PRM2. This realizes the correction for each region. The warp parameter selection unit 162 can determine whether the first region AR1 or the second region AR2 is being warped by the warp processing unit 130. In forward warping, it is determined whether a pixel of the warp target image on the input side belongs to the first region AR1 or the second region AR2. In inverse warping, it is determined whether a pixel of the display image on the output side corresponds to the first region AR1 or the second region AR2. The warp target image and the display image are associated by distortion correction, and the region can be determined from the correspondence.

[0024] Here, the first area AR1 is an area in the warp target image that includes the image IM20 of the first display object and does not include the image IM25 of the second display object. Specifically, the first area AR1 is an area in the warp target image other than the second area AR2. Note that, when the warp target image includes an image of a third display object and the third display object is an AR display, the first area AR1 is set so as to include the image IM20 of the first display object and the image of the third display object.

[0025] The second area AR2 is an area including the image IM25 of the second display object in the warp target image. Specifically, the second area AR2 is an area including the entire image IM25 of the second display object. Note that, when the warp target image includes an image of a third display object and the third display object is fixedly displayed, the second area AR2 is set so as to include the image IM25 of the second display object and the image of the third display object.

[0026] The first area AR1 and the second area AR2 may be defined in the display image. In this case, the first area AR1 includes an image IM20' of the first display object that has been corrected for distortion and vibration, and the second area AR2 includes an image IM25' of the second display object that has been corrected for distortion. Since the warp target image and the display image are associated with each other by distortion correction as described above, the second area AR2 can be set so as to include the image IM25' of the second display object, and the area other than the second area AR2 can be set as the first area AR1.

[0027] Distortion correction is a process of converting the coordinates of the warp target image according to the curvature of the display area. This coordinate conversion is a coordinate conversion that cancels the image distortion caused by the curvature of the screen. The display image is an image displayed on the display panel of the HUD 400, and the display image displayed on the display panel is projected onto the screen by the projection unit, so that it is displayed in the viewer's field of vision as an undistorted virtual image.

[0028] Vibration correction is a process for correcting a display position shift of a first display object that is displayed following a real object in AR display. This display position shift occurs due to a change in the position or attitude of a moving body on which the HUD is mounted, and vibration correction is performed according to the change in the position or attitude of the moving body. In the right diagram of Fig. 2, an automobile 30 is a moving body, and a position shift occurs between a leading vehicle 10 and a first display object 20 due to a change in the position or attitude of the automobile 30.

[0029] The vibration correction includes shift correction. The vibration correction may further include rotation correction, scale correction, or both. The shift is a horizontal or vertical positional shift of the first display object 20 with respect to the vehicle in front 10. The shift correction is a correction for translating the first area AR1 of the warp target image in the horizontal or vertical direction to cancel the shift. The rotation is a clockwise or counterclockwise rotation of the first display object 20 with respect to the vehicle in front 10. The rotation correction is a correction for rotating the first area AR1 of the warp target image in the clockwise or counterclockwise direction to cancel the rotation. The scale is an enlargement or reduction of the size of the first display object 20 with respect to the vehicle in front 10. The scale correction is a correction for enlarging or reducing the first area AR1 of the warp target image to correct the scale.

[0030] The vibration correction is, for example, latency compensation, which will be described later. The latency compensation is to compensate for a display position shift that occurs from the timing of rendering the rendering image to the timing of performing the warp process. Specifically, the difference between the display position of the first display object determined or predicted from the tracking information at the timing of rendering and the display position of the first display object determined from the tracking information at the timing of the warp process is compensated. The tracking information is acquired from a sensor, the details of which will be described later. Note that the vibration correction is not limited to latency compensation, and may be any process that corrects a change in the position or attitude of a moving object that cannot be followed by rendering. That is, instead of using the difference, for example, a low-frequency component of the tracking information may be used in rendering, and a high-frequency component of the tracking information may be used in the vibration correction of the warp process.

[0031] As shown in the center and right diagrams of FIG. 2, the warp processing unit 130 outputs the display image to the HUD 400. The HUD 400 projects the display image onto a screen, and the viewer looks at the screen, so that the display image appears to be superimposed on real space as a virtual image. The right diagram of FIG. 2 shows an AR display seen by the viewer, the driver, through the HUD 400. Through the windshield, the driver sees the vehicle in front 10, the road on which the vehicle in front 10 is running, and the scenery around them. The driver also sees a virtual image projected by the HUD 400 into a display area 40 on the windshield. The display area 40 indicates the range in which the HUD 400 can project a virtual image.

[0032] This virtual image includes the first display object 20 and the second display object 25. Specifically, in the display image in the middle diagram of FIG. 2, the parts other than the image IM20' of the first display object and the image IM25' of the second display object are black and transparently displayed on the HUD 400. Among the display images projected by the HUD 400, the first display object 20 is projected by the image IM20' of the first display object, and the second display object 25 is projected by the image IM25' of the second display object. In the virtual image, the parts other than the display objects are transparent, and only the display objects appear to be superimposed on the real space from the viewer. Here, the entire virtual image generated by the HUD 400 is called a "virtual image", and the part of the virtual image that is not transparent and is recognized by the viewer is called a "display object". Note that the background of the display object does not necessarily need to be transparent, and in that case, a specific part of the virtual image may be called a display object.

[0033] Among the virtual images projected by the HUD 400, the first display object 20 is vibration-compensated, and therefore tracks the vehicle 10 in front more accurately than in the case where vibration compensation is not performed. Moreover, the second display object 25 is not vibration-compensated, and therefore is displayed fixedly in the viewer's field of vision, and is easy to view without being affected by changes in the position or attitude of the automobile 30.

[0034] In the embodiment described above, the circuit device 160 is used in the HUD 400 that displays the first display object 20 corresponding to a real object in the real space and the second display object 25 in the display area 40. The circuit device 160 includes a storage unit 161 and a warp processing unit 130. The storage unit 161 stores a warp target image WTIM including an image IM20 of the first display object and an image IM25 of the second display object. The warp processing unit 130 performs warp processing on the warp target image WTIM to generate a display image DSIM to be displayed in the display area 40. The warp processing unit 130 performs warp processing on the image IM20 of the first display object, out of the warp target image WTIM, based on a first warp parameter PRM1. The first warp parameter PRM1 is a warp parameter that performs vibration correction of the display position deviation of the first display object 20 with respect to the real object and distortion correction according to the curvature of the display area 40. Furthermore, the warp processing unit 130 performs warping on the image IM25 of the second display object, out of the warp target image WTIM, based on the second warp parameters PRM2. The second warp parameters PRM2 are warp parameters that perform distortion correction.

[0035] In this way, vibration correction is performed on the first display object 20 that is made to follow the real object, and vibration correction is not performed on the second display object 25 that is not made to follow the real object. This allows AR display to be realized, and display objects that are not targets of AR display can be fixedly displayed in the field of view of the HUD viewer. Furthermore, according to this embodiment, since vibration correction is performed together with distortion correction in the warp process, vibration correction is performed at the timing of the warp process that is close to the timing of the HUD display. This allows the display position deviation between the real object and the first display object 20 to be corrected more accurately than in the conventional techniques in which the processing time from vibration correction to display is long as in the above-mentioned Patent Documents 1 to 3.

[0036] Moreover, in this embodiment, the circuit device 160 includes a warp parameter selection unit 162. When the warp processing unit 130 performs warp processing on the first area AR1 in which the first display object 20 is displayed, the warp parameter selection unit 162 outputs the first warp parameters PRM1 to the warp processing unit 130. Moreover, when the warp processing unit 130 performs warp processing on the second area AR2 in which the second display object 25 is displayed, the warp parameter selection unit 162 outputs the second warp parameters PRM2 to the warp processing unit 130.

[0037] In this way, warp processing using the first warp parameters PRM1 is performed on the first area AR1, so that distortion correction and vibration correction are performed on the image IM20 of the first display object. Also, warp processing using the second warp parameters PRM2 is performed on the second area AR2, so that distortion correction is performed on the image IM25 of the second display object, but vibration correction is not performed.

[0038] In this embodiment, the storage unit 161 is a line buffer.

[0039] The warping process is performed, for example, for each line or for each set of multiple lines. Therefore, the storage unit 161 only needs to store image data for the number of lines required for warping one line or multiple lines of the warp target image. Therefore, the storage unit 161 may be a line buffer rather than a frame memory.

[0040] The operation of the circuit device 160 may be executed as an image processing method for a HUD as follows. In this case, the subject of each step is not limited to the circuit device 160. For example, the subject of each step may be a display system of a HUD, which will be described later.

[0041] The image processing method is an image processing method for a head-up display that displays a first display object 20 corresponding to a real object in a real space and a second display object 25 in a display area 40. The image processing method performs warping on a warp target image WTIM including an image IM20 of the first display object and an image IM25 of the second display object, and generates a display image DSIM to be displayed in the display area 40. The image processing method performs warping on the image IM20 of the first display object, among the warp target images WTIM, based on a first warp parameter PRM1 that performs vibration correction of a display position deviation of the first display object 20 with respect to the real object and distortion correction according to the curvature of the display area 40. The image processing method performs warping on the image IM25 of the second display object, among the warp target images WTIM, based on a second warp parameter PRM2 that performs distortion correction.

[0042] 2. Second configuration example 3 shows a second configuration example of the circuit device 160. The circuit device 160 includes a storage unit 161, a warp processing unit 130, and a layer selection unit 163. The warp processing unit 130 includes a first correction unit 131, a second correction unit 132, and a synthesis unit 133. Note that descriptions of configurations and operations similar to those in the first configuration example will be omitted as appropriate.

[0043] 4 shows an explanatory diagram of the processing performed by the second configuration example of the circuit device 160. Note that the operation after the display image is output to the HUD is the same as in the first configuration example.

[0044] As shown in the left and center diagrams of Fig. 4, the first correction unit 131 performs warp processing using the first warp parameters PRM1 on the warp target image WTIM to generate a first corrected image CIM1. This warp processing is performed on the entire warp target image WTIM. In other words, the first corrected image CIM1 is an image in which the entire warp target image WTIM has been corrected for distortion and vibration. The first corrected image CIM1 includes an image IM20a of the first display object that has been corrected for distortion and vibration, and an image IM25a of the second display object that has been corrected for distortion and vibration.

[0045] The second correction unit 132 performs warping using the second warp parameters PRM2 on the warp target image WTIM to generate a second corrected image CIM2. This warping is performed on the entire warp target image WTIM. That is, the second corrected image CIM2 is an image in which the entire warp target image WTIM has been distortion-corrected. The second corrected image CIM2 includes a distortion-corrected image IM20b of the first display object and a distortion-corrected image IM25b of the second display object.

[0046] The first compensation unit 131 and the second compensation unit 132 are separate warp engines. That is, the warp processing of the first compensation unit 131 and the warp processing of the second compensation unit 132 are processed in parallel by each warp engine. The warp engine may be either a forward warp engine or an inverse warp engine.

[0047] As shown in the middle and right diagrams of FIG. 4, the synthesis unit 133 generates a display image DSIM by synthesizing the first corrected image CIM1 and the second corrected image CIM2. Here, the first area AR1 and the second area AR2 are defined in the display image DSIM. The synthesis unit 133 selects the first corrected image CIM1 in the first area AR1, and selects the second corrected image CIM2 in the second area AR2. As a result, as shown in the right diagram of FIG. 4, a display image DSIM including an image IM20a of the first display object that has been corrected for distortion and vibration, and an image IM25b of the second display object that has been corrected for distortion but not vibration, is output. Information specifying the first area AR1 and the second area AR2 is input from the layer selection unit 163 to the synthesis unit 133, and the synthesis unit 133 makes the above selection based on the information.

[0048] Alternatively, the synthesis unit 133 may perform synthesis by α blending. That is, the synthesis unit 133 blends the pixel values ​​of each pixel in the first region AR1 by setting the blending ratio of the first corrected image CIM1 to α1 and the blending ratio of the second corrected image CIM2 to 1-α1 in the first region AR1. Also, the synthesis unit 133 blends the pixel values ​​of each pixel in the second region AR2 by setting the blending ratio of the first corrected image CIM1 to 1-α2 and the blending ratio of the second corrected image CIM2 to α2 in the second region AR2. For example, if α1=α2=1, the synthesis is the same as the synthesis by the above-mentioned selection. Information specifying the first region AR1 and the second region AR2 is input from the layer selection unit 163 to the synthesis unit 133, and the synthesis unit 133 performs the above-mentioned α blending based on the information.

[0049] Note that the "composite" here does not mean a combination that has a large delay, such as one in which an entire frame is combined on a frame memory at once. In the above, selection and alpha blending are given as an example of combination. More specifically, the first correction unit 131 and the second correction unit 132 perform warping processing on one line at a time, and output the results on one line at a time. At this time, the composition unit 133 selects or alpha blends one line at a time, thereby outputting the display image on one line at a time. Note that this processing may be performed for multiple lines, but the number of lines is less than the number of lines in one frame.

[0050] In the embodiment described above, the warp processing unit 130 includes a first correction unit 131, a second correction unit 132, and a synthesis unit 133. The first correction unit 131 performs warping on the warp target image WTIM based on the first warp parameters PRM1, and outputs a first corrected image CIM1. The second correction unit 132 performs warping on the warp target image WTIM based on the second warp parameters PRM2, and outputs a second corrected image CIM2. The synthesis unit 133 synthesizes the first corrected image CIM1 and the second corrected image CIM2, and outputs a display image DSIM.

[0051] In this way, it becomes possible to perform parallel processing of the distortion correction and vibration correction by the first correction unit 131 and the distortion correction by the second correction unit 132. Then, the synthesis unit 133 synthesizes the first corrected image CIM1 obtained by the distortion correction and vibration correction with the second corrected image CIM2 obtained by the distortion correction, thereby generating a display image DSIM in which vibration-corrected areas and non-vibration-corrected areas are mixed.

[0052] In this embodiment, the synthesis unit 133 outputs the first corrected image CIM1 as the display image DSIM for the first region AR1 in which the first display object 20 is displayed. Moreover, the synthesis unit 133 outputs the second corrected image CIM2 as the display image DSIM for the second region AR2 in which the second display object 25 is displayed.

[0053] In this way, the first area AR1 that has been corrected for distortion and vibration and the second area AR2 that has been corrected for distortion but not vibration are synthesized and output as a display image DSIM. As a result, the vibration of the first display object 20 displayed in AR is corrected, and the vibration of the second display object 25 that is displayed in a fixed manner is not corrected. In addition, since synthesis is performed by selecting an area, the processing time from vibration correction to HUD display is shorter than that of conventional techniques such as synthesis on a frame-by-frame basis on a frame memory. This makes it possible to reduce display position deviation.

[0054] In this embodiment, the synthesis unit 133 performs alpha blending on the first corrected image CIM1 and the second corrected image CIM2 to output the display image DSIM.

[0055] In this way, the first corrected image CIM1 that has been corrected for distortion and vibration and the second corrected image CIM2 that has been corrected for distortion but not vibration are alpha-blended. This generates a display image DSIM that includes the first display object 20 that has been corrected for distortion and vibration and the second display object 25 that has been corrected for distortion but not vibration. In addition, since the synthesis is performed by alpha-blending, the processing time from vibration correction to HUD display is shorter than in conventional techniques such as synthesis on a frame-by-frame basis on a frame memory. This makes it possible to reduce display position deviation.

[0056] 3. Display System A display system including the circuit device 160 will be described below. Here, a case where the vibration correction is latency compensation will be described as an example, but as described above, the vibration correction is not limited to latency compensation. In addition, the first display object that is the target of the vibration correction will be mainly described, but as described above, the second display object that is fixedly displayed is also displayed on the HUD. Hereinafter, the first display object will be referred to as a virtual object. As described above, a virtual object is an object that is displayed on the HUD by following a real object in the real space.

[0057] As shown in the right diagram of Fig. 2, the display system of the HUD mounted on the automobile 30 uses a sensor such as Lidar to track the vehicle 10 ahead, renders a virtual object 20 based on the tracking result, warps the rendered image, and displays it on the HUD. At this time, there is a delay due to tracking processing, rendering processing, warping processing, data communication, etc. between the time when the sensor samples and the time when the virtual object 20 is displayed on the HUD. The delay caused by such processing or communication is called latency.

[0058] Latency causes a positional shift between the virtual object 20 and the vehicle in front 10, etc. In other words, a shift between the position of the vehicle in front 10, etc. when the display system renders the virtual object 20 and the position of the vehicle in front 10, etc. when the virtual object 20 is actually displayed on the HUD causes the virtual object 20 and the vehicle in front 10, etc. to be displayed out of alignment. For example, even if the position of the vehicle in front 10, etc. at a future display timing is predicted in the rendering process, there is a time lag between the prediction time and the display, so there is a possibility that the predicted position of the vehicle in front 10, etc. will be shifted from the position of the vehicle in front 10, etc. when the virtual object 20 is actually displayed on the HUD.

[0059] FIG. 5 shows an explanatory diagram of a display position shift due to pitch displacement or vertical displacement as an example. xyz is a coordinate system fixed to a car equipped with a HUD. The z direction is the traveling direction of the car, and the x and y directions are perpendicular to the z direction and perpendicular to each other. When the y direction coincides with the vertical direction, the x direction is the horizontal direction, and the y direction is the vertical direction or up-down direction. When the car is in a position where the y direction does not coincide with the vertical direction, the x direction and the y direction do not coincide with the horizontal and vertical directions, but for convenience, the x direction will be called the horizontal direction and the y direction will be called the vertical direction or up-down direction below. The arrow direction shown in the figure is the positive direction, and when it is desired to clearly indicate the positive and negative directions, they will be described as +x direction, -x direction, etc.

[0060] The first diagram in Fig. 5 shows a state where there is no positional deviation between the virtual object 21 and the real object 11. The real object 11 is an object that exists in real space and is an object that is a tracking target for AR display. For example, the car in front 10 in the right diagram in Fig. 2 is a real object. The projection unit 36 ​​of the HUD projects a virtual image onto the windshield 35, and the virtual object 21 in the virtual image is seen by the driver's eyes 50.

[0061] The second diagram in FIG. 5 shows the positional deviation between the virtual object 21 and the real object 11 when the windshield 35 and the projection unit 36 ​​shift by WSM in the +y direction. The vertical shift WSM occurs due to pitch rotation or vertical shift of the automobile 30. The pitch rotation is a rotation around an axis parallel to the x direction, and the vertical shift is a translation in the y direction. When the display system of the HUD renders the virtual object 21, the state of the first diagram is assumed, but since there is a time until it is displayed on the HUD, if the vertical shift WSM occurs during that time, the vertical shift WSM is not reflected in the rendered image. For this reason, the display position of the virtual object 21 is shifted in the +y direction by the vertical shift WSM of the windshield 35, etc., and the virtual object 21 appears to be shifted in the +y direction relative to the real object 11. For example, when the automobile 30 repeats pitch motion due to unevenness of the road, the virtual object 21 appears to be shifted relative to the real object 11 as if it is shaking up and down.

[0062] 5 shows the positional deviation between the virtual object 21 and the real object 11 when the driver's eyes 50 shift in the +y direction by the DEM. The windshield 35 does not move, but the eyes 50 shift in the +y direction relative to the windshield 35, so the display position of the virtual object 21 shifts in the -y direction. For this reason, the virtual object 21 appears to be shifted in the -y direction relative to the real object 11.

[0063] 5 shows the positional deviation between the virtual object 21 and the real object 11 when the real object 11 shifts by the RJM in the -y direction. The display position of the virtual object 21 does not shift because the windshield 35 and the eyes 50 do not move, but because the real object 11 shifts in the -y direction, the virtual object 21 appears to be shifted in the +y direction relative to the real object 11.

[0064] FIG. 6 is a diagram for explaining latency compensation in this embodiment. FIG. 6 shows a virtual object 21, a real object 11, and a virtual object 21 as seen by the driver. The left diagram in FIG. 6 shows a state in which the display positions of the real object 11 and the virtual object 21 match as seen by the driver. As shown in the middle diagram in FIG. 6, when the driver moves his head to the upper left, the real object 11 shifts to the upper left by MV. Since the display position of the virtual object 21 does not change, the real object 11 and the virtual object 21 shift by MV. As shown in the right diagram in FIG. 6, in this embodiment, the display positions of the real object 11 and the virtual object 21 are matched by compensating for the shift amount MV due to latency. By performing such latency compensation, when the driver moves his head to the upper left, the state transitions from the left diagram to the right diagram, and the position shift state as shown in the middle diagram can be avoided. Although details will be described later, in this embodiment, the position shift due to latency is reduced as much as possible by performing latency compensation in warp processing as close as possible to the display timing.

[0065] In order to consider what kind of latency exists in a HUD display system, a typical display system in the prior art and the latency that occurs in that display system will be described.

[0066] 7 shows an example of the configuration of a display system 90 in the prior art. The display system 90 includes a sensor 65, a processing device 60, a HUD controller 70, and a HUD 80.

[0067] The sensor 65 is a sensor that detects the position, attitude or motion of the vehicle, the driver or a real object.

[0068] The processing device 270 includes a sensor interface 61, a tracking processing unit 62, and a rendering image generation unit 63. The sensor interface 61 is an interface circuit that receives an output signal from a sensor 65. The tracking processing unit 62 is configured with a CPU, and tracks the position, attitude, or motion of the automobile, the driver, or a real object based on the output signal from the sensor 65. The rendering image generation unit 63 is configured with a GPU, and renders a virtual object so as to be displayed following the real object based on tracking information of the automobile, the driver, or the real object.

[0069] The HUD controller 70 includes a warp processing unit 71 that warps the rendering image. The HUD 80 includes a display 81 that displays the image after the warp processing. The display 81 is a display device such as a liquid crystal display device. The image displayed on the display 81 is projected onto the windshield by an optical system, thereby displaying the virtual object of the rendering image.

[0070] FIG. 8 shows a processing flow of the display system 90. Here, an example is shown in which an IMU and a camera are used as the sensor 65. The IMU performs sensing at a sampling rate that is sufficiently faster than the frame rate of the camera. Tck, Tck+1, and Tck+2 indicate the exposure timing of the camera. In other words, the image obtained by the camera indicates the position of the real object at the timing Tck, etc. Image signal processing is processing performed after the camera exposure ends, and includes signal reading from the image sensor, development processing, etc.

[0071] The tracking processing unit 62 tracks the position of the real object from the image after the image signal processing by visual inertial odometry. At this stage, the position of the real object at the camera exposure timing tck is detected. The tracking processing unit 62 tracks the position of the real object by inertial odometry based on the IMU output signal at timing tod after the visual inertial odometry ends and the result of the visual inertial odometry. This process compensates for the latency from the camera exposure timing tck to the inertial odometry timing tod, and the position of the real object at timing tod is detected.

[0072] The rendering image generation unit 63 renders a virtual object at a position corresponding to a real object detected by inertial odometry, and outputs the rendering image to the warp processing unit 71. At this time, a data communication time is generated from the GPU constituting the rendering image generation unit 63 to the HUD controller 70. The warp processing unit 71 performs warp processing on the rendering image, and the image after warp processing is displayed on the HUD 80. The first line to the last line are horizontal scanning lines of the display 81, and are scanned one line at a time. Therefore, the time required to scan the first line to the last line is required to display one frame of image, but here, the timing of displaying the first line is representatively referred to as display timing tdp.

[0073] The virtual object displayed on the HUD 80 corresponds to the position of the real object at the timing tod. Therefore, there is a latency between the timing tod and the display timing tdp, and a deviation between the real object and the virtual object may occur due to the motion of the car, the driver, or the real object during this period.

[0074] As described above, by displaying a virtual object rendered using tracking information detected before the display timing tdp, there is a problem that a display position shift occurs due to the latency between the timing at which the tracking information is detected and the display timing tdp. It is desirable to correct this display position shift as close as possible to the timing of the HUD display. As described above, in Patent Documents 1 to 3, the processing time from the correction of the position shift to the output to the head-up display becomes long, and there is a problem that the correction of the position shift becomes insufficient.

[0075] 9 shows an example of the configuration of a display system 100 in this embodiment. The display system 100 displays a virtual object corresponding to a real object on a HUD 400 based on an output signal from a sensor 500. The display system 100 includes a tracking processing unit 110, a rendering image generation unit 120, a warp processing unit 130, a parameter calculation unit 140, a storage unit 161, and a warp parameter selection unit 162.

[0076] Although circuit device 160 is not shown in Fig. 9, a circuit block included in circuit device 160 is included in display system 100. An example of a hardware configuration of display system 100 including circuit device 160 will be described later. Also, Fig. 9 shows an example in which the first configuration example of circuit device 160 in Fig. 1 is incorporated into display system 100, but when the second configuration example of circuit device 160 in Fig. 3 is incorporated into display system 100, a layer selection unit 163 is provided instead of warp parameter selection unit 162.

[0077] The sensor 500 is a sensor that detects the position, posture, or motion of a moving body, a viewer, or a real object. The sensor 500 is provided on a moving body, and includes, for example, Lidar, IMU, a camera, an eye tracking sensor, or a head tracking sensor. Lidar is an abbreviation for Light Detection and Ranging, and is a sensor that acquires three-dimensional information such as a z-map. IMU is an abbreviation for Inertial Measurement Unit, and is a sensor that detects motion on one axis or multiple axes. The IMU is, for example, composed of an acceleration sensor, a gyro sensor, or a combination thereof. The camera is a sensor that captures an image that is two-dimensional information. The eye tracking sensor is a sensor that detects the position of the viewer's eyes, the direction of the line of sight, or both. The head tracking sensor is a sensor that detects the position, posture, or both of the viewer's head.

[0078] The moving body is an object that moves in real space carrying the HUD 400, the viewer, and the sensor 500, such as an automobile, a motorcycle, an airplane, or a ship. The viewer is a user who views the virtual image projected on the HUD 400, and is the operator or passenger of the moving body. The real object is an object that exists in real space. The real object may be any object that changes the position or attitude of the real object in the HUD display area as seen by the viewer when the position or attitude of the moving body, the viewer, or the real object changes.

[0079] The tracking processing unit 110 tracks the position, posture, or motion of a moving body, a viewer, or a real object based on the output signal of the sensor 500, and outputs the result as tracking information. For example, the tracking processing unit 110 tracks a real object based on two-dimensional distance measurement information from a Lidar or a two-dimensional image from a camera. The tracking processing unit 110 also tracks a car based on acceleration or angular velocity information from an IMU. The tracking processing unit 110 also tracks the driver's eyes based on information on the eye position or gaze direction from an eye tracking sensor.

[0080] The tracking information may be in any form as long as it indicates the position, attitude, or motion of a moving body, a viewer, or a real object. For example, the tracking information may be coordinates indicating a position in real space, an angle indicating an attitude, a vector indicating a parallel movement, or an angular velocity indicating a rotation. Alternatively, the tracking information may be information obtained by converting the coordinates in the real space into coordinates, angles, vectors, or angular velocities on an image. The tracking information includes first tracking information of a moving body, second tracking information of a viewer, and third tracking information of a real object. However, it is sufficient that the tracking information includes at least one of the first to third tracking information, and for example, the second tracking information of the viewer may be omitted.

[0081] The rendering image generation unit 120 renders a virtual object based on tracking information of a moving body, a viewer, or a real object, and outputs a rendering image including the virtual object. Specifically, the rendering image generation unit 120 obtains a position where a real object is visible in the display area of ​​the HUD 400, and renders a virtual object at a position corresponding to the position of the real object. The rendering image generation unit 120 also renders a fixedly displayed second display object in addition to the virtual object that is the first display object.

[0082] The parameter calculation unit 140 obtains a latency compensation parameter based on the tracking information, corrects a warp parameter used for distortion correction based on the latency compensation parameter, and outputs a corrected warp parameter. This corrected warp parameter is a first warp parameter. The parameter calculation unit 140 outputs the warp parameter used for distortion correction as a second warp parameter. The tracking information used here is sampled later than the tracking information used for rendering processing. From the viewpoint of minimizing latency, it is desirable to use tracking information obtained immediately before or as close as possible to the parameter calculation.

[0083] The latency compensation parameter is a parameter that compensates for a difference between the position of a virtual object in a rendered image and the position of the virtual object at the display timing. The latency compensation parameter indicates a shift amount or a rotation angle on the image data. More specifically, the latency compensation parameter indicates a shift amount or a rotation angle in an undistorted image before warping processing.

[0084] The first warp parameter and the second warp parameter are parameters for coordinate conversion in the warp process, and are parameters for associating the coordinates of the input image with the coordinates of the output image in the warp process. The warp parameters are, for example, a matrix or a table that represents the coordinate conversion between the input image and the output image.

[0085] The warp processing unit 130 generates a display image by performing the warp processing explained in Fig. 1 to Fig. 4 on the rendering image. The display image is an image displayed on the display panel of the HUD 400, and the display image displayed on the display panel is projected onto a screen by the projection unit, so that a viewer looking at the screen sees a real space and a virtual image superimposed thereon.

[0086] The HUD 400 includes a projection unit for projecting a virtual image onto a screen. The projection unit includes a display panel that displays a display image, and an optical system that projects the display image displayed on the display panel onto a screen as a virtual image. The screen is a windscreen of a moving object or the like. Alternatively, the HUD 400 may include a dedicated screen onto which the virtual image is projected.

[0087] Fig. 10 shows a detailed processing flow of the tracking processing unit 110, and Fig. 11 shows a processing flow of the display system 100. Note that the tracking process shown in Fig. 10 is an example, and the method for tracking a real object or the like is not limited to this. In addition, the sensor used in the tracking process may be selected according to the method to be adopted.

[0088] The operation of the sensor shown in Fig. 11 is as described in Fig. 8. The tracking processing unit 110 performs visual inertial odometry processing 111 based on the image IMG(tck) captured by the camera at the timing tck and the motion information MTI(tck) of the moving body sampled by the IMU at the timing tck, and estimates the position and orientation Pr(tck) of the real object and the moving body at the timing tck.

[0089] The tracking processing unit 110 performs inertial odometry processing 112 based on the position and orientation Pr(tck) and the motion information MTI(tnow1) of the moving body sampled by the IMU at the timing tnow1, and estimates the position and orientation Pc(tck, tnow1) of the real object and the moving body at the timing tnow1. tnow1 is after tck and before the start of the inertial odometry processing 112. More preferably, tnow1 is after the end of the visual inertial odometry processing 111 and before the start of the inertial odometry processing 112.

[0090] The tracking processing unit 110 performs prediction processing 113 based on the position and orientation Pc(tck, tnow1), and predicts the position and orientation Pp(tck, tnow1, tfuture) of the real object and the moving body at a timing tfuture after tnow1. Fig. 11 shows an example in which tfuture=tdp and the position and orientation Pp(tck, tnow1, tdp) is predicted. tdp is the display timing of the HUD. The prediction processing 113 predicts the future position and orientation Pp, for example, based on the differential value of the position and orientation Pc obtained in time series.

[0091] The rendering image generating unit 120 renders a virtual object based on the position and orientation Pp (tck, tnow1, tdp) and outputs a rendering image. If this rendering image were displayed as is on the HUD 400, the virtual object would be displayed corresponding to the position of the real object indicated by the position and orientation Pp.

[0092] Here, the predicted position and orientation Pp(tck, tnow1, tdp) are predicted using information at tck and tnow1, and are not positions based on the actual sensor output at tdp. That is, if the actual movement that occurred between tnow1 and tdp differs from the movement predicted by the prediction process 113, a position shift due to latency may occur between the virtual object and the real object. That is, the latency to be compensated for in the example of FIG. 11 is the latency due to the processing time between tnow1 and tdp. Note that the latency to be compensated for is not limited to this. For example, if the inertial odometry and prediction process are omitted and the rendering process is performed using Pr(tck), the latency due to the processing time between tck and tdp is the latency to be compensated for.

[0093] In this embodiment, the position shift due to the latency is compensated for as follows. The tracking processing unit 110 performs visual inertial odometry processing 111 based on the image IMG(tck+1) captured by the camera at the timing tck+1 and the motion information MTI(tck+1) of the moving object sampled by the IMU at the timing tck+1, and estimates the position and orientation Pr(tck+1) of the real object and the moving object at the timing tck+1. tck+1 is the next camera exposure timing after tck.

[0094] The tracking processing unit 110 performs inertial odometry processing 112 based on the position and orientation Pr(tck+1) and the motion information MTI(tnow2) of the moving body sampled by the IMU at time tnow2, and estimates the position and orientation Pc(tck+1, tnow2) of the real object and the moving body at time tnow2.

[0095] The tracking processing unit 110 performs a prediction process 113 based on the position and orientation Pc(tck+1, tnow2) to predict the position and orientation Pp(tck+1, tnow2, tdp) of the real object and the moving body at tdp.

[0096] The parameter calculation unit 140 calculates a latency compensation parameter based on the position and orientation Pp(tck, tnow1, tdp) and the position and orientation Pp(tck+1, tnow2, tdp). Specifically, the parameter calculation unit 140 calculates a latency compensation parameter that cancels the position shift of the virtual object caused by the difference between the position and orientation Pp(tck, tnow1, tdp) and the position and orientation Pp(tck+1, tnow2, tdp). The parameter calculation unit 140 corrects the warp parameter using the latency compensation parameter and outputs the corrected warp parameter.

[0097] The warp processing unit 130 warps the rendering image using the corrected warp parameters to generate a display image. The corrected warp parameters correspond to the first warp parameters described with reference to Figs. 1 to 4, and the warp processing using the corrected warp parameters corresponds to warp processing including distortion correction and vibration correction. This warp processing is performed on a second region including a virtual object in the rendering image, which is the image to be warped. The warp processing unit 130 performs warp processing for distortion correction only on the first region of the rendering image.

[0098] The warp processing unit 130 sequentially performs warp processing on the first line to the final line in accordance with the scanning timing of the first line to the final line on the display panel of the HUD 400, and outputs the data of each line after the warp processing in sequence in accordance with the scanning timing to the HUD 400. In Fig. 8, the display timing of the first line is set to tdp, but tdp may be any timing in the frame period from when the first line to the final line are displayed.

[0099] In the above embodiment, the display system 100 is a display system of the HUD 400 that displays the first display object 20 corresponding to a real object in the real space and the second display object 25 in the display area 40. The display system 100 includes a storage unit 161 and a warp processing unit 130. The storage unit 161 stores a warp target image WTIM including an image IM20 of the first display object and an image IM25 of the second display object. The warp processing unit 130 performs warp processing on the warp target image WTIM to generate a display image DSIM to be displayed in the display area 40. The warp processing unit 130 performs warp processing on the image IM20 of the first display object, out of the warp target image WTIM, based on a first warp parameter PRM1. The first warp parameter PRM1 is a warp parameter that performs vibration correction of the display position deviation of the first display object 20 with respect to the real object and distortion correction according to the curvature of the display area 40. Furthermore, the warp processing unit 130 performs warping on the image IM25 of the second display object, out of the warp target image WTIM, based on the second warp parameters PRM2. The second warp parameters PRM2 are warp parameters that perform distortion correction.

[0100] Furthermore, in this embodiment, the display system 100 includes a parameter calculation unit 140 that calculates a first warp parameter PRM1 and a second warp parameter PRM2. The parameter calculation unit 140 calculates the first warp parameter PRM1 based on tracking information. The tracking information is at least one of first tracking information of a moving object on which the HUD 400 is mounted, second tracking information of a viewer of the HUD 400, and third tracking information of a real object.

[0101] In this way, the parameter calculation unit 140 can calculate the vibration correction parameters based on the tracking information, and can calculate the first warp parameters PRM1 based on the vibration correction parameters and output the first warp parameters PRM1 for performing vibration correction and distortion correction to the warp processing unit 130.

[0102] In this embodiment, the display system 100 also includes a rendering image generation unit 120 that renders the warp target image WTIM. The parameter calculation unit 140 calculates a latency compensation parameter based on the tracking information, and obtains the first warp parameter PRM1 using the latency compensation parameter as a parameter for vibration correction. The latency compensation parameter is a parameter that compensates for latency including the rendering process latency of the warp target image WTIM.

[0103] In this way, latency can be compensated for by vibration correction. That is, the position shift of the virtual object caused by the latency due to the rendering process between tnow1 and tdp is compensated for based on the latency compensation parameters. By performing latency compensation in the warp process performed immediately before display, the remaining latency can be minimized, realizing AR display in which the virtual object has high tracking ability with respect to the real object.

[0104] In this embodiment, the rendering image generating unit 120 generates a rendering image based on tracking information at a first timing. The parameter calculating unit 140 calculates a latency compensation parameter based on tracking information at a second timing that is later than the first timing and earlier than the image display timing tdp, and tracking information at the first timing. In the example of Fig. 8, tnow1 corresponds to the first timing, and tnow2 corresponds to the second timing.

[0105] In this way, the latency between the first timing and the display timing is shortened to the latency between the second timing and the display timing, and thus the position shift due to the latency is reduced. To explain using the example of FIG. 8, the latency compensation parameter that corrects the warp parameter reflects the motion information sampled by the IMU at tnow2, which is after tnow1. By correcting the display position of the virtual object using this latency compensation parameter, the latency between tnow1 and tdp is shortened to the latency between tnow2 and tdp. Since the warp process is performed immediately before the HUD display, tnow2 is close to tdp, and the latency can be reduced as much as possible to reduce the display position shift of the virtual object due to the latency.

[0106] In this embodiment, the parameter calculation unit 140 corrects the warp parameters of the coordinate transformation according to the curvature of the display area based on the latency compensation parameters, thereby outputting the first warp parameters. The warp processing unit 130 performs the latency compensation process by performing the warp process on the first area AR1 of the warp target image WTIM using the first warp parameters.

[0107] In this way, by performing warp processing using the first warp parameter based on the latency compensation parameter, latency compensation can be performed in the warp processing. For example, a configuration in which image processing for latency compensation is performed separately from warp processing is conceivable, but the processing load increases by the amount of image processing for latency compensation, and by performing warp processing after latency compensation, the latency of the warp processing remains. In this embodiment, latency compensation and warp processing are integrated, so that the latency can be minimized while suppressing an increase in the processing load.

[0108] 4. Detailed configuration example of the display system An example of a hardware configuration of the display system 100 will be described below, but the hardware configuration of the display system 100 is not limited to this. For example, in the following, the parameter calculation unit 140 is provided in the processing device 150, but the parameter calculation unit 140 may be provided in the circuit device 160. In the following, the circuit device 160 includes a warp parameter selection unit 162, but when the second configuration example of FIG. 3 is adopted, the circuit device 160 may include a layer selection unit 163 instead of the warp parameter selection unit 162.

[0109] 12 is a detailed configuration example of the display system 100. The display system 100 includes a processing device 150 and a circuit device 160. The processing device 150 is configured, for example, by an SoC, and includes an interface 151, a tracking processing unit 110, a rendering image generating unit 120, a parameter calculation unit 140, and a storage unit 145. SoC is an abbreviation for System on Chip. The circuit device 160 includes a storage unit 161, a warp processing unit 130, and a warp parameter selection unit 162. Note that components already described are denoted by the same reference numerals, and descriptions of those components will be omitted as appropriate.

[0110] The interface 151 is an interface for inputting an output signal of the sensor 500 to the processing device 150. The interface 151 is, for example, a receiving circuit that receives the output signal of the sensor 500. Alternatively, the interface 151 may be a terminal to which the output signal of the sensor 500 is input. In this case, the output signal of the sensor 500 input to the terminal is input to a processor constituting the tracking processing unit 110, and the processor receives and processes the output signal of the sensor 500.

[0111] The tracking processing unit 110 performs tracking processing based on an output signal of the sensor 500. The rendering image generation unit 120 renders a virtual object based on tracking information output by the tracking processing unit 110. The tracking processing unit 110 and the rendering image generation unit 120 are configured with a processor such as a CPU or a GPU. CPU is an abbreviation for Central Processing Unit. GPU is an abbreviation for Graphical Processing Unit. The tracking processing unit 110 and the rendering image generation unit 120 may be realized by one processor, or each may be realized by an individual processor.

[0112] The storage unit 145 is a semiconductor memory such as a RAM or a non-volatile memory, and stores the warp parameters before latency compensation. RAM is an abbreviation for Random Access Memory. As described above, the warp parameters are a matrix or a table. The warp parameters may be either forward map parameters or inverse map parameters. The forward map parameters are parameters that associate each pixel of the input image with a destination coordinate corresponding to each pixel, or parameters that associate each pixel of the input image with a relative movement amount to the destination coordinate corresponding to each pixel. The inverse map parameters are parameters that associate each pixel of the output image with a reference source coordinate corresponding to each pixel, or parameters that associate each pixel of the output image with a relative movement amount from the reference source coordinate corresponding to each pixel.

[0113] The parameter calculation unit 140 obtains latency compensation parameters based on the tracking information, corrects the warp parameters read from the storage unit 145 based on the latency compensation parameters, and outputs the corrected warp parameters as first warp parameters. The parameter calculation unit 140 also outputs the warp parameters read from the storage unit 145 as second warp parameters. The parameter calculation unit 140 may be realized by the same processor as the processors of the tracking processing unit 110 and the rendering image generation unit 120, or may be realized by a processor different from those.

[0114] The warp processing unit 130 generates a display image by warping the rendering image using the first warp parameter and the second warp parameter. Here, the rendering image corresponds to the image to be warped. The warp processing unit 130 is configured with a logic circuit, for example, an automatically placed and wired gate array or an automatically wired standard cell array. The HUD 400 includes a display 410, and causes the display image to be displayed on the display 410. The display 410 is a display device such as a liquid crystal display device. The HUD 400 includes a projection optical unit (not shown). The projection optical unit projects the image displayed on the display 410 onto a screen, thereby superimposing a virtual object on the real space.

[0115] 5. Warp Processing Details of the warp process will be described below using inverse warp as an example, although the warp process may be either forward warp or inverse warp, and the warp processor 130 may be either a forward warp engine or an inverse warp engine.

[0116] 13 is an explanatory diagram of the warp processing in this embodiment. The input image is a rendering image. The output image is a display image displayed on a liquid crystal display panel of a HUD or the like. (xsrc, ysrc) indicate the horizontal and vertical positions in the input image. (xdst, ydst) indicate the horizontal and vertical positions in the output image.

[0117] If the correction of the warp parameters using the latency compensation parameters is represented as g(), the position in the latency compensated image is expressed as the following equation (1).

[0118] (xlat,ylat)=g(xsrc,ysrc) ···(1)

[0119] If the distortion correction using the warp parameters according to the screen curvature is expressed as f(), the position in the output image is expressed as the following equation (2).

[0120] (xdst,ydst)=f(xlat,ylat) ···(2)

[0121] From the above equations (1) and (2), the coordinate transformation between the input image and the output image is expressed as the following equation (3).

[0122] (xdst,ydst)=f·g(xsrc,ysrc) ···(3)

[0123] Inverse warping uses the inverse transformation of equation (3) above, resulting in equation (4) below.

[0124] (xsrc,ysrc)=g -1 f -1 (xdst, ydst) ···(4)

[0125] In this embodiment, the parameter calculation unit 140 calculates the latency compensation parameter g -1 Find g -1 f -1 By calculating the warp parameter f -1After correction, the warp parameter g -1 f -1 The warp processing unit 130 outputs the corrected warp parameter g -1 f -1 The output image is generated by performing the warping process of the above equation (4) using

[0126] In addition, the warp parameter g after correction -1 f -1 The target of the warp processing by the warp parameter f is the first area AR1 described in FIG. 1 to FIG. 4, and the target of the warp processing by the warp parameter f is the second area AR2. -1 That is, the warp processing is performed by the corrected warp parameter g -1 f -1 is the first warp parameter, and the warp parameter f -1 is the second warp parameter.

[0127] 14 is a diagram showing the correspondence between tracking information of a moving object etc. and latency compensation based on the tracking information. Here, it is assumed that six-axis motion information is obtained as the tracking information.

[0128] The yaw displacement Δα is a rotational displacement with an axis parallel to the vertical y direction as the rotation axis. The yaw displacement Δα causes a horizontal shift in the real object seen by the viewer through the screen. The horizontal shift is a shift in the horizontal x direction. Therefore, the latency compensation method is a horizontal shift.

[0129] The pitch displacement Δβ is a rotational displacement around an axis parallel to the horizontal x direction. The pitch displacement Δβ causes a vertical shift in the real object seen by the viewer through the screen. The vertical shift is a shift in the vertical y direction. Therefore, the latency compensation method is a vertical shift.

[0130] The roll displacement Δγ is a rotational displacement with the axis parallel to the z-direction, which is the forward and backward direction of the moving object, as the rotation axis. The roll displacement Δγ causes rotation in the real object seen by the viewer through the screen. For this reason, the latency compensation method is rotation.

[0131] The horizontal shift Δx is a shift in the x direction, which causes a horizontal shift in the real object seen by the viewer through the screen. Therefore, the latency compensation technique is a horizontal shift.

[0132] The vertical shift Δy is a shift in the y direction, which causes a vertical shift in the real object that the viewer sees through the screen, so the latency compensation technique is a vertical shift.

[0133] The fore-aft shift Δz is a shift in the z-direction that causes a shrinking or enlargement of the real object that the viewer sees through the screen, so the latency compensation technique becomes a shrinking or zooming.

[0134] FIG. 15 shows an example of the latency compensation parameters that realize the above compensation method. The matrix on the right side of the equation shown in FIG. 15 is the matrix g explained in FIG. -1 m13 indicates a horizontal shift, and m23 indicates a vertical shift. m12 indicates a horizontal skew. A horizontal skew is a transformation that distorts a rectangle into a parallelogram, but is used here as an approximation of a rotation. m11 indicates a horizontal scale, and m22 indicates a vertical scale. m11 and m22 correspond to reduction or zoom.

[0135] For example, if only a vertical shift occurs, m11 = m22 = 1, and m12 = m13 = 0. In this case, the equation in FIG. 15 becomes the following equation (5).

[0136] (xsrc,ysrc)=(xlat,ylat+m23) ···(5)

[0137] In Fig. 13, the inverse transformation f -1() moves (xdst, ydst) to (xlat, ylat). The above equation (5) corresponds to shifting this amount by m23 in the vertical direction, and this shift realizes latency compensation.

[0138] FIG. 16 is a diagram for explaining a method for calculating a latency compensation parameter from tracking information.

[0139] From the geometrical positional relationship between the center of rotation of yaw, roll and pitch, the viewer's head, the HUD screen and the real object, the displacement occurring in the moving body, the viewer or the real object can be converted into the shift amount or rotation angle on the screen. Then, the matrix on the right side of Fig. 15 can be determined from the shift amount or rotation angle.

[0140] In Fig. 16, a method for obtaining a latency compensation parameter from tracking information will be described using m23 indicating a vertical shift in Fig. 15 as an example. Here, an example will be described in which the pitch displacement Δβ and vertical shift Δym of the moving object 32 and the vertical shift Δyp of the viewer 52 are used as tracking information.

[0141] If the distance between the pitch rotation center PTC and the screen 34 is DCF, the pitch displacement Δβ of the moving body 32 vertically shifts the screen 34 by DCF×Δβ. Also, the vertical shift Δym of the moving body 32 vertically shifts the screen 34 by Δym. Summing these, the vertical shift amount of the screen 34 is DCF×Δβ+Δym. Assuming that the head of the viewer 52 and the real object 12 do not move, for example, when the screen 34 is raised by +(DCF×Δβ+Δym), the position of the real object 12 on the screen 34 as seen by the viewer appears to be lowered by -(DCF×Δβ+Δym) relative to the screen 34. Therefore, the latency compensation parameter in this case is m23=-(DCF×Δβ+Δym).

[0142] The distance between the head of the viewer 52 and the real object 12 is DPT, and the distance between the screen 34 and the real object 12 is DFT. Assuming that the screen 34 and the real object 12 do not move, for example, when the viewer moves up by +Δyp, the position of the real object 12 on the screen 34 seen by the viewer 52 appears to move up by +(DFT / DPT)×Δyp relative to the screen 34. If it is assumed that the real object 12 is sufficiently far away, it can be approximated as DFT / DPT=1, so the vertical shift amount of the real object 12 is Δyp. Therefore, the latency compensation parameter in this case is m23=Δyp.

[0143] In the embodiment described above, the display system 100 includes a storage unit 145, which stores a table that associates coordinates in a display image with a movement amount indicating the movement destination of the coordinates as a warp parameter. The parameter calculation unit 140 shifts the movement amount of the table based on the latency compensation parameter, and outputs the table with the shifted movement amount as a first warp parameter PRM1.

[0144] In the example of Figure 13, f -1 () is stored as a table in the storage unit 145. In the example described in FIG. 15 and the above formula (5), the amount of movement of the table that moves (xdst, ydst) to (xlat, ylat) is shifted vertically by, for example, m23. That is, the table is corrected to move (xdst, ydst) to (xlat, ylat+m23).

[0145] Alternatively, the storage unit 145 may store, as warp parameters, parameters of a conversion equation that converts coordinates in a display image into coordinates of a destination. The parameter calculation unit 140 shifts the coordinates of the destination based on the latency compensation parameter, and outputs the parameters of the conversion equation in which the coordinates of the destination have been shifted as the first warp parameters PRM1.

[0146] The conversion equation is, for example, a polynomial, and the coefficients of the polynomial are stored as parameters of the conversion equation in the storage unit 145. In the example of FIG.-1 The parameters of the conversion equation indicating () are stored in the storage unit 145. In the example described in FIG. 15 and the above formula (5), the amount of movement of the conversion equation that moves (xdst, ydst) to (xlat, ylat) is shifted vertically by, for example, m23. That is, the conversion equation is corrected to move (xdst, ydst) to (xlat, ylat+m23).

[0147] Although the present embodiment has been described in detail as above, it will be easily understood by those skilled in the art that many modifications are possible without substantially departing from the novel matters and effects of the present disclosure. Therefore, all such modifications are intended to be included in the scope of the present disclosure. For example, a term described at least once in the specification or drawings together with a different term having a broader meaning or synonymy can be replaced with that different term anywhere in the specification or drawings. In addition, all combinations of the present embodiment and modifications are also included in the scope of the present disclosure. In addition, the configurations and operations of sensors, head-up displays, processing devices, circuit devices, display systems, electronic devices, and mobile objects are not limited to those described in the present embodiment, and various modifications are possible. [Explanation of symbols]

[0148] 10...vehicle in front, 11, 12...real object, 20...first display object, 21...virtual object, 25...second display object, 30...car, 32...moving object, 34...screen, 35...windshield, 36...projection unit, 40...display area, 50...eye, 52...viewer, 60...processing device, 61...sensor interface, 62...tracking processing unit, 63...rendering image generating unit, 65...sensor, 70...HUD controller, 71...warp processing unit, 81...display, 90...display system, 100...display system, 110...tracking processing unit, 111...visual inertial odometry processing, 112...inertial odometry processing, 113...prediction processing, 120...rendering Ring image generation unit, 130...warp processing unit, 131...first correction unit, 132...second correction unit, 133...combining unit, 140...parameter calculation unit, 145...storage unit, 150...processing unit, 151...interface, 160...circuit device, 161...storage unit, 162...warp parameter selection unit, 163...layer selection unit, 270...processing unit, 410...display, 500...sensor, AR1...first area, AR2...second area, CIM1...first corrected image, CIM2...second corrected image, DSIM...display image, IM20...image of first display object, IM25...image of second display object, PRM1...first warp parameter, PRM2...second warp parameter, WTIM...image to be warped

Claims

1. A circuit device used in a head-up display that displays a first display object corresponding to a real object in a real space and a second display object in a display area, a storage unit that stores a warp target image including an image of the first display object and an image of the second display object; a warp processing unit that performs warping on the warp target image using a first warp parameter and a second warp parameter to generate a display image to be displayed in the display area; Including, the first warp parameters are parameters of coordinate transformation including both vibration correction of a display position deviation of the first display object with respect to the real object and distortion correction according to a curvature of the display area, the second warp parameters are parameters of a coordinate transformation that is the distortion correction according to the curvature of the display area; The warp processing unit includes: a first correction unit that performs the coordinate transformation on the entire warp target image by using the first warp parameters, thereby simultaneously performing the vibration correction and the distortion correction, and outputs a first corrected image; a second correction unit that performs the coordinate transformation on the entire warp target image using the second warp parameters to perform the distortion correction and output a second corrected image; a synthesis unit that synthesizes the first corrected image and the second corrected image and outputs the display image; A circuit device comprising:

2. 2. The circuit device according to claim 1, The first warp parameter is a warp parameter used for the distortion correction is a parameter obtained by correcting the warp parameter with a latency compensation parameter that compensates for a latency including a rendering process latency of the warp target image; The second warp parameter is A circuit device characterized in that the warp parameters are used for the distortion correction.

3. 2. The circuit device according to claim 1, The synthesis unit is outputting the first corrected image as the display image for a first region in which the first display object is displayed; The circuit device is characterized in that the second corrected image is output as the display image for a second area in which the second display object is displayed.

4. 2. The circuit device according to claim 1, The synthesis unit is a circuit device which performs alpha blending of the first corrected image and the second corrected image to output the display image.

5. 5. The circuit device according to claim 1, The circuit device is characterized in that the memory unit is a line buffer.

6. A display system of a head-up display that displays a first display object corresponding to a real object in a real space and a second display object in a display area, a storage unit that stores a warp target image including an image of the first display object and an image of the second display object; a warp processing unit that performs warping on the warp target image using a first warp parameter and a second warp parameter to generate a display image to be displayed in the display area; Including, the first warp parameters are parameters of coordinate transformation including both vibration correction of a display position deviation of the first display object with respect to the real object and distortion correction according to a curvature of the display area, the second warp parameters are parameters of a coordinate transformation that is the distortion correction according to the curvature of the display area; The warp processing unit includes: a first correction unit that performs the coordinate transformation on the entire warp target image by using the first warp parameters, thereby simultaneously performing the vibration correction and the distortion correction, and outputs a first corrected image; a second correction unit that performs the coordinate transformation on the entire warp target image using the second warp parameters to perform the distortion correction and output a second corrected image; a synthesis unit that synthesizes the first corrected image and the second corrected image and outputs the display image; A display system comprising:

7. 7. The display system according to claim 6, a parameter calculation unit that calculates the first warp parameters and the second warp parameters; The parameter calculation unit is A display system characterized by calculating the first warp parameters based on tracking information which is at least one of first tracking information of a moving body on which the head-up display is mounted, second tracking information of a viewer of the head-up display, and third tracking information of the real object.

8. 8. The display system of claim 7, a rendering image generating unit that renders the warp target image, The parameter calculation unit is Calculating a latency compensation parameter based on the tracking information to compensate for a latency including a rendering process latency of the warped image; correcting a warp parameter used for the distortion correction with the latency compensation parameter to obtain the first warp parameter; A display system, characterized in that the warp parameters used for the distortion correction are output as the second warp parameters.

9. 1. An image processing method for a head-up display that displays a first display object corresponding to a real object in a real space and a second display object in a display area, the method comprising: performing warping processing on a warp target image including an image of the first display object and an image of the second display object, using first warp parameters which are parameters of coordinate transformation including both vibration correction of a display position deviation of the first display object with respect to the real object and distortion correction according to a curvature of the display area, and second warp parameters which are parameters of coordinate transformation for the distortion correction according to the curvature of the display area, to generate a display image to be displayed in the display area; In the warping process, performing the coordinate transformation on the entire warp target image using the first warp parameters, thereby simultaneously performing the vibration correction and the distortion correction, and outputting a first corrected image; performing the coordinate transformation using the second warp parameters on the entire warp target image to perform the distortion correction and output a second corrected image; an image processing method comprising: synthesizing the first corrected image and the second corrected image to output the display image;

10. 10. The image processing method according to claim 9, The first warp parameter is a warp parameter used for the distortion correction is a parameter obtained by correcting the warp parameter with a latency compensation parameter that compensates for a latency including a rendering process latency of the warp target image; The second warp parameter is The image processing method according to claim 1, wherein the warp parameters are used for the distortion correction.

Images