Image processing apparatus and image processing method

Abstract

To provide an image processing apparatus, a method, and a program for reducing character crushing and blurring due to reprojection while suppressing an increase in processing load.SOLUTION: In an image processing apparatus, when a viewpoint of a user changes, a CG drawing unit reprojects a first image viewed from a viewpoint before the change to a second image viewed from a viewpoint after the change, draws an object 404 of a UI including characters on a character layer 401, and draws a CG object not including characters on a CG layer 402, and a correction control unit performs homography conversion of the CG layer 402 based on information of rotation and movement acquired by an operation acquisition unit when the viewpoint of the user changes, moves the character layer 401 without rotation, and performs the reprojection of the first image viewed from the viewpoint before the change to a second image viewed from the viewpoint after the change.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to an image processing device and an image processing method. [Background technology]

[0002] In recent years, mixed reality (MR) technology has become known as a technology for seamlessly fusing the real world and the virtual world in real time. MR technology is realized, for example, by using a video see-through type HMD (Head Mounted Display). A video see-through type HMD captures an image of a subject that approximately matches the subject observed from the pupil position of a user wearing the HMD using a video camera or the like, and enables the user to view an image in which CG (Computer Graphics) is superimposed on the captured image.

[0003] When displaying an image in which a CG image is superimposed on a captured image, a delay occurs in the display of the CG image relative to the captured image, which may cause the user to feel uncomfortable. In order to reduce the sense of discomfort caused by the delay in displaying the CG image, the HMD performs keystone correction on the CG image so as to cancel out the change in posture between when the CG image starts to be drawn and when the CG image is displayed. This process of correcting the CG image so as to cancel out the change in posture of the HMD (the head of the user wearing the HMD) is called "reprojection" or "time warp."

[0004] When a CG image includes characters, the characters may be crushed or blurred due to reprojection. Patent Document 1 discloses a technique for excluding an overlay image that should be displayed at a certain position on the screen from the reprojection process. Patent Document 2 discloses a technique for rendering graphic data showing characters, etc., as a layer including information on edges and distances from the edges, in a layer separate from visual scene data, and for retaining information on edges to perform reprojection process. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] International Publication No. 2019 / 229816 [Patent Document 2] US Patent Application Publication No. 2017 / 0372457 Summary of the Invention [Problem to be solved by the invention]

[0006] However, the technology of Patent Document 1 is a technology that aims to exclude GUIs displayed at fixed positions on the screen from the reprojection target, and does not assume reprojection processing of GUIs that are fixed in space, etc. Also, in the method of Patent Document 2, extra memory is reserved to hold information about edges, and the processing load increases due to data transfer of the memory reserved in the reprojection processing and holding information about edges in the rendering processing.

[0007] SUMMARY OF THE PRESENT DISCLOSURE In view of the above, an object of the present invention is to provide an image processing device that reduces blurring and smearing of characters caused by reprojection processing while suppressing an increase in the processing load. [Means for solving the problem]

[0008] A first aspect of the present invention is a display device having a reprojection means for, when a user's viewpoint has changed, reprojecting a first image seen from a viewpoint before the change onto a second image seen from a viewpoint after the change, and when there is a specific change in the user's viewpoint, the reprojection means displays a display including characters. The image processing device is characterized in that, if a first object that does not satisfy a predetermined condition is included in the first image, the first object is rotated and moved, and, if a second object that satisfies the predetermined condition is included in the first image, the second object is moved without being rotated.

[0009] A second aspect of the present invention is an image processing method comprising a reprojection step of, when a user's viewpoint changes, reprojecting a first image seen from a viewpoint before the change onto a second image seen from a viewpoint after the change, and, when there is a specific change in the user's viewpoint, in the reprojection step, if the first image contains a first object that does not satisfy a predetermined condition indicating that it contains text, the first object is rotated and moved, and if the first image contains a second object that satisfies the predetermined condition, the second object is moved without being rotated.

[0010] A third aspect of the present invention is a program for causing a computer to function as each of the means of the image processing device described above. Effect of the Invention

[0011] According to the present invention, it is possible to reduce blurring and smearing of characters caused by reprojection processing while suppressing an increase in the processing load. [Brief description of the drawings]

[0012] [Figure 1] FIG. 2 is a diagram illustrating an example of a hardware configuration of an image processing device. [Diagram 2] 2 is a block diagram illustrating a functional configuration of an image processing device and an HMD. [Diagram 3] FIG. 13 is a diagram illustrating an example in which reprojection processing is not changed depending on the object. [Figure 4] FIG. 13 is a diagram illustrating an example of changing reprojection processing depending on an object. [Diagram 5] 1A and 1B are diagrams illustrating the occurrence of blurred and smeared characters. [Figure 6] 4 is a flowchart illustrating a reprojection process according to the first embodiment. [Figure 7] 10 is a flowchart illustrating a reprojection process according to a second embodiment. [Figure 8] 13 is a flowchart illustrating a reprojection process according to a third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] The present invention relates to a technique for sharpening characters in a reprojection process. When a CG image including characters is superimposed on a captured image, the reprojection process may cause the characters to become blurred and smudged. For example, when a user wearing an HMD operates a virtual user interface (UI) fixed in space, the user's head moves minutely even if the user intends to remain still. When minute reprojection is performed in response to head movement, the characters included in the CG image are likely to become blurred and smudged.

[0014] First Embodiment The hardware configuration of an image processing device 100 that performs character sharpening in reprojection processing will be described with reference to FIG.

[0015] The image processing device 100 includes a central processing unit (CPU) 101, a random access memory (RAM) 102, and a read only memory (ROM) 103. The image processing device 100 also includes an input I / F (interface) 104 and an output I / F (interface) 105.

[0016] The CPU 101 is a processor that controls the entire image processing device 100. 2 is used as a working area when the CPU 101 controls each functional block of the image processing device 100 shown in Fig. 2 and performs various processes. The ROM 103 stores a control program, various application programs, various data, etc. The CPU 101 can realize the processing of each functional block of the image processing device 100 by expanding the control program stored in the ROM 103 into the RAM 102 and executing it.

[0017] The input I / F 104 is an interface for inputting data from an external device as an input signal in a format processable by the image processing device 100. The data from the external device includes, for example, image data captured by the imaging unit 201 of the HMD 200, and data on the movement, tilt, rotation, etc. of the HMD 200 acquired by the sensing unit 202. The output I / F 105 is an interface for outputting a display image processed by the image processing device 100 to an external display device. The output I / F 105 outputs a display image in a format processable by the display unit 203 of the HMD 200, for example.

[0018] Fig. 2 is a block diagram illustrating functional configurations of the image processing device 100 and the HMD 200. The image processing device 100 communicates with the HMD 200 via an input I / F 104 and an output I / F 105. Note that, in the example of Fig. 2, the image processing device 100 and the HMD 200 are described as separate devices, but the image processing device 100 may be configured to be included in the HMD 200.

[0019] The HMD 200 is a video see-through type HMD that captures an image of a subject using a charge-coupled device such as a CCD to obtain digital image data of the subject, and displays an MR image (mixed reality image) on which a CG image is superimposed to the user who is wearing it.

[0020] The HMD200 captures an image of a space viewed by a user wearing the HMD200, and transmits the captured image to the image processing device 100. The image processing device 100 acquires information on the position and orientation (hereinafter also referred to as position and orientation) of the HMD200 from the captured image received from the HMD200. The image processing device 100 superimposes a CG image on the captured image based on the acquired information on the position and orientation of the HMD200, and transmits the superimposed image to the HMD200. The HMD200 displays the superimposed image received from the image processing device 100. By wearing the HMD200, a user wearing the HMD200 can experience an MR space in which reality and CG are seamlessly connected.

[0021] The HMD 200 includes an imaging unit 201, a sensing unit 202, and a display unit 203. The image processing device 100 includes an imaging processing unit 211, a position and orientation acquisition unit 212, a CG drawing unit 213, a motion acquisition unit 214, a correction control unit 215, a correction unit 216, an approximation correction unit 217, and an image synthesis unit 218. The functional blocks of the HMD 200 and the image processing device 100 will be described with reference to Figs. 2 to 5.

[0022] The imaging unit 201 of the HMD 200 includes an objective optical system and an image sensor for capturing an image of the outside world. The imaging unit 201 transmits a captured image of the outside world to an imaging processing unit 211 of the image processing device 100. The imaging unit 201 may capture an image of the eyes of a user wearing the HMD 200 in addition to the outside world. The user's eye image captured of the user's eyes is transmitted to the imaging processing unit 211 in the same manner as the captured image.

[0023] The sensing unit 202 senses the operation of the HMD 200 and acquires information such as the rotation and movement of the HMD 200. The sensing unit 202 includes, for example, an IMU (Inertial Measurement Unit), an acceleration sensor, and an angular velocity sensor. The sensing data output by the sensing unit 202 is transmitted to a operation acquisition unit 214 of the image processing device 100.

[0024] The display unit 203 includes an eyepiece optical system and a display device such as a liquid crystal display or an organic EL (Electro-Luminescence) display in order to present an image to a user wearing the HMD 200.

[0025] The imaging processing unit 211 of the image processing device 100 receives a captured image captured by the imaging unit 201 of the HMD 200. The imaging processing unit 211 can execute various image processing on the captured image. The image processing executed by the imaging processing unit 211 includes, for example, demosaic processing, shading correction, noise reduction, distortion correction, and stereo rectification. The image processing executed by the imaging processing unit 211 is mainly processing for making the captured image by the imaging unit 201 closer to the human eye's perception. The imaging processing unit 211 transmits the captured image that has been subjected to the image processing to the image synthesis unit 218 and the position and orientation acquisition unit 212.

[0026] Furthermore, the imaging processing unit 211 can acquire depth information in the real world from the captured image. The imaging processing unit 211 can acquire depth information from the captured image using a method for acquiring depth information from the time difference between irradiation of laser light and reflected light, for example, called LiDAR (Light Detection and Ranging). The imaging processing unit 211 can also acquire depth information from the captured image using a method for acquiring depth information from a parallax image using a stereo camera. The method for acquiring depth information is not particularly limited, and therefore details will be omitted.

[0027] The imaging processing unit 211 also receives the user's eye image captured by the imaging unit 201. The imaging processing unit 211 can execute various image processes on the user's eye image in the same way as on the captured image of the outside world. The imaging processing unit 211 estimates information related to the line of sight, such as the position of the user's eye relative to the HMD 200 and the direction of the line of sight, from the user's eye image. The imaging processing unit 211 transmits the estimated information related to the line of sight (hereinafter, referred to as line of sight information) to the correction control unit 215 and the correction unit 216. The line of sight information may include information on the position of the user's line of sight in the captured image.

[0028] The position and orientation acquisition section 212 acquires information on the position and orientation of the HMD 200 using the captured image that has been image-processed by the image capture processing section 211 and information on the operation of the HMD 200 acquired by the operation acquisition section 214 .

[0029] The position and orientation acquisition unit 212 can acquire information on the position and orientation of the HMD 200 based on the relationship between a world coordinate system that represents the real world and a camera coordinate system that represents an image observed through the HMD 200. Specifically, by arranging a marker or the like that serves as a reference in the real world, the position and orientation acquisition unit 212 can acquire information on the position and orientation of the HMD 200 based on the positional relationship of the marker in an image captured by the imaging unit 201 having a stereo camera configuration. In addition, the position and orientation acquisition unit 212 may acquire information on the position and orientation of the HMD 200 by using an external sensor that monitors the position and orientation of the HMD 200 in the world coordinate system without using a stereo camera. The method of acquiring the position and orientation of the HMD 200 is not limited to the above method. The position and orientation acquisition unit 212 transmits the acquired information on the position and orientation of the HMD 200 to the CG drawing unit 213.

[0030] The CG rendering unit 213 reads information on contents (objects) from a storage unit such as the RAM 102 or the ROM 103. The CG rendering unit 213 generates a CG image in which an object is rendered according to the user's viewpoint, based on the position and orientation information of the HMD 200 acquired by the position and orientation acquisition unit 212.

[0031] The CG drawing unit 213 superimposes an object on a captured image in any position and orientation of the HMD 200. The CG drawing unit 213 renders the objects read from the storage unit. The CG drawing unit 213 draws each object on one or more layers including at least one of a layer including an object including text and a layer including an object not including text.

[0032] The CG drawing unit 213 may divide the layers into layers of objects including text and layers of objects not including text, and may include multiple objects in one layer. By dividing and drawing objects into multiple layers, the CG drawing unit 213 can perform a reprojection process on layers including objects including text to reduce crushing and bleeding of the text.

[0033] In each embodiment, an object including characters may be regarded as an object that satisfies a predetermined condition indicating that the object includes characters, and an object that does not include characters may be regarded as an object that does not satisfy the predetermined condition indicating that the object includes characters. This is because when an object includes characters as a pattern, for example, when the characters included in the object are only a few characters or characters that are difficult to read, the crushed or smudged characters do not cause inconvenience to the user.

[0034] The predetermined condition indicating that an object contains characters is that the object contains more than a predetermined amount of characters. Examples of cases where the object contains more than a predetermined amount of characters include cases where the number of characters is greater than a predetermined number, and cases where the proportion of the display area of ​​the object that is occupied by the area displaying characters is greater than a predetermined proportion. Specifically, for example, an object that contains more than 10 characters may be treated as an object that contains characters, and an object that contains 10 or fewer characters as a pattern may be treated as an object that does not contain characters. In addition, the predetermined condition indicating that an object contains characters may be that the size of the characters contained in the object is greater than a predetermined size. In addition, the predetermined condition indicating that an object contains characters may be that both of the above two conditions are satisfied.

[0035] Fig. 3 is a diagram showing an example in which reprojection processing is not changed depending on whether an object includes text or not. In the example of Fig. 3, a UI object 303 including text and CG objects 304 and 305 including no text are rendered on the same CG layer 301. In an image of a CG layer 311 obtained by homography transformation of the CG layer 301, objects are rotated and moved, causing characters to be blurred and smudged. Therefore, a display image 320 obtained by combining an image of the CG layer 311 and a captured image 302 results in an image in which characters are blurred and smudged.

[0036] On the other hand, Fig. 4 is a diagram showing an example in which reprojection processing is changed depending on whether an object includes text or not. In the example of Fig. 4, CG drawing unit 213 draws UI object 404 including text on text layer 401, and draws CG object not including text on CG layer 402. Text layer 401 and CG layer 402 may hold alpha value and depth value information for each pixel in addition to color information.

[0037] If there is no object including text in the drawing data, the CG drawing unit 213 may not draw an object in the text layer 401. The CG drawing unit 213 may draw only text in the text layer 401, or may draw the entire UI object including text. The correction control unit 215 described later performs homography conversion on the CG layer 402 and moves the text layer 401 without rotating it. The image synthesis unit 218 described later can generate a display image 420 in which text is sharpened by synthesizing the text layer 411 to which the text-containing object 404 has been moved and the CG layer 412 after homography conversion with the captured image 403.

[0038] The motion acquisition unit 214 receives sensing data output by the sensing unit 202 of the HMD 200. The motion acquisition unit 214 uses the received sensing data to acquire information on the movement, inclination, rotation, and the like of the HMD 200 as motion information of the HMD 200.

[0039] The correction control unit 215 corrects the CG image generated by the CG drawing unit 213 based on information on the operation (rotation and movement) of the HMD 200 acquired by the operation acquisition unit 214. That is, when the user's viewpoint changes in response to the operation of the HMD 200, the correction control unit 215 reprojects a first image seen from the viewpoint before the change onto a second image seen from the viewpoint after the change.

[0040] Depending on the CG data drawn by the CG drawing unit 213, the processing load varies, and the captured image depicting the real world and the rendered CG image are mismatched in time. In the MR technology, the captured image depicting the real world and the rendered CG image are superimposed, and then the composite image is displayed. If the captured image and the CG image that are mismatched in time are superimposed, the user experiencing the MR image may feel uncomfortable. Therefore, in order to eliminate the mismatch in time between the captured image and the CG image, a reprojection process is performed.

[0041] The correction control unit 215 converts the CG image to match the captured image, taking into account the time difference between the captured image and the CG image. That is, the correction control unit 215 reprojects the CG image seen from the user's viewpoint before the change to an image seen from the viewpoint after the change, in accordance with the change in the user's viewpoint due to the time mismatch. This eliminates the time mismatch between the captured image and the CG image generated by the CG drawing unit 213, and generates a display image that does not look strange.

[0042] Fig. 5 is a diagram for explaining the occurrence of crushing and blurring of characters due to reprojection processing. As shown in Fig. 5, when a user wearing an HMD 200 operates a virtual user interface (UI) fixed in space, the user's head moves slightly even if the user intends to stay still. By executing minute reprojection in response to head movement, crushing and blurring of characters included in a CG image are likely to occur.

[0043] In this way, when an object rotates and moves due to a change in the user's viewpoint, the characters included in the object may become crushed or blurred. Therefore, the correction control unit 215 controls to perform different reprojection processing for an object including characters and an object not including characters when there is a specific change in the user's viewpoint such that the object rotates and moves. The specific change in the user's viewpoint corresponds to a specific change in the position and posture of the head of the user wearing the HMD 200. The specific change is a change in viewpoint or the like that causes the rotation of the object, which causes the characters to become crushed and blurred.

[0044] The correction control unit 215 receives information on the movement, inclination, rotation, etc. of the HMD 200 at the start of drawing the object and at the present time from the action acquisition unit 214. The correction control unit 215 calculates the amount of viewpoint change from the start of drawing the object to the present time based on the difference in the movement, inclination, rotation, etc. between the start of drawing and the present time. The amount of viewpoint change may be the amount of change in rotation only, or may be the amount of change in rotation and movement. The user's viewpoint changes (moves) according to at least one of the position and posture of the user's head.

[0045] The correction control unit 215 transmits a character layer 401, which is a rendering of an object including characters, to the approximation correction unit 217. The correction control unit 215 also transmits a CG layer 402, which is a rendering of an object not including characters, to the correction unit 216. The correction unit 216 and the approximation correction unit 217 respectively transmit the received CG layer 402 and character layer 401 to the approximation correction unit 217. To project.

[0046] When the character layer 401 is not drawn by the CG drawing unit 213, the correction control unit 215 does not need to transmit layer data to the approximation correction unit 217. When the image seen from the user's viewpoint before the change does not include an object including text, the character layer 401 is not drawn and is not transmitted to the approximation correction unit 217.

[0047] On the other hand, if the image seen from the user's viewpoint before the change includes an object including text, the correction control unit 215 performs different reprojection processing for the object including text and the object not including text. That is, the correction control unit 215 transmits the text layer 401 including the object including text to the approximation correction unit 217, and transmits the CG layer 402 including the object not including text to the correction unit 216.

[0048] The correction unit 216 generates a corrected CG layer 412 by reprojecting the image of the CG layer 402 so as to reproduce the amount of viewpoint change that changes according to at least one of the position and posture of the user's head. When correcting rotation by reprojection processing, the correction unit 216 performs reprojection by obtaining a homography transformation matrix that reproduces the rotational movement and calculating the correspondence relationship between the coordinates before and after the projection.

[0049] Furthermore, when correcting rotation and movement, the correction unit 216 performs reprojection using the depth information and rotation and movement information of the CG layer 402. For example, the correction unit 216 divides the image into several blocks, and projects the three-dimensional position calculated using the depth of the four vertices of each block by the amount of rotation and translation. The correction unit 216 generates the corrected CG layer 412 by performing trapezoidal transformation on the texture of the object (CG) in each block.

[0050] The approximation correction unit 217 calculates a movement component that approximates the viewpoint change amount, and moves the image of the character layer 401 without rotating it, thereby generating a corrected character layer 411. When the rotation around the optical axis (around the user's face direction) at the time of image capture is defined as the rotation in the roll direction, the approximation correction unit 217 approximates the rotation in the pitch direction and the yaw direction to a movement. The optical axis is a straight line perpendicular to the image plane from the focal point. The approximation correction unit 217 does not approximate the rotation in the roll direction to a movement, but rotates the object according to the rotation of the head around the user's face direction, and approximates only the rotation in the pitch direction and the yaw direction to a movement. That is, the approximation correction unit 217 rotates the object according to the rotation of the head around the user's face direction, regardless of whether the object includes a character or not.

[0051] A change in the viewpoint corresponds to a change in the position and posture of the HMD 200 (user's head), and an amount of viewpoint change corresponds to an amount of change in rotation and an amount of change in movement of the HMD 200. The approximation correction unit 217 can approximate the amount of viewpoint change to an amount of movement of an object by using the amount of change in rotation and the amount of change in movement of the HMD 200.

[0052] When approximating the viewpoint change amount to the object movement amount, it may be approximated to the movement amount in the area on the image corresponding to the principal point, or to the movement amount in the area where the text exists. Furthermore, the approximation correction unit 217 may approximate the movement amount in the area including the user's gaze position in the captured image based on the gaze information received from the imaging processing unit 211.

[0053] The approximation correction unit 217 moves the object including the text by the nearest neighbor method. By performing rendering after the movement by the approximated movement amount by the nearest neighbor method, the position where the object is drawn can be determined without changing the brightness of each pixel from the state before drawing. Therefore, the occurrence of blurring and smearing of characters due to the reprojection process is suppressed.

[0054] The image synthesis unit 218 generates a display image 420 by synthesizing the captured image 403 received from the imaging processing unit 211, the corrected CG layer 412 received from the correction unit 216, and the corrected character layer 411 received from the approximation correction unit 217. The image synthesis unit 218 may synthesize each image in a predetermined order, or may synthesize each image in consideration of the depth and alpha value, etc. of each image. In addition, when the application running on the image processing device 100 is a VR (Virtual Reality) application that displays only a virtual world, not MR, the image synthesis unit 218 may display the captured image 403 without synthesizing it. The image synthesis unit 218 transmits the synthesized image to the display unit 203 of the HMD 200. The display unit 203 displays the display image 420 received from the image synthesis unit 218.

[0055] 6 is a flowchart illustrating the reprojection process of the first embodiment. The reprojection process of FIG.

[0056] In step S601, the correction control unit 215 receives information on the movement, inclination, rotation, etc. of the HMD 200 at the start of drawing the object and at the present time from the action acquisition unit 214. The correction control unit 215 acquires the viewpoint change amount from the start of drawing the object to the present time based on the difference in the movement, inclination, rotation, etc. between the start of drawing and the present time. The viewpoint change amount is the amount of change in at least one of the position and orientation of the HMD 200 (user's head) corresponding to a specific change in the user's viewpoint. The change amount of the position and orientation of the HMD 200 can be expressed by the amount of change in the movement and rotation of the HMD 200. The specific change in the user's viewpoint is a change in viewpoint such that the object rotates and moves.

[0057] In loop processing L1, the correction control unit 215 executes the processing of steps S602 to S605 for each drawing layer (layer i: i=1,...,N) drawn by the CG drawing unit 213. In step S602, the correction control unit 215 determines whether or not the drawing layer is a character layer including an object including text. If the drawing layer is a character layer, the processing proceeds to step S603. If the drawing layer is not a character layer, the processing proceeds to step S605.

[0058] In step S603, the approximation correction unit 217 receives information on the character layer 401 and the viewpoint change amount from the correction control unit 215. The approximation correction unit 217 acquires the movement amount when the rotation of the HMD 200 corresponding to the change in viewpoint is approximated to the movement of the object. In step S604, the approximation correction unit 217 corrects and reprojects the character layer 401 by moving the object by the movement amount acquired in step S603 without rotating it. The approximation correction unit 217 generates the corrected character layer 411 by a reprojection process that corrects the character layer 401.

[0059] In step S605, the correction unit 216 receives information about the CG layer 402 and the amount of viewpoint change from the correction control unit 215. The correction unit 216 corrects and reprojects the CG layer 402 by rotating and moving the object so as to reproduce the rotation and movement of the HMD 200 corresponding to the change in viewpoint. The correction unit 216 generates the corrected CG layer 412 by the reprojection process.

[0060] When the loop process L1 for each drawing layer is completed, the reprojection process shown in FIG. 6 is completed.

[0061] In the first embodiment, the image processing device 100 reprojects objects including text and objects including no text using different methods. During the reprojection process, the image processing device 100 rotates and moves objects including no text in accordance with changes in the user's viewpoint, whereas it moves objects including text without rotating them. By not rotating objects including text, the image processing device 100 can suppress the occurrence of crushing and bleeding of text due to the reprojection process.

[0062] Furthermore, the image processing device 100 can sharpen characters for objects including characters by a simple process of approximating the rotational movement of the HMD 200 to the movement of the HMD 200. Therefore, the image processing device 100 can reduce the crushing and blurring of characters caused by the reprojection process while suppressing an increase in the processing load.

[0063] <Second embodiment> In the first embodiment, the image processing device 100 moves an object including text without rotating it when performing reprojection processing. In contrast, in the second embodiment, when the amount of viewpoint change is greater than a threshold, the image processing device 100 rotates and moves even an object including text in accordance with the change in the user's viewpoint. When the amount of viewpoint change is greater than a threshold, if an object including text is moved without rotating it, the object including text may shift from the position when actually rotated, or the relative position with respect to the object not including text and the captured image may change. When the amount of viewpoint change is greater than a threshold, the image processing device 100 according to the second embodiment performs normal reprojection of the object including text as well, thereby reducing the discomfort felt by the user due to the shift in position of the object including text and the change in relative position.

[0064] 7 is a flowchart illustrating the reprojection process of the second embodiment. The reprojection process of FIG.

[0065] In step S701, the correction control unit 215 receives information on the movement, inclination, rotation, etc. of the HMD 200 at the start of drawing the object and at the present time from the action acquisition unit 214. The correction control unit 215 acquires the viewpoint change amount from the start of drawing the object to the present time based on the difference in the movement, inclination, rotation, etc. between the start of drawing and the present time. The viewpoint change amount is the amount of change in at least one of the position and orientation of the HMD 200 (user's head) corresponding to a specific change in the user's viewpoint. The change amount of the position and orientation of the HMD 200 can be expressed by the amount of change in the movement and rotation of the HMD 200. The specific change in the user's viewpoint is a change in viewpoint such that the object rotates and moves.

[0066] In loop processing L2, the correction control unit 215 executes the processing of steps S702 to S705 for each drawing layer (layer i: i=1, . . . , M) drawn by the CG drawing unit 213. In step S702, the correction control unit 215 determines whether or not the viewpoint change amount is equal to or less than a predetermined threshold and the drawing layer is a character layer including an object including a character. When the change amount of at least one of the position and the posture of the HMD 200 (user's head) corresponding to a specific change of the user's viewpoint is equal to or less than a predetermined threshold, the correction control unit 215 can determine that the viewpoint change amount is equal to or less than the threshold. When the viewpoint change amount is equal to or less than the threshold and the drawing layer is a character layer, the processing proceeds to step S703. When the viewpoint change amount is greater than the threshold or the drawing layer is not a character layer, the processing proceeds to step S705.

[0067] In step S703, the approximation correction unit 217 receives information on the character layer 401 and the viewpoint change amount from the correction control unit 215. The approximation correction unit 217 performs correction on the HMD 2 corresponding to the change in the viewpoint. 00 rotation is approximated to the movement of the object, and the amount of movement is obtained. In step S704, the approximation correction unit 217 corrects and reprojects the character layer 401 by moving the object by the amount of movement obtained in step S703 without rotating the object. The approximation correction unit 217 generates the corrected character layer 411 by the reprojection process that corrects the character layer 401.

[0068] In step S705, the correction unit 216 receives information on the character layer 401 or CG layer 402 and the viewpoint change amount from the correction control unit 215. The correction unit 216 corrects and reprojects the received layer by rotating and moving the object so as to reproduce the rotation and movement of the HMD 200 corresponding to the change in the viewpoint. In this way, when the change amount of at least one of the position and the orientation of the HMD 200 is larger than a predetermined threshold, the correction unit 216 also corrects the object including the character by rotating and moving it. The correction unit 216 generates the corrected character layer 411 or the corrected CG layer 412 by the reprojection process.

[0069] When the loop process L2 for each drawing layer is completed, the reprojection process shown in FIG. 7 is completed.

[0070] When the viewpoint change amount is greater than a threshold, the characters may be crushed and blurred by rotating and moving an object including the characters and then reprojecting it. However, when the HMD 200 is rotated significantly, blurring may occur in the captured image, and the object may become blurred even after normal reprojection processing by rotation and movement. In this case, the crushing and blurring of the characters becomes less noticeable. In addition, it is not expected that the user will read the characters while rotating his / her head significantly. When it is not expected that the user will read the characters included in the object, the image processing device 100 can further reduce memory usage and processing load by not performing processing to sharpen the characters.

[0071] In the second embodiment described above, when the amount of change in viewpoint is equal to or less than a threshold and blurring and smearing of characters is a problem, the image processing device 100 moves the object including characters without rotating it. On the other hand, when the amount of change in viewpoint is greater than the threshold, the image processing device 100 rotates and moves the object including characters in the same manner as the object not including characters. In this way, the image processing device 100 can reduce blurring and smearing of characters if the amount of change in at least one of the position and the posture of the head is equal to or less than a threshold, and can maintain geometric consistency by normal reprojection if the amount of change is greater than the threshold.

[0072] <Third embodiment> In the first embodiment, when performing reprojection processing, the image processing device 100 moves an object including text without rotating it. In contrast, in the third embodiment, the image processing device 100 rotates and moves an object including text in accordance with a change in the user's viewpoint when the user is not looking at the object. That is, when the user is viewing an object including text such as a UI, the image processing device 100 performs reprojection processing to reduce crushing and blurring of the text, and when the user is not viewing the object, the image processing device 100 does not sharpen the text.

[0073] 8 is a flowchart illustrating the reprojection process of the third embodiment. The reprojection process of FIG.

[0074] In step S801, the correction control unit 215 receives the motion of the object from the motion acquisition unit 214. Information on the movement, tilt, rotation, etc. of the HMD 200 at the start of drawing and at the present time is received. The correction control unit 215 acquires the amount of viewpoint change from the start of drawing of the object to the present time based on the difference in the movement, tilt, rotation, etc. between the start of drawing and the present. The amount of viewpoint change is the amount of change in at least one of the position and orientation of the HMD 200 (user's head) corresponding to a specific change in the user's viewpoint. The amount of change in the position and orientation of the HMD 200 can be expressed by the amount of change in the movement and rotation of the HMD 200. The specific change in the user's viewpoint is a change in viewpoint such that an object rotates and moves.

[0075] In loop processing L3, the correction control unit 215 executes the processing of steps S802 to S806 for each drawing layer (layer i: i=1,...,K) drawn by the CG drawing unit 213. In step S802, the correction control unit 215 determines whether or not the drawing layer is a character layer including an object including text. If the drawing layer is a character layer, the processing proceeds to step S803. If the drawing layer is not a character layer, the processing proceeds to step S806.

[0076] In step S803, the correction control unit 215 receives gaze information from the image capture processing unit 211. The correction control unit 215 determines whether the user's gaze is directed to the character in the character layer 401 based on the gaze information. The gaze information includes information on the user's gaze position in the captured image. For example, the correction control unit 215 projects the gaze position onto the character layer, and when the distance between the gaze position and the character position or the area including the character is equal to or less than a predetermined threshold, it can determine that the user's gaze is directed to the character in the character layer 401. If the user's gaze is directed to the character, the process proceeds to step S804. If the user's gaze is not directed to the character, the process proceeds to step S806.

[0077] In step S804, the approximation correction unit 217 receives information on the character layer 401 and the viewpoint change amount from the correction control unit 215. The approximation correction unit 217 acquires the movement amount when the rotation of the HMD 200 corresponding to the change in viewpoint is approximated to the movement of the object. In step S805, the approximation correction unit 217 corrects and reprojects the character layer 401 by moving the object by the movement amount acquired in step S804 without rotating the object. The approximation correction unit 217 generates the corrected character layer 411 by a reprojection process that corrects the character layer 401.

[0078] In step S806, the correction unit 216 receives information on the character layer 401 or CG layer 402 and the viewpoint change amount from the correction control unit 215. The correction unit 216 corrects and reprojects the received layer by rotating and moving the object so as to reproduce the rotation and movement of the HMD 200 corresponding to the change in viewpoint. In this way, when the user's line of sight is not directed at the character in the character layer 401, the correction unit 216 also corrects the object including the character by rotating and moving it. The correction unit 216 generates the corrected character layer 411 or the corrected CG layer 412 by the reprojection process.

[0079] When the loop process L3 for each drawing layer is completed, the reprojection process shown in FIG. 8 is completed.

[0080] In the above third embodiment, when the user directs his / her gaze to an object such as a UI that includes text, the image processing device 100 moves the object including text without rotating it. On the other hand, when the user does not direct his / her gaze to an object including text, the image processing device 100 rotates and moves the object including text in the same manner as an object that does not include text. When the user does not direct his / her gaze to an object including text, the image processing device 100 does not perform processing to sharpen the text, thereby further reducing memory usage and The processing load can be reduced.

[0081] The above-described embodiment (including the modified examples) is merely an example, and the present invention also includes configurations obtained by appropriately modifying or changing the configurations of the above-described embodiment within the scope of the gist of the present invention. Also, the present invention also includes configurations obtained by appropriately combining the configurations of the above-described embodiment.

[0082] <Other embodiments> The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0083] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) a reprojection means for, when a user's viewpoint has changed, reprojecting a first image seen from a viewpoint before the change onto a second image seen from a viewpoint after the change; When there is a specific change in the user's viewpoint, the reprojection means If the first image contains a first object that does not satisfy a predetermined condition indicating that the first object contains text, the first object is rotated and moved; If a second object that satisfies the predetermined condition is included in the first image, the second object is moved without being rotated. 13. An image processing device comprising: (Configuration 2) The particular change corresponds to a particular change in the position and posture of the user's head. 2. The image processing device according to claim 1, (Configuration 3) 3. The image processing device according to claim 1, wherein the reprojection means moves the second object by a nearest neighbor method. (Configuration 4) the user's viewpoint changes according to at least one of a position and a posture of the user's head; The reprojection means rotates an object included in the first image in accordance with a rotation of the head around a facial orientation of the user, regardless of whether the object satisfies the predetermined condition. 4. The image processing device according to any one of configurations 1 to 3. (Configuration 5) The reprojection means rotates and moves the second object when a change amount of at least one of a position and a posture of the head of the user corresponding to the specific change is greater than a predetermined threshold value. 5. The image processing device according to any one of configurations 1 to 4. (Configuration 6) The reprojection means rotates and moves the second object when the user's line of sight is not directed at the second object. 5. The image processing device according to any one of configurations 1 to 4. (Configuration 7) The reprojection means performs control so that, when the first image includes an object that satisfies the predetermined condition, a reprojection process for the first object and a reprojection process for the second object are different from each other. 7. The image processing device according to any one of configurations 1 to 6. (Configuration 8) The predetermined condition is at least one of: the object contains more than a predetermined amount of characters; and the size of the characters contained in the object is larger than a predetermined size. 8. The image processing device according to any one of configurations 1 to 7. (Configuration 9) The first image is drawn on one or more layers including at least one of a layer including the first object and a layer including the second object. 9. The image processing device according to any one of configurations 1 to 8. (method) a reprojection step of, when a user's viewpoint has changed, reprojecting a first image seen from a viewpoint before the change onto a second image seen from a viewpoint after the change; In the event of a particular change in the user's viewpoint, the reprojection step comprises: If the first image contains a first object that does not satisfy a predetermined condition indicating that the first object contains text, the first object is rotated and moved; If a second object that satisfies the predetermined condition is included in the first image, the second object is moved without being rotated. 13. An image processing method comprising: (program) A program for causing a computer to function as each of the means of the image processing device according to any one of configurations 1 to 9. [Explanation of symbols]

[0084] 100: image processing device, 101: CPU 215: correction control unit, 216: correction unit, 217: approximation correction unit

Claims

1. a reprojection means for, when a user's viewpoint has changed, reprojecting a first image seen from a viewpoint before the change onto a second image seen from a viewpoint after the change; When there is a specific change in the user's viewpoint, the reprojection means If the first image includes a first object that does not satisfy a predetermined condition indicating that the first object includes text, the first object is rotated and moved; If a second object that satisfies the predetermined condition is included in the first image, the second object is moved without being rotated.

13. An image processing device comprising:

2. The particular change corresponds to a particular change in the position and posture of the user's head.

2. The image processing device according to claim 1,

3. 2. The image processing apparatus according to claim 1, wherein the reprojection means moves the second object by a nearest neighbor method.

4. the user's viewpoint changes according to at least one of a position and a posture of the user's head; The reprojection means rotates an object included in the first image in accordance with a rotation of the head around a facial orientation of the user, regardless of whether the object satisfies the predetermined condition.

2. The image processing device according to claim 1,

5. The reprojection means rotates and moves the second object when a change amount of at least one of a position and a posture of the head of the user corresponding to the specific change is greater than a predetermined threshold.

2. The image processing device according to claim 1,

6. The reprojection means rotates and moves the second object when the user's line of sight is not directed at the second object.

2. The image processing device according to claim 1,

7. The reprojection means performs control so that, when the first image includes an object that satisfies the predetermined condition, a reprojection process for the first object and a reprojection process for the second object are different from each other.

2. The image processing device according to claim 1,

8. The predetermined condition is at least one of: the object contains more than a predetermined amount of characters; and the size of the characters contained in the object is larger than a predetermined size.

2. The image processing device according to claim 1,

9. The first image is drawn on one or more layers including at least one of a layer including the first object and a layer including the second object.

2. The image processing device according to claim 1,

10. a reprojection step of, when a user's viewpoint has changed, reprojecting a first image seen from a viewpoint before the change onto a second image seen from a viewpoint after the change; In the event of a particular change in the user's viewpoint, the reprojection step comprises: If the first image includes a first object that does not satisfy a predetermined condition indicating that the first object includes text, the first object is rotated and moved; If a second object that satisfies the predetermined condition is included in the first image, the second object is moved without being rotated.

13. An image processing method comprising:

11. A program for causing a computer to function as each of the means of the image processing device according to any one of claims 1 to 9.

Images