Information processing apparatus, information processing method, and program

Abstract

To suppress jitter caused by a CG model that shows different motion from a camera image, and reduce discomfort of a user who views a composite image generated by synthesizing the CG model with the camera image.SOLUTION: An information processing apparatus includes: a determination means which determines position and attitude of a virtual camera; detection means which detects whether at least one of the position and attitude of the virtual camera determined by the determination means is stable or not; and acquisition means which acquires information on the stable position and attitude of the virtual camera, as reference position and attitude information. The determination means updates at least one of detected unstable position and attitude of the virtual camera, based on an attitude difference between the attitude of the virtual camera and the reference attitude.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] Conventionally, methods have been proposed for estimating the position and orientation (hereinafter also referred to as "position and orientation") of a camera that captures real space. One known method for estimating the position and orientation of a camera is the SLAM (Simultaneous Localization and Mapping) method. The SLAM method estimates the three-dimensional position and orientation of a camera by tracking feature points from successive images captured by the camera and estimating the positions of three-dimensional points of the subject in three-dimensional space. The SLAM method includes a method for estimating the positions of three-dimensional points of the subject from multiple images of the subject.

[0003] However, there is a possibility that errors may occur in the position of the subject's 3D points due to differences in the angle at which the subject is viewed in each image frame or due to feature points being hidden. If an error occurs in the position of the subject's 3D points, an error may occur in the estimated camera position or orientation. If the amount of error in the camera position or orientation fluctuates between successive image frames, fluctuations in the position or orientation that differ from the actual camera movement may be observed.

[0004] In addition to the SLAM method, it is also possible to use an external camera measurement method in which markers that are easy to track between images are attached to the surface of the camera to be measured, and a separate observation camera is used to estimate the position and orientation of the camera.Even when using the external camera measurement method, if the distance to the camera to be measured is far, fluctuations in the estimated position and orientation of the camera may be observed between frames due to noise including sampling errors.

[0005] The shaking of the camera (hereinafter referred to as the virtual camera) whose position and orientation have been estimated affects, for example, a mixed reality system that synthesizes an image of a computer graphics (CG) model with a camera image based on the position and orientation of the virtual camera. That is, since the image of the CG model to be synthesized with the camera image is drawn based on the position and orientation of the virtual camera, which includes shaking between frames, a difference occurs in the movement on the screen between the camera image and the CG model. The difference in the movement on the screen between the camera image and the CG model may cause a sense of discomfort to a user observing the synthesized image or may cause sickness in a user observing the synthesized image on a head-mounted display (HMD).

[0006] When the actual camera is substantially stationary, if the CG model moves on the image due to fluctuations in the position and orientation of the virtual camera, the sense of incongruity felt by the user is exaggerated. For example, in a scene where the camera image moves slowly while remaining substantially stationary, the greater the fluctuations in the estimated position and orientation of the virtual camera, the greater the movement of the CG model on the screen. Patent Document 1 discloses a method of detecting whether the camera is stationary, and if the camera is stationary, using the position and orientation information of the previous frame as is. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Japanese Patent Application Publication No. 2018-32155 Summary of the Invention [Problem to be solved by the invention]

[0008] In a scene where the camera image is almost still and moving slowly, the estimated virtual camera position The greater the shaking of the position, the greater the movement of the CG model on the screen compared to the camera image. The difference in the movement on the screen between the camera image and the CG model will be explained with reference to FIGS. 5(A) to 5(D). FIG. 5(A) shows the relative positions of the head of a user 500 wearing an HMD 510, a camera 503, a subject 530, and a CG model 520. FIGS. 5(B) to 5(D) are composite images of the camera image and the CG model 520, showing the state in which the CG model 520 moves in successive composite images 550 to 552 when the camera 503 is stationary. Crosses indicate the positions of feature points extracted in each camera image.

[0009] The movement of the CG model 520 on the screen is caused by fluctuations in the position and orientation of the virtual camera between frames of consecutive composite images 550 to 552. In the case of SLAM, for example, fluctuations in the position and orientation of the virtual camera occur when tracking of a feature point 560 fails due to sampling errors or when the same feature point 560 appears in different locations.

[0010] In this way, even if the camera is stationary, errors in estimating the position and orientation of the virtual camera may cause the CG model 520 to be drawn on the screen in a manner that differs from the camera image. The phenomenon in which the CG model 520 moves differently from the camera image is called "jitter."

[0011] If the position and orientation information of the previous frame is used without updating the position and orientation of the virtual camera, provided that the camera is stationary, the user may feel uncomfortable because the CG model 520 does not move even if the camera body moves slightly and movement occurs in the camera image.

[0012] The present invention aims to suppress jitter, which occurs when a CG model moves differently from a camera image, and reduce the sense of discomfort felt by a user viewing a composite image of a CG model and a camera image. [Means for solving the problem]

[0013] The information processing device of the present invention comprises a determination means for determining a position and attitude of a virtual camera, a detection means for detecting whether at least one of the position and attitude of the virtual camera determined by the determination means is stable, and an acquisition means for acquiring information on the position and attitude of the virtual camera in a stable state as information on a reference position and attitude, wherein when the determination means detects that at least one of the position and attitude of the virtual camera is not stable, the determination means updates at least one of the position and attitude of the virtual camera that is detected to be unstable based on an attitude difference between the attitude of the virtual camera and the reference attitude. [Effects of the Invention]

[0014] According to the present invention, it is possible to suppress jitter, which occurs when a CG model moves differently from a camera image, and reduce the sense of discomfort felt by a user viewing a composite image of a CG model and a camera image. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a block diagram illustrating a configuration of an information processing device according to a first embodiment. [Figure 2] FIG. 4 is a block diagram illustrating a detailed configuration of a stability detection unit. [Figure 3] FIG. 2 is a block diagram illustrating a detailed configuration of a stabilization unit. [Figure 4] FIG. 1 is a diagram illustrating an example of a hardware configuration of an information processing device. [Figure 5] 10A and 10B are diagrams illustrating the difference in movement on the screen between a camera image and a CG model. [Figure 6] 10A and 10B are diagrams illustrating a composite image according to the positional relationship between the imaging unit and the CG model [Figure 7] FIG. 10 is a diagram illustrating updating of the position of the virtual camera. [Figure 8] 10 is a flowchart illustrating a position and orientation update process according to the first embodiment. [Figure 9] 10 is a flowchart illustrating a process of a stability detection unit. [Figure 10] 4 is a flowchart illustrating processing by a stabilization unit according to the first embodiment. [Figure 11] 10A and 10B are diagrams illustrating a method for reducing shaking in the posture of the virtual camera. [Figure 12] 10 is a flowchart illustrating a process of a stabilization unit according to a modified example. [Figure 13] 10A and 10B are diagrams illustrating the effect on the display of a CG model when the imaging unit moves. [Figure 14] FIG. 10 is a block diagram illustrating the configuration of an information processing device according to a second embodiment. [Figure 15] 10 is a flowchart illustrating a position and orientation update process according to the second embodiment. [Figure 16] FIG. 10 is a diagram illustrating an example of a composite image of a camera image and a CG model image after conversion. DETAILED DESCRIPTION OF THE INVENTION

[0016] <Embodiment 1> Hereinafter, embodiments of the present invention will be described with reference to the drawings. The configuration of an information processing device 110 according to the present invention will be described with reference to Figs. 1 to 4. In the first embodiment, an HMD 100, which is an example of an electronic device, includes an information processing device 110, a display unit 102, and an imaging unit 103. Note that the information processing device 110 may be configured as an external device connected to the HMD 100.

[0017] In the scene of Fig. 5(A), when a user 500 wears the HMD 100 according to the first embodiment, the information processing device 110 can reduce shaking of the position of the virtual camera described in Figs. 5(B) to 5(D). Fig. 6(A) is a scene to which the first embodiment is applied, and shows the positional relationship between the head of a user 600 wearing the HMD 100, the imaging unit 103, a subject 630, and a CG model 620. The information processing device 110 can reduce shaking of the position of the virtual camera even in a situation where the camera (imaging unit 103) of the HMD 100 is substantially stationary and moves slowly or slightly.

[0018] 1 is a block diagram illustrating the configuration of an HMD 100 according to embodiment 1. The HMD 100 includes a display unit 102, an imaging unit 103, and an information processing device 110. The information processing device 110 includes a position and orientation determination unit 120, a stability detection unit 130, a stabilization unit 140, a storage unit 150, and an image generation unit 160.

[0019] The display unit 102 is a device for presenting an image to the user 600. The display unit 102 is, for example, a display mounted on the HMD 100 worn by the user 600. The display unit 102 is not limited to a display mounted on the HMD 100, but may also be an external monitor (display device). Furthermore, the display unit 102 may include a display of the HMD 100 and an external monitor, and may display the same image on each.

[0020] The imaging unit 103 may be a color camera that captures a scene, or a monochrome camera. The imaging unit 103 may also be a stereo camera that captures stereo images to be displayed on the display unit 102 of the HMD 100. The imaging unit 103 is fixedly disposed on the HMD 100 and moves in accordance with the movement of the user 600 wearing the HMD 100. The imaging unit 103 associates the captured image with the frame ID of the image and stores the image in the storage unit 150. In the example of FIG. 6(A), a stereo camera is used as the imaging unit 103 to display an image to each eye of the user 600.

[0021] The storage unit 150 stores data used for executing image generation processing by the information processing device 110. The storage unit 150 holds, for example, the following information: (a) CG model information (b) Virtual camera position and orientation information (c) Past virtual camera position and orientation information (d) Past tracking feature point coordinates (e) Virtual camera parameters (f) Camera image (g) Reference camera information (h) Rotation center information (i) Rotation matrix representing the pose difference (j) Composite image (k) Stability analysis execution flag (l) Stability detection results

[0022] The data held by the storage unit 150 is not limited to the above data, and data items may be added according to the functions implemented by the information processing device 110. Each functional unit of the information processing device 110 can read the data held by the storage unit 150 into a RAM (Random Access Memory) 407 described in Fig. 4 and use the data. Each functional unit of the information processing device 110 can also write the results of processing to the storage unit 150 via the RAM 407.

[0023] (a) CG model information includes vertex positions of 3D polygons in a model coordinate system unique to the CG model, edge information, and texture images. The model coordinate system is set in advance as a reference coordinate system that represents the coordinates of the vertices of 3D polygons on the surface of the CG model. The CG model information includes, for example, information for displaying a cubic CG model 620 having the same shape as the subject 630 (cube) shown in FIG. 6(B).

[0024] (b) The virtual camera position and orientation information includes parameters that represent the position and orientation of the virtual camera in a reference coordinate system (world coordinate system). The position and orientation of the virtual camera are determined according to the position and orientation of the image capture unit 103. The parameters that represent the position are X, Y, and Z coordinate values. The parameters that represent the orientation are rotation angles about three axes expressed as Euler angles. The orientation information may be stored as a 3x3 rotation matrix. The storage unit 150 stores the position and orientation information of the virtual camera in association with a frame ID.

[0025] (c) Past virtual camera position and orientation information includes parameters that represent the position and orientation of the virtual camera in a reference coordinate system determined from past camera images. The storage unit 150 stores information on the position and orientation of the virtual camera corresponding to multiple past frames, linked to the respective frame IDs. (d) Past tracking feature point coordinates are information on the coordinates of multiple feature points that were detected in past camera images and became tracked. The storage unit 150 stores information on the coordinates of multiple feature points that became tracked, linked to the frame IDs.

[0026] (e) The virtual camera parameters include a principal point position and a focal length. If the imaging unit 103 is a stereo camera, the virtual camera parameters include parameters that represent the relative positions and orientations of the left and right virtual cameras. (f) The camera images are stored in association with a frame ID. The storage unit 150 stores multiple camera images. If the camera images are stereo camera images, the storage unit 150 may manage the left and right images separately.

[0027] (g) The reference camera information includes information on the camera image, frame ID, and position and orientation of the virtual camera when the image capture unit 103 is determined to be substantially still by the stillness determination by the stability detection unit 130. The position and orientation of the virtual camera in the reference camera information is determined according to the position and orientation of the image capture unit 103 when it is determined to be in a substantially still state. When the image capture unit 103 is in a substantially still state, the stability detection unit 130 can determine that the virtual camera is in a stable state. Hereinafter, the position and orientation of the virtual camera when the virtual camera is in a stable state will be referred to as the reference position and reference orientation of the virtual camera, respectively.

[0028] (h) The rotation center information includes information about the rotation center position and rotation axis when rotating the reference position of the virtual camera in order to reduce shaking of the virtual camera position. The rotation center information includes, for example, offset information of the position and orientation in a camera coordinate system based on the virtual camera. If the imaging unit 103 is a stereo camera, the storage unit 150 may store offset information of the position and orientation in the camera coordinate system of either the left camera or the right camera as the rotation center information. (i) The rotation matrix representing the orientation difference is a rotation matrix representing the orientation difference between the orientation of the virtual camera in the current frame and the reference orientation. (j) The composite image is an image obtained by combining a rendering image of the CG model 620 and a camera image.

[0029] (k) The stability analysis execution flag is the output result of the fluctuation analysis unit 220 described in FIG. 2, and is a flag for setting whether or not to execute stability analysis by the frequency component analysis unit 230. (l) The stability detection result is the output result of the detection unit 240 described in FIG. 2, and includes a position correction flag and an attitude correction flag. The position correction flag is a flag for setting whether or not to correct the position of the virtual camera. The attitude correction flag is a flag for setting whether or not to correct the attitude of the virtual camera.

[0030] The position and orientation determination unit 120 estimates the position and orientation of the imaging unit 103, and determines the position and orientation of the virtual camera according to the estimated position and orientation. The position and orientation determination unit 120 can estimate the position and orientation of the imaging unit 103 using, for example, SLAM. The position and orientation determination unit 120 stores the position and orientation of the virtual camera determined according to the position and orientation of the imaging unit 103 in the storage unit 150.

[0031] Furthermore, the position and orientation determination unit 120 stores tracked feature point coordinates, which are the coordinates of feature points tracked in the camera image, in the storage unit 150. The tracked feature point coordinates are used by the rest-state determination unit 210 described in FIG. 2 to determine whether the imaging unit 103 is in a rest state.

[0032] The stability detection unit 130 detects whether at least one of the position and orientation of the virtual camera determined by the position and orientation determination unit 120 is stable. That is, the stability detection unit 130 detects shaking of the position and orientation of the virtual camera in past consecutive frames captured by the imaging unit 103. If the amount of shaking is greater than a predetermined threshold, the stability detection unit 130 determines that the position and orientation of the virtual camera are unstable. If the amount of shaking is equal to or less than the predetermined threshold, the stability detection unit 130 determines that the position and orientation of the virtual camera are stable. Details of the stability detection unit 130 will be described later with reference to FIG. 2.

[0033] Stabilization unit 140 updates at least one of the position and orientation of the virtual camera that is detected to be unstable based on the orientation difference between the orientation of the virtual camera in the current frame and the reference orientation of the reference camera information. Stabilization unit 140 determines the stabilized position and orientation of the virtual camera using the reference camera information, the position and orientation of the virtual camera in the current frame, rotation center information, and a rotation matrix representing the orientation difference, and outputs the determined position and orientation to storage unit 150. Details of stabilization unit 140 will be described later with reference to FIG. 3.

[0034] The image generation unit 160 renders the CG model 620 based on the CG model information, virtual camera position and orientation information, and virtual camera parameters stored in the storage unit 150. The image generation unit 160 combines the rendered image of the CG model 620 with the camera image stored in the storage unit 150. The image generation unit 160 stores the combined image, obtained by combining the image of the CG model 620 and the camera image, in the storage unit 150.

[0035] 2 is a block diagram illustrating a detailed configuration of stability detection section 130. Stability detection section 130 includes a rest state determination section 210, a fluctuation analysis section 220, a frequency component analysis section 230, and a detection section 240.

[0036] The stillness determination unit 210 acquires past tracked feature point coordinates stored in the storage unit 150 along with the frame ID. The stillness determination unit 210 calculates the standard deviation for each of the acquired multiple coordinate values ​​of the past tracked feature point coordinates (trajectories on the screen of feature points tracked in successive frames), and determines whether the calculated standard deviation is smaller than a predetermined threshold. If the standard deviation of the coordinate values ​​of the tracked feature points in successive frames is smaller than the predetermined threshold, the stillness determination unit 210 can determine that there is no fluctuation (shaking) in the position and orientation of the image capture unit 103, and that the image capture unit 103 is in a substantially still state (the virtual camera is in a stable state).

[0037] For example, the stillness determination unit 210 may determine fluctuations by going back 60 frames (frames per second if the update rate of the imaging unit 103 is 60 FPS) from the current frame among multiple coordinate values ​​of past tracking feature point coordinates. Note that the number of coordinate values ​​(number of frames) of past tracking feature point coordinates used to determine the still state is not limited to 60 frames, and may be more or less than 60 frames, as long as it is possible to determine whether the object is in a still state.

[0038] When it is determined that the imaging unit 103 is in a substantially still state, the stillness determination unit 210 stores reference camera information to be used in the stabilization process by the stabilization unit 140 in the storage unit 150. The reference camera information stored in the storage unit 150 includes, for example, the camera image, frame ID, and position and orientation of the virtual camera (reference position and reference orientation) when it is determined that the imaging unit 103 is in a substantially still state.

[0039] Whether the image capture unit 103 is in a substantially stationary state is not limited to being determined using the coordinate values ​​of past tracked feature points. The stationary state of the image capture unit 103 may be determined using a sensor capable of measuring the position and orientation of the HMD 100, such as an acceleration sensor fixed to the HMD 100. The stationary state determination unit 210 can detect whether the position and orientation of the virtual camera are stable based on the output result of the sensor fixed to the HMD 100.

[0040] Fig. 6(A) shows the positional relationship between the image capturing unit 103 and the CG model 620 when it is determined that the camera is in a substantially stationary state. Fig. 6(B) shows a composite image 650 obtained by combining a camera image and an image of the CG model 620 in the situation shown in Fig. 6(A). Fig. 6(C) shows the positional relationship between the image capturing unit 103 and the CG model 620 when the current frame is captured. Fig. 6(D) shows a composite image 610 obtained by combining a camera image of the current frame and an image of the CG model 620.

[0041] In the composite images 550 to 552 in Figures 5(B) to 5(D), jitter occurs in the CG model 520 even when the image capture unit 103 is stationary. However, in Figure 6(D), even though the position and orientation of the image capture unit 103 are changing slightly, the CG model 620 is displayed near the subject 630 without being affected by fluctuations in the position of the virtual camera. The process of combining the image of the CG model 620 with the camera image without being affected by fluctuations in the position of the virtual camera will be described later with reference to Figure 8.

[0042] The variation analysis unit 220 uses the CG model information and past virtual camera position and orientation information to analyze jitter (also referred to as variation) on the screen when the three-dimensional reference point of the reference CG model is displayed (projected onto the projection surface) based on the past virtual camera position and orientation. The amount of variation of the CG model on the screen is, for example, the standard deviation value of the display position (X, Y coordinate values) of the known three-dimensional reference point displayed on the screen based on the past virtual camera position and orientation.

[0043] If the acquired amount of variation exceeds a predetermined threshold (for example, 2 pixels), the variation analysis unit 220 sets the stability analysis execution flag to ON. If the acquired amount of variation is equal to or less than the predetermined threshold, the variation analysis unit 220 sets the stability analysis execution flag to OFF. The variation analysis unit 220 stores the value of the stability analysis execution flag in the storage unit 150 as the analysis result.

[0044] The known three-dimensional reference point can be, for example, the three-dimensional vertex on the CG model that is closest to the reference position of the virtual camera in the reference camera information. Note that the known three-dimensional reference point is not limited to the three-dimensional vertex that is closest to the reference position of the virtual camera, but can be any point at which fluctuations in the display position on the screen (projection surface) of the virtual camera in the reference camera information can be observed. For example, the known three-dimensional reference point can be the center of gravity of a three-dimensional vertex on the CG model.

[0045] The fluctuation analysis unit 220 can determine that the amount of fluctuation of the known three-dimensional reference point on the screen has increased, even though the stillness determination unit 210 has determined that the current frame is in a substantially still state. The fluctuation analysis unit 220 can determine that the increase in the amount of fluctuation of the three-dimensional reference point has caused shaking in the camera position or attitude determined by the position and attitude determination unit 120, and that the CG model is moving differently (jittering) relative to the camera image. In other words, the stability detection unit 130 can detect whether the position and attitude of the virtual camera are stable based on the amount of fluctuation in the display position of the known three-dimensional reference point.

[0046] The frequency component analysis unit 230 acquires information about the past virtual camera position and orientation from the storage unit 150 and analyzes the frequency components of the virtual camera position and orientation. The frequency component analysis unit 230 separates the position observation values ​​(XYZ values) in consecutive frames and the orientation observation values ​​(roll, pitch, and yaw values) in consecutive frames from the information about the past virtual camera position and orientation. The frequency component analysis unit 230 performs frequency analysis on the position observation values ​​in consecutive frames and the orientation observation values ​​in consecutive frames, respectively, and outputs the analysis results to the detection unit 240. The frequency component analysis unit 230 can perform frequency analysis on the position observation values ​​and the orientation observation values ​​using, for example, a fast Fourier transform (FFT).

[0047] The detection unit 240 individually detects the amount of positional fluctuation and the amount of posture fluctuation in the past position and posture of the virtual camera from the position frequency analysis results and posture frequency analysis results input from the frequency component analysis unit 230. The detection unit 240 detects the extent to which frequency components higher than a predetermined frequency are included in the frequency components of the virtual camera's position and posture. The predetermined frequency is a frequency that is unlikely to occur in human movement, for example, a frequency of 10 Hz or higher. The detection unit 240 can detect whether the position and posture of the virtual camera are stable based on the amount of frequency components higher than the predetermined frequency included in the frequency components of the virtual camera's position and posture.

[0048] The detection unit 240 detects the positional fluctuation and the posture fluctuation of the virtual camera separately based on the amount of frequency components higher than a predetermined frequency included in the frequency components of the past position and posture of the virtual camera. If the amount of positional fluctuation is greater than a threshold (third threshold), the detection unit 240 sets the position correction flag to on, and if the amount of positional fluctuation is equal to or less than the third threshold, the detection unit 240 sets the posture correction flag to on. If the amount of posture fluctuation is greater than a threshold (fourth threshold), the detection unit 240 sets the posture correction flag to on, and if the amount of posture fluctuation is equal to or less than the fourth threshold, the detection unit 240 sets the posture correction flag to off.

[0049] The detection unit 240 outputs the detection result to the storage unit 150. Note that the detection unit 240 is not limited to detecting both the amount of positional fluctuation and the amount of posture fluctuation, and may detect either the amount of positional fluctuation or the amount of posture fluctuation.

[0050] 3 is a block diagram illustrating a detailed configuration of stabilization unit 140. When stability detection unit 130 detects that at least one of the position and orientation of the virtual camera is unstable, stabilization unit 140 updates at least one of the position and orientation of the virtual camera that is detected to be unstable. Stabilization unit 140 includes rotation center determination unit 310, rotation detection unit 320, and conversion unit 330.

[0051] The rotation center determination unit 310 sets a predetermined rotation axis used to stabilize the measurement value of the virtual camera's position when the virtual camera's position is unstable between frames. The predetermined rotation axis passes through a position that has a predetermined positional relationship with the virtual camera at the position and orientation (reference position and reference orientation) when the reference camera information is acquired. The position that has a predetermined positional relationship with the virtual camera is, for example, the rotation center position of the head of the user wearing the HMD 100. The rotation center position of the user's head is set as a three-dimensional position in the camera coordinate system of the virtual camera at the reference position and reference orientation. Furthermore, the direction of the rotation axis that passes through the rotation center position can be, for example, the normal direction of a plane determined based on the rotation center position, the reference position, and the orientation of the virtual camera in the current frame.

[0052] The head rotation center position may be set using the results of measuring the approximate position of the user's head rotation center in advance using a ruler or the like. Furthermore, the head rotation center position does not necessarily have to be set by measuring it in advance, but may also be set by automatically measuring the shape of the user's head. For example, the rotation center determination unit 310 detects the contour of the user's head from an image of the user's head in a reference position and reference posture captured by a camera positioned in the direction of the head's top, and sets the head rotation center position based on the detected contour. The rotation center determination unit 310 can perform a process of fitting an elliptical shape to the detected head contour, and set the intersection point of the major axis and minor axis of the elliptical shape as the head rotation center position. The rotation center determination unit 310 stores the determined rotation center position in the storage unit 150.

[0053] The rotation detection unit 320 acquires the attitude difference between the attitude of the virtual camera in the current frame and the attitude of the virtual camera in the reference camera information (reference attitude). The attitude of the virtual camera in the current frame and the reference attitude are recorded using Euler angles based on the world coordinate system, and each is converted into a 3 × 3 matrix. The attitude difference between the attitude of the virtual camera in the current frame and the reference attitude is calculated by multiplying the inverse matrix of the attitude of the virtual camera in the current frame by the matrix of the reference attitude. The rotation detection unit 320 stores the rotation matrix indicating the calculated attitude difference in the storage unit 150.

[0054] The reference orientation of the virtual camera in the reference camera information and the orientation of the virtual camera in the current frame are each determined based on the orientation of the image capture unit 103 estimated by the position and orientation determination unit 120. Because the head of the user wearing the HMD 100 is substantially stationary, it is assumed that the position of the center of rotation of the head is substantially stationary. For this reason, it can be determined that the position of the center of rotation of the obtained orientation difference substantially coincides with the position of the center of rotation of the head.

[0055] The conversion unit 330 acquires a new virtual camera position that reduces the influence of shaking of the virtual camera position based on the rotation center position, a rotation matrix indicating the orientation difference, and the orientation of the virtual camera for the current frame, all stored in the storage unit 150. The conversion unit 330 acquires, as the new virtual camera position, a position obtained by rotating the virtual camera by the orientation difference indicated by the rotation matrix around a rotation axis that passes through the rotation center position. The conversion unit 330 updates the virtual camera position for the current frame, which is stored in the storage unit 150, to the acquired position.

[0056] Referring to FIG. 7, updating of the position of the virtual camera by the conversion unit 330 will be described. The reference position 730 of the virtual camera represents the reference position of the virtual camera in the reference camera information when it is determined that the image capturing unit 103 is substantially stationary. The conversion unit 330 acquires a new virtual camera position 735 for the current frame in which positional fluctuation has been reduced by rotating the reference position 730 of the reference camera information by a rotation amount 750 around a predetermined rotation axis passing through the rotation center position 700. The ellipse 760 indicates the head of the user 600 when the virtual camera is in the reference position and reference posture. The ellipse 765 indicates the head of the user 600 when the current frame is captured.

[0057] The rotation center position 700 is set based on the shape of an ellipse (ellipse 760 or ellipse 765) corresponding to the outline of the head of the user 600. For example, the rotation center position 700 can be the intersection point where the major axis and minor axis of the ellipse intersect. The rotation amount 750 is the amount of rotation angle when a rotation matrix indicating the posture difference stored in the storage unit 150 is converted into an expression using the rotation angle and the rotation axis. The orientation of the rotation axis can be the normal direction of a plane determined by the reference position 730, the rotation center position 700, and the posture of the virtual camera in the current frame.

[0058] The rotation axis and rotation amount 750 used by the conversion unit 330 are not limited to the example described in Fig. 7. For example, the rotation amount 750 may be an average of the orientation differences between the reference orientation and each of the orientations of the virtual camera in multiple past frames. Furthermore, the rotation amount 750 may be the orientation difference between the reference orientation and the orientation of the HMD 100 (image capture unit 103) measured by an angular velocity sensor or the like capable of measuring the position and orientation of the HMD 100, instead of the orientation of the virtual camera in the current frame.

[0059] The hardware configuration of the information processing device 110 will be described with reference to Fig. 4. Fig. 4 shows an example in which the information processing device 110 is configured integrally with the HMD 100. In the example of Fig. 4, 1, and is connected to an imaging unit 103 via an interface (I / F) 403. The information processing device 110 also includes a CPU (Central Processing Unit) 401, a ROM (Read Only Memory) 402, a storage medium drive 405, an external storage device 406, and a RAM 407. The devices constituting the information processing device 110 are connected to each other via a bus 410.

[0060] When the information processing device 110 is configured as a device separate from the HMD 100, the information processing device 110 may include input devices such as a mouse 408 and a keyboard 409, and may include a monitor 411 as a display device external to the HMD 100. In this case, the display unit 102 is provided in the HMD 100.

[0061] The CPU 401 controls the entire information processing device 110 using programs and data stored in the RAM 407 and the ROM 402. The CPU 401 executes each process of the information processing device 110 by operating as each functional unit described with reference to FIGS.

[0062] The ROM 402 stores setting data and a boot program for the computer. The RAM 407 temporarily stores programs and data loaded from the external storage device 406, the storage medium drive 405, etc. The RAM 407 also temporarily stores data received from an external device such as the imaging unit 103 via the I / F 403. The RAM 407 also functions as a work area used by the CPU 401 when executing various processes. For example, the RAM 407 functions as the storage unit 150 shown in FIG. 1.

[0063] The I / F 403 has an analog video port or a digital input / output port such as IEEE1394 for connecting the imaging unit 103. Data received via the I / F 403 is input to the RAM 407 or the external storage device 406.

[0064] The storage medium drive 405 reads out programs and data recorded on a storage medium such as a CD-ROM or DVD-ROM, and writes programs and data to the storage medium. Note that some or all of the programs and data described as being stored in the external storage device 406 may be recorded on the storage medium. The programs and data read out by the storage medium drive 405 from the storage medium are output to the external storage device 406 or RAM 407.

[0065] The external storage device 406 is a storage device capable of storing large amounts of information, such as a hard disk drive. The external storage device 406 stores an operating system (OS), programs for causing the CPU 401 to execute processing of each functional unit of the information processing device 110, and various data. The various data include data used in processing of each functional unit of the information processing device 110, such as information on CG models and information on the position and orientation of a virtual camera. The programs and various data stored in the external storage device 406 are loaded into the RAM 407 as appropriate under control of the CPU 401. The CPU 401 can realize processing of the information processing device 110 by executing the programs loaded into the RAM 407 using the various data loaded into the RAM 407. The external storage device 406 may function as the storage unit 150 shown in FIG. 1.

[0066] The mouse 408 and keyboard 409 are examples of input devices, and when operated by a user, various instructions can be input to the CPU 401. The monitor 411 is configured with a CRT (Cathode Ray Tube) or a liquid crystal panel, etc. The monitor 411 can display the same image as the display unit 102 of the HMD 100.

[0067] The process of updating the position and orientation of the virtual camera according to the first embodiment will be described with reference to Fig. 8. The information processing device 110 renders a CG model based on the updated position and orientation of the virtual camera, thereby generating a composite image with little discrepancy between the actual camera image and the CG image, thereby reducing the sense of discomfort felt by the user.

[0068] In step S810, the CPU 401 of the information processing device 110 reads the virtual camera parameters and CG model information from the ROM 402, and executes initialization for updating the position and orientation of the virtual camera.

[0069] In step S820, rotation center determination unit 310 of stabilization unit 140 determines the rotation center position. Rotation center determination unit 310 sets a rotation axis that serves as the center when rotating the reference position of the virtual camera and that passes through the determined rotation center position.

[0070] In step S830, the position and orientation determination unit 120 acquires a camera image captured by the imaging unit 103. In step S840, the position and orientation determination unit 120 estimates the position and orientation of the imaging unit 103 based on the camera image. The position and orientation determination unit 120 determines the position and orientation of the virtual camera according to the estimated position and orientation of the imaging unit 103.

[0071] In step S850, the stability detection unit 130 determines whether the number of times the position and orientation of the image capture unit 103 has been estimated in the past (the number of times the position and orientation of the virtual camera has been determined) in step S840 is greater than a predetermined threshold (first threshold). The first threshold is set, for example, based on whether there is enough information on the position and orientation of the virtual camera to determine in step S860 whether the image capture unit 103 is in a stationary state. If the number of times estimation is greater than the first threshold, the process proceeds to step S860. If the number of times estimation is equal to or less than the first threshold, the process proceeds to step S890.

[0072] In step S860, the stillness determination unit 210 determines whether the image capture unit 103 is still, based on information on past tracking feature point coordinates stored in the storage unit 150. If the image capture unit 103 is substantially still, the stillness determination unit 210 stores the camera image of the current frame, the frame ID, and the virtual camera position and orientation (reference position and reference orientation) as reference camera information in the storage unit 150. If it is determined that the image capture unit 103 is substantially still, the process proceeds to step S870. If it is determined that the image capture unit 103 is moving, the process proceeds to step S890.

[0073] In step S870, the stability detection unit 130 detects the past position and orientation of the virtual camera. Based on this, it is detected whether or not there is any shaking in the position or attitude of the virtual camera. The process of stability detection unit 130 in step S870 will be described with reference to FIG.

[0074] 9 is a flowchart illustrating the processing of the stability detection unit 130 in step S870. In step S910, the fluctuation analysis unit 220 uses information about the three-dimensional reference points extracted from the CG model 620 and information about the past virtual camera position and orientation to analyze whether jitter occurs in the CG model 620. The fluctuation analysis unit 220 acquires, as the amount of fluctuation, the standard deviation value of the display position (X, Y coordinate values) of the three-dimensional reference points of the CG model displayed on the screen based on the past virtual camera position and orientation.

[0075] In step S920, the fluctuation analysis unit 220 determines whether the amount of fluctuation acquired in step S910 is greater than a predetermined threshold (second threshold). The second threshold may be, for example, 2 pixels. If the amount of fluctuation is greater than the second threshold, the fluctuation analysis unit 220 sets the stability analysis execution flag to ON and proceeds to step S930. If the amount of fluctuation is equal to or less than the second threshold, the fluctuation analysis unit 220 sets the stability analysis execution flag to OFF and terminates the processing shown in FIG. 9.

[0076] In step S930, the stability analysis execution flag is set to ON, and frequency component analysis section 230 analyzes the frequency components of the past virtual camera position and past virtual camera orientation.

[0077] In step S940, detection unit 240 obtains the amount of frequency components unlikely to be generated by human movement from the frequency components of the past virtual camera position and past virtual camera posture, and detects whether components of position and posture fluctuation are included. The frequency components unlikely to be generated by human movement can be frequency components higher than a predetermined frequency.

[0078] The detection unit 240 sets the position correction flag to ON when the amount of position fluctuation is greater than a threshold (third threshold), and sets the position correction flag to OFF when the amount of position fluctuation is equal to or less than the third threshold. The amount of position fluctuation can be expressed as the amount or ratio of frequency components of the position that are higher than a predetermined frequency.

[0079] If the amount of posture fluctuation is greater than a threshold (fourth threshold), the detection unit 240 sets the posture correction flag to on, and if the amount of posture fluctuation is equal to or less than the fourth threshold, the detection unit 240 sets the posture correction flag to off. The amount of posture fluctuation can be expressed as the amount or proportion of frequency components of the posture that are higher than a predetermined frequency. The detection unit 240 stores the values ​​of the position correction flag and the posture correction flag in the storage unit 150 as detection results.

[0080] In step S880 of Fig. 8, stabilization unit 140 stabilizes the position of the virtual camera in the current frame based on the detection result in step S940 of Fig. 9. The processing of stabilization unit 140 in step S880 will be described with reference to Fig. 10.

[0081] FIG. 10 is a flowchart illustrating the processing of the stabilization unit 140 in step S880 according to the first embodiment. In step S1005, the rotation detection unit 320 determines whether the amount of positional shaking of the virtual camera is greater than a third threshold. The rotation detection unit 320 can determine that the amount of positional shaking of the virtual camera is greater than the third threshold when the position correction flag is on, based on the detection result of the detection unit 240 stored in the storage unit 150. If the amount of positional shaking of the virtual camera is greater than the third threshold, the processing proceeds to step S1010. If the amount of positional shaking of the virtual camera is equal to or less than the third threshold, the processing shown in FIG. 10 ends.

[0082] In step S1010, the rotation detection unit 320 detects the orientation of the virtual camera in the current frame and the base The posture difference from the posture (reference posture) of the virtual camera in the quasi-camera information is acquired. In step S1020, as described in FIG. 7, conversion unit 330 acquires a new virtual camera position that reduces the influence of the position fluctuation of the virtual camera, and updates the position information of the position and posture of the virtual camera in storage unit 150.

[0083] 8, image generation unit 160 renders a CG model based on the updated position and orientation of the virtual camera, and generates a composite image by combining the camera image of the current frame with the image of the CG model. Image generation unit 160 displays the generated composite image on display unit 102.

[0084] In step S895, the information processing device 110 (CPU 401) determines whether or not to end the process shown in Fig. 8. The CPU 401 can determine to end the process if the user performs an operation to instruct the end. If there is no operation to instruct the end, the process returns to step S830. If there is an operation to instruct the end, the process shown in Fig. 8 ends.

[0085] The user can issue an instruction to terminate, for example, to a controller for operating the HMD 100. Furthermore, if the information processing device 110 is configured as a device separate from the HMD 100, the user can issue an instruction to terminate via an input device such as the keyboard 909 described in FIG.

[0086] In the first embodiment described above, when the information processing device 110 detects, from past virtual camera position and orientation information, a fluctuation in the virtual camera position that induces jitter in the CG model, it acquires a virtual camera position that reduces the fluctuation in the position. Specifically, the information processing device 110 acquires a position obtained by rotating the reference position of the virtual camera by an orientation difference between the orientation of the virtual camera in the current frame and the reference orientation, around a predetermined rotation axis that passes through a rotation center position that has a predetermined positional relationship with the virtual camera. The information processing device 110 can generate a composite image in which the deviation between the actual camera image and the CG image is reduced by combining an image of the CG model rendered based on the acquired virtual camera position with the camera image in the current frame. Therefore, the information processing device 110 can reduce the sense of discomfort felt by the user.

[0087] <Modification> In the first embodiment, the information processing device 110 reduces shaking in the position of the virtual camera. In contrast, in the modified example, the information processing device 110 reduces shaking in the attitude of the virtual camera. The configuration of the information processing device 110 according to the modified example is the same as the configuration of the information processing device 110 according to the first embodiment described with reference to FIGS. 1, 2, and 4. The stabilization unit 140 according to the modified example does not need to include the rotation center determination unit 310 and the rotation detection unit 320 according to the first embodiment shown in FIG. 3. Below, the processing of the stabilization unit 140 that differs from that of the first embodiment will be described. Note that the processing of the other components is the same as that of the first embodiment, so description thereof will be omitted.

[0088] Stabilization unit 140 acquires the stabilized position and orientation of the virtual camera based on the reference camera information and the position and orientation information of the virtual camera for the current frame stored in storage unit 150, and updates the position and orientation of the virtual camera stored in storage unit 150. In the first embodiment, stabilization unit 140 updates the position of the virtual camera to reduce positional fluctuation. On the other hand, in a modified example, stabilization unit 140 updates the orientation of the virtual camera to reduce orientation fluctuation.

[0089] The conversion unit 330 of the stabilization unit 140 acquires a new virtual camera posture with reduced posture fluctuation, using the posture difference between the reference posture of the reference camera information and the posture of the virtual camera in the current frame. The conversion unit 330 can acquire the posture difference between the reference posture of the virtual camera and the posture of the virtual camera in the current frame, using the measurement values ​​(the posture of the image capture unit 103 in a substantially stationary state and the posture of the image capture unit 103 in the current frame) of the angular velocity sensor fixed to the HMD 100. It should be noted that the orientation of the image capturing unit 103 (HMD 100) is not limited to being measured using an angular velocity sensor fixed to the HMD 100, and may be measured using an external sensor.

[0090] FIG. 11 is a diagram illustrating a method for reducing shaking in the posture of the virtual camera. A projection plane 1140 is determined by a reference position 1130 of the virtual camera and the reference posture of the virtual camera when the reference camera information is acquired. On the other hand, a projection plane 1110 is a projection plane obtained by combining a virtual camera position 1135 when the current frame is captured and the reference posture of the virtual camera. An ellipse 1160 indicates the head of the user 600 when the virtual camera is in the reference position and posture. An ellipse 1165 indicates the head of the user 600 when the current frame is captured.

[0091] The conversion unit 330 acquires the orientation difference 1150 (amount of change in orientation) which is the double integral value of the angular velocity sensor value from the capture time of the camera image when the reference camera information was acquired to the capture time of the current frame. The conversion unit 330 adds the orientation difference 1150 to the reference orientation of the reference camera representing the projection surface 1110 to acquire the virtual camera orientation of the current frame representing the projection surface 1120.

[0092] Except for the stabilization process in step S880, the process of updating the position and orientation of the virtual camera according to the modified example is the same as the process according to the first embodiment described with reference to Fig. 8. A description of the same processes as those in the first embodiment will be omitted.

[0093] FIG. 12 is a flowchart illustrating the processing of stabilization unit 140 in step S880 according to a modified example. In step S1210, conversion unit 330 determines whether the amount of posture shake of the virtual camera is greater than a fourth threshold. Conversion unit 330 can determine that the amount of posture shake of the virtual camera is greater than the fourth threshold when the posture correction flag is on, based on the detection result of detection unit 240 stored in storage unit 150. If the amount of posture shake of the virtual camera is greater than the fourth threshold, the processing proceeds to step S1220. If the amount of posture shake of the virtual camera is equal to or less than the fourth threshold, the processing shown in FIG. 12 ends.

[0094] In step S1220, conversion unit 330 acquires a new virtual camera orientation that reduces the influence of shaking of the virtual camera orientation, as described with reference to FIG. 11, and updates the orientation information among the position and orientation of the virtual camera in storage unit 150.

[0095] In the above-described modification, when the information processing device 110 detects, from past virtual camera position and orientation information, a fluctuation in the virtual camera's orientation that induces jitter in the CG model, the information processing device 110 acquires a virtual camera orientation that reduces the fluctuation in orientation. Specifically, the information processing device 110 acquires an orientation obtained by rotating the reference orientation of the virtual camera by an orientation difference between the virtual camera's orientation in the current frame and the reference orientation. The information processing device 110 can generate a composite image in which the deviation between the actual camera image and the CG image is reduced by combining the image of the CG model rendered based on the acquired virtual camera orientation and the virtual camera position in the current frame with the camera image in the current frame. Therefore, the information processing device 110 can reduce the sense of discomfort felt by the user.

[0096] Note that the information processing device 110 is not limited to either the process of reducing the shaking of the position of the virtual camera described in the first embodiment or the process of reducing the shaking of the attitude of the virtual camera described in the modified example, and may execute both processes.

[0097] <Embodiment 2> In the first embodiment, the information processing device 110 reduces the shaking of the CG model image in the camera image by reducing the shaking of the position and posture of the virtual camera. In the second embodiment, from the time when the reference camera information is acquired (when the image capturing unit 103 is determined to be in a substantially stationary state), This is an embodiment for reducing the influence when the sensor 103 moves.

[0098] 13(A) to 13(D), the effect on the display of the CG model when imaging unit 103 moves from the time when reference camera information was acquired will be described. When imaging unit 103 moves, the movement of CG model 620 placed at the position of subject 630 will no longer match the movement of subject 630 on the camera image, which may cause the user to feel uncomfortable.

[0099] Fig. 13(A) shows a composite image 1350 when the reference camera information is acquired. Fig. 13(B) shows the positional relationship between the reference position 1330 of the virtual camera, the projection surface 1340, the CG model 620, and the subject 630 when the reference camera information is acquired. An ellipse 1310 indicates the head of the user 600 when the virtual camera is in the reference position and reference posture.

[0100] Fig. 13(C) shows a composite image 1351 obtained by combining the camera image of the current frame and the image of CG model 620 using the processing of embodiment 1. Fig. 13(D) shows the positional relationship between reference position 1330 of the virtual camera when reference camera information is acquired, virtual camera position 1335 when the current frame is captured, and CG model 620.

[0101] Virtual camera position 1325 is the position of the virtual camera updated by stabilization unit 140 in embodiment 1. In other words, virtual camera position 1325 is a position obtained by rotating reference position 1330 of the virtual camera around a predetermined rotation axis passing through rotation center position 1300 by the orientation difference between the orientation of the virtual camera in the current frame and the reference orientation. Ellipse 1315 indicates the head of user 600 when the current frame was captured.

[0102] Virtual camera position 1335 in the current frame has moved away from CG model 620 compared to when virtual camera reference position 1330 was acquired. Therefore, in composite image 1351, subject 630 shown in the camera image is smaller than CG model 620. If virtual camera position 1335 in the current frame moves from when reference camera information was acquired in accordance with movement of imaging unit 103, the generated composite image will give the user a sense of discomfort.

[0103] In the second embodiment, the information processing device 110 not only updates the position of the virtual camera as described in the first embodiment, but also applies transformation to the camera image in accordance with the movement of the imaging unit 103. Specifically, the information processing device 1410 transforms the camera image of the current frame into an image viewed from the virtual camera in the position and orientation updated by the stabilization unit 140, based on the positions of the known three-dimensional reference points. The camera image of the current frame is a camera image captured at the current position and orientation of the imaging unit 103 of the HMD 1400. By transforming the camera image of the current frame in accordance with the movement of the imaging unit 103, the CG model 620 moves in the camera image in accordance with the movement of the subject 630, reducing the sense of discomfort felt by the user.

[0104] 14 is a block diagram illustrating the configuration of an HMD 1400 according to embodiment 2. The same components as those in the HMD 100 according to embodiment 1 are denoted by the same reference numerals and will not be described. The information processing device 1410 according to embodiment 2 includes an image conversion unit 1420 in addition to the components included in the information processing device 110 according to embodiment 1.

[0105] Image conversion unit 1420 converts the camera image viewed from the position and orientation of the virtual camera for the current frame into a camera image viewed from virtual camera position 1335 updated by stabilization unit 140 and the orientation of the virtual camera for the current frame. In the example of Fig. 13(D) , image conversion unit 1420 converts the camera image displayed on projection plane 1345 for the current frame into a camera image displayed on projection plane 1320 viewed from virtual camera position 1325 and the orientation of the virtual camera for the current frame.

[0106] The image conversion unit 1420 converts the camera image displayed on the projection surface 1345 into a camera image displayed on the projection surface 1320, for example, by the following procedure. (1) The position difference is calculated from the double integral of the acceleration sensor value in the interval between when the reference camera information was acquired and when the current frame was captured. (2) The difference in position obtained in (1) is added to the reference position 1330 of the virtual camera in the reference coordinate system to obtain the camera position (virtual camera position 1335) of the image capturing unit 103 for the current frame. (3) Using the camera image of the current frame captured by the imaging unit 103 and the information on the virtual camera posture acquired by the position and posture determination unit 120, a group A of coordinate values ​​is acquired when multiple vertices of the CG model 620 are projected onto the projection surface 1345. The multiple vertices of the CG model 620 correspond to known three-dimensional reference points. (4) Using the virtual camera position 1325 updated by the stabilization unit 140 and the posture of the virtual camera in the current frame, a group of coordinate values ​​B is obtained when the multiple vertices of the CG model 620 projected onto the projection surface 1345 are projected onto the projection surface 1320. (5) Estimate a homography transformation matrix H that transforms the coordinate value group A into the coordinate value group B. (6) The image of the current frame captured by the image capturing unit 103 is transformed by the homography transformation matrix H estimated in (5) to generate a camera image on the projection plane 1320.

[0107] The process of updating the position and orientation of the virtual camera according to the second embodiment will be described with reference to Fig. 15. The same processes as those in the process of updating the position and orientation of the virtual camera according to the first embodiment in Fig. 8 are denoted by the same numbers, and the description thereof will be omitted. In the second embodiment, the information processing device 1410 performs the image conversion process in step S1510 after the stabilization process in step S880.

[0108] In step S1510, the image conversion unit 1420 converts the camera image of the current frame from the virtual camera position 1325 updated by the stabilization unit 140 into an image viewed from the virtual camera posture of the current frame, as described in steps (1) to (6) above.

[0109] 16 shows an example of a composite image 1600 of the camera image converted in step S1510 and the image of the CG model 620. The CG model 620 and the subject 630 are displayed in a state where the influence of the movement of the image capturing unit 103 is reduced.

[0110] Region 1610 is a region that does not correspond to the camera image of the current frame transformed by the homography transformation process in the image viewed from the position of the virtual camera updated by stabilization unit 140. Region 1610 may be filled with a preset background color, or may be filled with a color based on the luminance values ​​of the camera image of the current frame (for example, the average value of the luminance values), in order to reduce the sense of incongruity of the image presented to the user.

[0111] In the above-described second embodiment, even if the imaging unit 103 is determined to be substantially stationary and moves after the reference camera information is acquired, the information processing device 1410 converts the camera image of the current frame in accordance with the movement of the imaging unit 103. That is, the information processing device 1410 uses the stabilization unit 140 to reduce and stabilize at least one of the positional fluctuation and the attitude fluctuation of the virtual camera, and converts the camera image of the current frame to match the position and attitude of the stabilized virtual camera. This allows the information processing device 1410 to further reduce the sense of discomfort felt by the user regarding the appearance of the CG model 620.

[0112] Although the embodiments of the present invention (including modifications) have been described above, the present invention is not limited to these embodiments and various modifications and changes are possible within the scope of the gist of the present invention. Furthermore, the present invention also includes configurations obtained by appropriately combining the configurations of the above embodiments.

[0113] <Other embodiments> The present invention is realized by a computer (or a CPU or MPU) of a system or device reading and executing software program code from a storage medium on which the program code for implementing the functions of each embodiment is recorded. In this case, the program code read from the storage medium itself implements the functions of each of the above-described embodiments, and the storage medium storing the program code constitutes the present invention. When the present invention is applied to a storage medium, the storage medium stores program code corresponding to the flowcharts described above.

[0114] The functions of each of the above embodiments can also be realized by an operating system (OS) running on a computer performing some or all of the actual processing based on instructions in program code read by the computer.

[0115] Furthermore, the program code read from the storage medium may be written to a memory provided in a function expansion card inserted into the computer or a function expansion unit connected to the computer. The functions of each of the above embodiments may also be realized by a CPU provided in the function expansion card or function expansion unit performing some or all of the actual processing based on instructions from the program code.

[0116] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0117] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) a determining means for determining a position and orientation of a virtual camera; a detection means for detecting whether or not at least one of the position and the orientation of the virtual camera determined by the determination means is stable; an acquisition means for acquiring information on the position and orientation of the virtual camera in a stable state as information on the reference position and orientation; and When it is detected that at least one of the position and the orientation of the virtual camera is unstable, the determining means updates at least one of the position and the orientation of the virtual camera that is detected as unstable based on an orientation difference between the orientation of the virtual camera and the reference orientation. 1. An information processing device comprising: (Configuration 2) The determining means updates the position of the virtual camera to a position obtained by rotating the reference position by the attitude difference around a predetermined rotation axis passing through a position having a predetermined positional relationship with the virtual camera. 2. The information processing device according to configuration 1, (Configuration 3) The predetermined rotation axis is set based on the outline of the user's head detected from an image of the user taking the reference position and posture, taken from the direction of the top of the head. 3. The information processing device according to configuration 2. (Configuration 4) The determining means updates the orientation of the virtual camera to an orientation obtained by rotating the reference orientation by the orientation difference. 2. The information processing device according to configuration 1, (Configuration 5) The detecting means detects the virtual camera based on the amount of change in the display position of the known three-dimensional reference point. Detect whether the position and orientation of 5. The information processing device according to any one of configurations 1 to 4. (Configuration 6) the determining means determines a position and an orientation of the virtual camera in accordance with a position and an orientation of an electronic device; The detecting means detects whether the position and orientation of the virtual camera are stable based on an output result of a sensor capable of measuring the position and orientation of the electronic device. 5. The information processing device according to any one of configurations 1 to 4. (Configuration 7) The detection means detects whether the position and orientation of the virtual camera are stable based on the amount of frequency components higher than a predetermined frequency included in the frequency components of the position and orientation of the virtual camera. 5. The information processing device according to any one of configurations 1 to 4. (Configuration 8) the determining means determines a position and an orientation of the virtual camera in accordance with a position and an orientation of an electronic device; The present invention further includes a conversion means for converting a camera image captured at the current position and orientation of the electronic device into an image seen from the virtual camera after update by the determination means, based on the positions of known three-dimensional reference points. 8. The information processing device according to any one of configurations 1 to 7. (Configuration 9) The transformation means estimates a homography transformation matrix based on the coordinates of the three-dimensional reference points in the camera image and the coordinates of the three-dimensional reference points as seen from the virtual camera after being updated by the determination means, and transforms the camera image using the homography transformation matrix. 9. The information processing device according to configuration 8. (Configuration 10) The conversion means fills in an area in the image seen from the virtual camera after update by the determination means that does not correspond to the camera image with a preset background color or a color based on a luminance value of the camera image. 10. The information processing device according to configuration 8 or 9. (Configuration 11) the determining means determines a position and an orientation of the virtual camera in accordance with a position and an orientation of an electronic device; The attitude difference is a difference between the attitude of the electronic device and the reference attitude. 11. The information processing device according to any one of configurations 1 to 10. (method) a determining step of determining the position and orientation of the virtual camera; a detecting step of detecting whether or not at least one of the position and the orientation of the virtual camera determined in the determining step is stable; an acquisition step of acquiring information on the position and orientation of the virtual camera in a stable state as information on the reference position and orientation; and In the determining step, when it is detected that at least one of the position and the orientation of the virtual camera is unstable, at least one of the position and the orientation of the virtual camera that is detected as unstable is updated based on an orientation difference between the orientation of the virtual camera and the reference orientation. 1. An information processing method comprising: (program) 12. A program for causing a computer to function as each means of the information processing device according to any one of configurations 1 to 11. [Explanation of symbols]

[0118] 110: Information processing device, 120: Position and orientation determination unit, 130: Stability detection unit, 140: Stabilization unit, 210: Stillness determination unit, 330: Conversion unit

Claims

1. a determining means for determining a position and orientation of a virtual camera; a detection means for detecting whether or not at least one of the position and the orientation of the virtual camera determined by the determination means is stable; an acquisition means for acquiring information on the position and orientation of the virtual camera in a stable state as information on the reference position and orientation; and When it is detected that at least one of the position and the orientation of the virtual camera is unstable, the determining means updates at least one of the position and the orientation of the virtual camera that is detected as unstable based on an orientation difference between the orientation of the virtual camera and the reference orientation.

1. An information processing device comprising:

2. The determining means updates the position of the virtual camera to a position obtained by rotating the reference position by the attitude difference around a predetermined rotation axis passing through a position having a predetermined positional relationship with the virtual camera.

2. The information processing apparatus according to claim 1, wherein:

3. The predetermined rotation axis is set based on the outline of the user's head detected from an image of the user taking the reference position and posture, taken from the direction of the top of the head.

3. The information processing apparatus according to claim 2, wherein:

4. The determining means updates the orientation of the virtual camera to an orientation obtained by rotating the reference orientation by the orientation difference.

2. The information processing apparatus according to claim 1, wherein:

5. The detecting means detects whether the position and orientation of the virtual camera are stable based on the amount of fluctuation in the display position of a known three-dimensional reference point.

2. The information processing apparatus according to claim 1, wherein:

6. the determining means determines a position and an orientation of the virtual camera in accordance with a position and an orientation of an electronic device; The detecting means detects whether the position and orientation of the virtual camera are stable based on an output result of a sensor capable of measuring the position and orientation of the electronic device.

2. The information processing apparatus according to claim 1, wherein:

7. The detection means detects whether the position and orientation of the virtual camera are stable based on the amount of frequency components higher than a predetermined frequency included in the frequency components of the position and orientation of the virtual camera.

2. The information processing apparatus according to claim 1, wherein:

8. the determining means determines a position and an orientation of the virtual camera in accordance with a position and an orientation of an electronic device; The present invention further includes a conversion means for converting a camera image captured at the current position and orientation of the electronic device into an image seen from the virtual camera after update by the determination means, based on the positions of the known three-dimensional reference points.

2. The information processing apparatus according to claim 1, wherein:

9. The conversion means converts the coordinates of the three-dimensional reference point in the camera image and the determination means a homography transformation matrix is ​​estimated based on the coordinates of the three-dimensional reference point as seen from the virtual camera after the update by the step, and the camera image is transformed using the homography transformation matrix.

9. The information processing apparatus according to claim 8,

10. The conversion means fills in an area in the image seen from the virtual camera after the update by the determination means that does not correspond to the camera image with a preset background color or a color based on a luminance value of the camera image.

9. The information processing apparatus according to claim 8,

11. the determining means determines a position and an orientation of the virtual camera in accordance with a position and an orientation of an electronic device; The attitude difference is a difference between the attitude of the electronic device and the reference attitude.

2. The information processing apparatus according to claim 1, wherein:

12. a determining step of determining the position and orientation of the virtual camera; a detecting step of detecting whether or not at least one of the position and the orientation of the virtual camera determined in the determining step is stable; an acquisition step of acquiring information on the position and orientation of the virtual camera in a stable state as information on the reference position and orientation; and In the determining step, when it is detected that at least one of the position and the orientation of the virtual camera is unstable, at least one of the position and the orientation of the virtual camera that is detected as unstable is updated based on an orientation difference between the orientation of the virtual camera and the reference orientation.

1. An information processing method comprising:

13. A program for causing a computer to function as each of the means of the information processing device according to any one of claims 1 to 11.

Images