Information processing device, control method thereof, and program

Abstract

An information processing device that allows a user to obtain a recorded video of a highly accurate image of a virtual space, a control method therefor, and a program are provided. [Solution] When generating a recorded video of a virtual space image in which CG objects are rendered in conjunction with the movements of a user wearing an HMD (101), the HMD (101) acquires rendered images that are displayed at a constant framing rate on a display unit on which the user views the virtual space image, acquires recorded images as images of one frame of the recorded video, acquires image information corresponding to the recorded images, saves the recorded image information that associates the recorded images with the image information for each frame, and generates the recorded video using the recorded image information. When acquiring the rendered images, the HMD (101) generates and corrects rendering images of the CG objects using the acquired user position and orientation information, and acquires the rendering images as recorded images.

Description

[Technical Field]

[0001] The present invention relates to an information processing device, a control method thereof, and a program, and more particularly to an information processing device, a control method thereof, and a program that enable video recording of a virtual space. [Background technology]

[0002] In recent years, virtual reality (VR) technology has become popular, allowing users to experience a highly immersive experience by wearing a head-mounted display (HMD). In VR virtual spaces, users can immerse themselves in game worlds, communicate with other users, and participate in events such as live concerts. In addition, users can also record video of the virtual space.

[0003] On the other hand, the images of the virtual space displayed on the HMD are CG images, and high-quality and high-definition CG rendering is required. When users wear an HMD and view such CG images, they may experience visually induced motion sickness for various reasons. To mitigate this, various measures have been implemented in HMDs.

[0004] For example, in Patent Document 1, a composite image in which a virtual object is superimposed on a real-life image is displayed on an HMD for each frame period. However, if the user's head moves rapidly, the composite image becomes distorted, which can lead to visually induced motion sickness. Specifically, if the calculation accuracy of the position at which the virtual object should be superimposed in a certain frame period is below a threshold and the amount of movement of the user's head exceeds a threshold, a composite image generated in a frame period prior to that frame period is displayed on the HMD. This prevents a composite image with incorrect superimposition positioning from being displayed on the HMD, thereby reducing visually induced motion sickness.

[0005] Furthermore, the HMD disclosed in Patent Document 2 takes measures to prevent visually-induced motion sickness caused by a misalignment between the image and the orientation of the user's head when a rendering delay occurs in the image displayed on the HMD. Specifically, when a rendering delay occurs at a second time, the image at the first time is translated or rotated according to the difference between the position and orientation of the HMD at the first time before the rendering delay occurred and the position and orientation of the HMD at the second time, and displayed as the image at the second time. In this way, by correcting the image at a past time and displaying it as the image at the time when the rendering delay occurred, visually-induced motion sickness caused by the rendering delay is reduced. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Patent No. 5303685 [Patent Document 2] Patent No. 6944863 Summary of the Invention [Problem to be solved by the invention]

[0007] However, in Patent Document 1, frames from the past are used as recorded frames depending on the calculation accuracy of the superimposed display position of the virtual object and the amount of movement of the user's head, so the same frame may be displayed multiple times in succession in the video. In such cases, the virtual reality world appears to be stationary to the user wearing the HMD, resulting in a problem of reduced immersion.

[0008] Furthermore, in Patent Document 2, the image at the first time is moved or rotated in response to a change in the position of the HMD and displayed on the HMD as an image at the second time, and as a result of these processes, there may be areas in the image at the second time where no drawn image exists. In this case, there is a problem in that the areas in the image at the second time where no drawn image exists are displayed on the HMD as blacked out, for example.

[0009] Here, in order to reduce visually-induced motion sickness, images displayed on the HMD are required to be immediate, even if this causes problems such as those described in Patent Documents 1 and 2. On the other hand, when recording video of a virtual space, immediacy is not required, but accuracy is required to prevent problems such as those described in Patent Documents 1 and 2. Accuracy here refers to the absence of image quality degradation or image loss due to image processing in the video frames of the recorded video.

[0010] The present invention has been made in consideration of such problems, and aims to provide an information processing device, a control method thereof, and a program that allow users to obtain recorded video of highly accurate images of virtual space. [Means for solving the problem]

[0011] In order to solve the above problem, the information processing device according to claim 1 of the present invention is an information processing device that generates a recorded video of an image of a virtual space in which CG objects are drawn in conjunction with the movement of a user, and is characterized by comprising: a drawing image acquisition means that acquires a drawing image that is displayed at a constant framing rate on a display unit on which the user views the image of the virtual space; a recorded image acquisition means that acquires a recorded image as an image of one frame of the recorded video; an image information acquisition means that acquires image information corresponding to the recorded image; a recorded image information storage means that associates the recorded image and the image information for each frame and stores them as recorded image information; and a video generation means that generates the recorded video using the recorded image information, wherein the drawing image acquisition means comprises: a generation means that acquires position and orientation information of the user, and generates a rendering image of the CG object using the acquired position and orientation information; and a correction means that performs a correction process on the rendering image, and the recorded image acquisition means sets the rendering image as the recorded image. [Effects of the Invention]

[0012] According to the present invention, a user can obtain a recorded video of a highly accurate image of a virtual space. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a block diagram showing a hardware configuration of an HMD as an information processing device according to a first embodiment. [Figure 2] FIG. 2 is a functional block diagram showing the software configuration of the HMD. [Figure 3] 1A to 1C are diagrams illustrating a conventional method for generating drawn images and recorded images that is executed when a delay occurs in the drawing process while a user is viewing a virtual space with an HMD. [Figure 4] 10 is a flowchart showing a series of steps in a recording image generation process according to the first embodiment. [Figure 5A] 3A to 3C are diagrams illustrating a method for generating a drawn image and a recorded image according to the first embodiment. [Figure 5B] 3 is a diagram showing an example of recorded image information stored in a recorded image information storage unit in FIG. 2 according to the first embodiment. FIG. [Figure 6A] 10A to 10C are diagrams illustrating a method for generating a drawn image and a recorded image according to a second embodiment. [Figure 6B] FIG. 10 is a diagram showing an example of recorded image information stored in a recorded image information storage unit according to the second embodiment. [Figure 7] 10 is a flowchart showing a series of steps in a recording image generation process according to a second embodiment. [Figure 8A] 10A to 10C are diagrams illustrating a frame interpolation process according to a second embodiment. [Figure 8B] 10A and 10B are diagrams showing an example of recorded image information stored in a recorded image information storage unit after frame interpolation processing according to the second embodiment. [Figure 9] 8 is a flowchart of the moving image generation process in step S806 of FIG. 7. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, embodiments of the present invention will be described with reference to the drawings. The following embodiments do not limit the present invention, and not all of the combinations of features described in the present embodiments are necessarily essential to the solution of the present invention. The configurations of the embodiments may be modified or changed as appropriate depending on the specifications of the device to which the present invention is applied and various conditions (such as usage conditions and usage environment). Furthermore, a configuration may be achieved by appropriately combining parts of each of the embodiments described below. In the following embodiments, the same components will be described with the same reference symbols.

[0015] First Embodiment In the first embodiment, a case will be described with reference to the drawings in which the information processing device according to the present invention is the internal configuration of a standalone HMD (head mounted display) worn by a user (hereinafter simply referred to as HMD 101). However, the information processing device according to the present invention is not limited to the HMD 101 itself. For example, the information processing device according to the present invention may be a device included in equipment such as a PC or a small computer box that is connected to a standalone HMD and performs CG image generation processing.

[0016] FIG. 1 is a block diagram showing the hardware configuration of the HMD 101, and FIG. 2 is a functional block diagram showing the software configuration of the HMD 101.

[0017] First, a conventional method for generating drawn images and recorded images, which is executed when a delay occurs in CG drawing processing while the user is viewing a virtual space on the HMD 101, will be described with reference to FIG.

[0018] 3 shows the relationship between movement 350 of the head 301 of a user wearing the HMD 101, a timing chart 360, a drawn image 370, and a recorded image 380. As an example, the user wearing the HMD 101 rotates the head 301 from left to right while viewing a virtual space.

[0019] Head movement 350 indicates the position of HMD 101 as seen from above head 301. In this figure, the user is turning from left to right, and a drawing image 370 generated by CG drawing processing and correction processing linked to this is displayed on HMD 101.

[0020] In this figure, time progresses from left to right, and the head 301 is in rotational motion.

[0021] Timing chart 360 shows the execution timing of the CG rendering process and the correction process. Drawn image 370 is a collective term for drawn images 311-314, which are CG objects 303 and 304 rendered at a constant framing rate according to the position and orientation of head 301. Recorded image 380 is a collective term for a recorded image (identical to drawn images 311-314 in this case) representing one frame of a recorded video of a virtual space in which CG objects are rendered in conjunction with the movement of head 301. The correction process referred to here is a process involving image deformation and movement. Correction process may result in a decrease in image quality or loss of image. Because conventional technology is applied in Figure 3, when correction process is performed on drawn image 370, areas with image loss, such as drawn image 312, are displayed as blacked-out areas.

[0022] 3, if the CG rendering process is not completed within the time frame of one frame, a process called ATW (Asynchronous Timewarp) is performed as a correction process. This process cuts out the rendered image of a past frame according to the position and orientation information of the past frame and the position and orientation information of the frame to be corrected, and performs at least one of the following processes: cutting, transforming, and moving, and the resulting image is used as the rendered image of the frame to be corrected.

[0023] When the CG rendering process is completed within the frame period, as in frame 1, the image rendered by the CG rendering process (drawn image 311) is displayed on the HMD 101 as a drawn image 370.

[0024] However, when the CG rendering process is delayed and cannot be completed within the frame period, as in frame 2, the image rendered by the CG rendering process will not be ready in time for display as rendered image 370. In this case, an image corrected from rendered image 370 of the immediately preceding frame 1 (drawn image 311 corresponding to time (t0)) in accordance with the positions and orientations of the user's head 301 at time (t0) and time (t1) is generated as rendered image 312 and displayed on HMD 101. As a result, the area of ​​rendered image 312 outside rendered image 311 is blacked out due to image loss.

[0025] In addition, in the conventional example shown in Figure 3, when a user wearing HMD 101 records a video of the image they are experiencing in real time in a virtual space, an image identical to drawn image 370 (drawn images 311 to 314) is saved as recorded image 380 obtained by video recording.

[0026] In contrast to such conventional examples, in this embodiment, a highly accurate image that does not include an image with image defects such as the drawn image 312 is obtained as a recorded image.

[0027] The configuration and operation of the HMD 101 will be described below.

[0028] FIG. 1 is a block diagram showing the hardware configuration of the HMD 101.

[0029] The HMD 101 generates and transmits images of a VR space or an MR space in which virtual objects generated using computer graphics (CG) technology are placed on the HMD 101 worn by the user. Note that since VR technology and MR technology are existing technologies, detailed explanations of them will be omitted here.

[0030] In FIG. 1, the HMD 101 includes a CPU 102, a ROM 103, a RAM 104, a recording unit 105, an input detection unit 106, a communication I / F 107, a display unit 108, and an operation unit 109.

[0031] The CPU 102 is a system control unit that controls the entire HMD 101. The CPU 102 also executes an information processing program to realize various processes according to this embodiment.

[0032] The ROM 103 is a read-only memory that stores programs and parameters that do not require modification, such as basic programs and initial data.

[0033] The RAM 104 is a memory for temporarily storing input information, calculation results in information processing, image processing, and the like.

[0034] The recording unit 105 is a device capable of writing and reading various types of information. Examples of the recording unit 105 include a hard disk or memory card built into or external to the HMD 101, or a memory card, flexible disk, or IC card detachable from the HMD 101. The information processing program according to this embodiment is recorded in the recording unit 105, and the information processing program is read from the recording unit 105, loaded into the RAM 104, and executed by the CPU 102. The information processing program may also be stored in the ROM 103. The recording unit 105 can also record information on user operation instructions acquired by the operation unit 109, user position and orientation information detected by the external sensing unit 110, and recorded image information 205 (FIG. 2) described later.

[0035] The input detection unit 106 receives required data input. In this embodiment, the required data includes a video recording instruction input by operating a button on the operation unit 109.

[0036] The communication I / F 107 is an interface unit capable of transmitting and receiving data to and from an external operation device (not shown) or a cloud. In this embodiment, the communication I / F 107 can receive, via a network, position and orientation information of a user detected by an external sensing unit 110. The connection form of input and output by the communication I / F 107 includes both a local connection via USB, Bluetooth, or the like, and an internet connection via Ethernet or Wi-Fi.

[0037] The display unit 108 is an electronic display device such as a liquid crystal display device mounted on the HMD 101. The display unit 108 of this embodiment is configured integrally with an operation unit 109 that accepts user instructions on the display by hand tracking (not shown), and the CPU 102 displays an image of the VR space based on information from the recording unit 105. Note that the example in FIG. 1 is based on the premise that processing is performed by the HMD 101, and therefore the operation unit 109 and display unit 108 are also included as internal components of the HMD 101, but this is not limiting. For example, if a PC is the information processing device according to this embodiment, the HMD 101 including the operation unit 109 and display unit 108 is externally connected to the PC.

[0038] The operation unit 109 receives pointing operations by the user using a hand tracking function, receives input operations to an operation device capable of inputting various commands, etc., and acquires user instructions according to the received operations. In the present embodiment, an example of a user instruction acquired by the operation unit 109 is an instruction to record a video of an image in a virtual space. Information about this instruction to record a video is sent from the operation unit 109 to the input detection unit 106. In addition to the hand tracking function and the operation device, the operation unit 109 may also include a sound collection unit for acquiring user instructions by voice.

[0039] The sensing unit 110 is a device such as a sensor that is installed outside the HMD 101 and detects the rotation, tilt, and amount of movement of the head 301 of the user wearing the HMD 101, thereby obtaining position and orientation information of the user.

[0040] FIG. 2 is a functional block diagram showing the software configuration of the HMD 101.

[0041] The HMD 101 has, as its functional configuration, a drawn image acquisition unit 201, a recorded image acquisition unit 202, an image information acquisition unit 203, a recorded image information storage unit 204, a moving image generation unit 206, and an image display unit 207.

[0042] The drawn image acquisition unit 201 (drawn image acquisition means, generation means, correction means) generates a rendering image of the CG object using the position and orientation information acquired from the sensing unit 110, and then performs correction processing as necessary to generate drawn images, which are then sequentially transmitted to the display unit 108. As a result, an image of the virtual space is displayed in real time on the display unit 108 at a constant framing rate. In this embodiment, the drawn images are generated by internal processing in the drawn image acquisition unit 201 of the HMD 101, but they may also be generated outside the HMD 101, such as on a PC or the cloud, and the drawn image acquisition unit 201 may acquire them from outside.

[0043] When the correction process is performed, the recording image acquisition unit 202 (recording image acquisition means) acquires, as the recording image, the rendering image before the correction process, rather than the image displayed on the display unit 108. On the other hand, when the correction process is not performed, the recording image acquisition unit 202 acquires, as the recording image, the image displayed on the display unit 108.

[0044] The image information acquisition unit 203 (image information acquisition means) acquires, as image information, information related to the recorded image acquired by the recorded image acquisition unit 202. In this embodiment, the image information acquisition unit 203 acquires, as image information, the "image number, frame time, recorded image" of the recorded image.

[0045] A recorded image information storage unit 204 (recorded image information storage means) associates the recorded image acquired by the recorded image acquisition unit 202 with the image information acquired by the image information acquisition unit 203 for each frame, and stores the information as recorded image information 205.

[0046] The video generation unit 206 (video generation means) references the recorded image information 205, and if a necessary video frame is missing, interpolates that video frame to generate a recorded video. In this embodiment, the presence or absence of recorded images that can be used as video frames is determined from the recorded image information 205. If there are no usable recorded images, interpolating video frames is generated from recorded images corresponding to the previous and next acquisition times. Then, a video is generated by joining together the recorded images including the interpolated image. At this time, the interpolation process may be performed in accordance with the display frame rate of the HDM 101, or in accordance with a frame rate specified by the user.

[0047] The image display unit 207 displays the drawing images sequentially transmitted from the drawing image acquisition unit 201 on the display unit 108 in real time. In addition, in response to a user's instruction, the image display unit 207 can output the recorded video generated by the video generation unit 206 to an external device and play the recorded video on the display of the external device.

[0048] 4 is a flowchart showing the flow of the recorded image generation process according to this embodiment. The process shown in this flowchart is realized by the CPU 102 executing the information processing program according to this embodiment.

[0049] First, in step S401, it is determined whether or not a recording mode has been designated from a user operation acquired from the operation unit 109. If a recording mode has been designated (YES in step S401), the process proceeds to step S402.

[0050] Next, in step S402, the recording image acquisition unit 202 acquires, as a recording image, the image before correction processing by the drawing image acquisition unit 201. Note that if correction processing has not been performed on the drawing image displayed in real time on the display unit 108, the drawing image is acquired as is as a recording image.

[0051] In step S403, the image information acquisition unit 203 acquires the frame times of the recorded images acquired in step S402.

[0052] In step S404, the recorded image information storage unit 204 associates the recorded image acquired in step S402 with the frame time acquired in step S403 for each frame, and stores the associated information as recorded image information 205. In this embodiment, the recorded image information 205 is stored in units of frame numbers. The recorded image information storage unit 204 that executes step S403 will now be described with reference to FIGS. 5A and 5B.

[0053] Hereinafter, a method for generating a drawn image and a recorded image according to this embodiment will be described with reference to FIG. 5A.

[0054] In this embodiment, correction processing is performed on the drawn image of each frame. Also, Fig. 5B is a diagram showing an example of recorded image information 205 according to this embodiment, which is stored in the recorded image information storage unit 204.

[0055] Note that the correction process performed here differs from the correction process shown in FIG. 3 and the correction process shown in FIG. 6 (described later) by using a process called PTW (Positional Time Warp). In this correction process, CG rendering is performed for each frame, and the resulting rendered image is cropped in accordance with the latest movement of the HMD 101, and at least one of deformation and movement is performed to generate a rendered image. If PTW is not used, the CG rendering process is performed using the user's position and orientation at the start of each frame. However, CG rendering using this method requires a certain amount of processing time, which results in delays associated with the CG rendering process. PTW is a technology for reducing delays associated with this CG rendering process, but as this technology is well known, detailed explanations are omitted.

[0056] Head movement 550 indicates the position of HMD 101 as seen from above head 301. In this figure, the user is turning from left to right, and a drawing image 570 generated by CG drawing processing and correction processing linked to this is displayed on HMD 101.

[0057] In this figure, time progresses from left to right, and the head 301 is in rotational motion.

[0058] A timing chart 560 shows the execution timing of the CG drawing process and the correction process. A drawn image 570 is a collective term for drawn images 511 to 514 in which CG objects 303 and 304 are drawn at a constant framing rate according to the orientation of head 301.

[0059] Recorded image 580 is a collective term for recorded images 515 to 518, which are moving images recorded at a constant framing rate of the video that the user is experiencing in real time in the virtual space.

[0060] In this example, the drawn images 511-514 at each frame time (t1-t4) are images that have been subjected to the correction process. On the other hand, the recorded images 515-518 are images that correspond to the drawn images 511-514 at each frame time and that have not yet been subjected to the correction process, and are acquired by the recorded image acquisition unit 202 in response to a user instruction. The recorded images 515-518 are associated with the corresponding frame times (timestamps) and stored by the recorded image information storage unit 204 as recorded image information 205 (FIG. 5B).

[0061] Returning to FIG. 4, next, in step S405, the recorded image information storage unit 204 determines whether or not the recording of the moving image has finished. If the recording of the moving image has finished (YES in step S405), the process proceeds to step S406. On the other hand, if the recording of the moving image has not finished (NO in step S405), the process returns to step S402. Note that the determination that the recording of the moving image has finished may be made when the input detection unit 106 detects that an end instruction has been input by the user (for example, an instruction by the user to cancel the recording mode). Alternatively, the determination may be made when a preset time has elapsed and the system issues an end instruction.

[0062] Next, in step S406, the images stored in the recorded image information 205 are joined together to generate a moving image, and this process ends.

[0063] <Second embodiment> In the first embodiment, the generation of recorded images was described when correction processing is always performed on the drawn image of each frame. In contrast, in the present embodiment, correction processing, specifically, ATW, is performed on the drawn image of the corresponding frame only when the frame rate drops. In this case, the generation of recorded images is performed by saving only the drawn images of frames that have not been corrected as recorded images, and for frames that have been corrected, images are generated by frame interpolation when generating a moving image. This reduces the processing load on the moving image generation unit 206.

[0064] Hereinafter, a method for generating a drawn image and a recorded image according to this embodiment will be described, in which the ATW correction process is performed on the drawn image of the corresponding frame only when the frame rate drops.

[0065] 7 is a flowchart showing the flow of the recorded image generation process according to this embodiment. The process shown in this flowchart is realized by the CPU 102 executing the information processing program according to this embodiment.

[0066] First, in step S801, it is determined whether or not a recording mode has been designated based on a user operation acquired from the operation unit 109. If a recording mode has been designated (YES in step S801), the process proceeds to step S402.

[0067] Next, in step S802, the recorded image acquisition unit 202 acquires the drawn image from the drawn image acquisition unit 201.

[0068] Next, in step S803, the image information acquisition unit 203 acquires the frame time of each of the drawn images acquired in step S804.

[0069] Next, in step S804, the recorded image information storage unit 204 associates the drawn image acquired in step S802 with the frame time acquired in step S803 for each frame, and stores the associated data as recorded image information 205. Note that for frames of drawn images that have been corrected, only the frame time is stored, and the drawn image is not stored. In this embodiment, the recorded image information 205 is stored in units of frame numbers. The recorded image information storage unit 204 that executes step S804 will now be described in detail with reference to FIGS. 6A and 6B.

[0070] Hereinafter, a method for generating a drawn image and a recorded image according to this embodiment will be described with reference to FIG. 6A.

[0071] In this embodiment, correction processing is performed on the drawn image of the frame whose frame rate has decreased. Also, Fig. 6B is a diagram showing an example of recorded image information 205 according to this embodiment, which is stored in the recorded image information storage unit 204.

[0072] Head movement 750 indicates the position of HMD 101 as seen from above head 301. In this figure, the user is turning from left to right, and a drawing image 770 generated by CG drawing processing and correction processing linked to this is displayed on HMD 101.

[0073] In this figure, time progresses from left to right, and the head 301 is in rotational motion.

[0074] Timing chart 760 shows the execution timing of the CG drawing process and the correction process. Drawn image 770 is a collective term for drawn images 711 to 714, which are CG objects 303 and 304 drawn at a constant framing rate according to the orientation of head 301. In this example, the frame rate of drawn image 712 is reduced due to a delay in the CG drawing process, but the frame rates of the other drawn images 711, 713, and 714 are not reduced. Therefore, correction processing is not performed on drawn images 711, 713, and 714, while correction processing is performed on drawn image 712.

[0075] Recorded image 780 is a collective term for recorded images that are video recordings of the images the user is experiencing in real time in a virtual space at a constant framing rate. In this example, the recorded image information storage unit 204 associates drawn images 711, 713, and 714 with their respective frame times and stores them as drawn images 721, 723, and 724 in the recorded image information 205. On the other hand, for drawn image 712, the recorded image information storage unit 204 temporarily stores only its frame time as the recorded image information 205 ( FIG. 6B ). Note that in this embodiment, the drawn image that has undergone correction processing (drawn image 712 in this case) is not stored as recorded image 780 in the recorded image information 205, but the present invention is not limited to this. For example, the drawn image that has undergone correction processing may be stored as recorded image 780 in the recorded image information 205, and a flag indicating that the image is a target for frame interpolation may be added to the recorded image 780.

[0076] Returning to FIG. 7, next, in step S805, the recorded image information storage unit 204 determines whether or not the recording of the moving image has finished. If the recording of the moving image has finished (YES in step S805), the process proceeds to step S806. On the other hand, if the recording of the moving image has not finished (NO in step S805), the process returns to step S802. Note that the determination that the recording of the moving image has finished is the same as in the first embodiment, and therefore a duplicated explanation will be omitted.

[0077] Next, in step S806, the moving image generating unit 206 determines whether frame interpolation is necessary or not based on the information stored in the recorded image information 205, and executes frame interpolation processing and moving image generating processing to generate moving images according to the determination result. Here, if an image of the current frame is present in the recorded image information 205, it is determined that frame interpolation is not necessary. That is, a case in which frame interpolation is not necessary is when the drawn image is not corrected and is stored as is as the recorded image 780 in the recorded image information 205. On the other hand, if an image of the current frame is not present in the recorded image information 205, it is determined that frame interpolation is necessary. For example, an image that has been corrected, such as the drawn image 712, is not stored as the recorded image 780 in the recorded image information 205, and therefore the recorded image 780 does not contain an image of frame 2. In such a case, it is determined that frame interpolation is necessary. Note that since existing technology is used for frame interpolation, a detailed description thereof will be omitted here.

[0078] An example of the frame interpolation process of the moving image generating unit 206 in this embodiment will now be described with reference to Figures 8A and 8B. Details of the frame interpolation process of the moving image generating unit 206 will also be described with reference to the flowchart in Figure 9.

[0079] Fig. 8A is a diagram illustrating the frame interpolation process according to this embodiment, and Fig. 8B is a diagram illustrating an example of recorded image information 205 stored in the recorded image information storage unit 204 after the frame interpolation process according to this embodiment.

[0080] Recorded image (after frame interpolation) 910 indicates an image obtained after frame interpolation of recorded image 780. In this embodiment, since drawn image 712 is an image that has been subjected to correction processing, recorded image 780 is not stored in the recorded image information 205 of frame 2 in step S804 of FIG. 7. Therefore, the video generation unit 206 generates recorded image 911 by interpolation from recorded images 780 (recorded images 721 and 723) immediately before and after frame 2. In this case, the recorded image information storage unit 204 stores the generated recorded image 911 as recorded image (after frame interpolation) 910 in the recorded image information 205 of frame 2 (FIG. 8B).

[0081] FIG. 9 is a flowchart of the video generation process in step S806.

[0082] First, in step S1001, the moving image generating unit 206 acquires the recorded image information 205 stored in the recorded image information storing unit 204 in step S804 of FIG.

[0083] Next, in step S1002, it is determined whether the acquired recorded image information 205 is a target for frame interpolation in frame number units. In this embodiment, a frame for which a recorded image 780 is not saved in the recorded image information 205 is determined to be a target for interpolation. However, as in the above-described modified example, if a drawn image that has been corrected is also saved as a recorded image 780 in the recorded image information 205, a frame for which a flag indicating that the image is a target for frame interpolation has been added to the image in the recorded image information 205 is determined to be a target for interpolation. If the frame is a target for frame interpolation (YES in step S1002), the process proceeds to step S1003b. On the other hand, if the frame is not a target for frame interpolation (NO in step S1002), the process proceeds to step S1003a.

[0084] In step S1003a, the image that is not the target of frame interpolation is stored in the RAM 104, and the process proceeds to step S1004.

[0085] Next, in step S1003, frame interpolation processing is performed by reading the recorded images 780 of the frames before and after the frame determined to be the frame interpolation target from the recorded image information 205. Thereafter, the generated images are stored in the RAM 104 as the recorded images 780 of the frames determined to be the frame interpolation target.

[0086] Next, in step S1004, the images stored in the RAM 104 in steps S1003a and S1003b are joined together to generate a moving image.

[0087] Next, in step S1005, it is determined whether or not processing has been completed for all frame numbers of the recorded image information 205. If processing has not been completed (NO in step S1005), the process returns to step S1002. On the other hand, if processing has been completed (YES in step S1005), the video generation process ends.

[0088] As a result, when correction processing is performed only when the frame rate drops, frame interpolation processing is performed only on the drawing image to be corrected, making it possible to generate a moving image.

[0089] In this embodiment, the images are stored in the RAM 104 in steps S1003a and S1003b, but they may be stored in the recording unit 105 instead of the RAM 104.

[0090] As described above, in this embodiment, for frames in which correction processing has been performed on a drawn image, instead of the drawn image of that frame, an image obtained by frame interpolation using the recorded images of the frames before and after that frame is acquired as the recorded image of that frame, and a recorded moving image is generated. This makes it possible to generate a moving image with accurate images without any degradation in image quality or image loss due to the correction processing.

[0091] (Other embodiments) The present invention can also be realized by a process in which a program that realizes one or more functions of the present embodiment is supplied to a system or device via a network or a storage medium, and one or more general-purpose processors ASICs in the computer of the system or device read and execute the program. The present invention can also be realized by a dedicated processor (e.g., an ASIC or FPGA) that realizes one or more functions. Furthermore, the present invention can also be realized by a combination of a general-purpose processor and a dedicated processor. Note that the term "processor" used here refers to a processor in a broad sense and includes both general-purpose processors and dedicated processors. The process that realizes the present invention may be performed by a single processor alone, or may be performed by multiple processors located in physically separate locations in cooperation with each other.

[0092] In the first and second embodiments, an example in which the present invention is applied to the HMD 101, which is a VR device, has been described. However, the present invention is not limited to this as long as the drawing of a CG object in a virtual space can be linked to the movement of a user using the device. For example, the present invention may be applied to an AR device such as Pokémon GO, or an AR or MR device that uses optical see-through. However, in the case of these devices, the area of ​​the drawn image on the screen of the display unit 108 is only a part, and the area other than the drawn image (area where image loss occurs) is transparent.

[0093] Furthermore, when a user wearing the HMD 101 views a drawn image displayed in real time on the display unit 108, and later plays back and checks the recorded image, the user may feel uncomfortable that the drawn image and the recorded image after frame interpolation are different. Therefore, a notification flag may be attached to the recorded image of a frame (a frame to be corrected) that is different between the drawn image and the recorded image, and when a recorded image with a notification flag is displayed during playback of the recorded video, the user may be notified that the displayed recorded image is different from the drawn image.

[0094] Also, since the recorded image does not need to be displayed in real time, for example, in the case of Figure 6A, after the CG rendering process for frame 2 is completed, the obtained image can be saved as that of frame 2. However, since the CG rendering process and the subsequent correction and display processes are performed by different software and operate asynchronously, if the CG rendering process for frame 2 is completed beyond the frame period, adjustments must be made so that these processes can be synchronized.

[0095] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention.

[0096] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) An information processing device that generates a recorded video of an image of a virtual space in which CG objects are drawn in conjunction with the movements of a user, comprising: a drawn image acquisition means that acquires a drawn image that is displayed at a constant framing rate on a display unit on which the user views the image of the virtual space; a recorded image acquisition means that acquires a recorded image as an image of one frame of the recorded video; an image information acquisition means that acquires image information corresponding to the recorded image; a recorded image information storage means that associates the recorded image and the image information for each frame and stores them as recorded image information; and a video generation means that generates the recorded video using the recorded image information, wherein the drawn image acquisition means acquires position and orientation information of the user, and comprises: a generation means that generates a rendering image of the CG object using the acquired position and orientation information; and a correction means that performs correction processing on the rendering image, and the recorded image acquisition means sets the rendering image as the recorded image. (Configuration 2) The information processing device according to configuration 1, characterized in that the correction process is a process of performing at least one of cutting, transforming, and moving a rendering image of a frame to be corrected in accordance with the latest movement of the user, to obtain a drawing image of the frame to be corrected. (Configuration 3) The information processing device according to Configuration 2, wherein the correction process is performed every time the rendering image is generated by the generating means. (Configuration 4) An information processing device according to any one of configurations 1 to 3, characterized in that the correction process is a process of performing at least one of cutting, deformation, and movement on a drawn image of a frame prior to the frame to be corrected, in accordance with position and orientation information in the previous frame and position and orientation information in the frame to be corrected, and treating the resulting image as the drawn image of the frame to be corrected. (Configuration 5) The information processing device according to configuration 4, wherein the correction process is performed when a delay occurs in the generation of the rendering image by the generation means. (Configuration 6) The information processing device according to Configuration 5, wherein the video generating means does not use the recorded image of the frame to be corrected as the rendering image, but performs frame interpolation from the recorded images of the previous and next frames to generate the recorded image of the frame to be corrected. (Configuration 7) The information processing device according to Configuration 6, characterized in that the recorded image information includes a frame time, the recorded image information saving means temporarily saves only the image information as the recorded image information for the frame when the delay occurs, and the video generating means determines that the frame of the frame time of the recorded image information in which only the image information is saved is the frame to be corrected. (Configuration 8) The information processing device according to Configuration 6, characterized in that the recorded image information includes a frame time, the recorded image information saving means adds a flag to the recorded image included in the recorded image information of the frame when the delay occurs, indicating that the image is a target image for frame interpolation, and the video generating means determines that the frame of the frame time of the recorded image information to which the flag has been added is the frame to be corrected. (Configuration 9) The information processing device described in Configuration 7 or 8, characterized in that the recorded image information further includes a frame number, and the video generation means determines whether the recorded image information is a frame to be corrected in units of the frame number. (Configuration 10) An information processing device described in any one of configurations 4 to 9, characterized in that a notification flag is attached to the recorded image of the frame to be corrected, and when the recorded image with the notification flag is displayed during playback of the recorded video, the information processing device notifies the user that the displayed recorded image is different from the drawn image. (Configuration 11) The information processing device according to any one of configurations 1 to 10, characterized in that it is a head-mounted display worn by the user to view the image of the virtual space. (Configuration 12) An information processing device described in any one of configurations 1 to 10, characterized in that the user is a VR device that views images of the virtual space, and areas of image loss caused by the correction processing of the drawn image are painted black. (Configuration 13) An information processing device described in any one of configurations 1 to 10, characterized in that the information processing device is an AR device through which the user views images of the virtual space, and areas of image loss caused by the correction processing of the drawn image become transparent. (Configuration 14) An information processing device described in any one of configurations 1 to 10, characterized in that the user is an MR device through which an image of the virtual space is viewed, and areas of image loss caused by the correction processing of the drawn image become transparent. (Configuration 15) The information processing apparatus according to configuration 5, wherein the recorded image acquisition means also uses the rendering image as the recorded image for the correction target frame. (Method 1) A control method for an information processing device that generates a recorded video of an image of a virtual space in which CG objects are drawn in conjunction with the movements of a user, the control method comprising: a drawn image acquisition step of acquiring a drawn image that is displayed at a constant framing rate on a display unit on which the user views the image of the virtual space; a recorded image acquisition step of acquiring a recorded image as an image of one frame of the recorded video; an image information acquisition step of acquiring image information corresponding to the recorded image; a recorded image information saving step of associating the recorded image and the image information for each frame and saving them as recorded image information; and a video generation step of generating the recorded video using the recorded image information, wherein the drawn image acquisition step comprises a generation step of acquiring position and orientation information of the user and generating a rendering image of the CG object using the acquired position and orientation information; and a correction step of performing correction processing on the rendering image, wherein the rendered image is used as the recorded image in the recorded image acquisition step. (Program 1) A program for causing a computer to function as each means of the information processing device described in any one of configurations 1 to 15. [Explanation of symbols]

[0097] 101 HMD 102 CPU 103 ROM 104 RAM 105 Recording section 106 Input detection unit 107 Communication I / F 108 Display section 109 Operation section 201 Drawing image acquisition unit 202 Recorded image acquisition unit 203 Image information acquisition unit 204 Recorded image information storage unit 205 Recorded image information 206 Video Generation Unit 207 Image display unit

Claims

1. An information processing device for generating a recorded video of a virtual space in which CG objects are rendered in conjunction with a user's movements, a drawing image acquisition means for acquiring a drawing image to be displayed at a certain framing rate on a display unit on which the user views an image of the virtual space; a recorded image acquisition means for acquiring a recorded image as an image of one frame of the recorded moving image; image information acquisition means for acquiring image information corresponding to the recorded image; a recorded image information storage means for storing the recorded image and the image information in association with each other for each frame as recorded image information; a moving image generating means for generating the recorded moving image using the recorded image information; Equipped with The drawing image acquisition means a generation means for acquiring position and orientation information of the user and generating a rendering image of the CG object using the acquired position and orientation information; a correction unit that performs a correction process on the rendering image; Equipped with the recording image acquisition means sets the rendering image as the recording image; An information processing device characterized by:

2. The information processing device according to claim 1, characterized in that the correction process is a process of performing at least one of cutting, transforming, and moving a rendering image of the frame to be corrected in accordance with the latest movement of the user, to obtain a drawing image of the frame to be corrected.

3. 3. The information processing apparatus according to claim 2, wherein the correction process is performed every time the rendering image is generated by the generating means.

4. 2. The information processing device according to claim 1, wherein the correction process is a process of performing at least one of cutting, transforming, and moving a drawn image of a frame prior to the frame to be corrected, in accordance with position and orientation information in the previous frame and position and orientation information in the frame to be corrected, and setting the resulting image as the drawn image of the frame to be corrected.

5. 5. The information processing apparatus according to claim 4, wherein the correction process is performed when a delay occurs in the generation of the rendering image by the generation means.

6. The information processing device according to claim 5, wherein the video generating means generates the recorded image of the frame to be corrected by performing frame interpolation from the recorded images of previous and subsequent frames, rather than using the recorded image of the frame to be corrected as the rendering image.

7. the recorded image information includes a frame time; the recorded image information storage means temporarily stores only the image information for the frame when the delay occurs as the recorded image information; 7. The information processing apparatus according to claim 6, wherein the moving image generating means determines a frame of a frame time of recorded image information in which only the image information is stored as the frame to be corrected.

8. the recorded image information includes a frame time; the recorded image information storage means adds a flag indicating that the recorded image is an image to be subjected to frame interpolation to the recorded image included in the recorded image information of the frame when the delay occurs; 7. The information processing apparatus according to claim 6, wherein the moving image generating means determines that a frame of a frame time of the recorded image information to which the flag is added is the frame to be corrected.

9. The recorded image information further includes a frame number, 9. The information processing apparatus according to claim 7, wherein the moving image generating means determines whether or not the frame is a frame to be corrected for each of the frame numbers of the recorded image information.

10. The information processing device according to claim 4, characterized in that a notification flag is attached to the recorded image of the frame to be corrected, and when the recorded image with the notification flag is displayed during playback of the recorded video, the information processing device notifies the user that the displayed recorded image is different from the drawn image.

11. 2. The information processing device according to claim 1, wherein the information processing device is a head-mounted display worn by the user to view the image of the virtual space.

12. 2. The information processing apparatus according to claim 1, wherein the information processing apparatus is a VR device through which the user views an image of the virtual space, and an area of ​​an image defect caused by the correction process of the drawn image is painted black.

13. The information processing device according to claim 1 , wherein the device is an AR device through which the user views the image of the virtual space, and an area of ​​image loss caused by the correction process of the drawn image becomes transparent.

14. 2. The information processing apparatus according to claim 1, wherein the user is using an MR device to view the image of the virtual space, and an area of ​​the image defect caused by the correction process of the drawn image becomes transparent.

15. 6. The information processing apparatus according to claim 5, wherein the recording image acquisition means also uses the rendering image as the recording image for the correction target frame.

16. A control method for an information processing device that generates a recorded video of an image of a virtual space in which CG objects are rendered in conjunction with a user's movements, comprising: a drawing image acquisition step of acquiring a drawing image to be displayed at a certain framing rate on a display unit on which the user views an image of the virtual space; a recorded image acquisition step of acquiring a recorded image as an image of one frame of the recorded moving image; an image information acquisition step of acquiring image information corresponding to the recorded image; a recorded image information storing step of associating the recorded image with the image information for each frame and storing the image information as recorded image information; a moving image generating step of generating the recorded moving image using the recorded image information; and The drawing image acquisition step includes: a generating step of acquiring position and orientation information of the user and generating a rendering image of the CG object using the acquired position and orientation information; a correction step of performing a correction process on the rendering image; and In the recording image acquisition step, the rendering image is used as the recording image; A control method characterized by:

17. A program for causing a computer to function as each of the means of the information processing device according to claim 1.

Images