Information processing device, control method for information processing device, and program

Abstract

This technology enables the optimal display of computer graphics (CG) images even when user movements are difficult to predict. [Solution] The information processing device of the present invention has a determination means for determining a drawable area of ​​a CG object, and the determination means expands the drawable area in a direction that increases the area of ​​the CG object included in the drawable area when a part of the CG object is not included in the drawable area.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, a control method for the information processing apparatus, and a program.

Background Art

[0002] There is known a technique for superimposing a CG image (an image of a CG object) on a real image (an image of the real space) so that the CG object appears as if it exists in the real world. In the superimposition process, when the rendering (drawing) of the CG image is delayed with respect to the acquisition of the real image, the user's viewpoint (the position and orientation of the head) may be different between the time of acquiring the real image and the time of rendering the CG image. As a result, the intended superimposition result may not be obtained, and the display quality of the video may deteriorate.

[0003] In order to suppress the deterioration of the display quality of the video, a reprojection technique has been proposed to correct the drawing result (the position and orientation of the drawn CG image) so that the CG image is displayed in an appropriate position and orientation. However, the reprojection technique cannot complement the area of the CG image that has not been drawn. Therefore, depending on the correction amount of the drawing result, the CG image may be displayed in a state where a part is missing.

[0004] Patent Document 1 discloses a technique for expanding the drawing area of a CG image using the history information of the user's viewpoint.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] In the technology disclosed in Patent Document 1, the user's movements are predicted based on historical information, and the drawing area is expanded in the direction of those movements. Therefore, when the user's movements are difficult to predict, such as when the user starts moving from a stationary position, the drawing area may not be adequately expanded, and the CG image may be missing parts.

[0007] The present invention aims to provide a technology that enables the appropriate display of computer graphics images even when user movements are difficult to predict. [Means for solving the problem]

[0008] The information processing apparatus of the present invention has a determination means for determining a drawable area of ​​a CG object, and the determination means is characterized in that, if a part of the CG object is not included in the drawable area, it expands the drawable area in a direction that increases the area of ​​the CG object included in the drawable area. [Effects of the Invention]

[0009] According to the present invention, CG images can be displayed suitably even when the user's movements are difficult to predict. [Brief explanation of the drawing]

[0010] [Figure 1] This is a hardware configuration diagram for the HMD. [Figure 2] This is a functional configuration diagram of the HMD in Embodiment 1. [Figure 3] This is a functional configuration diagram of the second region determination unit in Embodiment 1. [Figure 4] This figure illustrates the drawing area in Embodiment 1. [Figure 5] This is a flowchart showing the region determination process in Embodiment 1. [Figure 6] This is a software configuration diagram of the HMD in Embodiment 1. [Figure 7] This is a functional configuration diagram of the HMD in Embodiment 2. [Figure 8]This is a functional configuration diagram of the next drawing area determination unit in Embodiment 2. [Figure 9] This figure illustrates the drawing area in Embodiment 2. [Figure 10] This is a flowchart showing the region determination process in Embodiment 2. [Figure 11] This is a software configuration diagram of the HMD in Embodiment 2. [Modes for carrying out the invention]

[0011] <Embodiment 1> Embodiment 1 of the present invention will be described below with reference to the drawings. Figure 1 is a hardware configuration diagram of the HMD (Head Mounted Display) 100 in Embodiment 1. The HMD 100 has a CPU 101, ROM 102, RAM 103, communication IF (interface) 104, position sensor 105, attitude sensor 106, camera 107, and display 108, and each component is connected to a bus 109 so that it can communicate with each other. Note that the present invention may be applied to a device other than an HMD. For example, the present invention may be applied to an information processing device such as a personal computer connected to the HMD by wire or wireless. The CPU 101, ROM 102, RAM 103, and communication IF 104 may be provided in the information processing device connected to the HMD.

[0012] In Embodiment 1, the HMD 100 is assumed to be a video see-through type HMD, but is not limited to this. The HMD 100 may be other head-mounted display devices such as, for example, an optical see-through type HMD (e.g., AR glasses) and a handheld display (HHD).

[0013] The CPU 101 controls the overall operation of the HMD 100 by loading the program stored in the ROM 102 into the RAM 103 and executing it.

[0014] The ROM 102 stores various data such as a boot program, firmware, various programs for the CPU 101 to implement the processes described later, and various image data.

[0015] The RAM 103 is a work memory that can temporarily store programs and data for the CPU 101 to perform processing. Various programs and various data are stored (deployed) in the RAM 103 by the CPU 101.

[0016] The communication IF 104 is an interface for communicating with an external device via a network. The HMD 100 performs data transmission and reception with the external device via the network connected to the communication IF 104.

[0017] The position sensor 105 and the attitude sensor 106 detect the position and attitude of the HMD 100 worn by the user on the head. The position sensor 105 and the attitude sensor 106 may be integrally configured, for example, an inertial measurement unit (IMU: Inertial Measurement Unit) or the like may also be used.

[0018] The camera 107 is a camera for imaging the real space. The display 108 is a display unit for displaying a composite image obtained by superimposing a CG image on a background image (real image) to the user of the HMD 100.

[0019] FIG. 2 is a functional configuration diagram of the HMD 100 in Embodiment 1. Each functional unit shown in FIG. 2 is realized, for example, when the CPU 101 executes a program.

[0020] The first position and attitude acquisition unit 200 acquires information on the user's viewpoint (the position and attitude of the head) based on the position detected by the position sensor 105 and the attitude detected by the attitude sensor 106 at the first timing when drawing the CG image.

[0021] The first region determination unit 201 determines the drawable area (drawing area) of the CG image in the virtual space based on the viewpoint information acquired by the first position and orientation acquisition unit 200. In Embodiment 1, the drawing area is the area on which the CG image can be drawn. The size of the drawing area determined by the first region determination unit 201 may be, for example, the same size as the display area of ​​the display 108.

[0022] The placement unit 202 determines the position and orientation of the CG image in the virtual space. For example, the CG image is a 2D image representing a CG object (3D model), and the position and orientation of the CG image determined here are the position and orientation of the CG object (3D model).

[0023] The second area determination unit 203 expands the drawing area determined by the first area determination unit 201. Based on the position and orientation of the CG image determined by the placement unit 202, the second area determination unit 203 expands the drawing area in a direction that increases the area of ​​the CG image included in the drawing area. The second area determination unit 203 may expand the drawing area to include the entire CG image. Depending on the processing power of the CPU 101 and the capacity of the RAM 103, the second area determination unit 203 may expand the drawing area so that the size of the drawing area does not exceed a predetermined size.

[0024] The drawing unit 204 draws the CG image in the drawing area determined by the second area determination unit 203, at the position and orientation determined by the placement unit 202.

[0025] The second position and orientation acquisition unit 205 acquires information about the user's viewpoint (head position and orientation) at a second timing (later than the first timing) when acquiring the real image. The second position and orientation acquisition unit 205 acquires information about the user's viewpoint based on the position detected by the position sensor 105 and the orientation detected by the orientation sensor 106.

[0026] The correction unit 206 performs a process (reprojection process) to correct the position and orientation of the CG image drawn by the drawing unit 204 to a position and orientation that does not cause discomfort to the user when displaying the CG image on the display 108. The position and orientation that does not cause discomfort to the user is the position and orientation that reproduces the view from the user's viewpoint acquired by the second position and orientation acquisition unit 205. In the reprojection process, the position and orientation of the CG image are corrected based on the user's viewpoint acquired by the first position and orientation acquisition unit 200 and the user's viewpoint acquired by the second position and orientation acquisition unit 205.

[0027] The background image acquisition unit 207 acquires real-world images captured by the camera 107. However, the background images acquired by the background image acquisition unit 207 are not limited to real-world images captured by the camera 107; for example, the HMD 100 may have a virtual space generation unit that generates CG images representing the virtual space as background images.

[0028] The synthesis processing unit 208 generates a composite image by superimposing the CG image corrected by the correction unit 206 with the real-world image acquired by the background image acquisition unit 207.

[0029] The display processing unit 209 controls the display 108 to display the composite image generated by the synthesis processing unit 208.

[0030] Figure 3 is a functional configuration diagram of the second region determination unit 203 in Embodiment 1. The first drawing region 300 is a drawing region determined by the first region determination unit 201. The placement information 301 is information regarding the position and orientation of the CG image determined by the placement unit 202.

[0031] The region detection unit 302 selects a CG image (target CG image) that is only partially included in the first drawing area 300, and detects the area of ​​the target CG image that is not included in the first drawing area 300. The region detection unit 302 may select target CG images only if they are of a specific type. The specific type may be, for example, a type that is likely to interest the user, and may be predetermined by the manufacturer or specified by the user. For example, among the CG images that are only partially included in the first drawing area 300, only CG images of virtual objects held by the user in the virtual space may be selected as target CG images. Hereinafter, the area of ​​the target CG image that is not included in the first drawing area 300 will be referred to as the out-of-range area 303.

[0032] The extended area determination unit 304 extends the first drawing area 300 to include the out-of-range area 303 and performs a process (area determination process) to determine the second drawing area 305. The extended area determination unit 304 may extend the first drawing area 300 to include the entire out-of-range area 303 (the entire target CG image), or it may extend the first drawing area 300 to include a part of the out-of-range area 303.

[0033] Figures 4A to 4C illustrate the drawing area in Embodiment 1. Below, the first drawing area 300 and the second drawing area 305 will be described using Figures 4A to 4C.

[0034] The placement information 301 includes information about the position and orientation of the CG image of the wrench 400 and information about the position and orientation of the CG image of the leader 401. In Figure 4A, the wrench 400 is positioned such that a portion of it is included in the first drawing area 300. The leader 401 is not included in the first drawing area 300. Therefore, the wrench 400 is selected as the target CG image. Then, the area of ​​the wrench 400 that is not included in the first drawing area 300 (the black area 402 in Figure 4B) is detected as the out-of-range area 303. The second drawing area 305 shown in Figure 4B is an extension of the first drawing area 300 shown in Figure 4A in the direction of the arrow shown in Figure 4B. The second drawing area 305 is an extended area of ​​the first drawing area 300 that includes the entire out-of-range area 303 (the entire wrench 400).

[0035] The drawing result 403 shown in Figure 4C is the drawing result of the drawing unit 204. The drawing result 403 shows the entire spanner 400 and a portion of the leader 401.

[0036] Figure 5 is a flowchart showing the region determination process of the second region determination unit 203. In S500, the region detection unit 302 detects an out-of-range region 303, which is a region of the target CG image that partially overlaps with the first drawing region 300 but is not included in the first drawing region 300. In S501, the expanded region determination unit 304 expands the first drawing region 300 to include the out-of-range region 303 and determines the second drawing region 305.

[0037] Figure 6 is a software configuration diagram of the HMD 100 in Embodiment 1. In Embodiment 1, the renderer 600 and HMD software 601 are programs stored in ROM 102 and executed by CPU 101.

[0038] The renderer 600 includes the processing performed by the second region determination unit 203 and the drawing unit 204. This is a RAM (rendering application). Renderer 600 obtains information about the first rendering area 300 from HMD software 601 and outputs the rendering result 403 to HMD software 601.

[0039] The HMD software 601 displays the rendering result 403 output by the renderer 600 on the display 108. The HMD software 601 also determines the user's viewpoint based on the position detected by the position sensor 105 and the orientation detected by the orientation sensor 106, and determines the first rendering area 300 to be output to the renderer 600.

[0040] The second area determination unit 203 determines the second drawing area 305 based on the first drawing area 300 and the arrangement information 301, and outputs information regarding the second drawing area 305 to the drawing unit 204.

[0041] Thus, in Embodiment 1, the renderer 600 performs the processing of the second region determination unit 203, enabling the display of CG images favorably even when the user's movements are difficult to predict.

[0042] <Embodiment 2> Embodiment 2 of the present invention will now be described. In Embodiment 1, the renderer 600 performed the processing performed by the second region determination unit 203 and the processing performed by the drawing unit 204 based on the placement information 301. In Embodiment 2, the processing performed by the second region determination unit 203 in Embodiment 1 is performed by the HMD software 601. The hardware configuration of the HMD in Embodiment 2 is the same as the hardware configuration of the HMD in Embodiment 1 (Figure 1).

[0043] Figure 7 is a functional configuration diagram of the HMD 100 in Embodiment 2. Each functional unit shown in Figure 7 is realized, for example, by the CPU 101 executing a program. The next drawing area determination unit 700 determines the drawing area for the next frame based on the drawing result of the current frame output from the drawing unit 204.

[0044] Figure 8 is a functional configuration diagram of the next drawing area determination unit 700 in Embodiment 2. Depth information 800 is depth information corresponding to the position and orientation of the CG image drawn by the drawing unit 204. Depth information 800 is information indicating the depth of each position of the CG object, and depth may be interpreted as coordinates (depth coordinates) in a direction perpendicular to the two-dimensional drawing area. In Embodiment 2, the drawing unit 204 determines the position and orientation of the CG image and generates depth information 800. The drawing unit 204 uses a predetermined viewpoint in the first frame, and from the second frame onward, it uses the viewpoint detected by the second position and orientation acquisition unit 205 in the previous frame. The correction unit 206 corrects the position and orientation of the CG image based on the viewpoint detected by the second position and orientation acquisition unit 205 in the current frame and the viewpoint detected by the second position and orientation acquisition unit 205 in the previous frame.

[0045] The third drawing area 801 is the current drawing area of ​​the CG image in the processing performed by the drawing unit 204.

[0046] The edge detection unit 802 detects the portion of the edge (contour) of the third drawing area 801 where a CG object exists (the portion where depth is indicated; the portion where the CG image is drawn) based on the depth information 800. The edge detection unit 802 then outputs the detection result (coordinates of the detected portion) as edge information 803.

[0047] The extended area determination unit 304 determines a fourth drawing area 804, which is an extended drawing area of ​​the third drawing area 801 and will be used in the next frame, based on the third drawing area 801 and the edge information 803. The specific area determination process will be described later.

[0048] Figures 9A to 9D illustrate the drawing area in Embodiment 2. Below, the third drawing area 801 and the fourth drawing area 804 will be described using Figures 9A to 9D.

[0049] Figure 9A shows the third rendering area 801 used in the first frame. In Figure 9A, the wrench 400 is positioned such that a portion of it is included in the third rendering area 801. The leader 401 is not included in the third rendering area 801. Therefore, the wrench 400 is selected as the target CG image. Figure 9B is a schematic diagram showing depth information 800. In Figure 9B, depth information 900 in the third rendering area 801 is shown as depth information 800. Depth information 900 indicates the depth of the portion of the wrench 400 included in the third rendering area 801.

[0050] In Figure 9C, the lower overlapping portion 901 is the portion of the lower end of the third drawing area 801 that overlaps with the wrench 400. The right overlapping portion 902 is the portion of the right end of the third drawing area 801 that overlaps with the wrench 400. In this case, the end information 803 indicates the lower overlapping portion 901 and the right overlapping portion 902. Then, as shown in Figure 9C, the fourth drawing area 804 is determined by extending the third drawing area 801 in the direction of the arrow (the direction in which the lower overlapping portion 901 and the right overlapping portion 902 exist).

[0051] Figure 9D shows the drawing result 903 for the fourth drawing area 804 in the second frame following the first frame. In the drawing result 903, the entire spanner 400 and a portion of the leader 401 are drawn.

[0052] Figure 10 is a flowchart showing the region determination process of the next drawing region determination unit 700. In S1000, the edge detection unit 802 detects the portion of the edge (contour) of the third drawing region 801 where a CG object exists, and acquires the detection result (coordinates of the detected portion) as edge information 803. In S1001, the expansion region determination unit 304 determines the fourth drawing region 804 by expanding the third drawing region 801 in the direction in which the edge information 803 was detected from the contour of the third drawing region 801. The amount by which the expansion region determination unit 304 expands the third drawing region 801 is not particularly limited. For example, it may be expanded by a predetermined expansion amount set in advance by the manufacturer, or the expansion amount may be determined based on history information of the viewpoint's change over time. If there is a tendency to move the viewpoint large (quickly), a large expansion amount may be determined, and if there is a tendency to move the viewpoint small (slowly), a small expansion amount may be determined.

[0053] Figure 11 shows the software configuration of the HMD 100 in Embodiment 2. In Embodiment 2, the renderer 600 and the HMD software 601 are programs stored in the ROM 102 and executed by the CPU 101.

[0054] The renderer 600 is a program (rendering application) that includes the processing performed by the drawing unit 204. The renderer 600 outputs the drawing results 903 and depth information 800 to the HMD software 601.

[0055] The HMD software 601 displays the rendering result 903 output by the renderer 600 on the display 108. The HMD software 601 determines the third rendering area 801 and outputs the fourth rendering area 804, determined by the next rendering area determination unit 700, to the renderer 600.

[0056] Thus, in Embodiment 2, the HMD software 601 executes the processing of the next drawing area determination unit 700, enabling the CG image to be displayed favorably even when the user's movements are difficult to predict.

[0057] Note that the various controls described above are performed by a single piece of hardware (e.g., a processor or circuit). This is acceptable, or it is not. Multiple pieces of hardware (for example, multiple processors, multiple circuits, or a combination of one or more processors and one or more circuits) may share the processing to control the entire device.

[0058] Furthermore, the above-mentioned processors are processors in a broad sense, including general-purpose processors and specialized processors. General-purpose processors include, for example, CPUs (Central Processing Units), MPUs (Micro Processing Units), and DSPs (Digital Signal Processors). Specialized processors include, for example, GPUs (Graphics Processing Units), ASICs (Application Specific Integrated Circuits), and PLDs (Programmable Logic Devices). Programmable logic devices include, for example, FPGAs (Field Programmable Gate Arrays) and CPLDs (Complex Programmable Logic Devices).

[0059] Furthermore, the embodiments described above (including modified examples) are merely examples, and configurations obtained by appropriately modifying or changing the above-described configurations within the scope of the gist of the present invention are also included in the present invention. Configurations obtained by appropriately combining the above-described configurations are also included in the present invention.

[0060] <Other Embodiments> The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit that implements one or more functions.

[0061] This embodiment includes the following configurations, methods, and programs. (Composition 1) It has a determination means for determining the drawable area of ​​a CG object, The determination means expands the drawable area in a direction that increases the area of ​​the CG object included in the drawable area if a part of the CG object is not included in the drawable area. An information processing device characterized by the following: (Configuration 2) The determination means extends the drawable area to include the entire CG object. The information processing device according to configuration 1, characterized by the above. (Composition 3) The determination means expands the drawable area so that the size of the expanded drawable area does not exceed a predetermined size. An information processing device according to configuration 1 or 2, characterized by the above. (Composition 4) The system further includes detection means for detecting the portion of the outline of the drawable region before expansion in which the CG object exists, The determination means determines a direction for expanding the drawable region in a direction in which a portion of the contour of the drawable region detected by the detection means is located. An information processing device according to any one of configurations 1 to 3. (method) The process includes a decision step to determine the drawable area of ​​a CG object, In the determination step, if a portion of the CG object is not included in the drawable area, the drawing is performed in a direction that increases the area of ​​the CG object included in the drawable area. Expand the area of ​​view. A control method for an information processing device characterized by the following features. (program) A program to cause a computer to function as one of the means of an information processing device described in any one of configurations 1 to 4. [Explanation of symbols]

[0062] 100: HMD 101: CPU 203: Second area determination unit 700: Next drawing area determination unit

Claims

1. It has a determination means for determining the drawable area of ​​a CG object, The determination means expands the drawable region in a direction that increases the area of ​​the CG object included in the drawable region, if a part of the CG object is not included in the drawable region. An information processing device characterized by the following:

2. The determination means extends the drawable area to include the entire CG object. The information processing apparatus according to feature 1.

3. The determination means expands the drawable area so that the size of the expanded drawable area does not exceed a predetermined size. The information processing apparatus according to feature 1.

4. The system further includes a detection means for detecting the portion of the outline of the drawable region before expansion in which the CG object exists. The determination means determines a direction for expanding the drawable region in a direction in which a portion of the contour of the drawable region detected by the detection means is located. The information processing apparatus according to feature 1.

5. The process includes a determination step to determine the drawable area of ​​a CG object. In the determination step, if a portion of the CG object is not included in the drawable area, the drawable area is expanded in a direction that increases the area of ​​the CG object included in the drawable area. A control method for an information processing device characterized by the following features.

6. A program for causing a computer to function as one of the means of an information processing apparatus described in any one of claims 1 to 4.

Images