Information processing device, information processing method and program

Abstract

To provide a technique that enables a user to observe the internal state of a CG model more properly.SOLUTION: An information processing device has: model generating means which generates a CG model for a gap area, which is an area encircled with a first CG model, based upon the first CG model; image generating means which renders the CG model for the gap area without rendering the first CG model to generate an image of a virtual space; and control means which controls display means to display an image based upon the image of the virtual space.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] Conventionally, there are systems (presentation systems for virtual reality and mixed reality, etc.) that display CG (Computer Graphics) models in response to the movement of a display device. In such systems, CG models may be superimposed on objects held by the user and objects that move in real space (hereinafter referred to as "movable objects").

[0003] For example, one possible use case is when a person puts their hand inside a large piece of equipment to perform maintenance. In such a use case, a CG model of a tool with the same shape as the "tool that the user actually holds in his / her hand" may be displayed. The user may then observe whether the CG model of the large equipment and the CG model of the tool do not come into contact with surrounding parts (i.e., whether sufficient working space is secured).

[0004] Patent Document 1 discloses a function for highlighting contact positions of a CG model of a part. By looking at the highlighted contact positions, the designer of the CG model can adjust the position of the CG model of the part so that contact does not occur between the CG model of the tool and the CG model of the part. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2004-178006 A Summary of the Invention [Problem to be solved by the invention]

[0006] As mentioned above, allowing users to observe the inside of large equipment plays an important role in maintenance. On the other hand, rendering (drawing) a CG model in virtual space and displaying the CG model requires a lot of processing. This can slow down the update speed (update rate) of images in the virtual space, and there are cases where users are unable to properly understand the internal state of the CG model of a large device.

[0007] Therefore, an object of the present invention is to provide a technique that allows a user to more appropriately observe the internal state of a CG model. [Means for solving the problem]

[0008] One aspect of the present invention is a method for producing a composition comprising the steps of: a model generating means for generating a CG model of a gap region, which is a region surrounded by a first CG model, based on the first CG model; an image generating means for generating an image of a virtual space by rendering a CG model of the gap region without rendering the first CG model; a control means for controlling a display means to display an image based on the image of the virtual space; The information processing device is characterized by having:

[0009] One aspect of the present invention is a method for producing a composition comprising the steps of: a model generating step of generating a CG model of a gap region, which is a region surrounded by a first CG model, based on the first CG model; an image generating step of generating an image of a virtual space by rendering a CG model of the gap region without rendering the first CG model; a control step of controlling a display means to display an image based on the image of the virtual space; The information processing method is characterized by having the following features. Effect of the Invention

[0010] According to the present invention, a user can more appropriately observe the internal state of a CG model. [Brief description of the drawings]

[0011] [Figure 1] 1 is a block diagram of an image generating device according to a first embodiment. [Diagram 2] 3 is an internal configuration diagram of an extraction unit according to the first embodiment. FIG. [Diagram 3] 4 is a flowchart of processing performed by the image generating apparatus according to the first embodiment. [Figure 4] 4 is a flowchart of a process for generating a model of a gap region according to the first embodiment. [Diagram 5] 4 is a flowchart of a virtual image generation process according to the first embodiment. [Figure 6] 13 is a flowchart of editing processing of a model of a gap region according to the third embodiment. [Figure 7] FIG. 4 is a diagram for explaining the processing of an extraction unit according to the first embodiment. [Figure 8] FIG. 13 is a diagram illustrating a contact area. [Figure 9] 1 is a diagram illustrating a hardware configuration of an image generating apparatus according to a first embodiment. [Figure 10] FIG. 13 is a diagram showing a virtual verification process. [Figure 11] FIG. 11 is a block diagram of an image generating device according to a second embodiment. [Figure 12] 10 is a flowchart of a process performed by an image generating apparatus according to a second embodiment. [Figure 13] FIG. 11 is a block diagram of an image generating device according to a third embodiment. [Figure 14] 11 is a flowchart of a process performed by an image generating apparatus according to a third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0013] <Embodiment 1> The hardware configuration and processing of an image generating device (information processing device; display control device) 100 according to the first embodiment will be described with reference to the hardware configuration diagram of FIG.

[0014] 9, the image generating device 100 has a CPU 901, a ROM 902, an I / F 903, a storage medium drive 905, an external storage device 906, a RAM 907, a mouse 908, a keyboard 909, a bus 910, and a monitor 911. The image generating device 100 is also connected to an HMD 10 having an imaging unit 103 and a display unit 110. The imaging unit 103 and the display unit 110 will be described later.

[0015] The CPU 901 is a control unit that controls each component of the image generating device 100. The ROM 902 is a ROM (Read Only Memory) that non-temporarily stores programs to be executed by the CPU 901. The I / F 903 is an interface that connects the imaging unit 103 to the bus 910. The storage medium drive 905 is a device that reads and writes data from and to a storage medium (a storage medium connectable to the image generating device 100).

[0016] The external storage device 906 is a device capable of storing various types of information. The RAM 907 is a random access memory (RAM) that provides a working area for the CPU 901. The mouse 908 and keyboard 909 are operation members that allow the user to input instructions. The bus 910 connects the various components of the image generating device 100. The monitor 911 displays various images.

[0017] Fig. 10 shows a state of verification in a virtual space (hereinafter referred to as "virtual verification"). In Fig. 10, a worker 1010 is a user wearing an HMD (head mounted display) 10 on his head. The worker 1010 holds a wrench 1020 in his hand as a movable object used in the work. The worker 1010 enters inside a device 700 in the virtual space and performs virtual verification. At this time, the worker 1010, the HMD 10, and the wrench 1020 exist not only in the virtual space but also in the real space.

[0018] Generally, a CG model generated from a CAD model has a large number of polygons and nodes. Therefore, when the worker 1010 observes (operates) the CG model, the frame rate of image (video; video) generation (display update) tends to decrease. Generally, if an image can be generated at 30 frames / second or more, an experience using the HMD 10 is certainly possible. However, when the CG model has a large number of polygons and nodes, the image generation speed decreases to, for example, 5 to 10 frames / second, making it difficult for the worker 1010 to realize (reproduce) virtual verification. Therefore, in the first embodiment, a method of realizing virtual verification using a CG model expressed with a small number of polygons by extracting only information necessary for the work before rendering a certain CG model is shown.

[0019] 1 shows a block diagram of an image generating device 100 that generates an image of a virtual space (hereinafter referred to as a “virtual image”) to be displayed on an HMD 10. The image generating device 100 includes a storage unit 150, a virtual image generating unit 160, an extracting unit 180, an estimating unit 190, and a position and orientation estimating unit 195.

[0020] The storage unit 150 manages information necessary to execute the image generating device 100. For example, the storage unit 150 transmits information read from the RAM 907 to other functional blocks. The storage unit 150 also receives information on the processing results of other functional blocks and writes the received information on the processing results to the RAM 907.

[0021] For example, the storage unit 150 stores (holds) CG model information, information on the position and orientation of a virtual camera, information on the position and orientation of a movable object, virtual camera parameters, gap area model information, necessary area information, and virtual images.

[0022] Here, the CG model information includes edge information and texture information (information such as surface color and pattern) as well as vertex positions of three-dimensional polygons in a model coordinate system unique to the CG model. The CG model information also includes information on the "CG model of device 700" and information on the "CG model of wrench 1020, which is a movable object," as shown in FIG.

[0023] The information on the position and orientation of the virtual camera is a parameter that indicates "the position and orientation of the virtual camera in the reference coordinate system." Here, the virtual camera is a camera (imaging device) that captures an image of the virtual space. The virtual camera is a virtual object of the imaging unit 103 of the HMD 10 (for example, an imaging unit including an imaging element that captures an image for the right eye and an imaging element that captures an image for the left eye). The reference coordinate system may be an arbitrary reference coordinate system in the virtual space.

[0024] The information on the position and orientation of a movable object is a parameter that represents the position and orientation of the movable object in a reference coordinate system. The virtual camera parameters are parameters that represent the principal point position, focal length, and the relative positions and orientations of the left and right image sensors (lenses).

[0025] The gap area model information includes information on the "CG model of the gap area 720" generated by the extraction unit 180 (see FIG. 7C). The gap area model information includes, for example, vertex positions of the 3D polygons of the CG model of the gap area 720, edge information, and texture information (information on the surface color and pattern, etc.). The "gap area" is a space surrounded by a certain CG model within a necessary area (area required in virtual verification) specified by the worker 1010. (For example, a space surrounded in at least three of six directions, i.e., up, down, left, right, front, back, etc.) Note that the "gap area" may be an area of ​​space surrounded by a certain CG model, regardless of whether it is within a range specified by the operator 1010.

[0026] The necessary area information is information that defines the necessary area (three-dimensional area). The necessary area information stores, for example, information on the three-dimensional coordinates of the vertices of a rectangular parallelepiped that represents the necessary area. The necessary area information is registered in advance by the worker 1010. The necessary area information is used by the extraction unit 180 when generating a CG model of the gap area 720. For example, the necessary area 705 indicated by the dashed line in FIG. 7A is the area specified by the worker 1010.

[0027] Note that storage unit 150 may have information other than the above, or may not have some of the above information, depending on the content to be used.

[0028] In order to reduce the amount of information of a CG model having a large number of polygons or nodes (such as the CG model of device 700 stored in storage unit 150), extraction unit 180 extracts only gap region 720 necessary for virtual verification from the CG model. In other words, extraction unit 180, as a model generation unit, generates a CG model of gap region 720 based on a certain CG model.

[0029] In the first embodiment, as shown in Fig. 10, a use case will be described in which a worker 1010 works in a virtual space by inserting his / her head and hands into the space inside the device 700. In this use case, the "area necessary for virtual verification" may be only the "area that may come into contact with the hands and head of the worker 1010." Note that the first embodiment is not limited to the "case of extracting the gap area 720 of the device 700" but can also be applied to the "case of handling the internal area surrounded by parts of the device 700."

[0030] The estimation unit 190 estimates the position and orientation of the HMD 10 (display unit 110) in real space. For example, an external camera that observes "multiple LEDs provided on the surface of the HMD 10" may be installed in the real space. In this case, the estimation unit 190 can estimate the position and orientation of the HMD 10 based on the coordinates of the multiple LEDs in the image acquired by the external camera.

[0031] The position and orientation estimation unit 195 estimates the position and orientation of a movable object in real space. For example, an external camera that observes "multiple LEDs provided on the surface of the wrench 1020, which is a movable object," may be installed in real space. In this case, the position and orientation estimation unit 195 can estimate the position and orientation of the wrench 1020 based on the coordinates of the images of the multiple LEDs in the image captured by the external camera.

[0032] The virtual image generating unit 160 generates a virtual image (renders a virtual image) based on the gap area model information, CG model information, virtual camera position and orientation information, movable object position and orientation information, and virtual camera parameters stored in the storage unit 150. More specifically, the virtual image generating unit 160 renders (draws) the CG model of the gap area 720 by referring to the gap area model information so as to express the CG model of the gap area 720 photographed by the virtual camera with image quality according to the virtual camera parameters. The virtual image generating unit 160 renders the CG model of the movable object based on the movable object information and the movable object position and orientation information of the movable object in the CG model information so as to express the CG model of the movable object photographed by the virtual camera with image quality according to the virtual camera parameters.

[0033] The virtual image generator 160 transmits the generated virtual image to the storage unit 150 .

[0034] In the past, rendering (drawing) of a CG model was performed by the device 700, so the rendering processing speed was slow, and virtual images were generated at 5 to 10 frames per second. However, In the first embodiment, the virtual image generating unit 160 does not render a CG model of the device 700, but renders a CG model of the gap region 720 as shown in Fig. 7C. This enables the virtual image generating unit 160 to generate a virtual image at a faster update rate than before.

[0035] Virtual image generating unit 160 also determines contact between the surface of the CG model of gap region 720 and the CG model of wrench 1020, which is a movable object. Virtual image generating unit 160 acquires shape information of the area where contact has occurred as shape information of the contact area. Virtual image generating unit 160 records the shape information of the contact area as part of the CG model of gap region 720 (or as associated information).

[0036] Fig. 8A shows a virtual image generated by virtual image generating unit 160. Contact area 810 shown in Fig. 8B shows an area where the CG model of device 700 (the surface of the CG model of gap area 720) and the CG model of wrench 1020 come into contact when worker 1010 is performing virtual verification. Note that contact area 810 may be a contact area between the surface of the CG model of gap area 720 and the CG model of wrench 1020, or may be an area of ​​the CG model of wrench 1020 that is not included in the CG model of gap area 720.

[0037] It should be noted that virtual image generator 160 may render only the CG model of interstitial region 720, or may further render parts or parts of the CG model of device 700.

[0038] The display unit 110 is incorporated in the HMD 10 worn by the worker 1010. The display unit 110 reads out a virtual image stored in the storage unit 150 and displays the virtual image. The display unit 110 may display (present) one virtual image corresponding to both the left and right eyes, or may display different virtual images for each of the left and right eyes.

[0039] 2 is a diagram showing the internal configuration of the extraction unit 180. The extraction unit 180 includes a voxel grid setting unit 210, a mesh generation unit 230, and an attribute information setting unit 240.

[0040] The voxel grid setting unit 210 sets a voxel grid (a voxel grid) based on a rectangular parallelepiped region indicated by the necessary region information stored in the storage unit 150. Specifically, the voxel grid setting unit 210 sets a voxel grid that can divide the rectangular parallelepiped region into a plurality of voxels (cubes) each having a predetermined length on one side. FIG. 7B shows an example of a voxel grid 710. Since the number of polygons in a mesh (polygon mesh) of a three-dimensional model is determined according to the size of one side of a voxel, it is desirable to set the size according to resources at the time of rendering. A "mesh" is one of the forms for expressing the three-dimensional shape of an object. In addition, when it is desired to improve the reproducibility of the shape of a CG model at the time of virtual verification, the reproducibility can be adjusted by setting the size of one side of a voxel to be small. The information on the set voxel grid is transmitted to the mesh generating unit 230.

[0041] Mesh generation unit 230 generates a CG model of gap region 720 by referring to vertex positions and edge information of the range included in the region of voxel grid 710 of the CG model of device 700. For example, mesh generation unit 230 detects three-dimensional vertex positions of the CG model of device 700 from the position of each voxel, and generates a mesh of the CG model of gap region 720 that can be defined by the three-dimensional vertex positions and edge information. Fig. 7C shows the CG model of gap region 720, which is a mesh model.

[0042] In addition, the number of polygons in the CG model of the device 700 is very large (exceeding a predetermined threshold). In this case, mesh generating unit 230 may refer to only the vertex position (and edge information) that is closest to each sampling point evenly distributed within each voxel, among all the vertex positions of the polygons of the CG model of device 700.

[0043] In the first embodiment, not only a mesh generation method using a voxel grid, but also any method of generating a CG model of the gap region 720 based on the CG model of the device 700 can be applied. For example, the marching cubes method may be used as the mesh generation method. For example, each of the eight vertices of each voxel is given an attribute of "0" indicating that it is inside the CG model of the device 700, or an attribute of "1" indicating that it is outside the CG model of the device 700. Then, one of 15 types of mesh shape patterns set in advance according to the attribute of the eight vertices of each voxel is automatically selected as the mesh shape of the CG model in the voxel. By assigning the selected mesh shape to each voxel, a mesh model suitable for the surface shape of the CG model can be automatically generated.

[0044] Furthermore, mesh generation unit 230 may generate a mesh of a CG model of gap region 720 that is defined not by the three-dimensional vertex positions of the CG model of device 700, but by "a position shifted a certain distance from the vertex positions away from device 700 (toward the gap region)." In this way, by generating a CG model of gap region 720 at a location a certain distance from the surface of the CG model of device 700, it is possible to provide a certain margin (clearance) for verification when determining contact during virtual verification.

[0045] The attribute information setting unit 240 copies "attribute information set in the CG model of the device 700" to "attribute information of the mesh of the CG model of the gap region 720 generated by the mesh generating unit 230." For example, the attribute information setting unit 240 selects "the vertex of the polygon of the CG model of the device 700" that is closest to the first vertex of the polygon of the CG model of the gap region 720. The attribute information setting unit 240 copies the attribute information (such as the UV coordinates of the texture and the alpha value of the polygon vertex) set to the vertex of the selected polygon to the attribute information of the first vertex.

[0046] In this way, the attribute information setting unit 240 copies the attribute information of the CG model of the device 700 to the attribute information of the CG model in the gap area 720. As a result, the virtual image displayed on the display unit 110 of the HMD 10 looks like a virtual image that is a rendering of the CG model of the device 700.

[0047] As described above, the extraction unit 180 can obtain a mesh with a small amount of information to be rendered by extracting the area required for virtual verification from the CG model of the device 700 having a large number of polygons and nodes. Here, the mesh is automatically outputted when the worker 1010 simply designates a three-dimensional area as the required area. For this reason, it is possible to shorten the work time of the worker 1010, compared to a case where the worker 1010 individually selects each part of the CG model to be displayed in virtual verification. In addition, the worker 1010 can adjust the update speed of the rendering of the virtual image to 30 frames / second or more by selecting (inputting) the size of the voxels of the voxel grid as a parameter.

[0048] The overall processing of the image generating device 100 according to the first embodiment will be described in detail with reference to the flowchart of FIG.

[0049] In step S310, the extraction unit 180 reads out the CG model of the device 700 and the CG model of the wrench 1020 from the storage unit 150.

[0050] In step S320, the extraction unit 180 extracts the necessary area information from the CG model of the device 700. Based on this, a CG model is generated for the gap region 720. The process of step S320 will be described in detail later.

[0051] In step S340, the virtual image generator 160 reads out from the storage unit 150 the virtual camera parameters required for rendering.

[0052] In step S350, the image generating device 100 generates a virtual image based on the CG model of the gap region 720, the virtual camera parameters, and the like. The image generating device 100 transmits the virtual image to the display unit 110. The display unit 110 then displays the virtual image. The process of step S350 will be described in detail later.

[0053] In step S360, storage unit 150 determines whether or not an end instruction has been issued by worker 1010. If it is determined that an end instruction has not been issued, the process returns to step S350. If it is determined that an end instruction has been issued, the process of this flowchart ends.

[0054] The flowchart in FIG. 4 shows details of the process of generating a CG model of the gap region 720 (the process of step S320).

[0055] In step S 410 , the voxel grid setting unit 210 sets a voxel grid based on the rectangular parallelepiped area of ​​the necessary area information stored in the storage unit 150 .

[0056] In step S420, mesh generator 230 generates a CG model of gap region 720 based on information about the CG model of the range of device 700 included in the voxel grid area (such as vertex position information and edge information).

[0057] In step S430, attribute information setting unit 240 copies (sets) the attribute information that was set in the CG model of device 700 to the attribute information of the CG model in gap area 720. In other words, attribute information setting unit 240 uses the attribute information that was set in the CG model of device 700 as the attribute information of the CG model in gap area 720 as well.

[0058] The flowchart in FIG. 5 shows details of the process for generating a virtual image (the process of step S350).

[0059] In step S510, the estimation unit 190 estimates the position and orientation of the display unit 110 of the HMD 10 in real space. Furthermore, the estimation unit 190 determines the position and orientation of the virtual camera based on the position and orientation of the display unit 110.

[0060] In step S520, the position and orientation estimation unit 195 estimates the position and orientation of the wrench 1020, which is a movable object in real space. In addition, the position and orientation estimation unit 195 determines the position and orientation of the CG model of the wrench 1020 in virtual space, based on the position and orientation of the wrench 1020 in real space.

[0061] In step S530, virtual image generation unit 160 generates a virtual image by rendering the CG model of gap region 720 and the CG model of wrench 1020. At this time, virtual image generation unit 160 performs these processes based on the CG model of gap region 720, CG model information, the position and orientation of the virtual camera, the position and orientation of wrench 1020, and virtual camera parameters.

[0062] In step S540, the virtual image generating unit 160 transmits the virtual image to the display unit 110. When the display unit 110 acquires the virtual image, it displays the virtual image. The image generator 160 controls the display 110 to display a virtual image.

[0063] According to the first embodiment, the image generating device 100 can quickly realize rendering of the CG model of the gap region 720. Therefore, the worker 1010 can properly observe the internal state of the CG model of the device 700 by looking at the CG model of the gap region 720 displayed at an appropriate update speed.

[0064] <Embodiment 2> In the first embodiment, a method has been described in which a virtual image of a CG model of the gap region 720 is presented to the HMD 10. On the other hand, an HMD for presenting a mixed reality space (a video see-through HMD or an optical see-through HMD) can also be used.

[0065] In the second embodiment, a case will be described where a video see-through HMD is used as the HMD 10. In the second embodiment, the setting of the rendering mode of the CG model of the gap region 720 will also be described.

[0066] 11 shows a block diagram of an image generating device 100 according to the second embodiment. In the second embodiment, the functional units having the same functions as those in the first embodiment are given the same numbers. The description of the functional units in the second embodiment that are the same as those in the first embodiment will be omitted. The HMD 10 has an imaging unit 103, an input unit 104, and a display unit 110. The image generating device 100 has an acquisition unit 105, a setting unit 155, and a synthesis unit 170 in addition to the configuration according to the first embodiment.

[0067] The imaging unit 103 is a camera arranged in front of the eyes (position in the line of sight direction) of the worker 1010 wearing the HMD 10. The imaging unit 103 captures "the real space seen by the eyes of the worker 1010 when the HMD 10 is not worn." The imaging unit 103 may be configured as a stereo camera so that images can be presented on the left and right display units of the HMD 10, respectively. The imaging unit 103 may be a monocular camera. The imaging unit 103 may also be a color camera that captures color images, or a monochrome camera that captures monochrome images.

[0068] The input unit 104 is an operation member (input device) such as a keyboard or a mouse. The setting unit 155 sets (switches) a rendering mode (a mode for switching a rendering method of the CG model of the gap region 720) in response to an operation of the operator 1010 on the input unit 104. The input unit 104 may be a keyboard 909 or a mouse 908 included in the image generating device 100.

[0069] The acquisition unit 105 acquires a camera image (captured image) captured by the imaging unit 103 of the real space, and stores the camera image in the RAM 907. The acquisition unit 105 also outputs the acquired camera image to the storage unit 150.

[0070] The storage unit 150 has information on a camera image, camera parameters, and a rendering mode in addition to the information managed in the first embodiment. The camera parameters are parameters of the imaging unit 103. The camera parameters are parameters representing the principal point position, focal length, lens distortion correction parameters, and the like of the imaging unit 103. When the imaging unit 103 has two imaging elements, one on the left and one on the right, the camera parameters may have parameters of the relative positions and orientations of the left and right imaging elements. Furthermore, the rendering mode information includes a flag that determines the rendering mode (display attribute) of the CG model of the gap region 720.

[0071] The virtual image generating unit 160 refers to the information stored in the storage unit 150 referred to in the first embodiment and the rendering mode information, and performs rendering of the CG model of the gap area 720. In this way, the virtual image generating unit 160 generates a virtual image. For example, When the dubbing mode is set to the "verification mode", the virtual image generation unit 160 renders the CG model of the gap region 720, similarly to the first embodiment. That is, the virtual image generation unit 160 renders the CG model of the gap region 720 based on the attribute information set in the CG model of the gap region 720 (= the attribute information of the CG model of the device 700).

[0072] On the other hand, when the rendering mode is the "contact check mode", the virtual image generating unit 160 uses "attributes that make the contact area 810 easily visible from a bird's-eye view" for rendering, instead of the attributes set in the CG model of the gap area 720 (see FIG. 8B). For example, when the rendering mode is the "contact check mode", rendering of the CG model of the gap area 720 is performed so that the entire contact area 810 is visible from the worker 1010 (virtual camera). For example, the virtual image generating unit 160 uses attributes that display only the sides of the CG model of the gap area 720 with dotted lines (or three-dimensional vertices with a point cloud), or attributes that express the CG model of the gap area 720 with a monochromatic semi-transparent texture. In this way, the virtual image generating unit 160 generates a virtual image so that the contact area 810 is easily observed when the worker 1010 looks down on it.

[0073] The synthesis unit 170 synthesizes the virtual image stored in the storage unit 150 with the camera image. The synthesized image is displayed on the display unit 110. The worker 1010 can visually recognize the image in which the gap area is synthesized with the camera image.

[0074] 12 is a flowchart showing details of the processing of the image generating device 100 according to embodiment 2. The same steps as those in embodiment 1 are given the same numbers, and the description of the processing of those steps will be omitted.

[0075] In step S1200, the acquisition unit 105 acquires a camera image from the imaging unit 103, and stores the camera image in the storage unit 150.

[0076] In step S1210, the virtual image generation unit 160 renders the CG model of the gap region 720 in accordance with the rendering mode by processing similar to the processing in step S530 in embodiment 1. In this way, the virtual image generation unit 160 generates a virtual image including the CG model of the gap region 720.

[0077] In step S1220, the synthesis unit 170 synthesizes the virtual image and the camera image to generate a synthetic image.

[0078] In step S1230, the composition unit 170 transmits the composite image to the display unit 110, thereby displaying the composite image on the display unit 110.

[0079] According to the second embodiment, even an HMD that presents a mixed reality space can display the CG model of the gap area 720, which has a high update speed. Therefore, the worker 1010 can properly observe the internal state of the CG model of the device 700 by looking at the CG model of the gap area 720.

[0080] <Embodiment 3> In the third embodiment, the image generating device 100 further includes an editing unit 185 and an output unit 1310 in addition to the configuration of the second embodiment. In the third embodiment, the image generating device 100 edits the results of the virtual verification by the worker 1010 and feeds them back to the designer of the CG model of the device 700.

[0081] FIG. 13 is a block diagram showing a configuration for realizing an image generating device 100 according to the third embodiment. The same numbers are used for functional units that realize the same functions as in the second embodiment. The description of the functional units in FIG. 13 that are the same as those in the second embodiment will be omitted.

[0082] The editing unit 185 provides an interface for editing the shape of the CG model of the gap region 720 (editing interface). By using the editing interface, the worker 1010 can edit (change) the shape of the CG model of the gap region 720 while observing the CG model of the gap region 720 displayed on the display unit 110. In addition, the CG model of the gap region 720 edited by the editing unit 185 is transmitted to the storage unit 150 after the editing is completed. Furthermore, the CG model of the gap region 720 stored in the storage unit 150 is updated. The editing unit 185 may display a plurality of options on the display unit 110 so that the worker 1010 can select one of a plurality of modes included in the editing mode of the CG model of the gap region 720.

[0083] Also, modes may be assigned in advance to specific keys on the keyboard 909, and the mode may be switched by pressing the specific key. The editing mode includes, for example, two modes: a "correction mode" (deletion mode) and a "confirmation mode." If necessary, the editing state may be managed by inputting an "undo command" and a "confirm command." Details of each process will be described later.

[0084] The output unit 1310 converts the CG model of the gap area 720 (the CG model of the gap area 720 edited by the editing unit 185) so as to be adapted to a data format for output to an external CAD software. The output unit 1310 outputs the converted CG model of the gap area 720. The CG model of the gap area 720 output from the output unit 1310 is drawn in the same coordinate system as the CG model of the device 700 on the CAD software side. This allows the designer to observe the difference between the CG model of the gap area 720 and the CG model of the device 700. By checking the difference between the CG model of the gap area 720 and the CG model of the device 700, the designer can accurately grasp the correction parts of the CG model of the device 700. This reduces the possibility of miscommunication between the worker 1010 and the designer.

[0085] In order to clearly indicate the edited area of ​​the CG model of the gap region 720 using CAD software, the edited area of ​​the CG model of the gap region 720 may be highlighted (for example, the color may be changed). Furthermore, text information (annotation) indicating the content of the edit may be added to the edited area of ​​the CG model of the gap region 720.

[0086] Fig. 14 is a flowchart showing details of the process of the image generating device 100 according to embodiment 3. In Fig. 14, the same steps as those in embodiment 2 are given the same numbers, and the following description will be omitted.

[0087] In step S1400, storage unit 150 determines whether or not an instruction to execute the edit mode has been given from worker 1010. If it is determined that an instruction to execute the edit mode has been given, the process proceeds to step S1410. If it is determined that an instruction to execute the edit mode has not been given, the process proceeds to step S1210.

[0088] In step S1410, editing unit 185 edits the shape of the CG model of gap region 720. Note that editing unit 185 edits the shape of the CG model of gap region 720 in response to an operation performed by "worker 1010 observing the CG model of gap region 720 displayed on display unit 110." When editing of the shape of the CG model of gap region 720 is completed, editing unit 185 outputs the CG model of gap region 720 to output unit 1310.

[0089] FIG. 6 shows details of the process in step S1410 (editing the CG model of the gap area 720). 14 is a flowchart showing the process of step S1410. The process of step S1410 is repeatedly executed while the edit mode is set. When the edit mode is changed to another mode, the process proceeds from step S1410 to step S1210. When the edit mode is set to the finalize mode, the process may proceed from step S1410 to step S1210.

[0090] In step S610, the editing unit 185 displays a "display item prompting the user to select one of a plurality of options in the edit mode" on the display unit 110. The plurality of options include, for example, options of a "correction mode," an "undo command," and a "confirm command." The worker 1010 may select one of the displayed plurality of options with the mouse 908 or a key on the keyboard 909. A three-dimensional cursor according to the position and orientation of the head may be presented on the display unit 110, and the worker 1010 may change the position and orientation of the head to select one of the plurality of options. In addition, when a device that detects the position of the hand (such as Ultraleap's LeapMotion) is used, the worker 1010 may select one of the plurality of options using the position of the hand.

[0091] In step S620, the editing unit 185 determines whether or not the "correction mode" has been selected. If it is determined that the "correction mode" has been selected, the process proceeds to step S650. If it is determined that the correction mode has not been selected, the process proceeds to step S630.

[0092] In step S630, the editing unit 185 determines whether or not the Undo command has been selected. If it is determined that the Undo command has been selected, the process proceeds to step S660. If it is determined that the Undo command has not been selected, the process proceeds to step S640.

[0093] In step S640, the editing unit 185 determines whether or not a confirmation command has been selected. If it is determined that a confirmation command has been selected, the process proceeds to step S670. If it is determined that a confirmation command has not been selected, the process of this flowchart ends.

[0094] In step S650, the editing unit 185 operates in a "correction mode" and corrects the CG model of the gap region 720 so that the surface of the CG model of the gap region 720 and the CG model of the wrench 1020 do not come into contact with each other. First, the editing unit 185 detects a contact area 810 between the surface of the CG model of the gap region 720 and the CG model of the wrench 1020. Then, the editing unit 185 corrects the shape of the CG model of the gap region 720 so as to have an area including the contact area 810. For example, the editing unit 185 translates the polygonal three-dimensional vertices of the CG model of the gap region 720 adjacent to the contact area 810 by a certain amount in a direction in which no contact occurs. That is, the editing unit 185 moves the polygonal vertices of the CG model of the gap region 720 so as to cut off a part of the CG model of the device 700. In the translation process, the editing unit 185 may set a small amount of translation toward the outline of the contact area 810, and may increase the amount of translation toward the center of the contact area 810 as the area approaches the center. By changing the translation distance between the outline and center of the contact area 810, the amount of erosion at the time of actual contact can be approximated.

[0095] An example of the process of step S650 will be described with reference to Figs. 8B and 8C. In Figs. 8B and 8C, the gap region 720 is a region surrounded by the CG model of the device 700. In Fig. 8B, a contact region 810 is recorded in the CG model of the gap region 720. When the correction mode is selected in this state, the editing unit 185 moves the three-dimensional vertices of the CG model of the gap region 720. Fig. 8C shows a state after the three-dimensional vertices of the CG model of the gap region 720 have been moved. In Fig. 8C, the CG model of the gap region 720 has a region 820 including the contact region 810. The editing unit 185 may store the movement amount of each of the moved three-dimensional vertices, and prepare to return the CG model of the gap region 720 to the shape before editing when a restore command is executed. In addition, the C of the gap region 720 in step S650 may be changed to a new shape by changing the C of the gap region 720. The user may be able to freely edit (correct) the G model according to the type and amount of operation performed by the user.

[0096] In step S660, editing unit 185 reverses the translation of the three-dimensional vertices of the CG model in gap area 720 so as to cancel the processing of the immediately previous correction mode. When reversing the translation of the three-dimensional vertices, editing unit 185 refers to the amount of movement for each three-dimensional vertex recorded in step S650.

[0097] In step S670, the editing unit 185 sets the “check mode” and outputs the CG model of the current gap area 720 to the output unit 1310.

[0098] As described above, the image generating device 100 according to the third embodiment edits the CG model of the gap region 720, and outputs the CG model of the gap region 720 whose shape has been edited to the designer's CAD software. This allows the designer to easily grasp the state of the CG model of the gap region 720, such as its shape.

[0099] It is noted that, although it has been described above that editing unit 185 can edit (change) the shape of the CG model in gap region 720, editing unit 185 may also be able to edit the size or attributes of the CG model in gap region 720. Alternatively, editing unit 185 may be able to edit the position and orientation of the CG model in gap region 720 (the CG model in gap region 720 can be moved and rotated) in response to a user operation.

[0100] Although the present invention has been described in detail based on the preferred embodiments, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Parts of the above-described embodiments may be combined as appropriate.

[0101] Also, in the above, "If A is equal to or greater than B, proceed to step S1, and if A is smaller (lower) than B, proceed to step S2" may be read as "If A is greater (higher) than B, proceed to step S1, and if A is equal to or less than B, proceed to step S2." Conversely, "If A is greater (higher) than B, proceed to step S1, and if A is equal to or less than B, proceed to step S2" may be read as "If A is greater (higher) than B, proceed to step S1, and if A is smaller (lower) than B, proceed to step S2." Therefore, unless a contradiction occurs, "equal to or greater than A" may be read as "equal to or greater than A (high; long; many)," and "equal to or less than A" may be read as "equal to or less than A (low; short; few)." And, "equal to or greater than A" may be read as "equal to or greater than A," and "equal to or less than A" may be read as "equal to or less than A."

[0102] Each functional unit in each of the above embodiments (variations) may or may not be individual hardware. The functions of two or more functional units may be realized by common hardware. Each of a plurality of functions of one functional unit may be realized by individual hardware. Two or more functions of one functional unit may be realized by common hardware. Furthermore, each functional unit may or may not be realized by hardware such as an ASIC, FPGA, or DSP. For example, the device may have a processor and a memory (storage medium) in which a control program is stored. Then, the functions of at least some of the functional units of the device may be realized by the processor reading and executing the control program from the memory.

[0103] (Other embodiments) The present invention provides a program for implementing one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and executes the program on a computer of the system or device. The present invention can be realized by a process in which one or more processors in a computer read and execute a program, or by a circuit (for example, an ASIC) that realizes one or more functions.

[0104] The disclosure of the above embodiments includes the following configurations, methods, and programs. (Configuration 1) a model generating means for generating a CG model of a gap region, which is a region surrounded by a first CG model, based on the first CG model; an image generating means for generating an image of a virtual space by rendering a CG model of the gap region without rendering the first CG model; a control means for controlling a display means to display an image based on the image of the virtual space; 13. An information processing device comprising: (Configuration 2) the gap region is an area within a range designated by a user and is surrounded by the first CG model; 2. The information processing device according to configuration 1. (Configuration 3) a synthesis means for generating an image by synthesizing a captured image of a real space with an image of the virtual space, The control means controls the display means to display the image generated by the synthesis means. 3. The information processing device according to configuration 1 or 2. (Configuration 4) the model generating means generates a CG model of the gap region based on a shape of the first CG model by using a voxel grid. 4. The information processing device according to any one of configurations 1 to 3. (Configuration 5) the model generating means, after generating the CG model of the gap region, sets attribute information of the first CG model to attribute information of the CG model of the gap region. 5. The information processing device according to any one of configurations 1 to 4. (Configuration 6) The method further includes acquiring a CG model of a movable object, the movable object being a moving object; the model generating means, after generating the CG model of the gap region, corrects the CG model of the gap region so that a surface of the CG model of the gap region and the CG model of the movable object do not come into contact with each other. 6. The information processing device according to any one of configurations 1 to 5. (Configuration 7) a setting means for setting a rendering mode of the CG model of the gap region to either a first rendering mode for rendering the CG model of the gap region using attribute information set for the CG model of the gap region, or a second rendering mode for rendering the CG model of the gap region without using attribute information set for the CG model of the gap region, the image generating means renders a CG model of the gap area in accordance with the rendering mode set by the setting means; 6. The information processing device according to any one of configurations 1 to 5. (Configuration 8) The method further includes acquiring a CG model of a movable object, the movable object being a moving object; the second rendering mode is a mode in which the CG model of the gap region is rendered so that a user can see an area of ​​the surface of the CG model of the gap region that is in contact with the CG model of the movable object. 8. The information processing device according to configuration 7. (Configuration 9) the second rendering mode is a rendering mode in which each side of the CG model in the gap region is displayed with a dotted line, or a rendering mode in which a monochromatic semi-transparent texture is used as a texture for the CG model in the gap region. 9. The information processing device according to configuration 8. (Configuration 10) The method further comprises an editing means for editing the shape of the CG model of the gap area generated by the model generating means in response to a user operation. 10. The information processing device according to any one of configurations 1 to 9. (Configuration 11) Further, an output unit for outputting a CG model of the gap area is provided. 11. The information processing device according to any one of configurations 1 to 10. (method) a model generating step of generating a CG model of a gap region, which is a region surrounded by a first CG model, based on the first CG model; an image generating step of generating an image of a virtual space by rendering a CG model of the gap region without rendering the first CG model; a control step of controlling a display means to display an image based on the image of the virtual space; 13. An information processing method comprising: (program) 12. A program for causing a computer to function as each of the means of the information processing device according to any one of configurations 1 to 11. [Explanation of symbols]

[0105] 100: Image generating device (information processing device), 180: Extraction unit (model generation unit), 160: Virtual image generation unit

Claims

1. A model generation means for generating a CG model of a gap region, which is a region enclosed by a first CG model, based on the first CG model, An image generation means that generates an image of a virtual space by rendering the CG model of the gap region without rendering the entire first CG model, A control means for controlling the display means to display an image based on the image of the virtual space, An information processing device characterized by having the following features.

2. The gap region is an area within a range specified by the user and is an area enclosed by the first CG model. The information processing apparatus according to feature 1.

3. The system further includes a synthesis means for generating an image by combining an image captured of the real space with an image of the virtual space, The control means controls the display means to display the image generated by the synthesis means. The information processing apparatus according to claim 1 or 2.

4. The model generation means generates a CG model of the gap region based on the shape of the first CG model using a voxel grid. The information processing apparatus according to claim 1 or 2.

5. The model generation means, after generating the CG model of the gap region, sets the attribute information of the first CG model to the attribute information of the CG model of the gap region. The information processing apparatus according to claim 1 or 2.

6. The system further includes means for acquiring a CG model of a movable object, which is a moving object. The model generation means, after generating the CG model of the gap region, corrects the CG model of the gap region so that the surface of the CG model of the gap region does not come into contact with the CG model of the movable object. The information processing apparatus according to claim 1 or 2.

7. The system further includes a setting means for setting the rendering mode of the CG model of the gap region to either a first rendering mode that renders the CG model of the gap region using attribute information set for the CG model of the gap region, or a second rendering mode that renders the CG model of the gap region without using attribute information set for the CG model of the gap region. The image generation means renders the CG model of the gap area according to the rendering mode set by the setting means. The information processing apparatus according to claim 1 or 2.

8. The system further includes means for acquiring a CG model of a movable object, which is a moving object. The second rendering mode is a mode in which the CG model of the gap region is rendered so that the user can see the portion of the surface of the CG model of the gap region that is in contact with the CG model of the movable object. The information processing apparatus according to feature 7.

9. The second rendering mode is a rendering mode in which each edge of the CG model of the gap area is displayed as a dotted line, or a rendering mode in which a single-color semi-transparent texture is used for the texture of the CG model of the gap area. The information processing apparatus according to feature 8.

10. The system further includes editing means for editing the shape of the CG model of the gap region generated by the model generation means in accordance with user operations. The information processing apparatus according to claim 1 or 2.

11. The system further includes output means for outputting a CG model of the gap region. The information processing apparatus according to claim 1 or 2.

12. A model generation step of generating a CG model of a gap region, which is a region enclosed by the first CG model, based on the first CG model, An image generation step that generates an image of a virtual space by rendering the CG model of the gap region without rendering the entire first CG model, A control step of controlling the display means to display an image based on the image of the virtual space, An information processing method characterized by having the following features.

13. A program for causing a computer to function as each of the means of the information processing apparatus described in claim 1 or 2.

Images