Image control system, image control program, storage medium having image control program recorded thereon, and image control method

Abstract

To provide an image control system, a structure, an image control program, a storage medium having an image control program recorded therein, and an image control method that can display a content image larger than an indoor space without discomfort.SOLUTION: An image control system comprises a control unit 3 that displays, on a display 42, CG images CG1, CG2 superimposed on a reality image G1 that is an image of a specific space 20. At least part of an area of a surfaces of a wall 21 and the ceiling 22 is applied with a specific pattern 6 to cover the area. The control unit 3 determines an area applied with the specific pattern 6 in the reality image G1 as an occlusion-off area 5, when superimposing the CG image CG1 on the occlusion-off area 5, does not execute occlusion, and when superimposing the CG image CG2 on an area other than the occlusion-off area 5 in the reality image G1, executes occlusion.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to a building for augmented reality or mixed reality, a design method for a building for augmented reality or mixed reality, an interior design for augmented reality or mixed reality, a design method for an interior design for augmented reality or mixed reality, an image control system, an image control program, a storage medium on which an image control program is recorded, and an image control method for displaying augmented reality or mixed reality. [Background technology]

[0002] In augmented reality (AR) and mixed reality (MR), when a real object is present in the foreground and a CG (Computer Graphics) object is located in the background, occlusion has been known, which renders the real object so that it appears in front of the CG object in the background (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2022-43909 Summary of the Invention [Problem to be solved by the invention]

[0004] In augmented reality and mixed reality, which display an actually captured real-world image overlaid with a CG content image, if a content image larger than the indoor space is displayed in a closed space such as indoors to perform occlusion, part of the content image will penetrate the ceiling or walls of the indoor space, and part of the content image will be hidden by the ceiling or walls.

[0005] An object of the present invention is to provide a building for augmented reality or mixed reality capable of displaying a content image larger than an indoor space, a method for designing a building for augmented reality or mixed reality, an interior for augmented reality or mixed reality, a method for designing an interior for augmented reality or mixed reality, an image control system, an image control program, and an image control program. [Means for solving the problem]

[0006] The image control system of the present invention includes a control unit that overlays a predetermined content image onto a real image, which is an image of a specific space having walls and a ceiling, captured by a camera in a display terminal for augmented reality or mixed reality equipped with a camera and a display, and displays the image on the display, and an occlusion-off area, which is at least a portion of the surface of the wall and ceiling, is given a specific pattern that is set in advance to cover the occlusion-off area, and the control unit determines that the area in the real image with the specific pattern is the occlusion-off area, and when the occlusion-off area and the content image are superimposed, occlusion is not performed, and when the content image and an area in the real image other than the occlusion-off area are superimposed, occlusion is performed.

[0007] In addition, the image control method of the present invention includes a mixed reality control step of superimposing a predetermined content image on a real image, which is an image of a specific space having walls and a ceiling, captured by a camera in a display terminal for augmented reality or mixed reality equipped with a camera and a display, and displaying the superimposed content image on the display, and an occlusion-off area, which is at least a portion of the surface of the wall and ceiling, is given a specific pattern that is set in advance so as to cover the occlusion-off area, and the mixed reality control step determines the area in the real image with the specific pattern to be the occlusion-off area, and when the occlusion-off area and the content image are superimposed, occlusion is not performed, and when the content image is superimposed on an area in the real image other than the occlusion-off area, occlusion is performed.

[0008] In addition, the image control system of the present invention includes a control unit that superimposes a predetermined content image onto a real image, which is an image of a specific space having walls and a ceiling, captured by a camera in a display terminal for augmented reality or mixed reality equipped with a camera and a display, and displays the image on the display, and when superimposing the content image onto the real image, the control unit erases lines corresponding to corners of the specific space from the real image without performing occlusion.

[0009] In addition, the image control system of the present invention includes a control unit that overlays a specified content image without performing occlusion onto a real image, which is an image of a specific space having walls and a ceiling, captured by a camera in a display terminal for augmented reality or mixed reality equipped with a camera and a display, and the inner surface of the specific space is plain.

[0010] According to these configurations, it is possible to display a content image larger than the specific space in the specific space.

[0011] It is also preferable that the control unit further erases the specific pattern from the real image.

[0012] According to this configuration, by erasing the specific pattern that is unnecessary for the user, it is possible to generate an image that does not feel strange to the user.

[0013] In addition, it is preferable that the specific pattern varies depending on the position at which it is applied within the occlusion off area, and the control unit further identifies the relative position of the display terminal with respect to the specific space from the specific pattern captured in the real image, and displays the content image according to the identified relative position.

[0014] According to this configuration, the specific pattern differs depending on the position where it is placed within the occlusion off area, so that the specific pattern in the real image makes it possible to identify the relative position of the display terminal that captured the real image with respect to the specific space.

[0015] It is also preferable that the control unit further erases lines corresponding to corners of the specific space from the real image.

[0016] According to this configuration, it is possible to reduce the sense of incongruity that occurs due to not performing occlusion.

[0017] It is also preferable that the display terminal be further provided.

[0018] According to this configuration, the display terminal is included in the image control system.

[0019] It is also preferable that the control unit is built into the display terminal.

[0020] According to this configuration, the display terminal functions as the control unit.

[0021] It is also preferable that the specific space is further provided.

[0022] According to this configuration, the specific space is included in the image control system.

[0023] Furthermore, a building for augmented reality or mixed reality according to the present invention comprises the specific space used in the above-mentioned image control system.

[0024] This augmented or mixed reality building constitutes a specific space used for the image control system described above.

[0025] In addition, the method for designing a building for augmented reality or mixed reality according to the present invention involves designing the above-mentioned building for augmented reality or mixed reality by applying the specific pattern to at least a portion of the surfaces of the walls and ceiling of the building for augmented reality or mixed reality.

[0026] This design method allows for designing buildings for augmented or mixed reality.

[0027] Furthermore, the interior design for augmented reality or mixed reality according to the present invention has the specific pattern applied to at least a portion of the inner surface of the specific space used in the above-mentioned image control system.

[0028] This augmented reality or mixed reality interior design constitutes a specific space used for the image control system described above.

[0029] Further, the method for designing an interior design for augmented reality or mixed reality according to the present invention designs the interior design for augmented reality or mixed reality by applying the specific pattern to at least a part of the inner surface thereof.

[0030] This design method allows interiors to be designed for augmented reality or mixed reality.

[0031] Further, an image control program according to the present invention causes a computer to function as the control unit used in the above-mentioned image control system.

[0032] This image control program causes a computer to function as a control unit used in the above-mentioned image control system.

[0033] Moreover, a storage medium according to the present invention stores the above-mentioned image control program.

[0034] This storage medium records the above-mentioned image control program. Effect of the Invention

[0035] A building for augmented reality or mixed reality, a method for designing a building for augmented reality or mixed reality, an interior for augmented reality or mixed reality, a method for designing an interior for augmented reality or mixed reality, an image control system, an image control program, a storage medium having an image control program recorded thereon, and an image control method configured as described above make it possible to display a content image larger than an indoor space. [Brief description of the drawings]

[0036] [Figure 1] 1 is a conceptual diagram showing an example of a configuration of an image control system according to an embodiment of the present invention. [Diagram 2] FIG. 1 is a conceptual explanatory diagram for explaining the positional relationship when a three-dimensional CG image is arranged in a specific space. [Diagram 3] 3 is an explanatory diagram illustrating an image of a specific space, a CG image, and a third person viewed by a user using a conventional display terminal in the state shown in FIG. 2.

[0023] FIG. [Figure 4] FIG. 4 is an explanatory diagram for explaining an image when occlusion is not performed in the state shown in FIG. 3. [Diagram 5] 4 is a flowchart showing an example of an operation of the image control system. [Figure 6] 1 is an image showing an example of a real image captured by a display terminal. [Figure 7] FIG. 11 is an explanatory diagram showing an example of a composite image G2 generated in step S5. [Figure 8] FIG. 13 is an explanatory diagram showing an example of a composite image G3 generated in step S6. [Figure 9] FIG. 13 is an explanatory diagram showing an example of a composite image G4 generated in step S7. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0037] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In each drawing, the same reference numerals are used to denote the same configuration, and the description thereof will be omitted. In each drawing, XYZ orthogonal coordinate axes are appropriately shown to clarify the directional relationship. FIG. 1 is a conceptual diagram showing an example of the configuration of an image control system 1 according to an embodiment of the present invention. The image control system 1 includes a building 2, a display terminal 4, and a control unit 3.

[0038] The building 2 has a specific space 20 having walls 21a, 21b, 21c, a ceiling 22, and a floor 23. A step 221 is provided in the ceiling 22. The building has an interior with a preset specific pattern 6 applied so as to cover the walls 21a, 21c, the ceiling 22, and a portion of the floor 23. The step 221 in the ceiling 22 and the entire surface of the wall 21b are covered with the specific pattern 6. The specific space 20 has a protrusion 24 that is entirely covered with the specific pattern 6. The protrusion 24 protrudes from the walls 21a, 21b and the ceiling 22.

[0039] It should be noted that the image control system 1 does not necessarily have to include the display terminal 4, and does not necessarily have to include the building 2 and the specific space 20.

[0040] The area covered with the specific pattern 6 is set as the occlusion-off area 5. The specific pattern 6 is a different pattern depending on the position where it is placed within the occlusion-off area 5. As the specific pattern 6, various patterns can be used, such as a pattern known as a random dot pattern or an AR marker similar to a QR code (registered trademark).

[0041] A method for designing the building 2 may be to apply a specific pattern to at least a portion of the surface areas of the walls and ceiling. A method for designing the interior of the specific space 20 may be to apply a specific pattern to at least a portion of the inner surface of the specific space 20.

[0042] The display terminal 4 is a terminal device equipped with a camera 41 and a display 42. As the display terminal 4, a so-called smartphone, a camera-equipped VR goggle such as a head-mounted display, a so-called AR glass, or the like can be used. The display terminal 4 is capable of communicating with the control unit 3. The display terminal 4 may communicate with the control unit 3 by wireless communication such as WiFi (registered trademark), may communicate with the control unit 3 via mobile communication or the Internet, or may communicate with the control unit 3 via a cable in a wired manner.

[0043] The display terminal 4 transmits a real image G1 captured by the camera 41 to the control unit 3, and displays an image on the display 42 in accordance with an instruction from the control unit 3.

[0044] The control unit 3 can be configured using a computer such as a server device or a game machine, etc. Alternatively, the control unit 3 may be built into the display terminal 4.

[0045] The control unit 3 is composed of, for example, a CPU (Central Processing Unit) that executes predetermined arithmetic processing, a RAM (Random Access Memory) that temporarily stores data, a non-volatile storage device, a communication circuit, and peripheral circuits thereof, etc. The control unit 3 operates, for example, by executing an image control program stored in the above-mentioned storage device.

[0046] The control unit 3 superimposes a predetermined content image on a real image G1, which is an image of the specific space 20 captured by the camera 41 of the display terminal 4, and displays the superimposed image on the display 42. The control unit 3 determines an area in the real image G1 to which a specific pattern 6 is applied as an occlusion-off area 5, and does not perform occlusion when the occlusion-off area 5 and the content image are superimposed, and performs occlusion when an area other than the occlusion-off area 5 in the real image G1 and the content image are superimposed.

[0047] Next, before describing the operation of the image control system 1, a usage scenario of mixed reality as envisioned by the present inventor will be described. In mixed reality, a user exists in a real space. Then, a CG image is superimposed on a real image actually captured by a camera and displayed on a mixed reality device such as a smartphone or a head-mounted display. Through the mixed reality device, the user experiences a real space that is, so to speak, extended by CG.

[0048] In this case, if it is possible to display a CG image that is larger than the size of the real space, for example the size of the room the user is in, it is thought that the user can be made to feel the expansion of the space, as if the real space had expanded.

[0049] 2 is a conceptual explanatory diagram for explaining the positional relationship when three-dimensional CG images CG1 and CG2 are placed in specific space 20. CG image CG1 is a three-dimensional CG image of a giraffe that is larger than specific space 20. When CG image CG1 is placed in specific space 20, the giraffe's neck penetrates through ceiling 22, as shown in FIG.

[0050] A three-dimensional CG image CG2 of a cow is placed in the -Y direction of the CG image CG1 in the specific space 20. A real third person R1 is standing in the -X direction of the CG image CG2, and a user U is standing in the -X direction of the third person R1. The user U uses the display terminal 4 to view the specific space 20 via the display 42.

[0051] Fig. 3 is an explanatory diagram for explaining an image of the specific space 20, CG images CG1, CG2, and a third person R1 viewed by a user U using a conventional display terminal in the state shown in Fig. 2. In this case, the walls, ceiling, and floor of the specific space 20, and the third person R1 become real images that are actually photographed, and the CG images CG1 and CG2 are superimposed on this real image.

[0052] In the display image of the conventional display terminal, the neck of the giraffe's CG image CG1, which penetrates the ceiling 22, is hidden by occlusion in the shadow of the ceiling 22. In addition, the cow's CG image CG2 is positioned further back than the third person R1, so part of the CG image CG2 is hidden by the shadow of the third person R1 due to occlusion.

[0053] In this way, when occlusion is performed using a conventional display terminal, it is not possible to display the entire CG image larger than the specific space 20.

[0054] Fig. 4 is an explanatory diagram for explaining an image when occlusion is not performed in the state shown in Fig. 3. When occlusion is not performed, unlike Fig. 3, the CG image CG1 of the giraffe can be displayed in its entirety. On the other hand, the CG image CG2 of the cow, which is positioned further back than the third person R1, is also displayed in its entirety, and the real image of the third person R1, who is in front of the CG image CG2, is covered up by the CG image CG2.

[0055] In this way, if the user U moves around while viewing the specific space 20 using a conventional display terminal with the real image of the third party R1 covered by the CG image CG2, the user U may bump into the third party R1 without noticing its presence. Therefore, even if the entire CG image CG1 of the giraffe can be displayed, it is not preferable not to perform occlusion.

[0056] The following describes the operation of the image control system 1. Fig. 5 is a flowchart showing an example of the operation of the image control system 1. As a premise, the following description will be given taking as an example a case where a user U and a third party R1 are standing at the positions shown in Fig. 2 in the specific space 20 shown in Fig. 1.

[0057] First, the user U captures an image of the specific space 20 shown in Fig. 1 with the camera of the display terminal 4. Then, the real image G1 captured by the display terminal 4 is acquired by the control unit 3 (step S1).

[0058] 6 is an image showing an example of a real image G1 captured by the display terminal 4. The real image G1 shows only the exterior of a specific space 20 and a third person R1 that actually exist.

[0059] Next, the control unit 3 identifies the relative position of the display terminal 4 with respect to the specific space 20 from the specific pattern 6 in the real image G1 (step S2). Since the specific pattern 6 is a different pattern depending on the position where it is placed within the occlusion-off area 5, it is possible to identify the relative position of the display terminal 4 displaying the real image G1 with respect to the specific space 20 from the specific pattern 6 in the real image G1.

[0060] Note that the present invention is not limited to an example in which the relative position of the display terminal 4 with respect to the specific space 20 is specified based on the specific pattern 6. As a method for specifying the relative position of the display terminal 4 with respect to the specific space 20, various known methods can be used.

[0061] Next, the control unit 3 determines the composite position and angle of the CG images CG1 and CG2 with respect to the real image G1 based on the relative positions identified in step S2 (step S3). The CG images CG1 and CG2 correspond to an example of a content image. Note that, although a giraffe and a cow are shown as an example of a content image, the content image is not limited to a specific image. The content image may be, for example, a starry sky or a cityscape.

[0062] Next, the control unit 3 determines the area in the real image G1 to which the specific pattern 6 is added as the occlusion-off area 5 (step S4).

[0063] Next, the control unit 3 does not perform occlusion when the occlusion-off area 5 is to be superimposed on the content image, and performs occlusion when the content image is to be superimposed on an area in the real image G1 other than the occlusion-off area 5 to generate a composite image G2 (step S5). In step S5, the control unit 3 generates a composite image G2 by superimposing the CG images CG1 and CG2, which are content images, on the real image G1 based on the composition position and angle determined in step S3.

[0064] 7 is an explanatory diagram showing an example of a composite image G2 generated in step S5. The CG image CG1 of the giraffe is superimposed on the occlusion-off area 5 covered with the specific pattern 6. Therefore, in step S7, no occlusion is performed, and although the CG image CG1 of the giraffe is larger than the specific space 20, the entire CG image CG1 is displayed, unlike in FIG.

[0065] In this way, by using the composite image G2 in which the CG image CG1 larger than the size of the actual space is displayed, it is possible to give the user U using the display terminal 4 a sense of spaciousness, as if the actual space had expanded.

[0066] On the other hand, the CG image CG2 of the cow is superimposed on the actual image of the third person R1. Since the specific pattern 6 is not applied to the image of the third person R1, in step S5, the third person R1 is in an area other than the occlusion-off area 5. Therefore, the image of the third person R1, which is in an area other than the occlusion-off area 5, and the CG image CG2 of the cow are superimposed by performing occlusion. As a result, the image of the third person R1 is not obscured by the CG image CG2 of the cow, and even if the user U moves while viewing the specific space 20 using the display terminal 4, the risk of the user U bumping into the third person R1 without noticing its presence is reduced, unlike in FIG. 4.

[0067] Next, the control unit 3 erases the specific pattern 6 from the real image G1 portion of the composite image G2 to generate a composite image G3 (step S6). Fig. 8 is an explanatory diagram showing an example of the composite image G3 generated in step S6.

[0068] In the image control system 1, the specific pattern 6 applied to the walls 21a, 21b, 21c, the ceiling 22, the floor 23, and the protrusions 24 allows for both display of the CG image CG1, which is a content image larger than the indoor space, and reduction in the risk of the user U bumping into a third party R1 or the like. However, the specific pattern 6 is an unnecessary pattern for the user U. Therefore, by erasing the specific pattern 6 from the composite image G2, a composite image G3 that is easy for the user U to see can be generated.

[0069] Next, the control unit 3 erases the corner line L, which is a line corresponding to the corner of the specific space 20, from the real image G1 portion of the synthetic image G3 to generate a synthetic image G4 (step S7). Fig. 9 is an explanatory diagram showing an example of the synthetic image G4 generated in step S7.

[0070] In the composite image G3 shown in Fig. 8, a CG image CG1 larger than the indoor space can be displayed, but the giraffe's neck, which should be behind the step 221, is displayed in front of the step 221, causing a sense of inconsistency in perspective and creating a sense of distorted space. By erasing the angular line L from the composite image G3, the sense of spatial distortion is eliminated, as in the composite image G4 shown in Fig. 9, and the sense of incongruity can be reduced.

[0071] Next, the control unit 3 displays the composite image G4 on the display 42 of the display terminal 4 (step S8). This allows the user U to view the specific space 20 via the display terminal 4 and experience a mixed reality in which the specific space 20 is expanded and the CG image CG1 is displayed without any sense of incongruity, and also reduces the risk of the user U bumping into a third party R1 or the like.

[0072] The control unit 3 may display the composite image G3 in step S8 without executing step S7, and may display the composite image G2 in step S8 without executing steps S6 and S7. The control unit 3 may also process the composite image G2 in step S7 without executing step S6.

[0073] Furthermore, the specific pattern 6 does not necessarily have to be a pattern that differs depending on the position where it is applied within the occlusion-off region 5, and the control unit 3 does not have to execute steps S2 and S3.

[0074] Also, the specific space 20 does not have to have the specific pattern 6, and the control unit 3 may not execute steps S2 to S4 and S6, and may superimpose the real image G1 and the content image in step S5 without constantly executing occlusion. Even if constant occlusion is not executed in step S5, by erasing the rectangular lines L in step S7, it is possible to reduce the sense of discomfort that the space is distorted while displaying a content image larger than the indoor space.

[0075] Alternatively, the inner surface of the specific space 20 may be plain, and the control unit 3 may not execute steps S2 to S4 and S6, and may superimpose the real image G1 and the content image to generate a composite image G2 without performing occlusion at all times in step S5, and may process the composite image G2 in step S7. Alternatively, the control unit 3 may not execute step S7, and may display the composite image G2 in step S8.

[0076] Even if constant occlusion is not performed in step S5, if the inner surface of the specific space 20 is plain, the user U will not easily perceive a sense of perspective from the composite image G4 displayed on the display 42, so that it is possible to display a content image larger than the indoor space while reducing the sense of discomfort that the space is distorted. [Explanation of symbols]

[0077] 1 Image Control System 2. Building 3. Control Unit 4 Display terminal 5 Occlusion off area 6 Specific Pattern 20 Specific space 21a, 21b, 21c Wall 22 Ceiling 23 Beds 24 Protrusion 41 Camera 42 Display 221 Step CG1, CG2 CG images (content images) G1 Real image G2, G3, G4 composite image L square wire R1 third party U User

Claims

1. a control unit that displays a specific content image larger than the specific space on a real image captured by the camera of a display terminal for augmented reality or mixed reality, the real image being an image of a specific space having walls and a ceiling, on the display; an occlusion-off region, which is at least a part of the surface of the wall and the ceiling, is provided with a predetermined specific pattern so as to cover the occlusion-off region; The control unit determines an area in the real image that has the specific pattern applied to it as the occlusion off area, and when the occlusion off area is superimposed on the content image, does not perform occlusion, and when the content image is superimposed on an area in the real image other than the occlusion off area, performs occlusion.

2. 2. The image control system according to claim 1, wherein the control unit further erases the specific pattern from the real image.

3. The specific pattern is a pattern that differs depending on a position where it is applied within the occlusion-off region, The image control system of claim 1 , wherein the control unit further identifies a relative position of the display terminal with respect to the specific space from the specific pattern captured in the real image, and displays the content image according to the identified relative position.

4. The image control system according to claim 1 , wherein the control unit further erases lines corresponding to corners of the specific space from the real image.

5. a control unit that displays a specific content image larger than the specific space on a real image captured by the camera of a display terminal for augmented reality or mixed reality, the real image being an image of a specific space having walls and a ceiling, on the display; The control unit of the image control system erases lines corresponding to corners of the specific space from the real image without performing occlusion when overlaying the content image on the real image.

6. 6. The image control system according to claim 1, further comprising the display terminal.

7. 7. The image control system according to claim 6, wherein the control unit is built in the display terminal.

8. The image control system according to any one of claims 1 to 5, further comprising the specific space.

9. An image control program that causes a computer to function as the control unit used in the image control system according to any one of claims 1 to 5.

10. A storage medium on which the image control program according to claim 9 is recorded.

11. a mixed reality control step of superimposing a predetermined content image larger than a specific space on a reality image captured by a camera in a display terminal for augmented reality or mixed reality having a wall and a ceiling, the reality image being an image of the specific space having a wall and a ceiling, and displaying the superimposed content image on the display; an occlusion-off region, which is at least a part of the surface of the wall and the ceiling, is provided with a predetermined specific pattern so as to cover the occlusion-off region; The mixed reality control process is an image control method in which an area in the real image to which the specific pattern is applied is determined to be the occlusion off area, and when the occlusion off area is superimposed on the content image, occlusion is not performed, and when the content image is superimposed on an area in the real image other than the occlusion off area, occlusion is performed.

Images