Image display device and image display method

Abstract

A drawing processing unit 10 affixes an omnidirectional image to the obverse surface of a three-dimensional object as a translucent texture, and affixes the omnidirectional image to the reverse surface of the three-dimensional object as an opaque texture after rotation and reversal for the purpose of positional alignment with the translucent texture affixed to the obverse surface. If a viewpoint lies outside of the three-dimensional object, a viewpoint movement unit 20 moves the viewpoint to the inside of the three-dimensional object upon selection of the three-dimensional object. The drawing processing unit 10 renders the three-dimensional object that is observed from the viewpoint.

Description

[Technical Field]

[0001] The present invention relates to image display technology. [Background technology]

[0002] An interface has been developed that allows users to virtually walk around indoors or outdoors using omnidirectional images taken with a 360-degree camera. Users can explore the virtual space represented by the omnidirectional images by changing their viewpoint and moving around. Summary of the Invention [Problem to be solved by the invention]

[0003] To achieve virtual movement in a virtual space using 360-degree images, it is necessary to take comprehensive 360-degree photographs of both indoor and outdoor areas, which requires a huge amount of data and takes time to process.

[0004] The present invention has been made in view of these problems, and its purpose is to provide an image display technology that allows easy movement in a lightweight three-dimensional space. [Means for solving the problem]

[0005] In order to solve the above problem, an image display device according to one embodiment of the present invention includes a rendering processor that applies a first omnidirectional image to a surface of a first three-dimensional object as a translucent texture, rotates and flips the first omnidirectional image to align with the translucent texture applied to the surface, and then applies the first omnidirectional image to a back surface of the first three-dimensional object as an opaque texture, or applies the first omnidirectional image to a back surface of the first three-dimensional object as a translucent texture, and rotates and flips the first omnidirectional image to align with the translucent texture applied to the back surface, and then applies the first omnidirectional image to a surface of the first three-dimensional object as an opaque texture, and a viewpoint moving unit that, when a viewpoint is outside the first three-dimensional object, moves the viewpoint to inside the first three-dimensional object by selecting the first three-dimensional object. The rendering processor renders the first three-dimensional object observed from the viewpoint.

[0006] Another aspect of the present invention is an image display method including the steps of: pasting a first omnidirectional image onto a surface of a first three-dimensional object as a semi-transparent texture, rotating and flipping the first omnidirectional image to align with the semi-transparent texture pasted on the surface, and pasting the first omnidirectional image onto a back surface of the first three-dimensional object as an opaque texture; or pasting the first omnidirectional image onto a back surface of the first three-dimensional object as a semi-transparent texture, rotating and flipping the first omnidirectional image to align with the semi-transparent texture pasted on the back surface, and pasting the first omnidirectional image onto the surface of the first three-dimensional object as an opaque texture; when a viewpoint is outside the first three-dimensional object, moving a viewpoint into the first three-dimensional object by selecting the first three-dimensional object; and rendering the first three-dimensional object observed from the viewpoint.

[0007] Any combination of the above components, and any transformation of the present invention into a method, device, system, computer program, data structure, recording medium, etc., are also valid aspects of the present invention. [Effects of the Invention]

[0008] According to the present invention, it is possible to provide an image display technique that allows easy movement in a lightweight three-dimensional space. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a configuration diagram of an image display device according to an embodiment of the present invention; [Figure 2] 1A and 1B are diagrams illustrating a method for applying a texture to the front and back surfaces of a 3D object. [Figure 3] 10A and 10B are diagrams illustrating a method for attaching a video to the front and back surfaces of a 3D object. [Figure 4] 10 is a flowchart showing a process for drawing a semi-transparent 3D object. [Figure 5] FIG. 10 is a diagram illustrating a nested structure of semi-transparent 3D objects. [Figure 6] 6(a) and 6(b) are diagrams illustrating an example in which an opaque 3D object is placed inside a semi-transparent 3D object. [Figure 7] 7(a) to 7(c) are diagrams illustrating a method for setting an entrance / exit in a semi-transparent 3D object by semantic segmentation. [Figure 8] 8(a) and 8(b) are diagrams illustrating the shape of a semi-transparent 3D object that matches the scene of the texture that is applied to the semi-transparent 3D object. [Figure 9] 10A and 10B are diagrams illustrating the movement of a semi-transparent 3D object when a device displaying the semi-transparent 3D object is tilted. [Figure 10] 10(a) and 10(b) are diagrams illustrating the hierarchical structure of a semi-transparent 3D object. [Figure 11] 11(a) and 11(b) are diagrams illustrating animation patterns of semi-transparent 3D objects. DETAILED DESCRIPTION OF THE INVENTION

[0010] 1 is a configuration diagram of an image display device 100 according to this embodiment. The image display device 100 includes a rendering processing unit 10, a viewpoint moving unit 20, a display control unit 30, a doorway setting unit 40, a layer setting unit 50, a 3D object storage unit 60, a texture storage unit 70, and a layer structure storage unit 80.

[0011] In this embodiment, multiple three-dimensional objects are arranged in a nested structure. The three-dimensional objects can be observed from outside the three-dimensional object, or the viewpoint can be moved inside the three-dimensional object to observe the three-dimensional object from the inside. As a user interface, multiple three-dimensional objects in a nested structure may be arranged in a virtual space, or multiple three-dimensional objects in a nested structure may be accessed via a browser or the like on a network such as the Web. Any user interface can be used to access multiple three-dimensional objects in a nested structure, and is not limited to the example shown here.

[0012] The same texture is applied to the front and back of the surface of a 3D object, and the front of the 3D object is translucent, while the back is opaque. Therefore, when you look at a 3D object from the outside, you can see inside the 3D object, but when you move your viewpoint inside the 3D object, you cannot see outside the 3D object.

[0013] The rendering processing unit 10 reads three-dimensional object data from the 3D object storage unit 60, reads an omnidirectional image (also called a "360-degree image") from the texture storage unit 70, pastes the omnidirectional image as a semi-transparent texture onto the front surface of the three-dimensional object, and pastes the same omnidirectional image as an opaque texture onto the back surface of the three-dimensional object after rotating and flipping it to align it with the semi-transparent texture pasted onto the front surface. In this case, a normal omnidirectional image is pasted onto the front surface of the three-dimensional object, and an inverted omnidirectional image is pasted onto the back surface of the three-dimensional object; however, as an example, if the three-dimensional object is likened to a globe with a map displayed on its front surface, it is desirable to paste a normal omnidirectional image onto the front surface of the three-dimensional object so that the map is not flipped.

[0014] Alternatively, the rendering processing unit 10 may paste an omnidirectional image as a semi-transparent texture onto the back surface of the three-dimensional object, and paste the same omnidirectional image as an opaque texture onto the front surface of the three-dimensional object after rotating and flipping it to align it with the semi-transparent texture pasted onto the back surface. In this case, an inverted omnidirectional image is pasted onto the front surface of the three-dimensional object, and a normal omnidirectional image is pasted onto the back surface of the three-dimensional object. As an example, if text is drawn on the omnidirectional image, it is desirable to paste a normal omnidirectional image onto the back surface of the three-dimensional object so that the text does not appear reversed when viewed from inside the three-dimensional object.

[0015] In this way, whether the front or back surface of the three-dimensional object is to be displayed as a normal omnidirectional image can be determined depending on the type of omnidirectional image, the purpose of the three-dimensional object, and the like.

[0016] The resolution of the translucent texture attached to the surface of a 3D object may be lower than the resolution of the opaque texture attached to the back surface of the 3D object. It is desirable to make the resolution of the translucent texture, which allows the interior of a 3D object to be seen from the outside, extremely low enough to allow the interior atmosphere to be seen sufficiently. Specifically, the difference in resolution should be at least one-tenth. A low-resolution omnidirectional image is prepared in advance by thinning out pixels from a high-resolution omnidirectional image, and the high-resolution omnidirectional image is used as an opaque texture attached to the back surface of the 3D object, while the low-resolution omnidirectional image is used as a translucent texture attached to the surface of the 3D object. This reduces the amount of calculation and memory required, speeds up drawing operations, and reduces power consumption.

[0017] When the viewpoint is outside the three-dimensional object, the viewpoint moving unit 20 moves the viewpoint to the inside of the three-dimensional object by selecting the three-dimensional object.

[0018] The rendering processor 10 renders a three-dimensional object that can be observed from a viewpoint, and the display controller 30 displays the rendering result on a display.

[0019] Three-dimensional objects with a semi-transparent texture attached to the front surface and an opaque texture attached to the back surface may be arranged in a nested structure.

[0020] In a nested structure in which a second three-dimensional object is placed inside a first three-dimensional object, the viewpoint moving unit 20 moves the viewpoint to the inside of the second three-dimensional object by selecting the second three-dimensional object when the viewpoint is inside the first three-dimensional object. By nesting three-dimensional objects, a lightweight system that allows instantaneous movement through three-dimensional space can be realized. By moving three-dimensional objects in a nested structure, an interface that allows instantaneous movement to another location can be provided.

[0021] The rendering processing unit 10 renders at least one of a first three-dimensional object and a second three-dimensional object in a nested structure observed from a viewpoint. When the viewpoint is inside the first three-dimensional object but outside the second three-dimensional object, the rear surface of the first three-dimensional object observed from the viewpoint and the front surface of the second three-dimensional object are rendered. When the viewpoint is inside the second three-dimensional object, the rear surface of the second three-dimensional object observed from the viewpoint is rendered.

[0022] When a 3D object in a nested structure is a terminal 3D object into whose interior the viewpoint cannot be moved, the rendering processing unit 10 sets the surface texture of the terminal 3D object to opaque. The viewpoint moving unit 20 does not move the viewpoint into the terminal 3D object even when the user selects the terminal 3D object. By making the surface of the terminal 3D object opaque, an interface that makes it easy to intuitively understand the end of the nested structure can be provided.

[0023] The hierarchy setting unit 50 determines a hierarchical structure based on the position information or meta information of the omnidirectional image to be attached to the three-dimensional object, sets a nesting structure of the three-dimensional object based on the hierarchical structure, and stores the nesting structure in the hierarchical structure storage unit 80. The drawing processing unit 10 refers to the hierarchical structure stored in the hierarchical structure storage unit 80 and arranges the three-dimensional objects in a nested structure for rendering. Because the nesting structure of the three-dimensional objects is determined based on the meta information of the omnidirectional image, it is intuitively easy to understand, and users can naturally move between the three-dimensional objects arranged in the nested structure.

[0024] The entrance / exit setting unit 40 sets a specific location in the omnidirectional image as an entrance / exit based on semantic information in the omnidirectional image attached to the three-dimensional object, and sets the entrance / exit to be completely transparent. The viewpoint moving unit 20 moves the viewpoint into the inside of the three-dimensional object by selecting an entrance / exit of the three-dimensional object when the viewpoint is outside the three-dimensional object, and moves the viewpoint out of the three-dimensional object by selecting an entrance / exit of the three-dimensional object when the viewpoint is inside the three-dimensional object.

[0025] By extracting semantic information from an image and using that information to make specific parts of a 3D object completely transparent, it is possible to express an exit or entrance, thereby increasing the user's motivation to move.In a nested structure of 3D objects, it is possible to make the entrances to and exits from 3D objects intuitively clear, providing an interface that makes it easy to move between nested 3D objects.

[0026] Semantic information in the image can be used to make specific parts of a 3D object not only fully transparent, but also replaced with a different image, for example, the ceiling of a building can be made fully transparent, or the ceiling can be replaced with an image of a blue sky or a night sky.

[0027] The rendering processor 10 can use a cube or a rectangular parallelepiped as the 3D object when the omnidirectional image is an image of a building, and can use a sphere or an ellipsoid as the 3D object when the omnidirectional image is an image of an outdoor space. While there are no restrictions on the 3D object, such as a cube or a cylinder, it is more intuitive for users to use cubes for buildings and spheres for outdoor spaces. This allows for the provision of an intuitively easy-to-understand interface using 3D models with simplified shapes that match the user's intuition.

[0028] The rendering processing unit 10 can also reflect the influence of a light source inside a three-dimensional object on the surface or interior of another three-dimensional object in a nested structure. For example, a light source inside a three-dimensional object at a higher level can cast light on the surface of another three-dimensional object at a lower level within the three-dimensional object, cast a shadow, or illuminate the interior of another three-dimensional object at a lower level. Furthermore, between adjacent three-dimensional objects at the same level, light from a light source of one three-dimensional object and its reflected light can affect the surface or interior of an adjacent three-dimensional object.

[0029] An attribute setting unit that sets the attributes of the three-dimensional objects stored in the 3D object storage unit 60 may further be provided.

[0030] FIG. 2 is a diagram illustrating a method for applying textures to the front and back surfaces of a 3D object.

[0031] The same textures 300 and 310 applied to the front and back surfaces of the 3D object 200 are, for example, equirectangular omnidirectional images generated by equirectangular projection. The omnidirectional image can be captured by a 360-degree camera, or a panoramic image stitched together from multiple images.

[0032] When the textures 300, 310 are applied to the 3D object 200, the texture 300 applied to the front surface of the 3D object 200 has a transparency of about 20 to 50%, and the texture 310 applied to the back surface of the 3D object 200 is opaque. This creates a visual effect in which the interior is visible when the 3D object 200 is viewed from the outside, but the outside is not visible when viewed from the inside of the 3D object 200. By providing an interface that allows the interior of the 3D object 200 to be viewed from the outside but the outside is not visible from inside the 3D object 200, the user is given a hint that they can enter the interior of the 3D object 200, and it is possible to motivate the user to move their viewpoint into the interior of the 3D object 200.

[0033] The 3D object 200 is, for example, an object with a simple three-dimensional shape such as a sphere, cube, or cylinder. By using a simple three-dimensional shape as the 3D object 200, it is possible to provide an interface that allows for an easy bird's-eye view of the entire image while keeping the amount of calculation and data required for the omnidirectional image rendering process low. This allows for a realistic yet lightweight display of three-dimensional space.

[0034] The texture 310 applied to the back surface of the 3D object 200 is a texture 300 applied to the front surface of the 3D object 200 that has been rotated and flipped so that the same subject overlaps in the same position on both the front and back surfaces of the 3D object 200.

[0035] Hereafter, we will refer to such simple 3D objects, in which the transparency of the textures applied to the front and back surfaces has been changed, as "semi-transparent 3D objects."

[0036] FIG. 3 is a diagram illustrating a method for attaching a video to the front and back surfaces of a 3D object.

[0037] As shown in Figure 3, it is also possible to apply video textures 320 and 330 to the 3D object 200. The video texture 320 is frame data of a translucent texture to be applied to the front surface of the 3D object 200. The video texture 330 is frame data of an opaque texture to be applied to the back surface of the translucent 3D object 200. The application process can be achieved using the same means as when applying a still image texture to the 3D object 200, but the video textures 320 and 330 have a structure in which frames are switched according to the display time. The display time is not limited, and anything from 30 fps to slow motion (240 fps) is possible.

[0038] FIG. 4 is a flowchart showing the process of drawing a semi-transparent 3D object.

[0039] Polygons are combined to generate a 3D object with a simple mesh structure (S10).

[0040] Next, the texture captured in 360-degree photography is applied to the front and back surfaces of the generated 3D object (S20). Textures captured in 360-degree photography and saved can be in various formats, such as equirectangular or two fisheye images side-by-side, and are then geometrically transformed to be applied to a 3D object with a simple shape.

[0041] The user's viewpoint is moved (S30). The user can move the viewpoint by using a mouse or touching the touch panel. The user can move the viewpoint from the outside to the inside of the semi-transparent 3D object by clicking the semi-transparent 3D object with the texture applied with the mouse or by touching the touch panel. The user can also move the viewpoint from the inside to the outside of the semi-transparent 3D object by right-clicking the mouse or other operations.

[0042] A rendering process is performed on the three-dimensional space seen from the user's camera viewpoint (S40). The frame rate of the rendering process may be changed depending on the update frequency of the texture information and the frequency of the user's operations.

[0043] When the user's viewpoint is outside the translucent 3D object, an image of the translucent 3D object as seen from the outside viewpoint is rendered. In this case, since the surface of the translucent 3D object is translucent, the objects inside the translucent 3D object can be seen through the texture attached to the surface of the translucent 3D object.

[0044] When the user's viewpoint is inside a semi-transparent 3D object, the image of the semi-transparent 3D object as seen from the viewpoint inside is rendered. In this case, the back surface of the semi-transparent 3D object is opaque, so the texture attached to the back surface of the semi-transparent 3D object is visible, but the outside is not.

[0045] Figure 5 is a diagram illustrating the nested structure of semi-transparent 3D objects. By overlapping multiple semi-transparent 3D objects of different sizes in a nested structure, the user can easily move between the semi-transparent 3D objects.

[0046] As shown in Figure 5, two translucent 3D objects 210, 210 are superimposed inside a translucent 3D object 200. The types and number of translucent 3D objects to be superimposed are arbitrary. For example, a translucent cubic 3D object may be superimposed inside a translucent spherical 3D object, and the number of translucent cubic 3D objects is not limited to one, and multiple translucent cubic 3D objects may be placed.

[0047] By placing a translucent 3D object inside another translucent 3D object, an interface can be realized that allows continuous entry into the interior of the 3D object, even when multiple 3D objects overlap. Note that it is also possible to place an opaque 3D object inside a translucent 3D object, but in that case, it is not possible to enter the interior of the opaque 3D object.

[0048] In a nested structure of translucent 3D objects, the user can move to a translucent 3D object at a higher or lower level. To move to a translucent 3D object at a lower level in the nested structure, perform an operation such as clicking on the translucent 3D object you want to move to. To move to a translucent 3D object at a higher level, perform a "back" operation such as right-clicking the mouse.

[0049] It is also possible to move to a translucent 3D object at the same level. While moving to a higher or lower level can be done simply by using the mouse or pinching on a touch panel, to move to another translucent 3D object at the same level, a warp 3D object, such as a warp hole or "anywhere door," is provided to represent the movement to another translucent 3D object at the same level, and the user can select the warp 3D object to move to another translucent 3D object at the same level.

[0050] 6(a) and 6(b) are diagrams illustrating an example in which an opaque 3D object is placed inside a semi-transparent 3D object.

[0051] As shown in Figure 6(a), a semi-transparent texture is applied to the surface of the semi-transparent 3D object 500, allowing the interior to be seen. It can be seen that an opaque 3D object 510 exists inside the semi-transparent 3D object 500.

[0052] 6(b) shows the three-dimensional space seen from a viewpoint inside the translucent 3D object 500. From a viewpoint inside the translucent 3D object 500, a 360-degree image based on an opaque texture attached to the back surface of the translucent 3D object 500 is visible. Because an opaque 3D object 510 exists inside the translucent 3D object 500, the opaque 3D object 510 is visible in the field of view.

[0053] Unlike semi-transparent 3D objects, it is not possible to go inside an opaque 3D object 510. The opaque 3D object 510 is the end point of a nesting structure in the sense that it is not possible to go inside it any further.

[0054] The final opaque 3D object can be a 3D object linked to the scene. For example, if the top-level nested object is a translucent 3D object with a 360-degree image of a department store attached, translucent 3D objects for numerous stores can be placed inside the translucent 3D object of the department store. If the bottom-level nested object is a translucent 3D object of a restaurant, a final opaque 3D object, such as a menu object or a dish object, can be placed inside the translucent 3D object of the restaurant. This provides an intuitive interface, allowing users to navigate through a department store rendered as a three-dimensional space, entering a specific store, and ultimately viewing the menu or selecting a specific dish.

[0055] 7(a) to 7(c) are diagrams illustrating a method for setting an entrance / exit in a semi-transparent 3D object by semantic segmentation.

[0056] By making a specific location in a semi-transparent 3D object completely transparent, the specific location becomes a "hole" and is explicitly represented as an "entrance / exit." The user recognizes the specific completely transparent location as an entrance / exit, and can move between nested semi-transparent 3D objects through the entrance / exit.

[0057] Semantic segmentation, which associates a label or category with each pixel in an image, may be used to automate the generation of entrances and exits.

[0058] Figure 7(a) is a diagram illustrating an example of an image labeled by semantic segmentation. Each pixel is labeled as "sky," "building," "window," "door," "car," "paved road," or "vegetation."

[0059] Semantic segmentation is performed on the texture attached to a semi-transparent 3D object, and for example, doorways can be automatically generated by "punching" pixels determined to be "doors." Since the doorways of the texture attached to the semi-transparent 3D object are set to transparent, the inside can be seen from the outside of the semi-transparent 3D object, and the outside can be seen from the inside of the object.

[0060] Referring to Figures 7(b) and 7(c), an example is described in which a translucent 3D object 530 has an entrance / exit 532 set in a nested structure in which another translucent 3D object 530 is inside a translucent 3D object 520.

[0061] 7(b) shows an image seen by a user when the user is inside the translucent 3D object 520. A user inside the translucent 3D object 520 can see the translucent 3D object 530 against the background of the opaque texture on the backside of the translucent 3D object 520. Because the doorway 532 of the translucent 3D object 530 is completely transparent, the interior of the translucent 3D object 530 can be seen through the doorway 532. The user can select the doorway 532 of the translucent 3D object 530 and enter the interior of the translucent 3D object 530 through the doorway 532.

[0062] 7(c) shows an image seen by a user when the user is inside the translucent 3D object 530. A user inside the translucent 3D object 530 can see the opaque texture on the backside of the translucent 3D object 530. Because the doorway 532 of the translucent 3D object 530 is completely transparent, the outside, i.e., the opaque texture on the backside of the translucent 3D object 520, can be seen through the doorway 532. The user can select the doorway 532 of the translucent 3D object 530 to exit the outside of the translucent 3D object 530 through the doorway 532.

[0063] 8(a) and 8(b) are diagrams illustrating the shape of a semi-transparent 3D object that matches the scene of the texture to be applied to the semi-transparent 3D object. The shape of the semi-transparent 3D object can be selected to match the scene of the 360-degree image.

[0064] As shown in Figure 8(a), when a 360-degree image taken indoors is pasted as a texture onto a translucent 3D object 540, since buildings are often similar to cubes, a cubic translucent 3D object 540 can be used.

[0065] As shown in FIG. 8(b), when a 360-degree image taken outdoors is pasted onto a translucent 3D object 550, a spherical translucent 3D object 550 can be used to represent an infinitely open space.

[0066] By using a translucent 3D object shaped to match the scene in this way, users can intuitively understand whether the interior of the translucent 3D object is indoors or outdoors, making it easier for them to decide whether or not to enter the interior of the translucent 3D object.

[0067] FIG. 9 is a diagram illustrating the movement of a semi-transparent 3D object when the device displaying the semi-transparent 3D object is tilted.

[0068] Suppose a semi-transparent spherical 3D object and a semi-transparent cubic 3D object are displayed on a display device such as a smartphone. The tilt of the display device can be calculated using information from the acceleration and gyro sensors installed on the display device. The movement of the displayed semi-transparent 3D objects can be made to differ depending on the tilt of the display device. For example, when the display device is tilted, the semi-transparent spherical 3D object will be animated as if it is rolling because of its small coefficient of friction, while the semi-transparent cubic 3D object will be animated as if it is not moving easily because of its large coefficient of friction.

[0069] Figures 10(a) and 10(b) are diagrams explaining the hierarchical structure of semi-transparent 3D objects. While there is no restriction on how semi-transparent 3D objects are layered or on the textures of semi-transparent 3D objects placed on the same layer, by applying object detection technology to the textures in advance, it is possible to output meta-information from the textures. The hierarchical structure of semi-transparent 3D objects can be determined using the meta-information of the textures applied to the semi-transparent 3D objects.

[0070] 10(a) shows the hierarchical structure of meta information. Texture 560b containing meta information called "Earth" is classified in folder 560a at the top level. Textures 562b and 564b containing meta information called "Water" and "Earth" are classified in folders 562a and 564a at the second level. Textures 566b and 568b containing meta information called "Sea" are classified in folder 566a at the lowest level under folder 562a called "Water" at the second level.

[0071] FIG. 10(b) shows the hierarchical structure of semi-transparent 3D objects corresponding to the hierarchical structure of the meta information in FIG. 10(a).

[0072] Texture 560b classified in folder 560a called "Earth" at the highest level is applied to semi-transparent 3D object 560c at the highest level.

[0073] Texture 562b classified in folder 562a called "water" in the second layer is applied to semi-transparent 3D object 562c in the second layer.

[0074] Texture 564b classified in folder 564a called "earth" in the second layer is pasted onto semi-transparent 3D object 564c in the second layer.

[0075] Textures 566b and 568b classified in a folder 566a called "sea" at the lowest level are applied to semi-transparent 3D objects 566c and 568c at the lowest level.

[0076] The "Earth" folder contains two folders, "Water" and "Earth," because textures whose meta information includes "Water" and textures whose meta information includes "Earth" are classified into different folders and are represented as separate semi-transparent 3D objects at the same level, which is expected to improve the user experience. The "Water" folder contains a folder called "Ocean," because "Ocean" is a sub-concept of "Water," and so placing a semi-transparent 3D object corresponding to "Ocean" inside a semi-transparent 3D object corresponding to "Water" makes it easier to understand intuitively.

[0077] Texture meta information is not limited to information extracted by image processing. GPS information indicating the location where the texture was captured may also be used as meta information. For example, a hierarchical structure of meta information, such as "Tokyo" under "Japan" and "Shinjuku" under "Tokyo," is created based on the texture's GPS information, and textures are classified using this structure. In accordance with the hierarchical structure of the texture meta information, a translucent 3D object corresponding to "Tokyo" is placed inside a translucent 3D object corresponding to "Japan," and a translucent 3D object corresponding to "Shinjuku" is placed inside a translucent 3D object corresponding to "Tokyo."

[0078] 11(a) and 11(b) are diagrams illustrating animation patterns of a semi-transparent 3D object. The animation of the semi-transparent 3D object can be changed depending on the texture applied to the semi-transparent 3D object or the status of the 3D object inside the semi-transparent 3D object.

[0079] For example, suppose a 360-degree image of a store is pasted onto a semi-transparent 3D object 550, and a character 3D object exists inside the semi-transparent 3D object 550. If the store is having a sale and the character is selling products, the semi-transparent 3D object 550 can be rotated as shown in Figure 11(a) or bounced as shown in Figure 11(b).

[0080] To indicate the attributes of a translucent 3D object, a shop sign or logo may be placed above the translucent 3D object. The sign or logo generally remains perpendicular to the line of sight and does not change orientation even when the translucent 3D object rotates or bounces. In addition, to indicate the business status of a shop, the sign or logo may rotate when the shop is open, stop when the shop is closed, and bounce during a sale.

[0081] To indicate the attributes of a semi-transparent 3D object, an animation showing an image of the products offered by the shop may be added near (for example, above) the semi-transparent 3D object. For example, if it is a ramen shop, an animation of steam may be added.

[0082] The texture to be applied may be switched depending on the time of day and the user's attribute information as well as the status of the translucent 3D object. For example, a blue sky texture may be displayed during the day and a starry sky texture may be displayed at night. Depending on the user's attributes such as age, gender, and hobbies, a texture suitable for children or adults, or a texture suitable for men or women, or a texture suitable for a user's hobbies may be selected.

[0083] To switch between multiple textures applied to a translucent 3D object, you can load multiple textures into memory, render them, and then overlay them. Then, set the transparency of the texture you want to select to 0% (i.e., opaque) and the transparency of the texture you don't want to select to 100% (i.e., fully transparent). For example, you can render and overlay two images, one for daytime and one for nighttime, and switch to the daytime texture by setting the transparency of the daytime texture to 0% and the transparency of the nighttime texture to 100% and the nighttime texture to 0% during the daytime. Similarly, you can switch between textures for spring, summer, autumn, and winter depending on the season, or by age or gender. Switching textures simply by changing the transparency eliminates the time it takes to load and render textures, enabling fast texture switching.

[0084] In addition, the effect of lighting may be reflected in the texture between semi-transparent 3D objects. For example, if there is an object that emits light inside a semi-transparent 3D object, the light emitted by that object may cast light or shadows on the texture of another semi-transparent 3D object inside the semi-transparent 3D object.

[0085] The present invention has been described above based on the embodiments. The embodiments are merely examples, and it will be understood by those skilled in the art that various modifications are possible in the combination of the components and treatment processes, and that such modifications are also within the scope of the present invention. It is used for this purpose. [Industrial Applicability]

[0086] The present invention relates to image display technology. [Explanation of symbols]

[0087] 10 drawing processing unit, 20 viewpoint movement unit, 30 display control unit, 40 entrance / exit setting unit, 50 hierarchy setting unit, 60 3D object storage unit, 70 texture storage unit, 80 hierarchical structure storage unit, 100 image display device.

Claims

1. A drawing processing unit that pastes an omnidirectional image as a semi-transparent texture onto one of the front and back surfaces of a three-dimensional object, and pastes the omnidirectional image as an opaque texture onto the other surface of the three-dimensional object after rotating and flipping the omnidirectional image to align it with the semi-transparent texture pasted onto the one surface; an entrance / exit setting unit that sets a specific location in the omnidirectional image labeled by semantic segmentation that associates a label with each pixel in the omnidirectional image attached to the three-dimensional object as an entrance / exit and sets the entrance / exit to be completely transparent; a viewpoint moving unit that moves the viewpoint to the inside of the three-dimensional object by selecting the entrance / exit of the three-dimensional object when the viewpoint is outside the three-dimensional object, and moves the viewpoint to the outside of the three-dimensional object by selecting the entrance / exit of the three-dimensional object when the viewpoint is inside the three-dimensional object, The image display device is characterized in that the drawing processing unit renders the three-dimensional object observed from the viewpoint.

2. 2. The image display device according to claim 1, wherein the rendering processing unit renders a three-dimensional object in a nested structure in which another three-dimensional object is placed inside the three-dimensional object, the three-dimensional object being observed from the viewpoint.

3. when the three-dimensional object in the nested structure is a terminal three-dimensional object into which the viewpoint cannot be moved, the rendering processing unit sets the surface texture of the terminal three-dimensional object to opaque; 3. The image display device according to claim 2, wherein the viewpoint moving unit does not move the viewpoint into the interior of the terminal three-dimensional object even when the terminal three-dimensional object is selected.

4. 3. The image display device according to claim 2, further comprising a hierarchical structure setting unit that determines a hierarchical structure based on position information or meta information of the omnidirectional image that is attached to the three-dimensional object in the nested structure, and sets the nested structure of the three-dimensional object based on the hierarchical structure.

5. 5. The image display device according to claim 2, wherein the drawing processing unit uses a cube or a rectangular parallelepiped as the three-dimensional object in the nested structure when the omnidirectional image is an image of a building, and uses a sphere or an ellipsoid as the three-dimensional object in the nested structure when the omnidirectional image is an image of an outdoor space.

6. 3. The image display device according to claim 2, wherein the rendering processing unit reflects the influence of a light source inside the three-dimensional object in the nested structure on the surface or inside of another three-dimensional object in the nested structure.

7. A step of pasting an omnidirectional image as a translucent texture on one of the front and back surfaces of a three-dimensional object, and pasting the omnidirectional image as an opaque texture on the other surface of the three-dimensional object after rotating and flipping the omnidirectional image to align it with the translucent texture pasted on the one surface; setting a specific location in the omnidirectional image labeled by semantic segmentation, which associates a label with each pixel in the omnidirectional image attached to the three-dimensional object, as a doorway, and setting the doorway to be completely transparent; moving a viewpoint into the interior of the three-dimensional object by selecting the entrance / exit of the three-dimensional object when the viewpoint is outside the three-dimensional object, and moving a viewpoint out of the three-dimensional object by selecting the entrance / exit of the three-dimensional object when the viewpoint is inside the three-dimensional object; and rendering the three-dimensional object as viewed from the viewpoint.

8. A step of pasting an omnidirectional image as a semi-transparent texture on one of the front and back surfaces of a three-dimensional object, and pasting the omnidirectional image as an opaque texture on the other surface of the three-dimensional object after rotating and flipping the omnidirectional image to align it with the semi-transparent texture pasted on the one surface; setting a specific location in the omnidirectional image labeled by semantic segmentation, which associates a label with each pixel in the omnidirectional image attached to the three-dimensional object, as a doorway, and setting the doorway to be completely transparent; moving a viewpoint into the interior of the three-dimensional object by selecting the entrance / exit of the three-dimensional object when the viewpoint is outside the three-dimensional object, and moving a viewpoint out of the three-dimensional object by selecting the entrance / exit of the three-dimensional object when the viewpoint is inside the three-dimensional object; and rendering the three-dimensional object observed from the viewpoint.

Images