Information processing device and information processing system

Abstract

The present invention provides an information processing device and information processing system that can display content representing objects and corresponding 3D objects on the live-action images of the live-action space, which are placed and displayed in a live-action space realized by switching and displaying live-action images in response to user movement operations, in response to user operations on the 3D objects. [Solution] The information processing device and information processing system according to the present invention comprises a reception unit and a display control unit. The reception unit receives the selection of content representing an object, which is placed and displayed in a real-world space realized by switching and displaying real-world images in response to the user's movement operation. The display control unit, when the selection of content is received by the reception unit, displays a three-dimensional object corresponding to the content on the real-world image of the real-world space.

Description

Technical Field

[0001] Embodiments of the present invention relate to an information processing apparatus and an information processing system.

Background Art

[0002] Conventionally, a technique for displaying a three-dimensional (3D) model space related to a space in which exhibits or artworks are displayed in a museum or art gallery is known. For example, a dedicated system (a system built specifically for that museum) for arranging and displaying a 3D model of an exhibit in a real space equipped with a walkthrough function that can move while looking around 360 degrees (for example, provided by Matterport) has been constructed.

Prior Art Documents

Non-Patent Documents

[0003]

Non-Patent Document 1

Non-Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] In a conventional exhibition system for museums and the like, a 3D model of an exhibit is arranged in a real space (for example, provided by Matterport) with a walkthrough function that can move while looking around 360 degrees, created using a 3D scan camera. However, in this real space, the user could not freely enlarge, reduce, or view the 3D model of the exhibit from an arbitrary direction. In order for the user to deepen their understanding of the exhibit, it is desirable that in a museum or the like constructed in this real space, the user can select the exhibit themselves, rotate, reduce, enlarge the 3D model of the exhibit, and view it from an arbitrary direction.

[0005] The problem that the present invention aims to solve is to provide an information processing device and information processing system that can display content representing an object and a corresponding 3D object on a live image of the live image space, in response to user operations on the 3D object. This content is displayed in a live image space, which is realized by switching and displaying live images in response to user movement operations. [Means for solving the problem]

[0006] The information processing device and information processing system according to the embodiment comprises a reception unit and a display control unit. The reception unit receives the selection of content representing an object, which is placed and displayed in a real-world space realized by switching and displaying real-world images in response to the user's movement operation. When the display control unit receives the selection of content from the reception unit, it causes the three-dimensional object corresponding to the content to be displayed on the real-world image of the real-world space. [Effects of the Invention]

[0007] According to an information processing device of an example embodiment, a 3D object corresponding to content representing an object, which is placed and displayed in a real-world space realized by switching and displaying real-world images in response to the user's movement operation, is displayed on the real-world image of the real-world space in response to the user's operation on the 3D object, allowing the user to perceive the object from any direction. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 shows an example of an overview of an information processing system according to an embodiment. [Figure 2] Figure 2 is a block diagram showing an example of the configuration of an information processing system according to the embodiment. [Figure 3] Figure 3 is a functional block diagram showing an example of the configuration of an information processing device according to the embodiment. [Figure 4] Figure 4 shows an example of object information according to the present invention. [Figure 5] Figure 5 shows an example of three-dimensional object information according to the embodiment. [Figure 6] Figure 6 shows an example of a walkthrough display according to the embodiment. [Figure 7] Figure 7 shows an example of object selection according to the embodiment. [Figure 8] Figure 8 shows an example of how object information is displayed according to the embodiment. [Figure 9] Figure 9 shows an example of how object information is displayed according to the embodiment. [Figure 10] Figure 10 shows an example of object selection according to the embodiment. [Figure 11] Figure 11 shows an example of how object information is displayed according to the embodiment. [Figure 12] Figure 12 shows an example of how object information is displayed according to the embodiment. [Figure 13] Figure 13 is a sequence diagram showing an example of the processing flow executed in the information processing system according to the embodiment. [Modes for carrying out the invention]

[0009] The following describes in detail, with reference to the drawings, the embodiments for implementing the information processing device and information processing system according to the present application (hereinafter referred to as "embodiments"). Note that these embodiments do not limit the information processing device and information processing system according to the present application. Furthermore, the same parts are denoted by the same reference numerals in the following embodiments, and redundant descriptions are omitted.

[0010] [1. An example of an information processing system overview] First, an overview of the information processing system according to the embodiment will be described using Figure 1. Figure 1 is a diagram showing an example of the overview of the information processing system according to the embodiment. The information processing device 1 of the information processing system S is configured using real-world space information 41 related to a real-world space realized by switching and displaying real-world images in response to the user's movement operation, and object (exhibits, etc.) information 42. When the selection of content indicating an object is received, the information processing device 1 superimposes the 3D object information 43 of the object corresponding to the content onto the real-world space information 41.

[0011] As illustrated in Figure 1, the information processing system S includes an information processing device 1, an imaging device 10, and a display device 100. The detailed configuration of the information processing system S will be described later in Figure 2.

[0012] First, the imaging device 10 takes photographs of the interior of the building (step S1). The interior of the building is not limited to indoor areas, but may also include outdoor spaces such as courtyards and outdoor exhibition areas. The imaging device 10 is, for example, a 3D scanning camera that combines LiDAR (Light Detection and Ranging) and a camera, and is capable of acquiring 360-degree panoramic images and 3D data of the interior of the building. The imaging device 10 acquires 360-degree panoramic images and 3D data at predetermined locations inside the building and transmits this imaging information to the information processing device 1.

[0013] Next, the information processing device 1 generates actual shooting space information 41 based on the shooting information received from the shooting device 10 (step S2). The information processing device 1 generates a 3D model of the inside of the building based on the 3D point cloud data of the inside of the building acquired by LiDAR, and generates shooting 3D space data by attaching the photo taken by the camera as a texture to the 3D model. The information processing device 1 can use the generated shooting 3D space data to display the inside of the building in a bird's-eye view mode in a 3D manner on the display device 100. On the other hand, when switching to the walkthrough mode, the user can move the actual shooting position within the shooting 3D space. Specifically, the user can select the shooting position by a predetermined viewpoint display, and the image taken at the selected viewpoint is displayed on the display device 100. Further, when the user selects a viewpoint within the image in which viewpoints adjacent or within a predetermined range are displayed, the image taken at the selected viewpoint is displayed on the display device 100. Thus, the information processing device 1 realizes a walkthrough on the display device 100 by switching the actually taken images according to the user's operation. The virtual two-dimensional space displayed in this walkthrough mode is taken as the actual shooting space, and the information for the walkthrough is taken as the actual shooting space information 41. In the following description, the description of the actual shooting space information 41 shall include the image displayed in the walkthrough mode (that is, the taken image), the content for selecting the destination of movement, and the like.

[0014] Next, the information processing device 1 causes the display device 100 to display the inside of the building as the actual shooting space information 41 (step S3). The information processing device 1 transmits control information for controlling the actual shooting space information 41 and the display of the display device 100 to the display device 100, and causes the display device 100 to display the inside of the building as the actual shooting space information 41. In this case, the display device 100 displays the actual shooting space information 41 based on the received control information. The display device 100 is, for example, a terminal device such as a PC (Personal Computer), a smartphone, a tablet, a digital camera, a goggle-type VR (Virtual Reality) device, or the like.

[0015] Subsequently, the information processing apparatus 1 receives a selection of an object in the museum displayed as the actual shooting space information 41 (step S4). The information processing apparatus 1 receives a selection of an object in the museum displayed as the actual shooting space information 41 from the user of the display device 100. Specifically, the information processing apparatus 1 receives a selection of content indicating the object. In addition to the selection of content indicating the object, the information processing apparatus 1 may receive a selection of content indicating audio information or text information.

[0016] Subsequently, when the information processing apparatus 1 receives a selection of content indicating an object, the information processing apparatus 1 superimposes and displays the three-dimensional object information 43 of the object corresponding to the content on the actual shooting space information 41 (step S5). When the information processing apparatus 1 receives a selection of content indicating an object, the information processing apparatus 1 transmits the three-dimensional object information 43 of the object corresponding to the content and control information to the display device 100. Then, the information processing apparatus 1 causes the display device 100 to superimpose and display the three-dimensional object information 43 on the actual shooting space information 41. In this case, the display device 100 superimposes and displays the three-dimensional object information 43 on the actual shooting space information 41 based on the received control information.

[0017] The information processing apparatus 1 superimposes and displays the three-dimensional object information 43 on the actual shooting space information 41 according to the display mode information 44. For example, when the selected object is a three-dimensional exhibit, the information processing apparatus 1 rotates, enlarges, and reduces the 3D model of the object according to the user's operation via the display device 100 and superimposes and displays it on the actual shooting space information 41. In addition, for example, when the selected object is a two-dimensional exhibit (such as a painting), the information processing apparatus 1 superimposes and displays it on the actual shooting space information 41 while imparting depth and three-dimensionality to the entire exhibit or a predetermined object included in the exhibit by a predetermined method. As the predetermined method, for example, a generation AI corresponding to the generation of a 3D model may be used.

[0018] When the information processing device 1 receives a selection of content indicating an object along with a selection of content indicating audio information from the user of the display device 100, it provides a description of the object as audio information 45. Furthermore, when the information processing device 1 receives a selection of content indicating an object along with a selection of content indicating text information from the user of the display device 100, it provides a description of the object as text information 46.

[0019] When the display device 100 is a terminal device and has a screen such as a liquid crystal display with touch panel functionality, the information processing device 1 superimposes 3D object information 43 onto the real-world spatial information 41 in response to tap operations, slide operations, etc., performed with a finger or stylus for the selected object.

[0020] When the display device 100 is a VR goggle and the VR controller is a glove-type controller, the information processing device 1 superimposes 3D object information 43 onto the real-world spatial information 41 for the selected object, according to the information from the sensors provided by the controller.

[0021] Thus, in a predetermined building configuration using real-world space information 41 and object (exhibit, etc.) information 42, which are realized by switching and displaying real-world images in response to the user's movement operations, when the information processing device 1 receives a selection of content indicating an object, it superimposes the 3D object information 43 of the object corresponding to the content onto the real-world space information 41, thereby allowing the user to understand the object.

[0022] [2. An example of the configuration of an information processing system] Next, the configuration of the information processing system according to the embodiment will be described using Figure 2. Figure 2 is a block diagram showing an example of the configuration of the information processing system according to the embodiment. As illustrated in Figure 2, the information processing system S includes an information processing device 1, a plurality of imaging devices 10, and a plurality of display devices 100. The information processing device 1, the plurality of imaging devices 10, and the plurality of display devices 100 are connected to each other via a predetermined network N so as to be communicative.

[0023] In the above example, the information processing system S includes multiple imaging devices 10 and multiple display devices 100, but it may also be configured to include a single imaging device 10 and a single display device 100. Furthermore, in the above example, the information processing device 1 is configured separately from the display device 100, but it may also be configured integrally with the display device 100.

[0024] The information processing device 1 is, for example, a server device that accepts the selection of an object displayed in the real-world spatial information 41 and overlays 3D object information 43 relating to the predetermined object onto the real-world spatial information 41.

[0025] The imaging device 10 is a device that photographs the interior of the building, and is, for example, a 3D scanning camera combined with LiDAR. The position of the imaging device 10 is not particularly limited as long as it is in a position that can photograph the interior of the building, for example, it may be in the position of the imaging device 10 itself, or it may be fixed to a wall surface such as the ceiling or floor. The imaging device 10 transmits the photographed information of the interior of the building to the information processing device 1.

[0026] The display device 100 is a display device that, for example, displays the interior of the building as real-world spatial information 41, and when the information processing device 1 receives a selection of content indicating an object, it superimposes the 3D object information 43 of the object corresponding to that content onto the real-world spatial information 41. The display device 100 is a terminal device such as a PC, smartphone, tablet, digital camera, or goggle-type VR device.

[0027] [3. An example of the configuration of an information processing device] Next, the configuration of the information processing device according to the embodiment will be described using Figure 3. Figure 3 is a functional block diagram showing an example of the configuration of the information processing device according to the embodiment. As illustrated in Figure 3, the information processing device 1 comprises a communication unit 2, a control unit 3, and a storage unit 4.

[0028] The communication unit 2 is a network device for performing wireless communication processing and wired communication processing. For example, the communication unit 2 transmits and receives various information to and from the imaging device 10 and the display device 100 via a predetermined network N.

[0029] The control unit 3 controls the information processing device 1 and includes, for example, a computer and various circuits having a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), flash memory, input / output ports, etc. As illustrated in Figure 3, the control unit 3 comprises an acquisition unit 31, a generation unit 32, a reception unit 33, and a display control unit 34.

[0030] The computer's CPU functions as an acquisition unit 31, a generation unit 32, a reception unit 33, and a display control unit 34, for example, by reading and executing a program stored in ROM.

[0031] Furthermore, at least one of the acquisition unit 31, generation unit 32, reception unit 33, and display control unit 34 of the control unit 3 can be configured with hardware such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array).

[0032] The memory unit 4 stores various types of information and corresponds to, for example, RAM or flash memory. These types of information include, for example, real-world spatial information 41, object information 42, 3D object information 43, display mode information 44, audio information 45, and text information 46.

[0033] The real-world spatial information 41 is information about the real-world space that is realized by switching and displaying real-world images in response to the user's movement operations. For example, the 3D object information 43 of an object is superimposed on the real-world spatial information 41.

[0034] Object information 42 is information about objects that are exhibits within the museum. As shown in Figure 4, object information 42 includes, for example, the tag number of the object shown in the content of the live-action spatial information 41, the object ID of that object, the object type (e.g., three-dimensional shape or two-dimensional shape), audio information, text information, etc.

[0035] The 3D object information 43 is 3D model information of an object that is superimposed on the real-world spatial information 41 when the reception unit 33 receives a selection of content indicating an object.

[0036] As shown in Figure 5, the 3D object information 43 includes, for example, the object ID, display mode, 3D scanner data, and image information of the object shown in the content of the real-world spatial information 41. If the object is three-dimensional, it includes, for example, 3D scanner data including image data and 3D shape data acquired using a 3D scanner; if the object is two-dimensional, it includes, for example, image information related to the object.

[0037] The display mode information 44 includes, for example, information indicating that if the object has a three-dimensional shape, the image data will be mapped to the three-dimensional shape data for display, and if the object has a two-dimensional shape, the depth and three-dimensionality will be added to the entire image of the object or a predetermined target image included in the object in a predetermined manner for display.

[0038] Audio information 45 is audio data relating to the description of the object. Text information 46 is text data relating to the description of the object.

[0039] Next, the functions of the control unit 3 (acquisition unit 31, generation unit 32, reception unit 33, and display control unit 34) will be described. The acquisition unit 31 acquires various types of information. For example, the acquisition unit 31 acquires photographic information of the building taken by the photographing device 10 via the communication unit 2.

[0040] The generation unit 32 generates various types of information. The generation unit 32 generates real-world spatial information 41 based on the shooting information acquired by the acquisition unit 31. For example, the generation unit 32 generates a 3D model based on 3D point cloud data of the building acquired by LiDAR, and generates captured 3D spatial data by applying photographs taken by a camera as textures to the 3D model. Using the generated captured 3D spatial data, the generation unit 32 can, via the display control unit 34, display the building in a bird's-eye view 3D display mode on the display device 100. The generation unit 32 stores the generated captured 3D spatial data in the storage unit 4. On the other hand, when switching to walk-through mode, the user can move to the actual shooting position within the captured 3D space. Specifically, the user can select a shooting position by displaying predetermined viewpoints, and the image taken at the selected viewpoint is displayed. Furthermore, if the user selects a viewpoint within the image where adjacent or predetermined viewpoints are displayed, the image taken at the selected viewpoint is displayed. In this way, a walk-through is realized by switching the actually captured images according to the user's operation. The virtual two-dimensional space displayed on the display device 100 in walkthrough mode is designated as the real-world space, and the information for the walkthrough is designated as real-world space information 41. The generation unit 32 stores the generated real-world space information 41 in the storage unit 4.

[0041] Furthermore, the generation unit 32 generates various types of information based on the 3D object information 43. For example, for objects with a three-dimensional shape, the generation unit 32 uses the 3D shape data and image data to generate 3D information that rotates and enlarges / reduces the object in response to user operations. Also, for example, for objects with a planar shape, the generation unit 32 generates 3D information that adds depth and a sense of three-dimensionality to the entire image of the object or a predetermined target image contained within the object, using a predetermined method.

[0042] In the example described above, the information processing device 1 acquires photographic information using the acquisition unit 31 and generates various types of information such as real-world spatial information 41 using the generation unit 32. However, the acquisition unit 31 may also acquire pre-generated information from other devices. In other words, the information processing device 1 may generate the information itself or acquire it from other devices.

[0043] The reception unit 33 accepts various selections. The reception unit 33 accepts the selection of content that represents an object. In addition to the selection of content that represents an object, the reception unit 33 may also accept the selection of content that represents audio information or text information.

[0044] The display control unit 34 transmits the real-world spatial information 41 and control information for controlling the display of the display device 100 to the display device 100 via the communication unit 2, causing the display device 100 to display the real-world spatial information 41.

[0045] When the reception unit 33 receives a selection of content indicating an object, the display control unit 34 superimposes the 3D information generated by the generation unit 32 onto the real-world spatial information 41 for the content and the corresponding object. Specifically, the display control unit 34 transmits the 3D information and control information for controlling the display of the display device 100 to the display device 100 via the communication unit 2, and the display device 100 superimposes the 3D information onto the real-world spatial information 41.

[0046] [4. Example of a walkthrough display] The following describes an example of a walkthrough display using Figure 6. Figure 6 is a diagram showing an example of a walkthrough display. The interior of the building is constructed using real-world spatial information 41 and object (exhibits, etc.) information 42, which are real-world spatial information 41 related to the real-world space that is realized by switching and displaying real-world images in accordance with the user's movement operations. When the information processing device 1 receives a selection of content indicating an object, it superimposes the 3D object information 43 of the object corresponding to the content onto the real-world spatial information 41 on the display device 100. The circle display is a predetermined viewpoint.

[0047] Next, in Figure 7, when the user clicks or touches a predetermined tag indicating the "horse" object in the live-action spatial information 41, as shown in Figure 8, the 3D model of the "horse" object is superimposed on the live-action spatial information 41, rotating, shrinking, and enlarging according to the user's operation. In the case of VR, as shown in Figure 9, the 3D model of the "horse" object can be rotated, shrunk, and enlarged by the user's hands in the virtual space.

[0048] Next, in Figure 10, when the user clicks or touches a predetermined tag indicating the object of the "cat" painting in the live-action spatial information 41, the entire painting or the "cat" contained within the painting is superimposed on the live-action spatial information 41 with depth and three-dimensionality added in a predetermined manner, as shown in Figure 11. Furthermore, if a 3D model of the "cat" can be generated in a predetermined manner, as shown in Figure 12, the 3D model of the "cat" is superimposed on the live-action spatial information 41, rotated, scaled, and enlarged according to the user's operation. In the case of VR, the 3D model of the "cat" can be rotated, scaled, and enlarged by the user's hands in the virtual space, just like the 3D model of the "horse".

[0049] [5. Processing Flow] Next, an example of the processing flow executed in the information processing system according to the embodiment will be described using Figure 13. Figure 13 is a sequence diagram showing an example of the processing flow executed in the information processing system according to the embodiment.

[0050] First, the camera 10 takes pictures of a designated area inside the building (step S101).

[0051] Next, the imaging device 10 transmits the imaging information to the information processing device 1 (step S102).

[0052] Next, the information processing device 1 generates real-world spatial information 41 based on the imaging information received from the imaging device 10 (step S103).

[0053] Next, the information processing device 1 displays the designated interior of the building as real-time spatial information 41 on the display device 100 (step S104).

[0054] Next, the display device 100 transmits the selection of content representing an object to the information processing device 1 (step S105).

[0055] Next, the information processing device 1 receives the selection of content indicating an object, transmitted from the display device 100 (step S106).

[0056] Next, the information processing device 1 overlays the three-dimensional object information 43 of the object corresponding to the content onto the real-world spatial information 41 on the display device 100 (step S107).

[0057] [6. Effects] As described above, the information processing device according to the present invention comprises a reception unit and a display control unit. The reception unit receives the selection of content representing an object, which is placed and displayed in a real-world space realized by switching and displaying real-world images in response to the user's movement operation. When the display control unit receives the selection of content from the reception unit, it causes the display control unit to display a three-dimensional object corresponding to the content on the real-world image of the real-world space.

[0058] The display control unit displays the three-dimensional object on a real-world image of the real-world space in response to user operations on the three-dimensional object.

[0059] This operation is performed by virtual contact with the 3D object.

[0060] The system further includes a generation unit that generates a 3D object using 3D shape data or image data.

[0061] This operation is performed while the user virtually moves through the live-action space.

[0062] The information processing system according to the present invention comprises a server device, a display device, and a 3D scanning camera, and includes a reception unit and a display control unit. The reception unit receives the selection of content representing an object, which is placed and displayed in a real-world space realized by switching and displaying real-world images in response to the user's movement operation. When the display control unit receives the content selection from the reception unit, it displays the 3D object corresponding to the content on the real-world image of the real-world space.

[0063] Through any or a combination of the above processes, a 3D object corresponding to content representing an object, which is placed and displayed in a live-action space realized by switching and displaying live-action images in response to the user's movement operations, can be displayed on the live-action image of the live-action space in response to the user's operation on the 3D object, allowing the user to perceive the object from any direction.

[0064] While embodiments of the present invention have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These embodiments can be carried out in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims and their equivalents. [Explanation of Symbols]

[0065] 1. Information Processing Device 2 Communications Department 3. Control Unit 31 Acquisition Department 32 Generation part 33 Reception Department 34 Display Control Unit 4 Storage section 41 Live-action spatial information 42 Object Information 43 3D Object Information 44 Display Pattern Information 45 Audio Information 46 Text Information 10 Imaging device 100 display device

Claims

1. A reception unit that accepts the selection of content representing objects, which is placed and displayed in a live-action space realized by switching and displaying live-action images in response to the user's movement operations, When the reception unit accepts the selection of content, a display control unit displays a three-dimensional object corresponding to the content on the live-action image of the live-action space, An information processing device characterized by comprising:

2. The display control unit causes the three-dimensional object to be displayed on the real-world image of the real-world space in response to user operations on the three-dimensional object. The information processing apparatus according to feature 1.

3. The operation described above is performed by virtual contact with the three-dimensional object. The information processing apparatus according to feature 2.

4. A generation unit that generates the three-dimensional object using three-dimensional shape data or image data, The information processing apparatus according to claim 3, further comprising the above.

5. The aforementioned operation is performed while the user virtually moves through the live-action space. The information processing apparatus according to feature 4.

6. An information processing system comprising a server device, a display device, and a 3D scanning camera, A reception unit that accepts the selection of content representing objects, which is placed and displayed in a live-action space realized by switching and displaying live-action images in response to the user's movement operations, When the reception unit accepts the selection of content, a display control unit displays a three-dimensional object corresponding to the content on the live-action image of the live-action space, An information processing system characterized by comprising the following features.

Images