Information processing system, attachment device, and program

Abstract

To provide an information processing system, an attachment device, and a program which devise how to show a work or the like as if it exists in the real world.SOLUTION: An attachment device 1 to be mounted on the head of a user comprises: a display unit 37; an imaging unit 36; an object recognition unit 12 which recognizes an object on the basis of an image captured by the imaging unit 36; an arrangement image generation unit 15 which arranges a work image related to the object recognized by the object recognition unit 12 in a cube object with reference to texts stored in a work information DB 4a storing therein work images with which the texts are associated, to generate an arrangement image being a stereoscopic image using binocular parallax; and an image output unit 16 which outputs a superimposed image in which the object is superimposed on the arrangement image generated by the arrangement image generation unit 15, to the display unit 37.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an information processing system, a mounting device, and a program. [Background technology]

[0002] BACKGROUND ART Conventionally, a system has been disclosed that distributes images of artworks such as paintings stored in a museum database to an unspecified number of customers using Web3D (for example, Patent Document 1). Furthermore, an image output device has been disclosed that is devised to display not only art pieces but also various other artworks in a way that is easy for users to understand (for example, Patent Document 2). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2003-296587 [Patent Document 2] Japanese Patent Publication No. 2021-043493 Summary of the Invention [Problem to be solved by the invention]

[0004] In the system described in Patent Document 1, the user selects a 3D museum model by selecting a background, building, interior, board pattern, etc., and then places an image of an artwork in the model. In addition, the image output device described in Patent Document 2 outputs a three-dimensional space. Furthermore, both documents placed works in virtual spaces, which provided a sense of immersion but were separate from the real world.

[0005] Therefore, an object of the present invention is to provide an information processing system, a wearing device, and a program that are devised to display artworks and the like as if they existed in the real world. [Means for solving the problem]

[0006] The present invention solves the above problems by the following means. The first invention is an information processing system comprising a display unit, a photographing unit, a memory unit that stores a plurality of item images associated with text, an object recognition means that recognizes an object based on an image photographed by the photographing unit, an image generation means that generates an arrangement image in which the item image related to the object recognized by the object recognition means is arranged in three-dimensional space by referring to the text, and an image output means that outputs to the display unit a superimposed image in which the object is superimposed on the arrangement image generated by the image generation means. A second invention is an information processing system according to the first invention, which is provided with an image output means for superimposing and outputting the layout image at a position near the object recognized by the object recognition means. A third invention is an information processing system according to the first invention, further comprising an image confirmation means for referring to the memory unit and confirming whether or not there is an item image corresponding to the object recognized by the object recognition means, and when the image confirmation means confirms the item image corresponding to the object, the image generation means retrieves the item image corresponding to the object and other item images related to the item image from the memory unit, and generates the arrangement image in which the item image and other related item images are arranged in the three-dimensional space. A fourth invention is an information processing system according to the third invention, wherein the image output means superimposes the item image corresponding to the object recognized by the object recognition means at the position of the object, and outputs the layout image. A fifth invention is an information processing system in which, in the information processing system of the fourth invention, the image output means hides the item image corresponding to the object that is superimposed on the position of the object, and outputs the placement image. A sixth invention is an information processing system according to any one of the third to fifth inventions, further comprising an information acquisition means for acquiring information about the object from an external device when the image confirmation means is unable to confirm the item image corresponding to the object, and the image generation means acquires other item images related to the object from the memory unit based on the information about the object acquired by the information acquisition means, and generates the arrangement image in which the other acquired item images are arranged based on their relationship with the object. The seventh invention is an information processing system that is any of the information processing systems from the first to sixth inventions, and is provided with a layout image operation receiving means that receives operations by the user on the layout image output by the image output means, and an operation image output means that outputs the layout image, the aspect of which has been changed in response to the operation received by the layout image operation receiving means, to the display unit. An eighth invention is an information processing system according to the seventh invention, wherein the operations on the layout image accepted by the layout image operation accepting means include operations related to moving the position of the layout image, rotating the layout image, or enlarging or reducing the layout image. A ninth invention is an information processing system that is any of the information processing systems from the first to eighth inventions, and is provided with an item selection means that accepts a selection operation by the user on the item image included in the layout image output by the image output means, and an explanatory image output means that outputs to the display unit an explanatory image including the text associated with the item image for which the item selection means accepted the selection operation. The tenth invention is an information processing system in which, in any of the first to ninth inventions, at least the photographing unit and the display unit are included in a wearing device that is worn on the head of the user, and the image generating means generates the layout image, which is a stereoscopic image that utilizes binocular parallax. An eleventh invention is an information processing system according to any one of the first to ninth inventions, wherein at least the photographing unit and the display unit are included in a mobile terminal. The 12th invention is an information processing system that is any of the first to 11th inventions, and that includes a plurality of the photographing units located at different positions and a plurality of the display units corresponding to each of the plurality of photographing units, and the image output means outputs the superimposed image in which the object is superimposed on the layout image to each display unit corresponding to each photographing unit. A thirteenth invention is a wearing device to be worn on the head of a user, comprising: a display unit; a photographing unit; an object recognition means for recognizing an object based on an image photographed by the photographing unit; an image generation means for arranging the item image related to the object recognized by the object recognition means in three-dimensional space by referring to text stored in a memory unit that stores item images with associated text, and generating an arrangement image that is a stereoscopic image utilizing binocular parallax; and an image output means for outputting to the display unit a superimposed image in which the object is superimposed on the arrangement image generated by the image generation means. The fourteenth invention is a program for causing a computer communicatively connected to an imaging device and a display device to function as an object recognition means for recognizing an object based on an image captured by the imaging device, an image generation means for generating an arrangement image in which the item image related to the object recognized by the object recognition means is arranged in three-dimensional space by referring to text stored in a memory unit that stores item images with associated text, and an image output means for outputting to the display device a superimposed image in which the object is superimposed on the arrangement image generated by the image generation means. [Effects of the Invention]

[0007] According to the present invention, it is possible to provide an information processing system, a wearing device, and a program that are devised to display works of art as if they existed in the real world. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is an overall schematic diagram of an image processing system according to an embodiment of the present invention. [Figure 2]FIG. 2 is a functional block diagram of the mounting device according to the present embodiment. [Figure 3] FIG. 2 is a diagram showing an example of data stored in a work information DB according to the present embodiment. [Figure 4] 1A and 1B are diagrams illustrating an example of wearing the wearing device according to the present embodiment and an example of display on the wearing device. [Figure 5] 10 is a flowchart showing a layout image output process of the mounting device according to the present embodiment. [Figure 6] 10A and 10B are diagrams illustrating examples of display on the wearing device according to the present embodiment. [Figure 7] 10A and 10B are diagrams illustrating examples of display on the wearing device according to the present embodiment. [Figure 8] 10 is a flowchart showing a layout image operation process of the mounting device according to the present embodiment. [Figure 9] 10A and 10B are diagrams showing examples of display and a list of operation methods in the mounting device according to the present embodiment. [Figure 10] 10A and 10B are diagrams illustrating examples of display on the wearing device according to the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, this is merely an example, and the technical scope of the present invention is not limited to this example. (Embodiment) <Image processing system 100> FIG. 1 is a schematic diagram of an entire image processing system 100 according to this embodiment. FIG. 2 is a functional block diagram of the mounting device 1 according to this embodiment.

[0010] The image processing system 100 (information processing system) shown in FIG. 1 is a system including a mounting device 1, a work server 4, and an external server 7 (external device). The image processing system 100 references text stored in the work information DB 4a of the work server 4 and generates an arrangement image in which work images (item images) related to the recognized object are arranged in a cube object (three-dimensional space), which is a 3D (3-Dimensions) image of a cube-shaped space. The image processing system 100 then outputs a superimposed image in which the object is superimposed on the generated arrangement image to the wearing device 1. In this way, the wearing device 1 allows the experiencer P wearing the wearing device 1 to see the work image in front of them as if it were present in the real world. In addition, the user P can be provided with an MR (Mixed Reality) environment in which the user P can touch the arranged images.

[0011] Below, the display screen for artworks will be explained using examples of clay figurines and earthenware that are held in museums, etc. However, this is just an example, and other artworks such as sculptures and paintings may also be used. The mounting device 1, the work server 4, and the external server 7 are each connected to each other so as to be able to communicate with each other via a communication network N. The communication network N is, for example, an internet line or the like, and may be wired or wireless. Also, although only one mounting device 1 is shown in FIG. 1, multiple mounting devices 1 may be connected to the communication network N.

[0012] <Attachment device 1> The wearable device 1 is, for example, a head mounted display (HMD) that the participant P wears on his / her head. As shown in FIG. 2, the wearing device 1 includes a control unit 10, a storage unit 30, a photographing unit 36, a display unit 37, and a communication interface unit 39. The control unit 10 is a central processing unit (CPU) that controls the entire mounting device 1. The control unit 10 appropriately reads and executes an operating system (OS) and application programs stored in the storage unit 30, thereby cooperating with the above-mentioned hardware and executing various functions.

[0013] The control unit 10 includes a captured image acquisition unit 11, an object recognition unit 12, an image confirmation unit 13, an information acquisition unit 14, a placement image generation unit 15, an image output unit 16, a placement image operation reception unit 17, an operation image output unit 18, a work image operation reception unit 19, and an explanatory image output unit 20. The photographed image acquiring unit 11 acquires photographed images from the photographing unit 36. The photographed image acquiring unit 11 constantly continues to acquire photographed images photographed by the photographing unit 36.

[0014] The object recognition unit 12 functions as an object recognition means. The object recognition unit 12 recognizes an object based on a captured image. The object recognition unit 12 can recognize an object from a captured image using a known object recognition method such as a deep learning convolutional neural network (CNN).

[0015] The image confirmation unit 13 functions as an image confirmation means. The image confirmation unit 13 refers to the work information DB (database) 4a (storage unit) and confirms whether or not there is a work image corresponding to the object recognized by the object recognition unit 12. The image confirmation unit 13 refers to the work information DB (database) 4a (storage unit) and can confirm whether or not there is a work image corresponding to the object recognized by the object recognition unit 12 based on the degree of match obtained by comparing the images. The information acquisition unit 14 functions as information acquisition means. When the image confirmation unit 13 cannot confirm a work image corresponding to the object in the work information DB 4a, the information acquisition unit 14 acquires information about the object from the external server 7. The information acquisition unit 14 acquires information associated with an image corresponding to the image of the object as information about the object.

[0016] The layout image generation unit 15 functions as an image generation means. The layout image generation unit 15 references text stored in the artwork information DB 4a and generates a layout image in which an artwork image related to the recognized object is placed on a cube object. Here, the text corresponding to the artwork image includes various information about the artwork indicated by the artwork image, such as the name of the artwork, explanatory text, the age of the artwork, and the latitude and longitude where the artwork was discovered (or exhibited). At this time, the layout image generating unit 15 generates, as the layout image to be generated, an image for the right eye and an image for the left eye, which are stereoscopic images utilizing binocular parallax.

[0017] Furthermore, when the image confirmation unit 13 confirms that a work image corresponding to the target object is found, the layout image generation unit 15 may obtain the work image corresponding to the target object and other work images related to the work of the work image from the work information DB 4a, and generate a layout image in which the work image corresponding to the target object and the other related work images are arranged on a cube object. In this case, other works related to a certain work can be obtained based on text associated with the work. Specifically, the layout image generation unit 15 may select, for example, works that match the category, works of a similar era, or works with similar latitude and longitude of birth.

[0018] Furthermore, if the image confirmation unit 13 is unable to confirm a work image corresponding to the target object, the arrangement image generation unit 15 may acquire other work images related to the target object from the work information DB4a based on information about the target object acquired by the information acquisition unit 14, and generate an arrangement image in which the other acquired work images are arranged based on their relationship with the target object.

[0019] The image output unit 16 functions as an image output unit. The image output unit 16 outputs a superimposed image, in which an object is superimposed on the layout image generated by the layout image generation unit 15, to the display unit 37. Furthermore, the image output unit 16 may output the layout image to the display unit 37 by superimposing it at a position near the recognized object. Furthermore, the image output unit 16 may superimpose an artwork image corresponding to the object at the position of the recognized object, and output the layout image to the display unit 37. In this case, the image output unit 16 may not display the artwork image corresponding to the object superimposed at the position of the object.

[0020] The layout image operation receiving unit 17 functions as a layout image operation receiving means. The layout image operation receiving unit 17 receives operations by the user P on the layout image output by the image output unit 16. Here, the operations on the layout image received by the layout image operation receiving unit 17 include, for example, operations related to moving the position of the layout image, rotating the layout image, and enlarging or reducing the layout image. The position of the layout image can be moved, for example, by the user P pinching any side of the cube object with his / her fingers and then sliding his / her hand up, down, left, right (horizontally or vertically).

[0021] The arrangement image can be rotated horizontally around the up-down (vertical) axis by, for example, having the participant P pinch one of the left-right sides of the cube object with his / her fingers and then slide his / her hand in the forward / backward direction (toward the depth or toward the user).Also, the arrangement image can be rotated vertically around the horizontal axis by, for example, having the participant P pinch one of the up-down sides of the cube object with his / her fingers and then slide his / her hand in the forward / backward direction (toward the depth or toward the user). The layout image can be enlarged or reduced by, for example, the user P pinching one of the vertices of the cube object with his / her fingers and then sliding his / her hand up, down, left, right (horizontally or vertically). The operation image output unit 18 functions as an operation image output unit. The operation image output unit 18 outputs to the display unit 37 the layout image whose appearance has been changed in response to the operation accepted by the layout image operation accepting unit 17.

[0022] The artwork image operation receiving unit 19 functions as an item selection means. The artwork image operation receiving unit 19 receives a selection operation by the participant P for an artwork image included in the layout image output by the image output unit 16. For example, the participant P can perform a selection operation for an artwork image by performing a touch operation on one of the artwork images arranged in the cube object. The explanatory image output unit 20 functions as an explanatory image output means. The explanatory image output unit 20 outputs an explanatory image including text associated with the work image corresponding to the selection operation accepted by the work image operation accepting unit 19 to the display unit 37. The explanatory image output unit 20 may output the explanatory image by superimposing it on the work image corresponding to the selection operation. The explanatory image may also include the work image.

[0023] The storage unit 30 is a storage area such as a semiconductor memory element for storing programs, data, etc. required for the control unit 10 to execute various processes. The storage unit 30 includes a program storage unit 31 . The program storage unit 31 is a storage area for storing various programs. The program storage unit 31 stores a processing program 31a for performing various functions executed by the control unit 10 of the attachment device 1.

[0024] The photographing unit 36 ​​is a camera that photographs the real space. The photographing unit 36 ​​is provided, for example, on the outer surface opposite to the surface facing the head of the user P wearing the wearing device 1, near positions corresponding to the left and right eyes of the user P wearing the wearing device 1. The display unit 37 is a display device such as an LCD (Liquid Crystal Display), an organic EL display, etc. The display unit 37 is provided on the inner surface of the wearing device 1, which is the surface facing the head of the experiencer P wearing the wearing device 1. The communication interface unit 39 is an interface for communicating with the work server 4, the external server 7, etc. via the communication network N.

[0025] <Works Server 4> The work server 4 is a server that registers data in the work information DB 4a, extracts data stored in the work information DB 4a, and so on. The work server 4 is directly connected to a work information DB 4a, which is a database. Although not shown, the work server 4 includes a control unit, a storage unit, and a communication interface unit.

[0026] The work information DB 4a is a database that stores a plurality of items related to works and the contents (text) of the works corresponding to each of the plurality of items. FIG. 3 is a diagram showing an example of data stored in the work information DB 4a according to this embodiment. The work information DB4a stores the content of the work, data name, etc. for each item such as work ID (IDentification), category, work name, explanatory text, work data name, work birth year, birth location information (latitude, longitude), location of collection, related information link URL, related information video content, and keywords.

[0027] The work ID is identification information for identifying a work, and may be arbitrarily determined by the work server 4. The category indicates the genre of the work, such as pottery, clay figurines, Japanese paintings, Western paintings, prints, sculptures, and crafts. The work name is the name of the work. The work name may be the name by which the work is generally known, or may be a name given by the data registrant, for example. The commentary is a text that explains the work. The artwork data name is the file name of the artwork image. The artwork image is placed in the cube object, and is, for example, image data of a photograph of the artwork. The artwork image data only needs to express the artwork in an easy-to-understand manner, and may be 2D (2-Dimensions) image data or 3D image data.

[0028] The birth year of a work indicates the year in which the work was created. Depending on the work, the birth year of a work may indicate an approximate time period. The birth location information (latitude, longitude) indicates the location information of the place where the work was created. The birth location information (latitude, longitude) may also be the location information of the place where the work was discovered or excavated. The storage location indicates the actual storage location of the work, such as the name of an art gallery or museum. The related information link URL is, for example, the URL of a web page that contains information about the work. The related information video content indicates content related to the work. The related content may be video content that explains the details of the work, or the file name of an audio file. The keyword is a word that indicates the characteristics of the work, and multiple keywords may be included.

[0029] The attribute information items in the work information DB 4a described above are merely examples, and other items may be included, or some items may not be included. Furthermore, in the work information DB 4a, for example, a data registrant accesses the work server 4 using a registration terminal (not shown) and registers work data from a registration screen (not shown).

[0030] <External Server 7> The external server 7 may be any of various servers on the Internet, including, for example, a server of a portal site. The external server 7 receives, for example, an image of an object recognized by the wearing device 1 from the wearing device 1, searches for the image of the object, and transmits information about the object to the wearing device 1. Although not shown, the external server 7 includes at least a control unit, a storage unit, and a communication interface unit.

[0031] Here, a computer refers to an information processing device equipped with a control unit, a memory device, etc., and the mounting device 1, the work server 4, and the external server 7 are each information processing devices equipped with a control unit, a memory device, etc., and are included in the concept of a computer.

[0032] <Processing of mounting device 1> Next, the processing in the mounting device 1 will be described. FIG. 4 is a diagram showing an example of how the wearing device 1 according to this embodiment is worn and an example of a display on the wearing device 1. In FIG. FIG. 5 is a flowchart showing the layout image output process of the wearing device 1 according to this embodiment. 6 and 7 are diagrams showing examples of display on the wearing device 1 according to this embodiment.

[0033] As shown in FIG. 4(A), the user P wears the wearing device 1 on his / her head so that the wearing device 1 covers his / her eyes. With the participant P wearing the wearable device 1 on his / her head, the wearable device 1 starts the processing shown in Fig. 5. Note that this flowchart will be described using an example realized by a video see-through display. In a video see-through display, images captured in real time by a camera unit 36 ​​installed in front of the wearable device 1 are displayed. In other words, the real world in front of the participant P is captured by the camera unit 36 ​​and visualized.

[0034] In step S (hereinafter simply referred to as "S") 11 of FIG. 5, the control unit 10 (photographed image acquisition unit 11) of the wearing device 1 acquires a photographed image via the photographing unit . In S12, the control unit 10 (image output unit 16) displays the acquired photographed image on the display unit 37 as is.

[0035] 4(B) shows an example of an image 50 that is first seen by the participant P wearing the wearing device 1. The image 50 is an image of the real world (real space), and the participant P can see, through the wearing device 1, a clay vessel 52 placed on a stand 51 in the participant P's line of sight. Furthermore, if there are no restrictions on the exhibit, the participant P can also touch the stand 51 and the clay vessel 52.

[0036] 5, the control unit 10 (object recognition unit 12) recognizes the object based on the captured image. If the control unit 10 cannot recognize the object, it cannot perform the subsequent processing and ends this processing. In S14, the control unit 10 (image confirmation unit 13) refers to the artwork information DB 4a and confirms whether or not there is an artwork image corresponding to the recognized object.

[0037] In S15, the control unit 10 (image confirmation unit 13) determines whether or not the corresponding work image has been confirmed. If the corresponding work image has been confirmed (S15: YES), the control unit 10 proceeds to S16. On the other hand, if the corresponding work image has not been confirmed (S15: NO), the control unit 10 proceeds to S17. In S16, the control unit 10 (layout image generation unit 15) acquires the confirmed work image and other work images related to the work from the work information DB 4a. After that, the control unit 10 moves the process to S19.

[0038] On the other hand, in S17, the control unit 10 (information acquisition unit 14) transmits an image of the object to the external server 7, and acquires information about the object from the external server 7. In S18, the control unit 10 (layout image generating unit 15) acquires other artwork images related to the object from the artwork information DB 4a based on the information about the object acquired in the process of S17.

[0039] In S19, the control unit 10 (arrangement image generation unit 15) generates an arrangement image in which the acquired work information is arranged on a cube object. The control unit 10 refers to each text corresponding to the acquired plurality of work images and arranges the plurality of work images on the cube object. At this time, it is desirable to arrange each work image so that it faces forward as viewed from the experiencer P. In particular, if the work image is a 2D image, it is arranged so that the work image is viewed from the front by the experiencer P. Even if the work image is a 3D image, it is arranged so that the front part of the work faces forward as viewed from the experiencer P. Furthermore, the control unit 10 generates, as the arrangement image to be generated, an image for the right eye and an image for the left eye, which are stereoscopic images utilizing binocular parallax. In S20, the control unit 10 (image output unit 16) superimposes the object on the generated layout image and outputs it to the display unit 37.

[0040] FIG. 6 shows an image 60 in which a layout image 61 is superimposed in a position adjacent to the earthenware 52. In this example, the control unit 10 recognizes the earthenware 52 as an object, and generates and outputs an arrangement image 61 near the earthenware 52 as an artwork in the real world. The layout image 61 includes a plurality of artwork images 62 and a cube object 63. In this example, the layout image 61 uses the Z-axis (vertical direction) of the cube object as the era. The X-axis (horizontal direction) and Y-axis (forward / backward direction) are not specifically described, but may be, for example, latitude and longitude or category.

[0041] FIG. 7 shows an image 70 in which a layout image 71 is superimposed on a pottery vessel 52. In this example, the control unit 10 recognizes the earthenware 52 as an object, and generates and outputs the earthenware 52 as a work of art that exists in the real world as part of the layout image 71. The arrangement image 71 includes a plurality of artwork images 72 and a cube object 73. The arrangement image 71 arranges the artwork images 72 based on their relationship with the earthenware 52. By outputting such an arrangement image 71, the relationship between the earthenware 52 and the artwork images 72 is more clearly shown. Returning to FIG. 5, the control unit 10 then ends this process.

[0042] Next, the operation of the layout image will be described. FIG. 8 is a flowchart showing the layout image operation process of the wearing device 1 according to this embodiment. FIG. 9 is a diagram showing a list 85 of display examples and operation methods of the wearing device 1 according to this embodiment. FIG. 10 is a diagram showing a display example on the wearing device 1 according to this embodiment.

[0043] As shown in Figure 9(A), an image 80 including a layout image 81 is displayed in front of the participant P, and when the participant P extends his / her hand to the layout image 81, the control unit 10 recognizes the participant P's hand and displays the virtual hand VH in conjunction with the participant P's hand. 8, the control unit 10 (layout image operation receiving unit 17) determines whether or not an operation on the layout image has been received by the virtual hand VH linked to the hand of the user P. If an operation on the layout image has been received (S31: YES), the control unit 10 shifts the process to S32. On the other hand, if an operation on the layout image has not been received (S31: NO), the control unit 10 shifts the process to S34. In S32, the control unit 10 changes the appearance of the layout image in accordance with the operation.

[0044] Here, the layout image modes and the operation methods will be described using the list 85 in FIG. 9(B) as an example. For example, if the user P wants to move the position of the layout image, the user P pinches the edge of the cube object with his / her fingers, grabs the cube object, and moves his / her hand in the direction of movement on the plane. In this way, the layout image can be moved in accordance with the movement of the user P's hand. For example, when the user P pinches an edge 81a of a cube object shown in the layout image 81 of FIG. 9(A) with his / her hand, the virtual hand VH moves as if grasping the layout image 81. Furthermore, when the participant P wants to rotate the layout image 81, he / she grasps the cube object by pinching the side 81a of the cube object with his / her fingers and moves his / her hand back and forth. In this way, the layout image can be rotated in accordance with the movement of the participant P's hand.

[0045] If the participant P wishes to enlarge the layout image, he / she pinches a vertex of the cube object with his / her fingers and moves his / her hand toward the outside of the cube object. In this way, the layout image can be enlarged. When the layout image is enlarged, the artwork image may also be enlarged accordingly. Note that by enlarging the layout image to a size of, for example, about 5 m, the layout image can be output in such a way that the participant P appears as if he / she has entered the cube object, and the participant P can experience walking around inside the cube object. On the other hand, if the user P wants to reduce the size of the arranged image, the user P pinches the vertex of the cube object with his / her fingers and moves his / her hand inward toward the cube object. In this way, the arranged image can be reduced.

[0046] In S33 of FIG. 8, the control unit 10 (operation image output unit 18) outputs the changed layout image to the display unit 37. In S34, the control unit 10 (artwork image operation receiving unit 19) determines whether or not a work image selection operation has been received. If a work image selection operation has been received (S34: YES), the control unit 10 proceeds to S35. On the other hand, if a work image selection operation has not been received (S34: NO), the control unit 10 ends this process. The operation for selecting an image of a work of art is to touch the image of the work of art with a virtual hand VH linked to the hand of the participant P.

[0047] In S35, the control unit 10 acquires each piece of information about the selected work image from the work information DB 4a. In S36, the control unit 10 generates explanation information based on the acquired information. In S37, the control unit 10 (explanatory image output unit 20) outputs the generated explanatory information to the display unit 37 with the explanatory information further superimposed.

[0048] FIG. 10(A) shows an image 90 as an example of a state in which a user P enters the enlarged layout image. With image 90 displayed in front of the viewer P, the viewer P extends his / her hand to artwork image 91 and touches artwork image 91 with virtual hand VH. 10(B), an explanation window 95 (explanatory information) is displayed on the image 90. The explanation window 95 contains explanatory information about the artwork image 91. Returning to FIG. 8, the control unit 10 ends this process.

[0049] As described above, the wearing device 1 of this embodiment has the following advantages. (1) A layout image is generated in which a work image related to the recognized object is placed on a cube object by referring to the text stored in the work information DB4a, and a superimposed image in which the object is superimposed on the generated layout image is output to the display unit 37. Therefore, the artwork images stored in the artwork information DB 4a can be superimposed on the real world and shown to the viewer P. Furthermore, the placement information is obtained by placing artwork images related to the recognized objects on cube objects by referring to the text, making it easy to understand the relationships between the artworks and presenting a new way of displaying artworks. Furthermore, since the artworks are placed on cube objects, the positioning of the artworks can be easily grasped based on their positions. Furthermore, since the objects are superimposed on the placement images, the placement images can be confirmed together with the objects. (2) The layout image is output superimposed at a position close to the object, so that the object and the layout image can be checked side by side.

[0050] (3) The work information DB4a is referred to to check whether there is a work image corresponding to the recognized object, and if a work image corresponding to the object is confirmed, the work image corresponding to the object and other work images related to the work image are obtained from the work information DB4a, and an arrangement image is generated in which the work image and other related work images are arranged on the cube object. Therefore, an image of a work related to an object can be placed on a cube object and displayed, allowing the viewer to see the work related to the object in the real world.

[0051] (4) When an object is recognized, the artwork image corresponding to the object is superimposed on the object's position, and a layout image is output. Therefore, it is possible to check the object and the artwork image, including their mutual relationship. (5) The artwork image corresponding to the object superimposed on the position of the object is hidden, and the layout image is output. This allows the object to appear as if it is part of the layout image.

[0052] (6) If no artwork image corresponding to the target object is found, information about the target object is obtained from the external server 7, and based on the obtained information about the target object, other artwork images related to the target object are obtained from the artwork information DB 4a, and an arrangement image is generated in which the other obtained artwork images are arranged based on their relationship with the target object. Therefore, even if there is no work information corresponding to the object in the work information DB 4a, it is possible to use the external server 7 to display an arrangement image including other work images that have a relationship with the object.

[0053] (7) An operation by the user P on the layout image is accepted, and the layout image whose appearance has been changed in response to the accepted operation is output to the display unit 37. Therefore, the user P can operate the layout image and have the output corresponding to the operation be produced, so that it is possible to create a performance that will interest the user P. (8) Operations on a placed image include operations related to moving the position of the placed image, rotating the placed image, or enlarging or reducing the placed image. Therefore, various types of layout images can be provided to the experiencer P through the experiencer P's operation.

[0054] (9) A selection operation by the user P on a work image included in the layout image is accepted, and an explanatory image including text associated with the accepted work image is output to the display unit 37. Therefore, the viewer P can see the explanation of the work corresponding to the work image, and can learn more about the work. In addition, the viewer P can see the explanation of the work corresponding to the work image by operating the viewer P, so the viewer P can learn about various works independently.

[0055] (10) The image capturing unit 36 ​​and the display unit 37 are provided in the wearable device 1 that is worn on the head of the participant P, and generate a layout image that is a stereoscopic image utilizing binocular parallax. Therefore, a three-dimensional layout image can be made to appear in front of the eyes of the user P wearing the wearing device 1.

[0056] Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments. Furthermore, the effects described in the embodiments are merely a list of the most preferable effects resulting from the present invention, and the effects of the present invention are not limited to those described in the embodiments. Note that the above-described embodiments and the modified embodiments described below can be used in appropriate combinations, but detailed description thereof will be omitted.

[0057] (Variations) (1) In this embodiment, the embodiment has been described with reference to an example in which the wearable device 1 employs a video see-through display, but is not limited to this. For example, the embodiment may employ a wearable device employing an optical see-through display. In an optical see-through display, the user can view the surrounding scenery through a lens, and the image of the electronic display can be superimposed using an optical system such as a prism or half mirror. When this mechanism is employed, for example, the process of S12 in FIG. 5 is not necessary.

[0058] (2) In the present embodiment, the present invention is described with reference to an example realized by the wearing device 1, but is not limited thereto. For example, the present invention may be realized using the wearing device and a computer such as an information processing server communicably connected to the wearing device. In this case, processing other than processing related to receiving operations and outputting images may be performed by the wearing device or the information processing server.

[0059] (3) In the present embodiment, the wearing device is described as having a photographing unit and a display unit, but is not limited to this. The wearing device may be a computer such as a mobile terminal or tablet that is equipped with a photographing unit and a display unit.

[0060] (4) In this embodiment, an example of a layout image in which an object is recognized based on a captured image and other artwork information related to the recognized object is acquired is described. However, the present invention is not limited to the above-described example. For example, an object creator may specify one or more artwork images to be placed in advance, and then an layout image in which multiple artwork images are placed on a cube object based on the relevance of text corresponding to the specified artwork images is created and stored in a storage unit. Then, when displaying the layout image, the object may be superimposed on the layout image stored in the storage unit and output to a display unit. In this way, it is only necessary to output the layout image that has been generated in advance. Since the process of generating the layout image is performed in advance, the processing efficiency for display is improved, and the time required for outputting the image to the display unit can be shortened. In addition, when generating the placement image, the control unit of the mounting device may select the optimal axis based on the text of each acquired work image, and place each work image on a cube object based on the selected axis.

[0061] (5) In this embodiment, the example has been described using one participant P as an example, but this is not limiting. Multiple participants may be able to simultaneously view one layout image. In this case, each participant wears a wearable device equipped with a photographing unit and a display unit, and the image output means of the image distribution system superimposes an object on the layout image and outputs it to the display unit. This allows each participant in a remote location to share a single layout image and operate and view it.

[0062] (6) In the present embodiment, the manner of the arranged image is changed or an explanatory image of the artwork image is output in response to the operation of the experiencer P, but this is not limiting. For example, when an artwork image is selected, an artwork image of the same real size as the artwork itself may be output. Furthermore, the relevance between the artwork images may be indicated by connecting the artwork images with lines and displaying common keywords. Furthermore, the experiencer P may be allowed to place new artwork images or change the positions of placed artwork images by operation.

[0063] (7) In the present embodiment, examples have been described in which clay figurines, earthenware, and the like held in art galleries and museums are used as the target, but the present invention is not limited to this. For example, other artworks, paintings, and the like may also be used. Furthermore, the present invention is not limited to artworks, and may also be used, for example, in corporate portfolios that enable comparison and reference of multiple companies. In such cases, the image may be, for example, a corporate logo. [Explanation of symbols]

[0064] 1. Mounting device 4. Works Server 4a Work information DB 7 External Servers 10 Control Unit 11. Image acquisition unit 12 Object Recognition Unit 13 Image confirmation section 14 Information acquisition department 15 Placement image generation unit 16 Image output unit 17 Placement image operation reception section 18 Operation image output section 19. Work Image Operation Reception 20 Explanation image output section 30 Storage section 31a Processing Program 36 Photography Department 37 Display section 39 Communication interface section 50, 60, 70, 80, 90 images 51 units 52 Pottery 61, 71, 81 Placement images 62, 72 Work images 63, 73 Cube Object Around 81a 85 List 91 Work images 95 Explanation window 100 Image Processing System N Communication Network P Experiencer VH Virtual Hand

Claims

1. A display unit; The photography department and a storage unit that stores a plurality of item images associated with text; an object recognition means for recognizing an object based on the image captured by the imaging unit; an image generating means for generating an arrangement image in which the item images related to the object recognized by the object recognizing means are arranged in a three-dimensional space with reference to the text; an image output means for outputting a superimposed image, in which the object is superimposed on the layout image generated by the image generation means, to the display unit; An information processing system comprising:

2. 2. The information processing system according to claim 1, An information processing system comprising: an image output means for outputting the layout image by superimposing it at a position near the object recognized by the object recognition means.

3. 2. The information processing system according to claim 1, an image confirmation means for referring to the storage unit and confirming whether or not the item image corresponding to the object recognized by the object recognition means exists; An information processing system in which, when the image confirmation means confirms the item image corresponding to the target object, the image generation means retrieves the item image corresponding to the target object and other item images related to the item image from the memory unit, and generates the arrangement image in which the item image and the other related item images are arranged in the three-dimensional space.

4. 4. The information processing system according to claim 3, The image output means outputs the layout image by superimposing the item image corresponding to the object at a position of the object recognized by the object recognition means.

5. 5. The information processing system according to claim 4, The image output means outputs the layout image while hiding the item image corresponding to the object superimposed on the position of the object.

6. 4. The information processing system according to claim 3, an information acquisition means for acquiring information about the object from an external device when the image confirmation means cannot confirm the item image corresponding to the object; The image generation means acquires other item images related to the object from the memory unit based on information about the object acquired by the information acquisition means, and generates the arrangement image in which the other acquired item images are arranged based on their relationship with the object.

7. 2. The information processing system according to claim 1, an arrangement image operation receiving means for receiving an operation by an experiencer on the arrangement image output by the image output means; an operation image output means for outputting, to the display unit, the layout image whose aspect has been changed in response to the operation accepted by the layout image operation accepting means; An information processing system comprising:

8. 8. The information processing system according to claim 7, An information processing system, wherein the operation on the layout image accepted by the layout image operation accepting means includes an operation related to moving the position of the layout image, rotating the layout image, or enlarging or reducing the layout image.

9. 2. The information processing system according to claim 1, an item selection means for accepting a selection operation by a user for the item image included in the layout image output by the image output means; an explanatory image output means for outputting, to the display unit, an explanatory image including the text associated with the item image selected by the item selection means; An information processing system comprising:

10. 2. The information processing system according to claim 1, At least the photographing unit and the display unit are included in a wearable device that is worn on the head of the user, The image generating means generates the layout image, which is a stereoscopic image utilizing binocular parallax.

11. 2. The information processing system according to claim 1, An information processing system, wherein at least the photographing unit and the display unit are included in a mobile terminal.

12. 2. The information processing system according to claim 1, a plurality of the photographing units provided at different positions; and a plurality of the display units corresponding to the plurality of photographing units, The image output means outputs the superimposed image, in which the object is superimposed on the layout image, to each display unit corresponding to each photographing unit.

13. A wearing device to be worn on the head of the user, A display unit; The photography department and an object recognition means for recognizing an object based on the image captured by the imaging unit; an image generating means for arranging the item images related to the object recognized by the object recognizing means in a three-dimensional space with reference to the text stored in a storage unit that stores item images associated with the text, and generating an arrangement image that is a stereoscopic image utilizing binocular parallax; an image output means for outputting a superimposed image, in which the object is superimposed on the layout image generated by the image generation means, to the display unit; A mounting device comprising:

14. a computer communicably connected to the imaging device and the display device; an object recognition means for recognizing an object based on an image captured by the imaging device; an image generating means for generating an arrangement image in which the item images related to the object recognized by the object recognizing means are arranged in a three-dimensional space with reference to the text stored in a storage unit that stores item images associated with the text; an image output means for outputting a superimposed image, in which the object is superimposed on the layout image generated by the image generation means, to the display device; A program to make it function as such.

Images