Program, mounting device, and information processing system

Abstract

We provide a program, mounting device, and information processing system that can apply images with texture to objects to meet user needs. [Solution] The mounting device 1, which includes a shooting unit 36 ​​and a display unit 37, comprises a hand shape detection unit 12 that detects the shape of the user's hand, including the fingers, from an image captured by the shooting unit 36 ​​while the device is mounted on the user's head; a sample image output unit 17 that outputs a sample material image with texture to the display unit 37; an object output unit 19 that outputs an object stored in the object storage unit 34 to the display unit 37; a contact detection unit 14 that detects when the detected user's finger comes into contact with the output object; and a texture image application unit 20 that applies the output sample material image to the contact surface of the object where contact has been detected. The texture image application unit 20 applies the sample material image to the contact surface of the object so that the orientation of the pattern represented as the texture of the sample material image is the same.

Description

Technical Field

[0001] The present invention relates to a program, a mounting device, and an information processing system.

Background Art

[0002] Conventionally, various efforts using VR (Virtual Reality) and MR (Mixed Reality) technologies have been made. For example, VR and MR are increasingly being used when performing simulations using 3DCG (3 Dimensional Computer Graphics). As an example, a fitting simulation system for adjusting the color and pattern of fittings to match the color and pattern of the interior walls and floors has been disclosed (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] There are cases where it is desired to perform a simulation that satisfies the user's needs by applying an image having a texture such as color and pattern to an object, which is a model generated by computer graphics (CG).

[0005] Therefore, an object of the present invention is to provide a program, a mounting device, and an information processing system that can apply an image having a texture to an object so as to satisfy the user's needs.

Means for Solving the Problems

[0006] The present invention solves the above problems by the following means. The first invention is a program that causes a computer to function as a first detection means for detecting the shape of the user's hand, including the fingers, from an image captured by a camera unit in a wearable device attached to the user's head; a sample image output means for outputting a sample material image having a texture to a display unit in the wearable device; a contact detection means for detecting when the user's fingers, detected by the first detection means, have come into contact with an object visible through the display unit; and a texture application means for pasting the sample material image output by the sample image output means onto the contact surface of the object where the contact detection means has detected contact, wherein the texture application means is configured to paste the sample material image onto the contact surface of the object so that the orientation of the pattern represented as the texture of the sample material image is the same. The second invention is a program in which, in the program of the first invention, the sample image output means is configured to output the sample member image placed on the user's hand detected by the first detection means, and the computer is configured to function as a sample image linking means that tilts the sample member image in conjunction with the movement of the user's hand. The third invention is a program of the first or second invention that causes the texture application means to function such that, when the sample member image is placed in the user's hand by the sample image output means, the contact detection means detects contact and applies the sample member image to the contact surface of the object. The fourth invention is a program in which, in any of the programs of the first to third inventions, the computer is made to function as an object output means that outputs the object, which is an object image obtained by visualizing an object, to the display unit, and the contact detection means is made to function to detect when the user's finger detected by the first detection means has come into contact with the object output by the object output means. The fifth invention is a program of the fourth invention in which the object is composed of a plurality of partial images, and the texture application means is configured to apply the sample member image to the contact surface of the partial image of the object that has been contacted by the contact detection means. The sixth invention is a program in which, in any of the programs of the first to third inventions, the computer is made to function as an object detection means for detecting the object from the captured image, and the contact detection means is made to function to detect that the user's finger detected by the first detection means has come into contact with the object detected by the object detection means. The seventh invention is a wearable device comprising: a shooting unit; a display unit; and a storage unit that stores an object which is an object image obtained by imaging an object, the wearable device comprising: a first detection means for detecting the shape of the user's hand, including the fingers, from a captured image taken by the shooting unit while the device is worn on the user's head; a sample image output means for outputting a sample material image having a texture to the display unit; an object output means for outputting the object from the storage unit to the display unit; a contact detection means for detecting that the user's fingers detected by the first detection means have come into contact with the object output by the object output means; and a texture application means for attaching the sample material image output by the sample image output means to the contact surface of the object where the contact detection means has detected contact, wherein the texture application means attaches the sample material image to the contact surface of the object such that the orientation of the pattern represented as the texture of the sample material image is the same. The eighth invention is an information processing system comprising: a mounting device equipped with a shooting unit and a display unit; and an information processing device communicatively connected to the mounting device, wherein the information processing device comprises: a storage unit that stores an object which is an object image obtained by imaging an object; a first detection means that detects the shape of the user's hand, including the fingers, from a shooting image taken by the shooting unit while the user is wearing the mounting device on their head; a sample image output means that outputs a sample member image having a texture to the display unit; an object output means that outputs the object from the storage unit to the display unit; a contact detection means that detects that the user's fingers detected by the first detection means have come into contact with the object output by the object output means; and a texture application means that applies the sample member image output by the sample image output means to the contact surface of the object where the contact detection means has detected contact, wherein the texture application means applies the sample member image to the contact surface of the object such that the orientation of the pattern represented as the texture of the sample member image is the same. [Effects of the Invention]

[0007] According to the present invention, it is possible to provide a program, a mounting device, and an information processing system that can apply images with texture to objects to meet the needs of the user. [Brief explanation of the drawing]

[0008] [Figure 1] This is a functional block diagram of the mounting device according to this embodiment. [Figure 2] This flowchart shows the main processing of the mounting device according to this embodiment. [Figure 3] This figure shows an example of mounting and display of the mounting device according to this embodiment. [Figure 4] This flowchart shows the process for acquiring a sample member image of the mounting device according to this embodiment. [Figure 5] This figure shows an example of the display of the mounting device according to this embodiment. [Figure 6]This is a diagram showing a display example of the wearable device according to the present embodiment. [Figure 7] This is a diagram showing a display example of the wearable device according to the present embodiment. [Figure 8] This is a flowchart showing the master member image application process of the wearable device according to the present embodiment. [Figure 9] This is a diagram showing a display example of the wearable device according to the present embodiment. [Figure 10] This is a diagram for explaining the master member image application process according to the present embodiment.

Mode for Carrying Out the Invention

[0009] Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings. Note that this is merely an example, and the technical scope of the present invention is not limited thereto. (Embodiment) <Wearable device 1> FIG. 1 is a functional block diagram of the wearable device 1 according to the present embodiment. The wearable device 1 shown in FIG. 1 is devised to give the user P a sense of operation even with intuitive operations. Further, the wearable device 1 can apply an image with a texture to an object so as to satisfy the needs of the user P.

[0010] The wearable device 1 is, for example, an HMD (Head Mounted Display) worn by the user P on the head. The wearable device 1 includes a control unit 10, a storage unit 30, a photographing unit 36, and a display unit 37. The control unit 10 is a central processing unit (CPU) that controls the entire wearable device 1. The control unit 10 appropriately reads out and executes the operating system (OS) and application programs stored in the storage unit 30, and cooperates with the above-described hardware to execute various functions.

[0011] Before explaining the control unit 10, the storage unit 30 will be explained. The storage unit 30 is a storage area such as a semiconductor memory element for storing programs, data, etc. necessary for the control unit 10 to execute various processes. The storage unit 30 includes a program storage unit 31, a GUI (Graphical User Interface) storage unit 32, a reference member image storage unit 33, and an object storage unit 34. The program storage unit 31 is a storage area for storing various programs. The program storage unit 31 stores a processing program 31a for performing various functions executed by the control unit 10 described later.

[0012] The GUI storage unit 32 is a storage area for storing a GUI (virtual member image). In this example, the GUI stored in the GUI storage unit 32 has a plurality of button members. Each button member is associated with an image having, for example, a different texture. Each image having a different texture corresponds to the reference member image stored in the reference member image storage unit 33. The reference member image storage unit 33 is a storage area for storing reference member images. In this example, the reference member images stored in the reference member image storage unit 33 have different textures, and are, for example, kamaboko-shaped images of a size that can be placed on the palm of user P. Here, the texture includes, in addition to color and pattern, gloss, metallic feeling, unevenness, etc. And the reference member image has a shader and a texture for expressing the texture.

[0013] The object storage unit 34 is a storage area for storing objects. Here, the object is a 3DCG (3 Dimensional Computer Graphics) of an object for performing a simulation of the texture on the surface. The object used in the example described below is a model of a part of the front door which is the interior of a car. Also, the object may be composed of a plurality of partial images (parts).

[0014] Next, the control unit 10 will be described. The control unit 10 includes a captured image acquisition unit 11, a hand shape detection unit 12, a plane detection unit 13, a contact detection unit 14, a GUI output unit 15, an operation execution processing unit 16, a sample image output unit 17, an image linkage processing unit 18, an object output unit 19, and a texture image application unit 20. The image acquisition unit 11 acquires images from the shooting unit 36. The image acquisition unit 11 continuously acquires images captured by the shooting unit 36.

[0015] The hand shape detection unit 12 functions as a first detection means. The hand shape detection unit 12 detects the shape of the user P's hand, including its fingers, from the captured image. The hand shape detection unit 12 can detect the shape of the hand from a series of captured images, for example, using known hand tracking techniques. Hand tracking techniques include, for example, those that use image-based posture inference using machine learning (ML) techniques, or those that use infrared sensors to capture hand movements with an infrared camera and perform image analysis, and any of these may be used.

[0016] The plane detection unit 13 functions as a second detection means. The plane detection unit 13 detects the plane of an object from the captured image. The plane detection unit 13 can detect a plane using, for example, known plane detection or object detection techniques. Examples of plane detection techniques include those using visual-inertial odometry (VIO), those that make a judgment by comprehensively analyzing the image with various sensor information such as acceleration and gyroscope data, and those using SLAM (Simultaneous Localization and Mapping). Furthermore, the plane detection unit 13 determines whether the plane is a desk and detects the desk's plane. In addition to the plane detection technology described above, object recognition can also be used to detect the desk's plane.

[0017] The contact detection unit 14 functions as a contact detection means. The contact detection unit 14 detects, for example, contact by the user P's fingers detected by the hand shape detection unit 12. More specifically, the contact detection unit 14 detects when the fingers of user P, as detected by the hand shape detection unit 12, come into contact with the desk, as detected by the plane detection unit 13. The contact detection unit 14 also detects when the fingers of user P, as detected by the hand shape detection unit 12, come into contact with a GUI button component output by the GUI output unit 15. Furthermore, the contact detection unit 14 detects when the fingers of user P, as detected by the hand shape detection unit 12, come into contact with an object output by the object output unit 19.

[0018] The GUI output unit 15 functions as a component image output means. The GUI output unit 15 places the GUI on the flat surface of the desk detected by the flat surface detection unit 13, and on the flat surface of the desk near the position of the user P's hand detected by the hand shape detection unit 12, and outputs it to the display unit 37. More specifically, the GUI output unit 15 places and outputs the GUI at the position where the user P's finger touched, as detected by the contact detection unit 14. As a result, the GUI output by the GUI output unit 15 will appear as if it were placed on the desk. The operation execution processing unit 16 functions as a means for executing processing. The operation execution processing unit 16 executes processing associated with the GUI button component at the position where the user P's finger, detected by the contact detection unit 14, touches.

[0019] The sample image output unit 17 functions as both a result image output means and a sample image output means. The sample image output unit 17 outputs a sample component image. The sample image output unit 17 outputs the image obtained as a result of the processing performed by the operation execution processing unit 16 to the display unit 37. In this example, the image obtained as a result of the processing output by the sample image output unit 17 is a sample component image with texture. The sample image output unit 17 then places the sample component image on the hand of user P detected by the hand shape detection unit 12 and outputs it. The image synchronization processing unit 18 functions as a sample image synchronization means. The image synchronization processing unit 18 performs a process to tilt the sample member image output by the sample image output unit 17 in conjunction with the hand movements of the user P.

[0020] The object output unit 19 functions as an object output means. The object output unit 19 outputs objects stored in the object storage unit 34 to the display unit 37. The texture image application unit 20 functions as a texture application means. The texture image application unit 20 applies a sample material image output by the sample image output unit 17 to the contact surface of an object whose contact has been detected by the contact detection unit 14. More specifically, the texture image application unit 20 applies the sample material image to the contact surface of an object, provided that the sample material image is placed in the hand of user P by the sample image output unit 17 and the contact detection unit 14 has detected contact. The texture image application unit 20 also applies the sample material image to the contact surface of the image of the contact portion of the object that has been touched by the contact detection unit 14. At that time, the texture image application unit 20 applies the sample material image to the contact surface of the object so that the orientation of the pattern represented as the texture of the sample material image is the same.

[0021] The imaging unit 36 ​​is a camera that captures real space. Multiple imaging units 36 are provided in positions that allow them to capture the surroundings, for example, on the outer surface opposite to the surface facing the head of user P wearing the mounting device 1, and near the positions corresponding to the left and right eyes of user P wearing the mounting device 1. Therefore, the images captured by the imaging unit 36 ​​are images of approximately the same range as what user P's left and right eyes can see.

[0022] The display unit 37 is a display device such as an LCD (Liquid Crystal Display) or an organic EL display. The display unit 37 is provided on the inner surface that faces the head of the user P wearing the mounting device 1. Here, "computer" refers to an information processing device equipped with a control unit, memory device, etc., and the mounting device 1 is an information processing device equipped with a control unit 10, a memory unit 30, etc., and is included in the concept of a computer.

[0023] <Processing of mounting device 1> Next, we will explain the processing performed by the mounting device 1. Figure 2 is a flowchart showing the main processing of the mounting device 1 according to this embodiment. Figure 3 shows an example of mounting and display of the mounting device 1 according to this embodiment. Figure 4 is a flowchart showing the sample member image acquisition process of the mounting device 1 according to this embodiment. Figures 5 to 7 show examples of displays for the mounting device 1 according to this embodiment.

[0024] As shown in Figure 3(A), user P attaches the attachment device 1 to their head so that their eyes are covered by the attachment device 1. With user P wearing the device 1 on their head, the device 1 begins the process shown in Figure 2. The example described here uses a video see-through display. The video see-through display shows images captured in real time by the camera unit 36, which is installed in front of the device 1. In other words, the real world in front of user P is captured and visualized by the camera unit 36.

[0025] In step S (hereinafter simply referred to as "S") 11 of Figure 2, the control unit 10 (image acquisition unit 11) of the mounting device 1 acquires an image via the imaging unit 36 ​​and displays the acquired image directly on the display unit 37. Figure 3(B) shows an example of an image 50 visible to user P wearing the device 1. Image 50 is an image of the real world (real space), and user P can see the desk 50a in the user P's field of view through the device 1. In step S12 of Figure 2, the control unit 10 performs the sample member image acquisition process.

[0026] Here, the process for acquiring sample component images will be explained based on Figure 4. In step S31 of Figure 4, the control unit 10 (hand shape detection unit 12) detects the shape of the user P's hand, including its fingers, from the captured image. In S32, the control unit 10 (plane detection unit 13) detects the plane of an object from the captured image and detects the desk. Image 51 in Figure 5(A) includes a virtual hand 51b corresponding to the hand of user P, which was detected because user P's hand was captured in the image. Image 51 also includes the detection state 51c of the desk 50a, which was detected when a plane was detected and a desk 50a was detected within it. Note that the order of processing S31 and S32 in Figure 4 does not matter. Processing S31 and S32 may be performed simultaneously, or processing S31 may be performed after processing S32. Furthermore, while image 51 in Figure 5(A) displays the virtual hand 51b and detection status 51c, this display is not mandatory. It may or may not be displayed.

[0027] In S33 of Figure 4, the control unit 10 (contact detection unit 14) determines whether the detected finger of user P touched the detected surface of the desk. The control unit 10 determines whether the detected finger of user P touched the detected surface of the desk based on the relationship between the position of user P's finger and the position of the desk surface. If the detected finger of user P touched the detected surface of the desk (S33: YES), the control unit 10 moves the process to S34. On the other hand, if the detected finger of user P did not touch the detected surface of the desk (S33: NO), the control unit 10 moves the process to S31 and repeatedly performs the detection process from the captured image.

[0028] In S34, the control unit 10 (GUI output unit 15) places and displays the GUI at the position touched by user P's finger. Image 52 in Figure 5(B) includes a virtual hand 52b corresponding to the detected hand of user P. Image 52 also shows the GUI 52d positioned as if it were placed on a desk 50a. The GUI 52d is an image of an operating member on which multiple button members 52d1 to 52d4 are arranged.

[0029] In S35 of Figure 4, the control unit 10 (contact detection unit 14) determines whether the detected finger of user P has touched a button component of the GUI output by the GUI output unit 15. The control unit 10 determines whether the detected finger of user P has touched a button component of the GUI based on the relationship between the position of user P's finger and the position of each button component. If the detected finger of user P has touched a button component of the GUI output by the GUI output unit 15 (S35: YES), the control unit 10 moves the process to S36. On the other hand, if the detected finger of user P has not touched a button component of the GUI output by the GUI output unit 15 (S35: NO), the control unit 10 remains in that process.

[0030] Image 53 in Figure 6(A) includes a virtual hand 53b corresponding to the detected hand of user P. Image 53 also includes a GUI 53d that appears to be placed on a desk 50a, showing user P's right index finger pressing the button component 53d3. At this time, user P's right index finger is performing a pressing motion, and the pad of user P's right index finger is touching the desk 50a. Therefore, user P can feel that they have pressed the button component 53d3.

[0031] In S36 of Figure 4, the control unit 10 (operation execution processing unit 16) executes a process associated with the button component touched by user P's finger, and obtains a sample component image as an execution result. After that, the control unit 10 moves the process to S13 of Figure 2. In step S13 of Figure 2, the control unit 10 (hand shape detection unit 12) detects the shape of the user P's hand, including its fingers, from the captured image. Image 54 in Figure 6(B) includes virtual hands 54b and 54e, which correspond to the detected hands of user P. Virtual hand 54b corresponds to user P's right hand, and virtual hand 54e corresponds to user P's left hand. Image 54 also includes a GUI 54d, which appears to be placed on a desk 50a.

[0032] In S14 of Figure 2, the control unit 10 (sample image output unit 17) displays a sample component image on the detected user P's hand. Image 55 in Figure 7(A) includes virtual hands 55b and 55e corresponding to the detected hand of user P. Image 55 also shows a GUI 55d displayed as if it were placed on a desk 50a. Furthermore, Image 55 shows a sample component image 55f displayed as if it were placed on the virtual hand 55e corresponding to the palm of user P's left hand. The sample component image 55f is an image of the same pattern as the button component that user P's right index finger was touching.

[0033] In the above explanation, it was described that after acquiring the sample component image, the user P's left hand is detected, and the sample component image is displayed so as to be placed on the palm of the left hand. However, this is not limited to this. For example, after acquiring the sample component image, the sample component image may be displayed so as to be placed on a desk, and then, upon detecting the user P's left hand, the position of the sample component image, which was displayed so as to be placed on the desk, may be moved so as to be placed on the left hand. Alternatively, for example, the sample component image may be displayed so as to be placed on a desk, the user P's hand may be detected, and the sample component image may be moved by a pinching motion.

[0034] In S15 of Figure 2, the control unit 10 (object output unit 19) displays the object to which the texture is to be applied. Here, the control unit 10 may display the object to which the texture is to be applied in accordance with the display of the sample member image. Alternatively, the control unit 10 may display another GUI (not shown) for displaying objects as if it were placed on the surface of the desk, and the object may be displayed by operating the GUI. Here, as an example of displaying the other GUI, the control unit 10 may display the GUI having the above-mentioned button member when the user P's index finger touches the surface of the desk, and display another GUI when the user P's middle finger touches the surface of the desk. In addition, the above-mentioned GUI may have a button member for displaying objects, and the object may be displayed when the user P presses the button member.

[0035] Image 56 in Figure 7(B) includes virtual hands 56b and 56e corresponding to the detected hand of user P. Image 56 also displays a sample component image 56f as if it were placed on top of the virtual hand 56e, which corresponds to user P's left hand. Furthermore, Image 56 shows an object 56h in its display configuration. In this example, object 56h is a part of the car's interior, specifically a section of the front door.

[0036] In S16 of Figure 2, the control unit 10 (contact detection unit 14) determines whether the detected finger of user P has touched the object output by the object output unit 19. The control unit 10 determines whether the detected finger of user P has touched the object based on the relationship between the position of user P's finger and the position of the object. If the detected finger of user P has touched the object (S16: YES), the control unit 10 moves the process to S17. On the other hand, if the detected finger of user P has not touched the object (S16: NO), the control unit 10 moves the process to S13. In S17, the control unit 10 performs the sample member image application process and then terminates the process.

[0037] Next, we will explain the process of applying the sample component image. Figure 8 is a flowchart showing the process of applying a sample member image to the mounting device 1 according to this embodiment. Figure 9 shows an example of the display of the mounting device 1 according to this embodiment. Figure 10 is a diagram illustrating the sample member image application process according to this embodiment. In S41 of Figure 8, the control unit 10 identifies the part of the object from the position where user P's finger touched it. Image 57 in Figure 9(A) includes virtual hands 57b and 57e corresponding to the detected hand of user P. Image 57 also shows a configuration in which the sample member image 57f is displayed as if it were placed on the virtual hand 57e corresponding to user P's left hand. Furthermore, Image 57 shows an object 57h being displayed, with the virtual hand 57b corresponding to user P's right hand touching the object 57h.

[0038] In step S42 of Figure 8, the control unit 10 acquires texture data of the sample material image. In S43, the control unit 10 (image linkage processing unit 18) acquires the inclination of the sample member. Here, we will explain how to obtain the tilt of the sample component image using Figure 10. Figure 10(A) shows image 70 as viewed by user P, with the sample component image 71 placed on user P's palm. Vector 72 indicates the orientation of the sample component image 71. The angle of this vector 72, which represents the orientation of the sample component image 71, is calculated using a coordinate system.

[0039] The coordinate system 74 shown in Figure 10(B) represents the vectors of the sample member image when the user P's head is the origin 75 and the direction the user P is facing is the positive Y-axis. In coordinate system 74, vector 72 shown in image 70 of Figure 10(A) corresponds to vector 76. Figure 10(C) shows a manifestation 78 in which vector 76 of coordinate system 74 in Figure 10(B) is projected onto the XZ plane. In manifestation 78, vector 76 shown in coordinate system 74 corresponds to vector 79. The angle θ between the vector perpendicular to the user P's line of sight (positive Z-axis direction) and vector 79 can be used as the inclination of the sample member image 71. Note that if vector 79 is perfectly horizontal to the XY plane, the angle θ becomes 0 according to the above explanation. In that case, for example, if the projection plane is the XY plane, you can use the angle between the normal vector of the XZ plane (i.e., the vector in the positive Y direction) and the vector of the sample member image when projected onto the XY plane.

[0040] In S44 of Figure 8, the control unit 10 (image linkage processing unit 18) tilts the texture data to the same tilt as the acquired tilt. In S45, the control unit 10 (texture image application unit 20) applies texture data to the parts of the identified object. Image 58 in Figure 9(B) includes a virtual hand 58e corresponding to the detected hand of user P. Image 58 also shows a sample component image 58f placed as if it were placed on the virtual hand 58e corresponding to user P's left hand. Furthermore, Image 58 displays an object 58h, showing that the part touched by user P's finger is attached with the same pattern orientation as the sample component image 58f.

[0041] Furthermore, Image 59 in Figure 9(C) shows an example of application when the orientation of the sample component image is changed. Image 59 shows a virtual hand 59e corresponding to the detected hand of user P. Image 59 also shows a state where the sample component image 59f is displayed as if it were placed on the virtual hand 59e corresponding to user P's left hand. Furthermore, the sample part image 59f has a different inclination than the sample part image 58f in Figure 9(B). When user P's finger touches the part in the state of sample part image 59f, the part of object 59h is attached with the same pattern orientation as sample part image 59f, as shown in Figure 59.

[0042] In this way, when the orientation of the sample part image is changed and user P touches the object with their finger, the texture of the sample part image at the time of touch is directly reflected in the object's parts. Therefore, the sample part image can be attached to the object's parts at various angles, and user P can check the differences depending on the angle at which it is attached. This technology allows user P to apply images with various textures to parts simply by using the mounting device 1, enabling them to see the parts as if they were the real thing, which is beneficial. Furthermore, it reduces the time and cost involved in manufacturing and transporting actual parts with various textures.

[0043] Thus, the mounting device 1 of this embodiment has the following advantages. (1) The camera unit 36, which is attached to the head of user P, captures an image and detects the shape of user P's hand, including its fingers. A plane is then detected, and a GUI with button members stored in the storage unit 30 is placed on the detected plane near the detected location of user P's hand, and the result is output to the display unit 37 of the attachment device 1. Therefore, an image of the GUI can be output as if the GUI were placed on the detected plane. As a result, the GUI can be shown to user P as if it were placed on the plane.

[0044] (2) When it is detected that the detected user P's finger has come into contact with the detected plane, the GUI is placed and output at the position of the plane that user P's finger touched. Therefore, the GUI can be placed when user P touches the plane with their finger. As a result, the GUI can be placed at the position on the plane specified by user P.

[0045] (3) When it is detected that the detected user P's finger has come into contact with a GUI button component, the system executes the process associated with the button component at the position where user P's finger touched. Therefore, the GUI, arranged on a flat surface, allows user P to feel the sensation of pressing a button component, corresponding to the user P's action of pressing the button component. As a result, user P can feel that they have pressed the button component.

[0046] (4) The image obtained from the result of the processing is output to the display unit 37. Therefore, the user P can be informed that a process has been executed by pressing the button component. (5) The system was modified to detect the desk as a flat surface. Therefore, the GUI can be presented to user P as if it were sitting on a desk.

[0047] (6) The camera unit 36, which is attached to the head of user P, captures an image and detects the shape of the user's hand, including their fingers. A sample material image with a texture is output to the display unit 37, which is attached to the head of user P. When the detected fingers of user P come into contact with an object visible through the display unit 37, the output sample material image is attached to the contact surface of the object where contact was detected. In addition, the sample material image is attached to the contact surface of the object so that the orientation of the pattern represented as the texture of the sample material image is the same. Therefore, a sample image of a material with a texture can be reflected onto an object touched by user P's hand. Moreover, it can be reflected onto the object in the same orientation as the sample image.

[0048] (7) The sample component image is placed on the detected user P's hand and output, and the sample component image is tilted in conjunction with the movement of user P's hand. Therefore, when user P places a sample component image on their hand and moves their hand, the sample component image can be tilted in conjunction with the direction of the user P's hand movement.

[0049] (8) When contact is detected with the sample component image placed in the user P's hand, the sample component image is attached to the contact surface of the object. Therefore, when user P touches an object with a sample component image in their hand, the sample component image can be reflected onto the object with the same tilt as the sample component image at that moment. Thus, user P can check the appearance of the object with the tilt of the sample component image changed.

[0050] (9) The display unit 37 outputs an object, which is an image of the object, and it is set up to detect when the detected user's finger comes into contact with the output object. Therefore, sample material images can be applied to objects that are images of various objects. As a result, simulations can be performed in which the texture of the sample material image is applied to objects of various objects.

[0051] (10) The object is composed of multiple partial images, and the sample member image is attached to the contact surface of the object partial image that comes into contact. Therefore, the texture of the sample material image can be applied to the parts that make up the object.

[0052] Although embodiments of the present invention have been described above, the present invention is not limited to the embodiments described above. Furthermore, the effects described in the embodiments are merely a list of the most preferred effects arising from the present invention, and the effects of the present invention are not limited to those described in the embodiments. The embodiments described above and the modified forms described later can be used in combination as appropriate, but a detailed explanation is omitted.

[0053] (Transformed form) (1) In this embodiment, an example has been described using a mounting device 1 that applies a video see-through type display, but it is not limited to this. For example, a mounting device that applies an optical see-through type display may also be used. With an optical see-through type display, the user can view the surrounding scenery through the lens and see the image of the electronic display superimposed using an optical system such as a prism or a half mirror.

[0054] (2) In this embodiment, the implementation using the mounting device 1 has been described as an example, but it is not limited thereto. For example, it may be implemented using a mounting device and a computer such as an information processing device that is connected to the mounting device in a communicative manner. The information processing device may be, for example, a personal computer (PC), a mobile terminal, a server, etc. In that case, the mounting device is equipped with a communication network unit for communicating with the information processing device. The processing related to shooting and display is performed by the mounting device, and other processing may be performed by the mounting device or by the information processing device. In addition, the GUI storage unit, the sample member image storage unit and the object storage unit may be provided by the information processing device or by the mounting device.

[0055] (3) In this embodiment, an example was described in which a sample image of a component is output as a result of processing performed by pressing a button component of the GUI, but the embodiment is not limited to this. The processing performed by pressing a button component of the GUI may be any kind of processing.

[0056] (4) In this embodiment, the sample member image was described using a semi-circular shape as an example, but it is not limited to this. For example, it may be in the shape of paper or a board, and there are no limitations on the shape or size. (5) In this embodiment, the example described is that the sample component image is placed in the palm of the user P's hand, but the embodiment is not limited to this. The sample component image may be held in the hand of the user P. (6) In this embodiment, the object was described using a part of the interior of a car as an example, and the example described was applying a sample material image with texture to the part of the interior, but it is not limited to this. The object on which the texture simulation is performed may be anything.

[0057] (7) In this embodiment, an example of applying texture to an object that has been imaged has been described, but the invention is not limited to this. Texture may also be applied to an object in real space. In that case, the control unit of the mounting device (object detection means) detects an object that is in real space (for example, the desk in this embodiment) from the captured image, and the control unit of the mounting device (contact detection means) detects that the detected user's finger has come into contact with the detected object. [Explanation of Symbols]

[0058] 1. Mounting device 10 Control Unit 11 Image acquisition unit 12 Hand shape detection unit 13 Planar detection unit 14 Contact detection unit 15 GUI output section 16 Operation Execution Processing Unit 17 Sample Image Output Unit 18 Image Linkage Processing Unit 19. Object Output Section 20 Texture Image Application Section 30 Storage section 31a Processing program 32 GUI storage 33 Sample component image storage unit 34 Object Storage Unit 36 Photography Department 37 Display section Images 50-59 P User

Claims

1. Computers, A first detection means detects the shape of the user's hand, including their fingers, from an image captured by a camera unit attached to a device worn on the user's head, A sample image output means for outputting a sample member image with a texture to a display unit provided in the mounting device, A contact detection means that detects when the user's finger, detected by the first detection means, comes into contact with an object visible through the display unit, A texture application means that applies the sample member image output by the sample image output means to the contact surface of the object in which the contact detection means has detected contact, and make it work A program that causes the texture application means to function by pasting the sample member image onto the contact surface of the object such that the orientation of the pattern represented as the texture of the sample member image is the same.

2. In the program described in claim 1, The sample image output means is configured to output the sample member image placed on the user's hand detected by the first detection means. A program that causes the computer to function as a sample image linking means that tilts the sample member image in conjunction with the user's hand movements.

3. In the program described in claim 2, A program that causes the texture application means to function such that, when the sample member image is placed in the user's hand by the sample image output means, the contact detection means detects contact and applies the sample member image to the contact surface of the object.

4. In the program described in claim 1, The computer is configured to function as an object output means for outputting the object, which is an image of an object, to the display unit. A program that causes the contact detection means to function to detect when the user's finger, detected by the first detection means, has come into contact with the object output by the object output means.

5. In the program described in claim 4, The aforementioned object is composed of multiple partial images, A program that causes the texture application means to function by pasting the sample member image onto the contact surface of the partial image of the object that has come into contact with the contact detection means.

6. In the program described in claim 1, The computer is made to function as an object detection means for detecting the object from the captured image. A program that causes the contact detection means to function to detect when the user's finger, detected by the first detection means, has come into contact with the object, detected by the object detection means.

7. The photography department, Display unit and A storage unit that stores an object, which is an image of an object, A mounting device equipped with, A first detection means for detecting the shape of the user's hand, including their fingers, from an image captured by the imaging unit while the device is attached to the user's head, A sample image output means for outputting a sample material image with a texture to the display unit, Object output means for outputting the object from the storage unit to the display unit, A contact detection means that detects when the user's finger detected by the first detection means comes into contact with the object output by the object output means, A texture application means that applies the sample member image output by the sample image output means to the contact surface of the object in which the contact detection means has detected contact, Equipped with, The texture application means is a mounting device that attaches the sample member image to the contact surface of the object such that the orientation of the pattern represented as the texture in the sample member image is the same.

8. A mounting device equipped with an imaging unit and a display unit, An information processing device that is communicatively connected to the aforementioned mounting device, An information processing system equipped with, The aforementioned information processing device is A storage unit that stores an object, which is an image of an object, A first detection means for detecting the shape of the user's hand, including their fingers, from an image captured by the imaging unit while the user is wearing the attachment device on their head, A sample image output means for outputting a sample material image with a texture to the display unit, Object output means for outputting the object from the storage unit to the display unit, A contact detection means that detects when the user's finger detected by the first detection means comes into contact with the object output by the object output means, A texture application means that applies the sample member image output by the sample image output means to the contact surface of the object in which the contact detection means has detected contact, Equipped with, The texture application means is an information processing system that applies the sample member image to the contact surface of the object such that the orientation of the pattern represented as the texture in the sample member image is the same.

Images