Information processing system, information processing method, and program

Abstract

This information processing system comprises: a sensor unit to be attached to a user's body; and an information acquisition unit that acquires information pertaining to an object on the basis of a detection result from the sensor unit in a state in which the sensor unit or a part of the user's body has come into contact with, or proximity to, the object.

Description

Information processing system, information processing method, and program

[0001] The present technology relates to an information processing system, an information processing method, and a program.

[0002] Conventionally, there have been technologies that detect the position of an object touched by a user and provide the user with information or feedback based on the detection result. For example, a technology has been proposed in which a device is attached to a finger and outputs vibrations or other signals based on information corresponding to the display content of the position on a touch panel touched by the finger (Patent Document 1).

[0003] Japanese Patent Application Laid-Open No. 2020-42324

[0004] However, users' fingers do not only touch touch panels but also various other objects. Furthermore, there are various ways in which fingers can come into contact with an object, such as touching with fingers, pinching with fingers, or grasping with hands. Therefore, there is a demand for technology that can acquire information about an object when it is touched, regardless of the various objects and contact methods.

[0005] The present technology has been developed in consideration of these points, and aims to provide an information processing system, an information processing method, and a program that can acquire information about a part of a user's body or an object that a sensor has come into contact with.

[0006] In order to solve the above-mentioned problems, the first technology is an information processing system that includes a sensor unit that is attached to the user's body, and an information acquisition unit that acquires information about an object based on the detection results of the sensor unit when the sensor unit or a part of the user's body is in contact with or close to the object.

[0007] The second technology is an information processing method that acquires information about an object based on the detection results of a sensor unit attached to a user's body or a part of the user's body when the sensor unit is in contact with or close to the object.

[0008] The third technology is a program that causes a computer to execute an information processing method for acquiring information about an object based on the detection results of a sensor unit attached to a user's body or a part of the user's body when the sensor unit is in contact with or close to the object.

[0009] 1 is an external view showing the configuration of a device as an information processing system 100 in a first embodiment. FIG. 2 is a block diagram showing the configuration of the information processing system 100 in the first embodiment. FIG. 3 is an explanatory diagram of a vibration pattern information storage unit 121. FIG. 4 is a block diagram showing the hardware configuration of the information processing system 100. FIG. 5 is a first example of a mechanism for generating vibrations in an object OBJ. FIG. 6 is a second example of a mechanism for generating vibrations in an object OBJ. FIG. 7 is a third example of a mechanism for generating vibrations in an object OBJ. FIG. 8 is a flowchart showing the processing of the information processing system 100 in the first embodiment. FIG. 9 is an external view showing another example of the configuration of a device as the information processing system 100. FIG. 10 is an external view showing another example of the configuration of a device as the information processing system 100. FIG. 11 is an external view showing another example of the configuration of a device as the information processing system 100. FIG. 12 is a diagram showing another example of a vibration sensor 111. FIG. 13 is an explanatory diagram of an application example of the first embodiment. FIG. 14 is an explanatory diagram of an application example of the vibration pattern information storage unit 121. FIG. 15 is an external view showing an object OBJ and the configuration of a device as an information processing system 200 in a second embodiment. 1 is a block diagram showing the configuration of a 3D model processing device 1000 for manufacturing a mockup having a dot pattern. FIG. 1 is a block diagram showing the configuration of an information processing system 200 according to a second embodiment. FIG. 2 is a flowchart showing the processing of the information processing system 200 according to the second embodiment. FIG. 3 is an explanatory diagram of dot pattern detection by an optical sensor 211. FIG. 4 is an external view showing the configuration of a device as an information processing system 300 according to a third embodiment. FIG. 5 is a block diagram showing the configuration of an information processing system 300 according to the third embodiment. FIG. 6 is an explanatory diagram of an object information holding unit 323. FIG. 7 is a flowchart showing the processing of the information processing system 300 according to the third embodiment. FIG. 8 is an explanatory diagram of bone detection. FIG. 9 is a diagram showing a state in which a user is holding an object OBJ with his fingers. FIG. 10 is a diagram showing a state in which a user is holding an object OBJ with his fingers. FIG. 11 is a block diagram showing the configuration of a second example of the information processing system 300 according to the third embodiment. FIG. 12 is a flowchart showing the processing of the second example of the information processing system 300 according to the third embodiment. FIG. 13 is an explanatory diagram of the position and orientation estimation of an object OBJ.

[0010] Hereinafter, embodiments of the present technology will be described with reference to the drawings. The description will be made in the following order: <First embodiment> [Configuration of information processing system 100] [Configuration of object OBJ] [Processing in information processing system 100] <Second embodiment> [Manufacturing of mockup having dot pattern] [Configuration of information processing system 200] [Processing in information processing system 200] <Third embodiment> [Configuration of information processing system 300] [Processing in information processing system 300] <Modification>

[0011] First Embodiment [Configuration of Information Processing System 100] In the first embodiment, a sensor detects vibrations that occur when a user's fingers, which are body parts, come into contact with an object OBJ. Then, the information processing system 100 acquires the positions and movements of the user's fingers relative to the object OBJ as information about the object OBJ based on the detection results of the sensor.

[0012] In addition, "fingers touching object OBJ" includes not only cases where fingers directly touch object OBJ, but also cases where a device or sensor as information processing system 100 is attached to the fingertip and the device or sensor touches object OBJ, i.e., cases where fingers touch object OBJ via a device or sensor.

[0013] The following describes the configuration of the information processing system 100. In the example of Fig. 1, the information processing system 100 is configured as a device worn on the fingertip of a user.

[0014] Next, referring to FIG. 2, the information processing system 100 is made up of a sensor unit 110, an information processing unit 120, and an output unit .

[0015] The vibration sensor 111 is a sensor for detecting vibrations generated when a user's finger touches an object OBJ. The vibration sensor 111 corresponds to a sensor unit that is worn on the user's finger, that is, that senses while in contact with the user's body. For example, an IMU (Inertial Measurement Unit) can be used as the vibration sensor 111. However, any sensor that can detect vibrations can be used.

[0016] The vibration pattern information storage unit 121 stores a vibration pattern table in which information on a plurality of vibration patterns is associated in advance. In the vibration pattern table, each of the plurality of vibration patterns is associated with information on vibration, the position of the mechanism or configuration that generates the vibration pattern on the object OBJ, the type of action that generates the vibration pattern (tracing, pressing, etc.), etc.

[0017] For example, if the object OBJ has three button-type vibration generating mechanisms that generate different vibration patterns, namely vibration pattern A, vibration pattern B, and vibration pattern C, as shown in Figure 3, each vibration pattern is associated with information about the vibration, the position of the button, the function assigned to the button, the type of processing to be executed by the application processing unit 123, etc.

[0018] The information about vibration includes the period, amplitude, wavelength, number of vibrations per finger touch, length of vibration, magnitude of vibration, length of time vibration occurs, time interval between vibrations, etc. The information about vibration may be any information that can be detected by the vibration sensor 111. Furthermore, the information about vibration may be preset in advance, or may be information that the user can add, delete, or change as desired.

[0019] The information acquisition unit 122 acquires information about the object OBJ based on the vibration detection result by the vibration sensor 111 and the vibration pattern table stored in the vibration pattern information storage unit 121. The information about the object OBJ is at least one of the position and the movement of the user's fingers relative to the object OBJ.

[0020] The application processing unit 123 performs processing related to the application according to the information about the object OBJ acquired by the information acquisition unit 122, i.e., the position and movement of the user's fingers relative to the object OBJ. The application processing unit 123 performs processing such as starting up an application, executing an application, stopping and resuming application processing, and switching the application to be executed.

[0021] The processing executed by the application processing unit 123 includes display processing on a display provided in hardware such as an HMD (Head Mounted Display), an HUD (Head-Up Display), or a smartphone, and audio output processing from a speaker provided in hardware such as an HMD or a smartphone. The type of HMD is not particularly limited. For example, the HMD may be an optical see-through type or a video see-through type, or may be a glasses-type device or AR (Augmented Reality) glasses. Furthermore, the HMD may be of a type other than the types described above.

[0022] The display unit 131 is a liquid crystal display, an organic EL (Electroluminescence) display, or the like that displays videos, images, GUIs (Graphical User Interfaces), messages, and the like based on the processing in the application processing unit 123 .

[0023] The audio output unit 132 is a speaker that outputs audio based on the processing in the application processing unit 123 .

[0024] It should be noted that the information processing system 100 does not necessarily need to include both the display unit 131 and the audio output unit 132 as the output unit 130, and may include only one of them, or neither.

[0025] The information processing system 100 of the first embodiment is configured as described above. The information processing system 100 may represent each of the sensor unit 110, the information processing unit 120, and the output unit 130 individually, or may include a combination of two or more of them. The information processing system 100 may be configured by a single device or apparatus, or may be configured by multiple devices or apparatuses.

[0026] Next, the hardware configuration of the information processing system 100 other than the sensor unit 110 will be described with reference to FIG.

[0027] The CPU (Central Processing Unit) 151 functions as an arithmetic processing unit that performs various processes and controls the entire information processing system 100 and each unit. The CPU 151 executes various processes according to a program stored in a ROM (Read Only Memory) 152 or a program loaded from a storage unit 159 to a RAM (Random Access Memory) 153. The RAM 153 stores data and the like necessary for the CPU 151 to execute various processes as appropriate. The information processing unit 120 can be realized by a processor constituted by the CPU 151, the ROM 152, and the RAM 1533 executing a program. The information processing method of the present technology may be realized by executing the program.

[0028] The CPU 151 , ROM 152 , and RAM 153 are interconnected via a bus 154 , and the bus 154 is connected to a bridge 155 .

[0029] An interface 157 is connected to the bridge 155 via a bus 156 .

[0030] The interface 157 is connected to an input unit 158 ​​, an output unit 130 , a storage unit 159 , a drive 160 , a connection port 161 , and a communication unit 162 .

[0031] The input unit 158 ​​is, for example, various types of operators or operation devices such as a keyboard, a mouse, keys, a dial, a touch panel, a touch pad, a remote controller, etc. The input unit 158 ​​detects a user operation, and the CPU 151 interprets a signal corresponding to the input operation.

[0032] The output unit 130 is a liquid crystal display or organic EL display that displays videos, images, GUI, messages, etc., a speaker that outputs audio, a vibration generating mechanism that outputs feedback to the user, or the like.

[0033] The storage unit 159 is a large-capacity storage medium such as a hard disk, a flash memory, etc. The storage unit 159 stores various applications, data, information, etc.

[0034] A removable storage medium 163 can be connected to the information processing system 100 via a drive 160. The removable storage medium 163 includes a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, and the like.

[0035] The drive 160 can read data files such as programs used in various processes from the removable storage medium 163. The read data files are stored in the storage unit 159. The programs read from the removable storage medium 163 are installed in the storage unit 159 as needed. Furthermore, the information processing system 100 may transfer information and data to and from external devices via the removable storage medium 163.

[0036] An external device 164 can be connected to the information processing system 100 via a connection port 161 .

[0037] The communication unit 162 includes various communication terminals and communication modules that perform communication processing via a network NW such as the Internet. External devices can be connected to the information processing system 100 via the communication unit 162. The communication method may be either wired or wireless. Examples of communication methods include cellular communication, 4G, 5G, Wi-Fi, Bluetooth (registered trademark), NFC (Near Field Communication), Ethernet (registered trademark), HDMI (High-Definition Multimedia Interface), and USB (Universal Serial Bus).

[0038] The sensor unit 110 and other external devices can be connected via the connection port 161 and the communication unit 162 .

[0039] In the information processing system 100, for example, a program for processing of the present technology can be installed via network communication by the communication unit 162 or a removable storage medium 163. The program may also be stored in advance in the ROM 152, the storage unit 159, or the like.

[0040] 2 may be configured to be included in a single device, or the information processing unit 120 may be realized by a processing block of hardware having a dedicated function.

[0041] The sensor unit 110 and the information processing unit 120 may be configured to be included in a device that comes into contact with the user's body, and the output unit 130 may be configured to be included in another device. Alternatively, the device that comes into contact with the user's body may include only the vibration sensor 111, and the information processing unit 120 and the output unit 130 may be configured to be included in another device. Alternatively, the application processing unit 123 and the output unit 130 may be configured to be included in another device. The other device may be, for example, an HMD, HUD, smartphone, personal computer, smart watch, wearable device, headphones, etc.

[0042] In this case, each device may have a communication function to transmit and receive information and signals between different devices. The communication method may be either wired or wireless. Communication methods include cellular communication, 4G, 5G, Wi-Fi, Bluetooth (registered trademark), NFC, Ethernet (registered trademark), HDMI (registered trademark), USB, etc.

[0043] [Configuration of Object OBJ] In the first embodiment, it is preferable to configure the object OBJ so that vibration occurs when the user touches, presses, traces, etc. the object OBJ with a finger. Such a configuration of the object OBJ can be realized by various methods.

[0044] The first method is to provide an object OBJ with a button BT equipped with a mechanism that generates vibrations when pressed, as shown in FIG. 5 . In the example of FIG. 5 , buttons BT1 and BT2 that generate vibrations are attached to the housing of a camera serving as the object OBJ. When a user presses buttons BT1 and BT2, vibrations are generated, and the vibrations are detected by a vibration sensor 111, thereby enabling estimation of the position and movement of the user's fingers relative to the object OBJ. The vibration-generating mechanism may be any mechanism capable of generating vibrations, such as a mechanical mechanism, a mechanism using a microswitch, or a mechanism using a vibrator that vibrates when energized. As shown in FIG. 5 , a single object OBJ may be provided with multiple buttons BT, each of which generates a different vibration pattern. Note that the buttons BT may be configured to be pressed or may be slide buttons.

[0045] The second method is to provide an object OBJ with a dial DL that generates vibrations when rotated, as shown in FIG. 6 . In the example of FIG. 6 , a camera serving as the object OBJ is configured to include a dial DL that generates vibrations. In FIG. 6A , the camera serving as the object OBJ includes a dial DL1 whose rotation axis is in the height direction (up and down). In FIG. 6B , the camera serving as the object OBJ includes a dial DL2 whose rotation axis is in the depth direction (front and back). When a user rotates the dial DL with their fingers, vibrations are generated. By detecting the vibrations with the vibration sensor 111, the position and movement of the fingers relative to the object OBJ can be estimated. The mechanism that generates vibrations may be any mechanism that can generate vibrations, such as a mechanism using gears or a mechanism using a vibrator that vibrates when energized. In a mechanism using a dial DL, the amount of rotation of the dial DL can be estimated by detecting the number of vibrations of the dial with the vibration sensor 111.

[0046] The third method is to provide an uneven surface BM having continuous unevenness on the surface of the object OBJ, as shown in FIG. 7A . When a user traces this uneven surface BM with their finger, vibrations are generated, and by detecting these vibrations with the vibration sensor 111, the position and movement of the finger relative to the object OBJ can be estimated. When multiple different vibrations need to be detected, it is preferable to provide the object OBJ with multiple uneven surfaces BM having different uneven shapes, such as uneven surfaces BM1, BM2, BM3, and BM4 shown in FIGS. 7A to 7D . In the example of FIG. 7E , a bicycle grip, which serves as the object OBJ, is provided with a first region AR1 and a second region AR2, with uneven surface BM2 provided in the first region AR1 and uneven surface BM3 provided in the second region AR2.

[0047] A fourth method is to arrange a convexo-concave surface BM having continuous concaves and convexes in a straight line or a ring, as shown in FIGS. 8A and 8B . When a user traces this convexo-concave surface BM with their finger, vibrations are generated. By detecting these vibrations with the vibration sensor 111, the position and movement of the finger relative to the object OBJ can be detected. When multiple different vibrations need to be detected, it is preferable to provide multiple convexo-concave surfaces BM1 and BM2 with different convexo-convex shapes and sequences on the object OBJ, as shown in FIGS. 8A and 8B . In the example of FIG. 8C , a convexo-concave surface BM1 and a convexo-concave surface BM2 are provided on the operation piece of a controller for editing video or music. The arrangement of the convexo-concave surfaces is not limited to a straight line or a ring, and any shape, such as an L-shape, a semicircle, an S-shape, or a free shape, may be used.

[0048] In the third and fourth methods, the vibration-generating irregularities may be formed by linear protrusions as shown in Fig. 7, or by dot-like protrusions as shown in Fig. 8. Furthermore, the irregularities may be formed by combining dot-like protrusions and linear protrusions.

[0049] Furthermore, a figure may be formed by protrusions that form an uneven surface. This allows the uneven pattern to be part of the design of the object OBJ as a product, thereby achieving both design quality and convenience using this technology.

[0050] If a mechanism that does not require a power supply or external communication to generate vibrations is adopted, this technology can be introduced easily and at low cost.

[0051] It should be noted that the specific objects OBJ shown in FIGS. 5 to 8 are merely examples, and any object OBJ may be used as long as it can be provided with a mechanism for generating the vibrations described above.

[0052] [Processing in Information Processing System 100] Next, processing in the information processing system 100 will be described with reference to FIG.

[0053] In step S11, the vibration sensor 111 detects vibrations generated when the user's fingers come into contact with the object OBJ.

[0054] Next, in step S12, the information acquisition unit 122 estimates the position and movement of the user's fingers relative to the object OBJ based on the vibration detection result by the vibration sensor 111 and the vibration pattern table stored in the vibration pattern information storage unit 121. If the vibration detection result matches the vibration pattern in the vibration pattern table, it can be estimated that the position and movement associated with the vibration pattern in the vibration pattern table are the position and movement of the user's fingers relative to the object OBJ. This makes it possible to obtain the position and movement of the user's fingers relative to the object OBJ as information about the object OBJ. If the vibration detection result matches the vibration pattern, the process proceeds to step S13 (Yes in step S12).

[0055] 5, when the object OBJ has a plurality of buttons BT that generate vibrations, the vibration patterns generated by the buttons BT are associated in advance with the positions and types of the buttons BT in a vibration pattern table. Then, by determining whether the vibration detection results match the vibration patterns in the vibration pattern table, it is possible to estimate which button BT the user has pressed with their fingers. This makes it possible to estimate the positions and movements of the user's fingers relative to the object OBJ.

[0056] As shown in Fig. 6, when the object OBJ has a rotating dial DL, the vibration pattern generated by the dial DL is associated in advance with the position and type of the dial DL, the amount of rotation of the dial DL, etc. in a vibration pattern table. Then, by determining whether the vibration detection result matches the vibration pattern in the vibration pattern table, it is possible to estimate whether the user has touched the dial DL with their finger. It is also possible to estimate the amount of rotation of the dial DL. This makes it possible to estimate the position and movement of the user's finger relative to the object OBJ.

[0057] 7, when the object OBJ has an uneven surface BM that generates vibrations when traced, the vibration pattern generated by the uneven surface BM is associated in advance with the position and range of the uneven surface BM on the object OBJ in a vibration pattern table. Then, by determining whether the vibration detection result matches the vibration pattern in the vibration pattern table, it is possible to estimate where on the object OBJ the user has touched their finger to perform the tracing motion.

[0058] As shown in Fig. 8, when a continuous uneven surface BM is provided on the surface of an object OBJ in a straight line or a ring shape, the vibration patterns generated by the uneven surface BM are associated in advance with the amount of finger movement on the uneven surface BM in a vibration pattern table. Since the vibration pattern (vibration length) varies depending on the amount of finger movement on the uneven surface BM even for a single uneven surface BM, it is advisable to associate multiple patterns of finger movement amounts with vibration patterns in the vibration pattern table. Then, by determining whether the vibration detection result matches the vibration pattern in the vibration pattern table, the movement of the user's fingers on the object OBJ, i.e., the amount of movement of the user's fingers on the uneven surface BM, can be estimated.

[0059] Next, in step S13, the application processing unit 123 performs application processing in accordance with the position and movement of the user's fingers relative to the object OBJ. The processing of the application processing unit 123 results in display on the display unit 131 and audio output from the audio output unit 132.

[0060] When the object OBJ is provided with a mechanism for generating a plurality of different vibrations, by previously associating the types of applications and the types of application operations with the different vibrations, it is possible to perform application processing according to the position and movement of the user's fingers relative to the object OBJ. For example, in the example of Fig. 7E, when the user's fingers touch the first area AR1, application A is started, and when the user's fingers touch the second area AR2, application B is started.

[0061] Information on the position and movement of the user's fingers relative to the object OBJ may be transmitted to an external device.

[0062] Then, in step S14, it is confirmed whether the processing is to be terminated. The processing is to be terminated when the power supply of the device of the information processing system 100 is turned off, for example. Unless the processing is to be terminated, steps S11 to S15 are repeated (No in step S14).

[0063] The processing in the first embodiment is performed as described above. According to the first embodiment, the position and movement of a user's fingers relative to an object OBJ can be estimated easily and with high accuracy. A low-cost, space-saving, and highly customizable user operation detection system can be realized by using a mechanism for generating vibrations when a user brings their fingers into contact with an object OBJ and a mechanism for detecting the vibrations.

[0064] In technologies that combine virtual and reality, such as AR (Augmented Reality) and MR (Mixed Reality), there is a demand for detecting user operations on objects OBJ around the user in order to realize a variety of applications. In the first embodiment, a user operation detection system can be realized at low cost without incorporating sensors, communication functions, batteries, etc. into the objects OBJ.

[0065] The device serving as the information processing system 100 may be worn not only at the tip of the index finger but also at the base of the finger as shown in FIG. 10A , or on another finger such as the thumb as shown in FIG. 10B . Also, multiple devices may be worn on one finger as shown in FIG. 10C , or multiple fingers as shown in FIG. 10D . Furthermore, devices may be worn on all five fingers as shown in FIG. 10E . Furthermore, devices may be worn on the palm as shown in FIG. 10F . Thus, the manner in which the device serving as the information processing system 100 is worn is not particularly limited and may be arbitrary.

[0066] When a user wears multiple devices as the information processing system 100 on his or her fingers, each device may perform the process shown in Fig. 9, or an intermediate value or average value of the vibration detection results detected by each vibration sensor 111 constituting each device may be calculated and used to perform the process shown in Fig. 9. Alternatively, any one of the vibration detection results detected by each vibration sensor 111 constituting each device may be selected and the process shown in Fig. 9 may be performed.

[0067] The device serving as information processing system 100 is not limited to being worn on the tip of a finger, and may be configured as a ring-type device as shown in Fig. 11A. Also, it may be configured as a watch-type device or a bracelet-type device worn on the wrist as shown in Fig. 11B.

[0068] 12A, the device serving as the information processing system 100 may be configured as a pen-shaped (rod-shaped) device held by the user, or as a cane-shaped device as shown in Fig. 12B. In this case, the user touches the tip of the pen-shaped or cane-shaped device to the object OBJ, or presses or traces the object OBJ with the tip.

[0069] By configuring a white cane with a cane-shaped device, it becomes possible to trace the tactile paving blocks with the tip of the cane and have information corresponding to the type of tactile paving block presented to the user through processing by the application processing unit 123. Furthermore, by providing blocks with unevenness different from existing tactile paving blocks on the road, it becomes possible to present further information, such as the presence or absence of traffic lights and route guidance.

[0070] As shown in Fig. 13A, the vibration sensor 111 may be configured as a device attached to the object OBJ. Alternatively, as shown in Fig. 13B, the vibration sensor 111 may be configured as a device built into the object OBJ. Even if the vibration sensor 111 is configured in this manner, it can detect vibrations generated by a vibration generating mechanism such as the button BT1 or the button BT2. In these cases, the vibration sensor 111 is connected via a network to a device such as a smartphone that functions as the information processing unit 120 and transmits the detection results.

[0071] The information processing system 100 of the first embodiment can be applied as follows.

[0072] By associating the type of object OBJ with a pattern of a concave-convex surface in advance and providing a concave-convex surface on the surface of the object OBJ, the type of object OBJ can be estimated simply by tracing the concave-convex surface with a finger. This makes it possible to present the type of object OBJ to people who are blind, have poor eyesight, or are in a situation where they cannot see what is in front of them (darkness, eyes closed, etc.).

[0073] For example, suppose there are objects OBJ, such as a container for holding shampoo as shown in Fig. 14A and a container for holding conditioner as shown in Fig. 14B. Then, a textured surface BMS corresponding to the shampoo is provided in advance on the surface of the shampoo container, and a textured surface BML corresponding to the conditioner is provided on the surface of the conditioner container.

[0074] Furthermore, in the vibration pattern table stored in the vibration pattern information storage unit 121, information that the object OBJ is a shampoo container is associated with a vibration pattern generated by tracing the uneven surface BMS with a finger. Also, in the vibration pattern table, information that the object OBJ is a rinse container is associated with a vibration pattern generated by tracing the uneven surface BML with a finger.

[0075] When the user traces the uneven surface BMS of the shampoo container or the uneven surface BML of the conditioner container, the information processing system 100 determines whether the vibration detection result matches the vibration pattern in the vibration pattern table, and can estimate whether the user has traced the shampoo container or the conditioner container. This makes it possible to present to the user information as to whether the user has traced the shampoo container or the conditioner container. Note that the specific examples of shampoo and conditioner are merely examples, and any contents may be placed in the container. The object OBJ is not limited to a container for placing contents inside, and may be any object that can have an uneven surface on its surface.

[0076] The user may be made to touch or operate the object OBJ multiple times, and the accuracy of estimating the position and movement of the fingers relative to the object OBJ may be improved using the vibration detection results from the multiple times. To allow the user to touch or operate the object OBJ multiple times, instructions or feedback to the user may be presented to the user by display on the display unit 131 or audio output from the audio output unit 132 through processing by the application processing unit 123. Furthermore, the display content or audio output may be changed depending on the number of touches or operations, and when estimation of the contact position or movement is completed, a display or audio indicating completion may be output.

[0077] The accuracy of the determination of whether the vibration detection result by the information acquisition unit 122 matches the vibration pattern may be improved by using an algorithm such as dynamic time warping.

[0078] When the action of the user's fingers on the object OBJ is a tracing action, it is conceivable that the tracing speed will differ depending on the user. Therefore, the tracing speed of the fingers may be acquired by the IMU, and the vibration detection algorithm may be switched depending on the speed so that vibrations can be detected without being affected by the tracing speed.

[0079] When the user's finger motion on the object OBJ is a tracing motion, the magnitude of the tracing force may differ depending on the user. The vibration generated by the tracing force also changes. Therefore, a gain adjustment of the amplitude of the vibration detected for each user may be performed.

[0080] When a user is wearing an HMD equipped with a camera and a VST (Video See-Through) function, when an object OBJ captured by the camera is displayed on the HMD, the portion of the object OBJ that the user should touch with their fingers to use the object OBJ may be highlighted. Examples of highlighting methods include displaying an icon, mark, or message indicating the portion of the object OBJ, flashing the portion of the object OBJ, coloring the portion of the object OBJ, or displaying a line surrounding the portion of the object OBJ. Any one of these methods may be used, or a combination of two or more may be used. Furthermore, any display method that can highlight the portion of the object OBJ may be used. This allows the user to easily understand where on the object OBJ to touch in order to operate the object OBJ, even while wearing the HMD. VST is a function that allows the user to see the outside world even while wearing the HMD by displaying an image of the outside world captured by the HMD's camera on the HMD's display.

[0081] In addition, if the user is wearing an HMD with a VST function, virtual fingers may be displayed on the HMD to guide the user on how to touch the object OBJ with the fingers, the contact position, how to press, the pressing position, tracing speed, how to trace, the tracing position, etc.

[0082] Furthermore, when an object OBJ is displayed on an HMD equipped with a VST function, the contact position of the user's fingers on the object OBJ may be highlighted, allowing the user to easily grasp the part of the object OBJ that they are touching even while wearing the HMD.

[0083] The method of determining whether the vibration detection result and the vibration pattern are consistent with each other may be switched based on information about the user or information about the object OBJ by the information acquiring unit 122. For example, when the user or the object OBJ is in a specific state, the method may be configured to determine whether the vibration detection result and the vibration pattern are consistent with only a specific vibration pattern among a plurality of vibration patterns in the vibration pattern table stored in the vibration pattern information storing unit 121.

[0084] 15, the vibration pattern information storage unit 121 may store a plurality of vibration pattern tables in which different application processes are associated with each vibration pattern, and may switch the vibration pattern table to be used according to predetermined information. The predetermined information is information about the user and information about the object OBJ, such as the age of the user, the sex of the user, the user's level of proficiency in using the object OBJ (professional or amateur), the user's posture, the user's position, the type of the object OBJ, the area of ​​the object OBJ, etc.

[0085] This allows the application processing corresponding to the vibration detection result to be different depending on the user or object OBJ, even with the same vibration pattern, making it possible to execute more application processing than the number of vibration patterns according to the vibration detection result.

[0086] For example, when the user is sitting, vibration pattern table A shown in Fig. 15A is used, and if the vibration detection result matches vibration pattern A, processing A is executed. On the other hand, when the user is standing, vibration pattern table B shown in Fig. 15B is used, and if the vibration detection result matches vibration pattern A, processing B is executed. The user's posture can be detected by an IMU or an inertial sensor.

[0087] Also, for example, if the user is a professional, vibration pattern table A shown in Fig. 15A is used, and processing A is executed when the vibration detection result matches vibration pattern A. On the other hand, if the user is an amateur, vibration pattern table B shown in Fig. 15B is used, and processing B is executed when the vibration detection result matches vibration pattern A. In order to switch the vibration pattern table to be used depending on the user's attribute, whether professional or amateur, the attribute of each user input to the information processing system 100 by the user may be used.

[0088] In the above-described embodiment, the vibration sensor 111 was used to detect vibrations that occur when the user's fingers come into contact with the object OBJ, but a sensor other than the vibration sensor 111 may also be used to detect information other than vibrations.

[0089] A microphone may be used as a sensor, and the information detected by the sensor may be sound. In this case, the object OBJ may be provided with a mechanism that generates sound in response to the position and movement of the user's fingers relative to the object OBJ. However, if the object OBJ is an object that generates sound when touched, such a mechanism is not necessary. Furthermore, a table that associates information with sound patterns may be prepared in advance. Then, by detecting the sound generated in response to the position and movement of the user's fingers relative to the object OBJ with the microphone and identifying the sound pattern by referring to the table, the position and movement of the user's fingers relative to the object OBJ can be estimated as in the embodiment.

[0090] Alternatively, a temperature sensor may be used, and the information detected by the sensor may be temperature. In this case, the object OBJ may be provided with a mechanism that generates heat in response to the position and movement of the user's fingers relative to the object OBJ. However, if the object OBJ is inherently heat-generating and has different temperatures depending on the part of the object OBJ, such a mechanism is not necessary. Furthermore, a table that associates information related to temperature may be prepared in advance. Then, by detecting the temperature of the object OBJ with the temperature sensor and identifying the temperature by referring to the table, the position and movement of the user's fingers relative to the object OBJ can be estimated, as in the embodiment.

[0091] Alternatively, a magnetic sensor may be used, and the information detected by the sensor may be magnetic. In this case, the object OBJ may be provided with a mechanism such as RFID (Radio Frequency Identification) that generates magnetic fields in response to the position and movement of the user's fingers relative to the object OBJ. Furthermore, a table that associates information with magnetic patterns may be prepared in advance. Then, the magnetic field generated in response to the position and movement of the user's fingers relative to the object OBJ is detected by the magnetic sensor, and the magnetic pattern is identified by referring to the table, thereby estimating the position and movement of the user's fingers relative to the object OBJ, as in the embodiment.

[0092] Alternatively, a pressure sensor may be used, and the information detected by the sensor may be pressure. The pressure sensor can detect the pressure generated when the user's fingers come into contact with the object OBJ. In this case, a table may be prepared in advance that associates information with pressure patterns. Then, the pressure generated in accordance with the position and movement of the user's fingers relative to the object OBJ is detected by the pressure sensor, and the pressure pattern is identified by referring to the table, thereby making it possible to estimate the position and movement of the user's fingers relative to the object OBJ, as in the embodiment.

[0093] Alternatively, a camera may be used as a sensor, and the information detected by the sensor may be a two-dimensional code such as a QR (Quick Response) Code (registered trademark). In this case, the two-dimensional code may be attached to the object OBJ in advance. Furthermore, a table may be prepared in which information relating to the pattern of the two-dimensional code is associated. Then, when a user's fingers contact or approach the object OBJ, the camera detects the two-dimensional code, and by referring to the table and identifying the pattern of the two-dimensional code, the position and movement of the user's fingers relative to the object OBJ can be estimated, as in the embodiment. In addition to the two-dimensional code, color or character strings can also be used as information detected by the camera.

[0094] These sensors other than the vibration sensor 111 may be used alone or in combination with the vibration sensor 111 .

[0095] The vibration sensor 111 can reduce power consumption by entering sleep mode when there is no object OBJ near the user's fingers. The performance of the sensor can also be varied depending on the finger on which it is worn. A high-resolution sensor can be used for the index finger, which is used frequently, and a low-resolution sensor can be used for the middle finger, which is used relatively infrequently. Priorities can also be assigned to each finger. For example, a high priority can be assigned to the thumb and index finger, which are used frequently, and a low priority can be assigned to the middle finger, ring finger, and little finger, which are used relatively infrequently. Parameters such as the resolution of the sensor attached to each finger can be changed based on the priority of each finger. For example, a sensor attached to a finger assigned a high priority can be set to output high-resolution data, while a sensor attached to a finger assigned a low priority can be set to output low-resolution data to reduce power consumption. The priority of each finger can be preset or can be set arbitrarily by the user.

[0096] Second Embodiment In the second embodiment, when a user's fingers, which are body parts, come into contact with the object OBJ, a sensor detects a dot pattern provided on the surface of the object OBJ. Then, the information processing system 200 acquires the position and movement of the user's fingers relative to the object OBJ as information about the object OBJ based on the detection result of the sensor. In the second embodiment, the object OBJ is an assembly-type mockup (hereinafter referred to as a mockup), as shown in Fig. 16 . A dot pattern consisting of a plurality of dots is formed on the surface of the mockup.

[0097] Note that "fingers touching object OBJ" includes not only cases where fingers directly touch object OBJ, but also cases where a device or sensor as information processing system 200 is attached to the fingertip and the device or sensor touches object OBJ, i.e., cases where fingers touch object OBJ via a device or sensor.

[0098] [Manufacturing a mockup with a dot pattern] First, a 3D model processing device 1000 for manufacturing a mockup as an object OBJ having a dot pattern formed on its surface will be described with reference to Fig. 17. A manufacturing device 2000 for manufacturing the mockup is a 3D printer or a laser processing machine.

[0099] 3D printers are slower to create than laser processing machines, but offer higher modeling accuracy. Dot patterns can be printed on the surface of a mockup using a multi-material FDM (Fused Deposition Modeling) 3D printer or a powder sintering 3D printer. If a high-precision mockup is required, it is recommended to use a 3D printer.

[0100] Laser processing machines have the advantage of being faster than 3D printers, but with lower modeling accuracy. Using the laser marking function, it is possible to engrave a fine dot pattern onto the surface of a mockup. For example, when using a laser processing machine to create 3D models for product development such as cameras, simple mockups can be produced more quickly than with a 3D printer, speeding up the product design cycle.

[0101] The 3D model input unit 1001 inputs 3D model data of a mockup that has been created in advance by another device or application.

[0102] The 3D model development unit 1002 develops the input 3D model into a 2D texture (two-dimensional plane) using a known algorithm or AI. At this time, the position information of the 3D model and the position information of the 2D texture are associated with each other.

[0103] The correspondence data storage unit 1003 stores data that associates the position information of the 3D model with the position information of the 2D texture.

[0104] The dot pattern rendering unit 1004 renders a dot pattern on the 2D texture to generate dot pattern data for forming a dot pattern on the surface of the mockup by the manufacturing apparatus 2000. The 3D model processing apparatus 1000 outputs the dot pattern data to the manufacturing apparatus 2000.

[0105] The manufacturing apparatus 2000, to which the 3D model and dot pattern data have been input, then manufactures the mockup. A dot pattern can be formed on the surface of the mockup based on the dot pattern data using the printing function of a 3D printer or the engraving function of a laser processing machine, which serves as the manufacturing apparatus 2000. This allows the three-dimensional position of the 3D model of the mockup to automatically correspond to the position of the dot pattern.

[0106] It should be noted that the dot pattern formed on the surface of the mockup has a larger number of dots and a narrower spacing between the dots, which increases the detection accuracy of the sensor.

[0107] The mockup can also be manufactured by combining a 3D printer and a laser processing machine, for example, by using a 3D printer to create the parts of the mockup that users will particularly touch with their fingers, and a laser processing machine to create the other parts. Also, the dot pattern can be engraved in areas that users frequently touch (for example, by carving deeply) in a way that makes it less likely to fade. Also, the resolution of dot pattern detection can be changed by varying the density of the dot pattern depending on the part of the mockup.

[0108] It is also possible to form a dot pattern only in the portion of the mockup where the user will touch with their fingers, and not form a dot pattern in the portion where the user will not touch with their fingers.

[0109] In addition, if a dot pattern is missing or the dot pattern data does not match the 3D model, the dot pattern data can be updated by matching it with sensing data such as hand tracking data or IMU data, or the user can be notified that there is no longer any consistency.

[0110] The 3D model processing device 1000 may be configured as a personal computer, smartphone, tablet terminal, server, etc., or may be configured as dedicated hardware. In this case, it is preferable that the 3D model processing device 1000 and the manufacturing device 2000 are connected via a wired or wireless network. Furthermore, the manufacturing device 2000 may have the functions of the 3D model processing device 1000.

[0111] [Configuration of Information Processing System 200] Next, the configuration of the information processing system 200 will be described. The information processing system 200 is configured as a device worn on the user's finger as shown in Fig. 16. As shown in Fig. 18, the information processing system 200 is configured with a sensor unit 210, an information processing unit 220, and an output unit 230.

[0112] The optical sensor 211 is a sensor worn on the user's finger, and is used to detect a dot pattern on the surface of the object OBJ. A camera can be used as the optical sensor 211. Alternatively, a sensor that includes a light-emitting unit and a light-receiving unit and detects the shape or state of an object by irradiating the object with light and detecting changes in the reflected light may be used as the optical sensor 211. The optical sensor 211 corresponds to a sensor unit that is worn on the user's finger, i.e., that performs sensing while in contact with the user's body.

[0113] The proximity sensor 212 is a sensor for detecting a change caused by the user's finger approaching the object OBJ. For example, a camera, an optical distance measuring sensor, a capacitance sensor, or the like can be used as the proximity sensor 212. Any sensor can be used as long as it can detect a change caused by the user's finger approaching the object OBJ.

[0114] The proximity determination unit 221 performs processing to determine whether or not the user's fingers have approached the object OBJ based on the detection result of the proximity sensor 212. For example, if the amount of change caused by the user's fingers approaching the object OBJ is equal to or greater than a predetermined threshold, or if the distance between the user's fingers and the object OBJ is equal to or greater than a predetermined threshold, it can be determined that the user's fingers have approached the object OBJ.

[0115] The correspondence information storage unit 222 stores correspondence information between the position information of the 3D model and the position information of the 2D texture generated by the 3D model processing device 1000. For this purpose, it is preferable to supply the correspondence information from the 3D model processing device 1000 to the information processing system 200 in advance.

[0116] The information acquisition unit 223 acquires information about the object OBJ based on the detection result of the dot pattern by the optical sensor 211 and correspondence information between the position information of the 3D model and the position information of the 2D texture. The information about the object OBJ is the three-dimensional contact position and movement of the user's fingers on the object OBJ.

[0117] The feedback control unit 224 controls the operation of the feedback output unit 231 to return feedback to the user according to the position and movement of the user's fingers relative to the object OBJ.

[0118] The feedback output unit 231 is configured with hardware for outputting feedback to the user, and is, for example, a vibration generating mechanism for presenting a tactile sensation by vibration such as Haptics, or a speaker for providing feedback by audio output.

[0119] The information processing system 200 of the second embodiment is configured as described above. Note that the information processing system 200 may also include an application processing unit, a display unit, and an audio output unit similar to those of the first embodiment.

[0120] 18 may be configured to be included in a single device. In this case, the information processing unit 220 is realized by a processor including a CPU, RAM, ROM, etc., executing a program. Alternatively, the information processing unit 220 may be realized by a processing block of hardware having a dedicated function.

[0121] The sensor unit 210 and the information processing unit 220 may be configured to be included in a device that comes into contact with the user's body, and the output unit 230 may be configured to be included in another device. Alternatively, the device that comes into contact with the user's body may include only the sensor unit 210, and the information processing unit 220 and the output unit 230 may be configured to be included in another device. Alternatively, the feedback control unit 224 and the output unit 230 may be configured to be included in another device. The other device may be, for example, an HMD, a smartphone, a personal computer, a smartwatch, a wearable device, headphones, etc.

[0122] In this case, each device may have a communication function to transmit and receive information and signals between different devices. The communication method may be either wired or wireless. Communication methods include cellular communication, 4G, 5G, Wi-Fi, Bluetooth (registered trademark), NFC, Ethernet (registered trademark), HDMI (registered trademark), USB, etc.

[0123] The information processing system 200 may be configured by a single device or apparatus, or may be configured by a plurality of devices or apparatuses.

[0124] The hardware configuration of the information processing system 100 is the same as that of the first embodiment shown in FIG.

[0125] [Processing in Information Processing System 200] Next, processing in the information processing system 200 will be described with reference to FIG.

[0126] In step S21, the proximity sensor 212 detects a change that occurs when the user's finger approaches the object OBJ.

[0127] Next, in step S22, the proximity determination unit 221 determines whether or not the user's fingers have approached the object OBJ based on the detection result of the proximity sensor 212. For example, it can be determined that the user's fingers have approached the object OBJ when the detection result of the proximity sensor 212 is equal to or greater than a predetermined threshold. If the user's fingers have approached the object OBJ, the process proceeds to step S23 (Yes in step S22).

[0128] Next, in step S23, the dot pattern of the object OBJ is detected by an optical sensor 211 attached to the user's finger, as shown in Fig. 20. In Fig. 20, the dot pattern is detected by a camera serving as the optical sensor 211.

[0129] Next, in step S24, the information acquisition unit 223 estimates the position and movement of the user's fingers relative to the object OBJ based on the dot pattern as the detection result and the correspondence information between the position information of the 3D model and the position information of the 2D texture. By detecting the dot pattern, it is possible to estimate where on the object OBJ the user's fingers are touching based on the characteristics of the dots, such as the positional relationship between the multiple dots that make up the dot pattern, the distance between the dots, and the coordinates of the dots. Because the position information of the 2D texture on which the dot pattern is drawn corresponds to the position information of the 3D model, it is possible to estimate the position of the user's fingers relative to the object OBJ by referring to the correspondence information between the position information of the 2D texture and the position information of the 3D model. In addition, it is possible to estimate the movement of the user's fingers relative to the object OBJ from multiple consecutive changes in the contact position.

[0130] Next, in step S25, the feedback control unit 224 outputs feedback by controlling the operation of the feedback output unit 231 in accordance with the position and movement of the user's fingers relative to the object OBJ. Note that information on the position and movement of the user's fingers relative to the object OBJ may be transmitted to an external device.

[0131] When the feedback method is haptic presentation using vibrations such as Haptics, and the feedback output unit 231 is a vibration generating mechanism, the feedback control unit 224 controls the vibration generating mechanism to generate vibrations of a predetermined pattern. This allows feedback of the feel of touching an actual product when the object OBJ is a mockup. Vibrations may be generated in response to a specific position or movement relative to the object OBJ, or the type of vibration may be changed or the length or magnitude of the vibration may be adjusted in response to a specific position or movement relative to the object OBJ. This allows the user to determine where on the object OBJ their fingers are touching based on the type, magnitude, and length of the vibration.

[0132] When the feedback method is audio output and the feedback output unit 231 is a speaker, the feedback control unit 224 controls the speaker to output a predetermined audio. Audio may be output only at a specific contact position, or the type of audio to be output may be changed depending on the contact position, or the volume and length of the audio may be adjusted. This allows the user to know where on the object OBJ their fingers are touching based on the type, volume, and length of the audio.

[0133] For example, if the object OBJ is a camera mockup, feedback can be output only when the user touches an operation button, allowing the user to easily understand that they have touched the button with their finger.

[0134] The processing in the second embodiment is performed as described above. According to the second embodiment, the optical sensor 211 detects the dot pattern on the surface of the object OBJ, thereby making it possible to estimate the position and movement of the user's fingers relative to the object OBJ.

[0135] In hand tracking using images captured by a camera provided in an HMD worn by a user, the positions and movements of the fingers may not be detected depending on the position and angle of the camera. In the second embodiment, the optical sensor 211 attached to the user's fingers detects the dot pattern on the surface of the object OBJ, so that the positions and movements of the fingers relative to areas of the object OBJ that cannot be detected by the camera provided in the HMD can also be estimated.

[0136] By using a dot pattern, it is possible to estimate the position and movement of the fingers relative to the object OBJ with a precision that exceeds that of the hand tracking of an HMD, making it possible to reproduce the feel of using a detailed mockup.

[0137] Furthermore, by detecting the dot pattern on the surface of the object OBJ, it is possible to achieve contact detection and position estimation with accuracy that exceeds the hand tracking accuracy of the HMD, making it possible to detect the user's finger manipulation of the mockup and reproduce the experience of using the product.

[0138] When the user touches the object OBJ, the dot pattern may become dirty, but the detection result of the dot pattern may be corrected by detecting the dirt from an image acquired by the camera serving as the optical sensor 211. It is also possible to notify the user that the dot pattern is dirty. Furthermore, if the dot pattern cannot be detected, the detection result may be stored as a probability distribution, and a mechanism may be considered in which the accuracy improves depending on the number of touches, such that the distribution converges as the user traces the object OBJ multiple times.

[0139] The optical sensor 211 and the proximity sensor 212 can reduce power consumption by entering sleep mode when no object OBJ is near the user's finger. The performance of the sensors can also be varied depending on the finger on which they are worn. A high-resolution sensor can be used for the index finger, which is used frequently, and a low-resolution sensor can be used for the middle finger, which is used relatively infrequently. Priorities can also be assigned to each finger. For example, a high priority can be assigned to the thumb and index finger, which are used frequently, and a low priority can be assigned to the middle finger, ring finger, and little finger, which are used relatively infrequently. Parameters such as the resolution of the sensors worn on each finger can be changed based on the priority of each finger. For example, a sensor worn on a finger set to a high priority can be set to output high-resolution data, while a sensor worn on a finger set to a low priority can be set to output low-resolution data to reduce power consumption. The priority of each finger can be preset or can be set arbitrarily by the user.

[0140] The optical sensor 211 and the proximity sensor 212 can also be mounted on the tip of an object OBJ that the user holds with their fingers, such as a pen or a VR controller.

[0141] A color pattern may be used instead of the dot pattern. Even if the color pattern is recognized by the optical sensor 211, the position and movement of the finger relative to the object OBJ can be estimated in the same way as when a dot pattern is used. Also, a dot pattern and a color pattern may be used together.

[0142] Depending on the properties of the object OBJ, a vibration sensor, a microphone, a temperature sensor, a magnetic sensor, or the like can be used as the proximity sensor 212.

[0143] When a user wears an HMD equipped with a camera function and a VST function, when an object OBJ photographed by the camera is displayed on the HMD, the part of the object OBJ that the user is touching with his / her fingers may be highlighted, thereby enabling the user to easily grasp the part of the object OBJ that he / she is touching even while wearing the HMD.

[0144] The object OBJ may include a feedback output unit 231. In this case, it is preferable that the information processing unit 220 including the feedback control unit 224 and the feedback output unit 231 are configured to be able to communicate with each other via a wired or wireless network. Even if the feedback output unit 231 included in the object OBJ outputs feedback in accordance with the position and movement of the user's fingers relative to the object OBJ, the user can know where on the object OBJ he or she is touching, as in the embodiment.

[0145] The object OBJ in the second embodiment is not limited to a mockup, but may be any object on whose surface a dot pattern can be provided.

[0146] Instead of the dot pattern, a plurality of markers may be provided on the surface of the object OBJ. By detecting the markers with the optical sensor 211, the position and movement of the user's fingers relative to the object OBJ can be estimated in the same way as with the dot pattern. The dot pattern and the markers may be used together.

[0147] Third Embodiment [Configuration of Information Processing System 300] In the third embodiment, a sensor detects the shape of a user's fingers when the user holds an object OBJ with the fingers, which are body parts. Then, the information processing system 300 acquires the type of object OBJ held by the user as information about the object OBJ based on the detection result of the sensor. Note that holding an object OBJ with the hands includes all manners of holding the object OBJ with the fingers, such as pinching, squeezing, or gripping the object OBJ with the hands.

[0148] Note that "holding an object OBJ with fingers" includes not only cases where the fingers hold the object OBJ by directly contacting it, but also cases where a device or sensor serving as the information processing system 300 is attached to the fingertips and the device or sensor comes into contact with the object OBJ, i.e., cases where the fingers hold the object OBJ via the device or sensor.

[0149] The configuration of the information processing system 300 will be described. In the example of Fig. 21 , the information processing system 300 is configured as a device worn on the fingertip of a user. As shown in Fig. 22 , the information processing system 300 is configured with a sensor unit 310, an information processing unit 320, and an output unit 330.

[0150] The contact sensor 311 is a sensor for detecting a change that occurs when the user's fingers touch the object OBJ when holding the object OBJ. For example, a pressure sensor can be used as the contact sensor 311, assuming that the change that occurs when the user's fingers touch the object OBJ is a change in pressure. By detecting the pressure generated when the user's fingers touch the object OBJ with the pressure sensor, it is possible to know the timing when the fingers touch the object OBJ.

[0151] The camera 312 is used to capture an image of the user's fingers holding the object OBJ. The camera 312 may be, for example, provided in an HMD worn by the user, or may be a wearable camera worn by the user. The camera 312 corresponds to a sensor unit that senses while in contact with the user's body.

[0152] The contact determination unit 321 performs processing to determine whether or not the user's fingers have come into contact with the object OBJ based on the detection result of the contact sensor 311 .

[0153] The hand posture recognition unit 322 recognizes the posture of the user's hand by detecting bones in an image of the user's hand captured by the camera 312. Bone detection can be performed, for example, by a deep neural network (DNN). It can also be performed by machine learning, artificial intelligence, or the like, such as a convolutional neural network (CNN) or a random forest (RF). For example, neural networks and deep learning are used as learning methods for machine learning.

[0154] The object information storage unit 323 stores in advance an object information table in which information relating to a plurality of objects OBJ is associated with each other. For example, as shown in Fig. 23, the object information table associates, for a plurality of objects OBJ, the type of object OBJ, three-dimensional bone information of the fingers in a state in which the object OBJ is being held, sensor information that can be detected by various sensors in a state in which the object OBJ is being held, application processing, and the like.

[0155] The types of object OBJ that can be estimated by the information processing system 300 are those object OBJs that are pre-registered in the object information table of the object information storage unit 323. Therefore, by pre-registering a large amount of object information in the object information table, estimation accuracy can be improved. Furthermore, in addition to the pre-registered object OBJs, the user may add information about a new object OBJ of his / her choice and link it to the object information table.

[0156] The information acquisition unit 324 acquires information about the object OBJ based on the hand posture recognition result and the object information table stored in the object information storage unit 323. The information about the object OBJ is the type of object OBJ held by the user.

[0157] The application processing unit 325 executes processing related to the application depending on the type of the object OBJ.

[0158] The display unit 331 is a liquid crystal display, an organic EL display, or the like that displays videos, images, GUI, messages, and the like based on the processing in the application processing unit 325 .

[0159] The audio output unit 332 is a speaker that outputs audio based on the processing in the application processing unit 325 .

[0160] It should be noted that the information processing system 300 does not necessarily have to include both the display unit 331 and the audio output unit 332, and may include only one of them, or neither.

[0161] The information processing system 300 of the third embodiment is configured as described above.

[0162] 22 may be configured to be included in a single device. In this case, the information processing unit 320 is realized by a processor including a CPU, RAM, ROM, etc., executing a program. Alternatively, the information processing unit 320 may be realized by a processing block of hardware having a dedicated function.

[0163] Alternatively, the sensor unit 310, contact determination unit 321, hand shape recognition unit 322, object information storage unit 323, and information acquisition unit 324 may be configured as being included in a device that comes into contact with the user's body, and the application processing unit 325 and output unit 330 may be configured as being included in another device. Alternatively, the device that comes into contact with the user's body may be configured to include only the sensor unit 310, and the information processing unit 320 and output unit 330 may be configured as being included in another device. The other device may be, for example, an HMD, a smartphone, a personal computer, a smartwatch, a wearable device, headphones, etc.

[0164] In this case, each device may have a communication function to transmit and receive information and signals between different devices. The communication method may be either wired or wireless. Communication methods include cellular communication, 4G, 5G, Wi-Fi, Bluetooth (registered trademark), NFC, Ethernet (registered trademark), HDMI (registered trademark), USB, etc.

[0165] The information processing system 300 may be configured by a single device or apparatus, or may be configured by a plurality of devices or apparatuses.

[0166] The hardware configuration of the information processing system 100 is the same as that of the first embodiment shown in FIG.

[0167] [Processing in Information Processing System 300] Next, processing in the information processing system 300 will be described with reference to FIG.

[0168] In step S31, the contact sensor 311 detects a change that occurs when the user's finger touches the object OBJ.

[0169] Next, in step S32, the contact determination unit 321 determines whether or not the user's fingers have contacted the object OBJ based on the detection result from the contact sensor 311. For example, if a pressure sensor is used as the contact sensor 311, it can be determined that the user's fingers have contacted the object OBJ if the pressure value detected is equal to or greater than a predetermined value. If the user's fingers are in contact with the object OBJ, the process proceeds to step S33 (Yes in step S32).

[0170] Next, in step S33, an image of the user's fingers holding the object OBJ is acquired by photographing the camera 312. An image of the fingers holding the object OBJ can be acquired by photographing the camera 312 at the timing when the user's fingers touch the object OBJ, which can be identified by the contact sensor 311 and the contact determination unit 321, that is, at the timing when the user holds the object OBJ with their fingers. Alternatively, the camera 312 may continuously photograph the object OBJ for a certain period of time, and multiple consecutive images may be stored in a buffer or the like, and an image may be acquired by selecting an image whose photographing timing coincides with the timing when the user held the object OBJ with their fingers, which can be identified by the contact determination unit 321.

[0171] Next, in step S34, the hand posture recognition unit 322 performs bone detection on the image captured by the camera 312, thereby recognizing the posture of the hand holding the object OBJ as the hand posture recognition result, as shown in FIG.

[0172] Next, in step S35, the information acquisition unit 324 estimates the type of object OBJ being held by the user based on the hand posture recognition result and the object information table stored in the object information storage unit 323. If the hand posture recognition result matches the three-dimensional bone information of the fingers in the object information table, it can be estimated that the type of object OBJ associated with the three-dimensional bone information of the fingers in the object information table is the type of object OBJ being held by the user. If the hand posture recognition result matches the three-dimensional bone information of the fingers, the process proceeds to step S36 (Yes in step S35).

[0173] Next, in step S36, the application processing unit 325 executes processing related to the application in accordance with the type of the object OBJ.

[0174] If the estimation result is a specific object OBJ, the application processing unit 325 may process the HMD to display a notification according to the type of object OBJ. Notification display methods include, for example, displaying a message, an icon, or a mark. If the HMD has a VST function, a notification such as a message may be displayed superimposed on or near the object OBJ displayed on the HMD. Furthermore, a notification such as a message may be displayed superimposed on or near the object OBJ displayed on the HMD after the user's fingers have left the object OBJ. The removal of the user's fingers from the object OBJ can be determined based on the detection result of the contact sensor 311. A specific object OBJ is, for example, a dangerous object OBJ such as a cutter or a kitchen knife, and the notification message, icon, or mark indicates that the object OBJ is dangerous. Furthermore, a specific sound may be output from a speaker provided in the HMD instead of or in addition to a display on the display. Note that the object OBJ is not limited to dangerous objects, and the notification is not limited to a notification of danger. The object OBJ may be any object, and the notification may be any notification relating to the object OBJ.

[0175] It should be noted that information on the type of object OBJ may be transmitted to an external device.

[0176] Then, in step S37, it is confirmed whether the process is to be completed. The process may be completed when the device is turned off, etc. Steps S31 to S37 are repeated unless the process is completed (No in step S37).

[0177] As shown in Figures 26A and 26B, a user may hold the same object OBJ with different finger shapes. Figure 26A shows a shape with the middle finger, ring finger, and little finger spread apart, while Figure 26B shows a shape with the middle finger, ring finger, and little finger closed. The same applies to Figures 27A and 27B and Figures 28A and 28B. In order to be able to estimate the type of object OBJ even when the object is held in different ways, three-dimensional bone information for multiple fingers may be associated with one object OBJ in an object information table.

[0178] The information processing system 300 may also include a position and orientation estimation unit 326 as shown in FIG. 29 . The position and orientation estimation unit 326 estimates the position and orientation of the object OBJ based on the positions and orientations of the fingers of the user holding the object OBJ, the positions and orientations of the palm and back of the hand, the midpoint of the fingers (the midpoint between the index finger and thumb), and the like. In this way, the position and orientation of the object OBJ is acquired as information about the object OBJ. The information processing system 300 may acquire either the type of object OBJ or the position and orientation of the object OBJ, or may acquire both, as information about the object OBJ.

[0179] In this case, the position and orientation estimation unit 326 estimates the position and orientation of the object OBJ held by the user, as shown in step S38 of the flowchart in Fig. 30. Then, in step S39, the application processing unit 325 executes processing related to the application in accordance with the type and position and orientation of the object OBJ.

[0180] For example, as shown in FIG. 31A, when the user pinches an object OBJ between their thumb and index finger, the position and orientation estimation unit 326 can estimate the position and orientation of the object OBJ based on the positions of the tips of the thumb and index finger, or the line connecting the tips of the thumb and index finger.

[0181] Furthermore, as shown in FIG. 31B, when the user is holding an object OBJ with his or her fingers, the position and orientation of the object OBJ can be estimated based on the positions of the palm, back of the hand, and fingers.

[0182] The processing in the third embodiment is performed as described above. According to the third embodiment, it is possible to estimate the type of object OBJ that the user is holding with his / her fingers. Furthermore, even if part of the user's fingers is hidden by the object OBJ, causing occlusion, it is possible to estimate the type of object OBJ that the user is holding.

[0183] In the above description, the type of object OBJ is estimated based on the shape of the user's fingers at the time the user holds the object OBJ. However, as the fingers approach the object OBJ, the accuracy of finger depth detection decreases, which may result in a decrease in the accuracy of finger shape recognition. Therefore, the type of object OBJ may be estimated based on the shape of the fingers before the user holds the object OBJ, for example, just before the user holds the object OBJ (N seconds before holding the object OBJ). This can be achieved by using the camera 312 to continuously capture images of the object OBJ at predetermined time intervals for a certain period of time starting at least N seconds before the user holds the object OBJ, storing multiple consecutive images in a buffer, or the like, and performing bone detection on the images captured N seconds before the user holds the object OBJ. N is a value preset in the information processing system 300 and may be set to any value by the user.

[0184] In the third embodiment, the detection result of the contact sensor 311 is used to determine whether or not the user's fingers have contacted the object OBJ, but the type of the object OBJ can also be estimated comprehensively based on the detection result of the contact sensor 311 and the hand and finger shape recognition result.

[0185] In addition, the type of object OBJ can be estimated not only from the detection result of the contact sensor 311 at the time when the user holds the object OBJ with his / her fingers, but also from the change in the sensor value from the time the object is first held until the sensor value as the detection result stabilizes.

[0186] For example, if the object OBJ is a doll, the change in pressure from when the user's fingers touch the object OBJ to when the user grasps it will be different between a hard doll made of plastic or the like as shown in Fig. 27 and a soft doll (stuffed toy) made of fur as shown in Fig. 28. Therefore, the type of object OBJ can be estimated based on the amount of change in pressure when the user holds the object OBJ. For this purpose, it is advisable to associate information on the amount of change in pressure and information indicating whether the object is hard or soft with the type of object OBJ in the object information storage unit 323 in advance.

[0187] The type of the object OBJ can be estimated using a bending sensor in the same way as when a pressure sensor is used.

[0188] Furthermore, sensors other than the camera 312 can be used, or the camera 312 can be used in combination with other sensors. By using other sensors in combination, the accuracy of estimating the type of the object OBJ can be improved. Examples of other sensors include the following:

[0189] By using a temperature sensor to detect the surface temperature of the object OBJ when the user holds the object OBJ, the type of the object OBJ can be estimated based on the surface temperature of the object OBJ. For this purpose, the type of the object OBJ may be associated with the temperature information of the object OBJ in advance in the object information table of the object information storage unit 323. The temperature information may be a specific temperature value or may be a temperature category such as high temperature, normal temperature, or low temperature.

[0190] Furthermore, the way in which a user holds an object OBJ may change depending on the temperature of the object OBJ. For example, if the object OBJ is at room temperature, the user may grasp it with all of the fingers, but if the object OBJ is hot, the user may pinch it with only the index finger and thumb. Therefore, the type of object OBJ can be estimated based on the temperature of the object OBJ detected using a temperature sensor and the way in which the user holds the object OBJ. For this purpose, the type of object OBJ may be associated with temperature information of the object OBJ in advance in the object information table of the object information storage unit 323. The temperature information may be a specific temperature value or may be a temperature category such as high temperature, room temperature, or low temperature.

[0191] Furthermore, by using a vibration sensor, the type of object OBJ can be estimated from changes in acceleration and displacement of the finger movement when the user holds the object OBJ. For this purpose, the type of object OBJ may be associated with the capacitance value of the object OBJ in advance in the object information table of the object information storage unit 323.

[0192] Furthermore, by using a capacitance sensor, the type of object OBJ can be estimated from the capacitance value. For example, if the capacitance value is high, it can be estimated that the object OBJ is made of a material with a high dielectric constant. For this purpose, the capacitance value of the object OBJ may be associated with the type of the object OBJ in advance in the object information table of the object information storage unit 323.

[0193] When using sensors other than these cameras 312, the accuracy of estimating the type of object OBJ can be improved by narrowing down the candidates from among multiple objects OBJ in the object information table of the object information storage unit 323 based on the detection results of the other sensors.

[0194] Furthermore, different objects OBJ may have the same three-dimensional bone information, which is the shape of the fingers when held, but by using other sensor information as described above, it may be possible to estimate the type of object OBJ as a different object even if the three-dimensional bone information is the same.

[0195] When using other sensors in combination with a pressure sensor, the pressure sensor and the other sensor may be attached to one finger, or the pressure sensor and the other sensor may be attached to different fingers. Furthermore, the pressure sensor and the other sensor may be attached to different positions on the hand. For example, a pressure sensor is attached to the index finger, which is frequently used when grasping an object OBJ, and the above-mentioned temperature sensor, vibration sensor, capacitance sensor, etc. are attached to the other fingers.

[0196] In bone detection performed by the hand posture recognition unit 322, the tip of each finger can be detected from the detection results, and therefore the distance between fingers when holding an object OBJ, for example, the distance between the index finger and thumb, can also be detected from the bone detection results. Since this distance between fingers is approximate to the size of the object OBJ, such as the thickness and length of the object OBJ, the type of the object OBJ can also be estimated using this distance between the fingers. For this purpose, the type of object OBJ may be associated with size information of the object OBJ in advance in the object information table of the object information storage unit 323.

[0197] In bone detection performed by the hand posture recognition unit 322, the tip of each finger can be detected from the detection results, and therefore the amount of change in the fingertip position can be detected from multiple consecutive bone detection results. The information acquisition unit 324 can estimate whether the object OBJ is soft or hard based on the amount of change in the fingertip position. This is because, with a soft object OBJ, the fingertip position changes after the fingertip touches the object OBJ, and with a hard object OBJ, the fingertip position does not change after the fingertip touches the object OBJ. For this purpose, the object information storage unit 323 may previously associate information indicating whether the object OBJ is soft or hard with the type of object OBJ.

[0198] Before estimating the type of object OBJ, the accuracy of estimating the type of object OBJ can be improved by narrowing down candidates from among multiple object OBJs registered in the object information table of the object information storage unit 323 based on the context, the location of the user, the time, the height of the object OBJ that can be determined from the height of the fingers, the detection results by the camera 312, etc.

[0199] Furthermore, the accuracy of estimating the type of object OBJ can be improved by narrowing down the candidates from among the multiple objects OBJ registered in the object information storage unit 323 according to the attributes of the user. For example, for a child, the candidates are narrowed down to stuffed toys and the like, and for an adult, the candidates are narrowed down to everyday items and the like.

[0200] Furthermore, the way an object OBJ is held may vary depending on the size of the user's fingers. For example, an adult may be able to hold an object OBJ by pinching it with their index finger and thumb, but a child may need to grip it with their entire hand. The same object OBJ may be held differently by children and adults, and may also be held differently by men and women. Therefore, the accuracy of estimating the type of object OBJ can be improved by setting user attributes such as age (child or adult) and gender in advance. To this end, the object information table in the object information storage unit 323 may previously associate the type of object OBJ with user attributes and three-dimensional bone information of the fingers of a user with that attribute holding the object OBJ.

[0201] If the type of object OBJ is not successfully estimated the first time, the user may be prompted to hold the object OBJ again by a display on the display unit 331 or a voice from the voice output unit 332. At this time, the user may be prompted to hold the object OBJ in a different way, to hold it with the opposite hand, to change the position or angle of the object OBJ and fingers relative to the camera 312, etc. This can improve the accuracy of estimating the type of object OBJ. When urging the user to hold the object OBJ again by a display or a voice, a desirable holding way and the shape and position of the fingers may be presented. In this case, multiple desirable holding ways and shapes and positions of the fingers may be presented so that the user can select from them.

[0202] If the result of estimating the type of object OBJ is incorrect, the user may be allowed to input feedback. Examples of methods for inputting feedback include performing specific gestures within the field of view of the camera 312, such as shaking the held object OBJ from side to side, tracing the object OBJ with a finger, or making a cross with a finger. When the hand posture recognition unit 322 identifies these specific gestures from the image captured by the camera 312, it uses the feedback to re-learn DNN, machine learning, artificial intelligence, or the like. This can improve the accuracy of hand posture identification, thereby improving the accuracy of estimating the type of object OBJ.

[0203] The information processing system 300 can also be configured as a device shaped like pliers or tongs, which can pinch or hold the object OBJ. In this case, a sensor such as a pressure sensor can be provided inside the pinching part (holding part), and the type of the object OBJ can be estimated based on the detection result of the sensor, the position and orientation of the device, etc.

[0204] In the third embodiment, the information processing system 300 can be configured as a pen-type device that can be held by a user. In this case, a sensor such as a pressure sensor is provided at the tip of the pen-type device, and the tip is brought into contact with an object OBJ. Then, the type of object OBJ can be estimated based on the detection result of the pressure sensor and the position and orientation of the pen-type device.

[0205] Furthermore, when the object OBJ is placed on a desk or the like, the initial orientation is fixed in the pitch and roll directions. Therefore, the yaw direction may be estimated from the way the user holds the object OBJ with their fingers.

[0206] In the third embodiment, an example in which the object OBJ is held in one hand has been described, but the object OBJ may be held with both hands. In this case, the type of object OBJ may be associated with three-dimensional bone information of the fingers in a state in which the object OBJ is held with both hands in the object information table of the object information storage unit 323.

[0207] The accuracy of the match determination between the hand posture recognition result by the information acquisition unit 324 and the three-dimensional bone information of the fingers may be improved by using an algorithm such as dynamic time warping.

[0208] 15 in the first embodiment, a plurality of object information tables in which different application processes are associated with each object OBJ may be stored in the object information storage unit 323, and the object information table to be used may be switched according to information about the user and information about the object OBJ. The information about the user and the information about the object OBJ may include the user's age, the user's sex, the user's proficiency in using the object OBJ (professional or amateur), the user's posture, the user's position, the type of object OBJ, the area of ​​the object OBJ, etc. This makes it possible to make the application processes to be executed differently depending on the user and the object OBJ even for the same type of object OBJ, and makes it possible to execute more application processes than the number of types of object OBJ according to the estimation result of the type of object OBJ.

[0209] The contact sensor 311 and camera 312 can reduce power consumption by going into sleep mode when there is no object OBJ near the user's finger. The performance of the sensors can also be changed depending on the finger on which they are worn. A fine-resolution sensor could be used for the index finger, which is used frequently, and a coarse-resolution sensor for the middle finger.

[0210] <Modifications> Although the embodiments of the present technology have been specifically described above, the present technology is not limited to the above-described embodiments, and various modifications based on the technical ideas of the present technology are possible.

[0211] The first embodiment and the second embodiment may be combined to estimate the position and movement of fingers relative to the object OBJ. Also, the first embodiment, the second embodiment, and the third embodiment may be combined to estimate the type of the object OBJ and also estimate the position and movement of fingers relative to the object OBJ.

[0212] The present technology can also be configured as follows. (1-1) An information processing system including: a sensor unit attached to a user's body; and an information acquisition unit that acquires information about an object based on a detection result from the sensor unit when the sensor unit or a body part of the user is in contact with or proximity to the object. (1-2) The information processing system according to (1-1), wherein the information about the object is a position of the user's fingers relative to the object. (1-3) The information processing system according to (1-1) or (1-2), wherein the information about the object is a movement of the user's fingers relative to the object. (1-4) The information processing system according to any of (1-1) to (1-3), wherein the sensor unit is a vibration sensor, and the detection result is vibrations generated when the sensor unit or the body part comes into contact with the object. (1-5) The information processing system according to any of (1-1) to (1-4), wherein the information acquisition unit acquires information about the object based on the detection result and a table that associates vibration patterns with information about the object. (1-6) The information processing system according to any one of (1-1) to (1-5), further comprising a feedback output unit that outputs feedback to the user according to information about the object. (1-7) The information processing system according to any one of (1-1) to (1-6), wherein the information about the object is at least one of the type of the object and the position and orientation of the object. (1-8) The information processing system according to (1-7), wherein the sensor unit is a camera, and the detection result is a body part of the user or a shape of the body part when the sensor unit is in contact with the object. (1-9) The information processing system according to (1-8), wherein the body part of the user detected by another sensor or the shape of the body part is acquired from an image captured by the camera when the sensor unit is in contact with the object. (1-10) The information processing system according to (1-8) or (1-9), wherein the information acquisition unit acquires information about the object by referring to the detection result and a table that associates the shape of the body part with information about the object. (1-11) The information processing system according to any one of (1-1) to (1-10), wherein the part is a finger of the user.(1-12) The information processing system according to any one of (1-1) to (1-11), wherein the sensor unit is worn on the user's finger or arm. (1-13) The information processing system according to any one of (1-1) to (1-12), further comprising an application processing unit that performs application processing according to information about the object. (1-14) The information processing system according to any one of (1-1) to (1-13), wherein, when the object is displayed on a head-mounted display worn by the user, a part of the object that the user should touch with their fingers in order to use the object is highlighted. (1-15) The information processing system according to any one of (1-1) to (1-14), wherein, when the object is displayed on a head-mounted display worn by the user, virtual fingers are displayed to guide the user's fingers on the object. (1-16) The information processing system according to any one of (1-1) to (1-15), wherein, when the object is displayed on a head-mounted display worn by the user, a contact position of the user's fingers on the object is highlighted. (1-17) The information processing system described in (1-5), wherein the information acquisition unit switches between and uses a plurality of tables, each of which associates a different application process with a vibration pattern, according to predetermined information. (1-18) The information processing system described in (1-7), wherein, when the object is displayed on a head-mounted display worn by the user, a notification related to the object is displayed on the head-mounted display according to the type of the object. (2-1) An information processing method that acquires information related to the object based on a detection result of a sensor unit worn on the user's body or a state in which the sensor unit is in contact with or close to an object. (2-2) The information processing method described in (2-1), wherein the information related to the object is the position of the user's fingers relative to the object. (2-3) The information processing method described in (2-1) or (2-2), wherein the information related to the object is the movement of the user's fingers relative to the object.(2-4) The information processing method according to any one of (2-1) to (2-3), wherein the sensor unit is a vibration sensor, and the detection result is vibration generated when the sensor unit or the part comes into contact with the object. (2-5) The information processing method according to any one of (2-1) to (2-4), wherein the information acquisition unit acquires information about the object based on the detection result and a table that associates vibration patterns with information about the object. (2-6) The information processing method according to any one of (2-1) to (2-5), wherein the information acquisition unit includes a feedback output unit that outputs feedback to the user according to the information about the object. (2-7) The information processing method according to any one of (2-1) to (2-6), wherein the information about the object is at least one of the type of the object and the position and orientation of the object. (2-8) The information processing method according to (2-7), wherein the sensor unit is a camera, and the detection result is a part of the user or a shape of the part when the sensor unit comes into contact with the object. (2-9) The information processing method according to (2-8), wherein the shape of the user's part detected by another sensor or the part is acquired from an image captured by the camera when the sensor unit comes into contact with the object. (2-10) The information processing method according to (2-8) or (2-9), wherein the information acquisition unit acquires information about the object by referring to the detection result and a table that associates the shape of the part with information about the object. (2-11) The information processing method according to any of (2-1) to (2-10), wherein the part is a finger of the user. (2-12) The information processing method according to any of (2-1) to (2-11), wherein the sensor unit is worn on the finger or arm of the user. (2-13) The information processing method according to any of (2-1) to (2-12), further comprising an application processing unit that performs application processing in accordance with information about the object. (2-14) The information processing method according to any one of (2-1) to (2-13), wherein, when the object is displayed on a head-mounted display worn by the user, a part of the object that the user should touch with their fingers in order to use the object is highlighted.(2-15) The information processing method according to any one of (2-1) to (2-14), wherein, when the object is displayed on a head-mounted display worn by the user, virtual fingers are displayed to guide the user's fingers relative to the object. (2-16) The information processing method according to any one of (2-1) to (2-15), wherein, when the object is displayed on a head-mounted display worn by the user, a contact position of the user's fingers relative to the object is highlighted. (2-17) The information processing method according to (2-5), wherein the information acquisition unit switches between and uses a plurality of tables, in which different application processes are associated with the vibration patterns, according to predetermined information. (2-18) The information processing method according to (2-7), wherein, when the object is displayed on a head-mounted display worn by the user, a notification related to the object is displayed on the head-mounted display according to the type of the object. (3-1) A program causing a computer to execute an information processing method for acquiring information about an object based on a detection result of a sensor unit worn on a user's body or a body part of the user in contact with or proximity to an object. (3-2) The program according to (3-1), wherein the information about the object is the position of the user's fingers relative to the object. (3-3) The program according to (3-1) or (3-2), wherein the information about the object is the movement of the user's fingers relative to the object. (3-4) The program according to any of (3-1) to (3-3), wherein the sensor unit is a vibration sensor and the detection result is vibration generated when the sensor unit or the body part comes into contact with the object. (3-5) The program according to any of (3-1) to (3-4), wherein the information acquisition unit acquires information about the object based on the detection result and a table that associates vibration patterns with information about the object. (3-6) The program according to any of (3-1) to (3-5), wherein the program includes a feedback output unit and outputs feedback to the user according to the information about the object.(3-7) The program according to any one of (3-1) to (3-6), wherein the information about the object is at least one of the type of the object and the position and orientation of the object. (3-8) The program according to (3-7), wherein the sensor unit is a camera, and the detection result is a body part of the user or the shape of the body part when the sensor unit is in contact with the object. (3-9) The program according to (3-8), wherein the body part of the user detected by another sensor or the shape of the body part is acquired from an image captured by the camera when the sensor unit is in contact with the object. (3-10) The program according to (3-8) or (3-9), wherein the information acquisition unit acquires information about the object by referring to the detection result and a table that associates the shape of the body part with information about the object. (3-11) The program according to any one of (3-1) to (3-10), wherein the body part is a finger of the user. (3-12) The program according to any one of (3-1) to (3-11), wherein the sensor unit is worn on the finger or arm of the user. (3-13) The program according to any one of (3-1) to (3-12), including an application processing unit that performs application processing according to information about the object. (3-14) The program according to any one of (3-1) to (3-13), wherein, when the object is displayed on a head-mounted display worn by the user, a part of the object that the user should touch with their fingers to use the object is highlighted. (3-15) The program according to any one of (3-1) to (3-14), wherein, when the object is displayed on a head-mounted display worn by the user, virtual fingers are displayed to guide the user's fingers on the object. (3-16) The program according to any one of (3-1) to (3-15), wherein, when the object is displayed on a head-mounted display worn by the user, a contact position of the user's fingers on the object is highlighted. (3-17) The program according to (3-5), wherein the information acquisition unit switches between and uses a plurality of the tables, each of which associates a different application process with a vibration pattern, according to predetermined information.(3-18) The program according to (3-7), wherein, when the object is displayed on a head-mounted display worn by the user, a notification regarding the object is displayed on the head-mounted display depending on the type of the object.

[0213] REFERENCE SIGNS LIST 100, 200, 300... Information processing system 110, 210, 310... Sensor unit 111... Vibration sensor 122, 223, 324... Information acquisition unit 123, 325... Application processing unit 211... Optical sensor 312... Camera OBJ... Object

Claims

1. An information processing system comprising: a sensor unit attached to a user's body; and an information acquisition unit that acquires information about an object based on the detection results of the sensor unit when the sensor unit or a part of the user's body is in contact with or in close proximity to the object.

2. The information processing system according to claim 1, wherein the information about the object is the position of the user's fingers relative to the object.

3. The information processing system according to claim 1, wherein the information about the object is the movement of the fingers relative to the object.

4. An information processing system according to claim 1, wherein the sensor unit is a vibration sensor, and the detection result is vibrations that occur when the sensor unit or the part comes into contact with the object.

5. An information processing system according to claim 1, wherein the information acquisition unit acquires information about the object based on the detection result and a table that associates vibration patterns with information about the object.

6. The information processing system according to claim 1, further comprising a feedback output unit that outputs feedback to the user in accordance with information relating to the object.

7. The information processing system according to claim 1, wherein the information about the object is at least one of the type of the object and the position and orientation of the object.

8. An information processing system according to claim 7, wherein the sensor unit is a camera, and the detection result is a part of the user's body or the shape of the part when the sensor unit comes into contact with the object.

9. The information processing system according to claim 8, wherein the shape of the user's body part detected by another sensor or the shape of the body part is acquired from an image captured by the camera when the sensor unit comes into contact with the object.

10. An information processing system according to claim 8, wherein the information acquisition unit acquires information about the object by referring to a table that matches the detection results with the shape of the part and information about the object.

11. The information processing system according to claim 1, wherein the part is a finger of the user.

12. The system according to claim 1, wherein the sensor unit is attached to the user's finger or arm.

13. The information processing system according to claim 1, further comprising an application processing unit that performs application processing in accordance with information about the object.

14. An information processing system as described in claim 1, wherein when the object is displayed on a head-mounted display worn by the user, the part of the object that the user should touch with their fingers to use the object is highlighted.

15. An information processing system according to claim 1, wherein when the object is displayed on a head-mounted display worn by the user, virtual fingers are displayed to guide the user's fingers relative to the object.

16. The information processing system according to claim 1, wherein when the object is displayed on a head-mounted display worn by the user, the contact position of the user's fingers on the object is highlighted.

17. The information processing system according to claim 5, wherein the information acquisition unit switches between a plurality of tables in which different application processes are associated with the vibration patterns, depending on predetermined information.

18. An information processing system according to claim 7, wherein when the object is displayed on a head-mounted display worn by the user, a notification regarding the object is displayed on the head-mounted display according to the type of the object.

19. An information processing method for acquiring information about an object based on the detection results of a sensor unit worn on a user's body or a part of the user's body when the sensor unit is in contact with or in close proximity to the object.

20. A program that causes a computer to execute an information processing method for acquiring information about an object based on the detection results of a sensor unit worn on a user's body or a part of the user's body when the sensor unit is in contact with or close to the object.

Images