Information processing device, tactile sensation presentation method, and computer program

Abstract

To provide a tactile sensation presentation method which enables presentation of tactile sensation of an object viewed by a user.SOLUTION: A tactile sensation presentation method is provided, comprising identifying an object viewed by a user in real space, acquiring tactile data for presenting tactile sensation of the identified object, and transmitting the acquired tactile data to a tactile sensation presentation device to present the tactile sensation of the object viewed by the user.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, a tactile presentation method, and a computer program. [Background technology]

[0002] Various technologies for presenting tactile sensations (haptics) have been proposed. Based on knowledge of magnetorheological fluids, the inventors have proposed a tactile sensation presentation device that can be operated by a user and that reproduces different sensations depending on the object (Patent Document 1, etc.). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6906275 Summary of the Invention [Problem to be solved by the invention]

[0004] An object of the present disclosure is to provide an information processing device, a tactile presentation method, and a computer program that can present the tactile sensation of an object visually recognized by a user. [Means for solving the problem]

[0005] An information processing device according to one aspect of the present disclosure is an information processing device that includes a processing unit that executes processing related to tactile presentation using a tactile presentation device, and the processing unit executes processing to identify an object that a user views in real space, acquire tactile data for presenting the tactile sensation of the identified object, and transmit the acquired tactile data to the tactile presentation device.

[0006] A tactile presentation method according to one aspect of the present disclosure identifies an object viewed by a user in real space, acquires tactile data for presenting the tactile sensation of the identified object, and transmits the acquired tactile data to a tactile presentation device, thereby presenting the tactile sensation of the object viewed by the user.

[0007] A computer program according to one aspect of the present disclosure is a computer program for causing a computer to execute processing related to tactile presentation using a tactile presentation device, and causes the computer to execute processing to identify an object that a user views in real space, acquire tactile data for presenting the tactile sensation of the identified object, and transmit the acquired tactile data to the tactile presentation device. [Effects of the Invention]

[0008] According to the present disclosure, it is possible to present a tactile sensation to an object that a user is viewing. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a schematic diagram showing a tactile presentation system according to the first embodiment. [Figure 2] FIG. 2 is a block diagram showing the internal configuration of an MR device. [Figure 3] FIG. 1 is a conceptual diagram illustrating the configuration of a learning model. [Figure 4] FIG. 10 is an explanatory diagram showing an example of the contents of a sensory DB. [Figure 5] FIG. 1 is a block diagram showing a configuration of a tactile presentation device. [Figure 6] FIG. 2 is a block diagram showing the internal configuration of the server device. [Figure 7] 10 is a flowchart illustrating an example of a processing procedure of the MR device and the server device. [Figure 8] FIG. 1 is a schematic diagram showing mixed reality. [Figure 9] FIG. 10 is a schematic diagram showing a tactile presentation system according to a second embodiment. [Figure 10]FIG. 10 is a block diagram showing the internal configuration of a tablet terminal according to a second embodiment. [Figure 11] 10 is a flowchart illustrating an example of a processing procedure of the tablet terminal and the server device. [Figure 12] FIG. 2 is a schematic diagram showing a display screen of a tablet terminal. [Figure 13] FIG. 10 is a schematic diagram showing a tactile presentation system according to a third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] An information processing device, a tactile presentation method, and a computer program according to embodiments of the present disclosure will be described below with reference to the drawings. In the following embodiments, a tactile presentation system that implements the tactile presentation method according to the present embodiments will be described. Note that the present disclosure is not limited to these examples, but is defined by the claims, and is intended to include all modifications within the meaning and scope equivalent to the claims. Furthermore, at least some of the embodiments described below may be combined in any manner.

[0011] (Embodiment 1) FIG. 1 is a schematic diagram showing a tactile presentation system 100 in the first embodiment. The tactile presentation system 100 includes an MR (Mixed Reality) device 1, which is an information processing device, a tactile presentation device 2, and a server device 3. The MR device 1 and the tactile presentation device 2 are communicatively connected via short-range wireless communication and exchange data with each other. The MR device 1 and the server device 3 exchange data with each other via a network N. The network N includes the Internet N1, a carrier network N2, a base station BS, and an access point AP.

[0012] The tactile presentation system 100 according to the first embodiment constructs a mixed reality for the real space visually recognized by the user, and reproduces the tactile sensation of an object 4 (see FIG. 8) that actually exists in the real space visually recognized by the user. In particular, this system makes it possible to reproduce the tactile sensation of an object 4 that the user cannot directly touch. Examples of objects 4 that cannot be touched include art exhibits, dangerous animals, high-temperature substances, and toxic substances. The user can experience the tactile sensation of an object 4 in real space that he or she would not normally be able to touch.

[0013] The MR device 1 is a see-through head-mounted display that performs spatial mapping of the real space that the user is viewing through the see-through display 15 and projects a hologram, which is a virtual object, to create a mixed reality in which the virtual object is visualized in the real space. As the MR device 1, HoloLens (registered trademark) by Microsoft Corporation, MagicLeap (registered trademark) by MagicLeap Corporation, etc. can be used. The MR device 1 is a type of augmented reality device that creates an augmented reality, and the augmented reality device also includes an AR (Augmented Reality) device (not shown).

[0014] The tactile presentation device 2 is a device that a user can operate by moving their finger while holding it along the displacement unit 202. The tactile presentation device 2 reads the position of the displacement unit 202, which is displaced by the user's finger movement, and controls the built-in MRF (Magneto-Rheological Fluid) device 24 according to the position to generate a force sensation due to a reaction force (rotational resistance) against the user's operation on the displacement unit 202, thereby presenting a tactile sensation. The form of the displacement unit 202 of the tactile presentation device 2 is not limited to that shown in FIG. 1 , but may be a button, a stick, a cushion-like object covered with a cover, or a glove-type object. The tactile presentation device 2 may be an active tactile presentation device 2 that uses a motor, a piezoelectric element, an ultrasonic oscillator, or the like instead of the MRF device 24 and generates a force sensation through rotational force or vibration in response to the user's operation, and may be combined with devices that present vibration, warmth, coldness, or electrical stimulation in addition to the displacement unit 202. It is also possible to combine an active type with a passive type tactile presentation device 2 such as that of the present invention. It may also be configured to be installed on the ground or a wall and operated by the user's palms, feet, etc.

[0015] In the tactile presentation system 100, an MR device 1 having installed therein a computer program P1 for presenting the tactile sensation of an object 4 visually recognized by a user in real space communicates with a server device 3 to realize the tactile presentation of the object 4. The MR device 1 acquires tactile data of the object 4 from an object database (DB: Data Base) 311 of the server device 3 via a network N.

[0016] 2 is a block diagram showing the internal configuration of the MR device 1. The MR device 1 is a see-through head-mounted display, and application software (not shown) for constructing mixed reality is pre-installed on it. By executing the application software, the MR device 1 can construct mixed reality by projecting required virtual objects into the real space visually recognized by the user.

[0017] The MR device 1 includes a processing unit 10, a storage unit 11, a first communication unit 12, a second communication unit 13, an imaging unit 14, a see-through display 15, and an eye tracking sensor 16. The processing unit 10 is a processor using a CPU (Central Processing Unit) and / or a GPU (Graphics Processing Unit). The processing unit 10 uses application software for constructing MR stored in the storage unit 11, a computer program (program product) P1, to construct mixed reality and execute processing for presenting the visual appearance of an object 4 in real space that the user can view.

[0018] The storage unit 11 is a non-volatile memory such as a flash memory, and stores a computer program P1, a learning model 110 for detecting an object 4 recognized by a user, a sensory DB 111, and other data referenced by the processing unit 10.

[0019] The computer program P1 may be provided by a non-transitory recording medium M1 on which the computer program is readably recorded. The storage unit 11 stores the computer program read from the recording medium M1 by a reading device (not shown). The recording medium M1 may be, for example, a magnetic disk, an optical disk, or a semiconductor memory. Alternatively, the computer program may be downloaded from an external server connected to a communications network and stored in the storage unit 11. The computer program P1 may be a single computer program or may be composed of multiple computer programs, and may be executed on a single computer or on multiple computers interconnected by a communications network.

[0020] FIG. 3 is a conceptual diagram illustrating the configuration of the learning model 110. The learning model 110 is an object detection model including, for example, a convolutional neural network (CNN) trained by deep learning. A general-purpose object detection model capable of detecting any general object and recognizing its type may be used as the learning model 110. The learning model 110 includes an input layer 110a to which image data captured by the imaging unit 14 is input, an intermediate layer 110b that extracts features of the image data, and an output layer 110c that outputs an object detection result. The learning model 110 is, for example, a YOLO model. The object detection result includes, for example, the center coordinate position and horizontal and vertical sizes of a bounding box surrounding the object 4, an object detection score indicating the likelihood that the image surrounded by the bounding box is an image of the object 4, and a class score indicating the likelihood that the object 4 belongs to a specific type of class. The class of the object 4 indicates the type of object 4 and is associated with the object ID. The class may be the object ID itself.

[0021] Although YOLO has been described as an example of the learning model 110, the learning model 110 may be configured using R-CNN, Fast R-CNN, Faster R-CNN, other CNNs, etc. Also, the learning model 110 may use algorithms such as decision trees, random forests, and SVMs (Support Vector Machines). Furthermore, the learning model 110 may be configured by combining multiple of the above-mentioned algorithms.

[0022] The sensory DB 111 is a storage that stores tactile data of an object 4 that is visually recognized by the user. Each time an object 4 is detected, the tactile data of the object 4 may be acquired from the object DB 311 of the server device 3 and stored in the sensory DB 111, or the tactile data of multiple objects 4 acquired in advance may be stored in the sensory DB 111.

[0023] The first communication unit 12 includes a communication circuit that realizes communication with the server device 3 via a network N including the Internet N1 or a carrier network N2. Specifically, the first communication unit 12 may be a wireless communication device that connects to the carrier network N2, or may be a wireless communication device for Wi-Fi. The processing unit 10 transmits and receives data to and from the server device 3 via the first communication unit 12.

[0024] The second communication unit 13 is a communication module for short-range wireless communication, for example, Bluetooth (registered trademark). The communication module includes a communication circuit. The processing unit 10 transmits and receives data to and from the tactile presentation device 2 via the second communication unit 13.

[0025] The imaging unit 14 is a camera that captures an image of the field of view of a user wearing the MR device 1, and includes an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) or a CCD (Charge Coupled Device). The imaging unit 14 captures a moving image based on a control instruction from the processing unit 10.

[0026] The see-through display 15 has a see-through holographic lens and a projection device, and creates a mixed reality in which virtual objects are visualized in the real space that the user sees through the see-through display 15.

[0027] The eye tracking sensor 16 includes an infrared camera that captures an image of the user, and measures the direction of the user's gaze based on the captured infrared image.

[0028] 4 is an explanatory diagram showing an example of the contents of the sensory DB 111. The sensory DB 111 stores a plurality of pieces of tactile data that differ depending on the type of object 4. The tactile data includes an object ID, the name of the object 4, and a tactile data table. The tactile data table stores the displacement amount (angle) of the displacement unit 202 and the value of the current supplied to the MRF device 24 of the tactile presentation device 2 in association with each other.

[0029] Fig. 5 is a block diagram showing the configuration of the tactile presentation device 2. The tactile presentation device 2 includes a gripping body 201 that is gripped and operated by a user as shown in Fig. 1, a control unit 20, a tactile data storage unit 21, a communication unit 22, a power supply unit 23, an MRF device 24, and a displacement sensor 25, all of which are shown in Fig. 5. Note that circuits related to control and power supply of the control unit 20, the power supply unit 23, etc. shown in Fig. 2 may be configured integrally with the gripping body 201, or may be configured separately and connected by wire or wirelessly.

[0030] The gripper 201 includes a hollow, disk-shaped casing, a displacement unit 202, a binder 203, and a link mechanism 204. The casing houses a magnetorheological fluid and a rotor having a rotating shaft. The rotor is a disk-shaped member that rotates inside the casing filled with magnetorheological fluid. The rotor is subjected to rotational resistance force by the magnetorheological fluid. The gripper 201 is also provided with an electromagnet (not shown) for applying a magnetic field to the magnetorheological fluid. The electromagnet, together with the magnetorheological fluid and the rotor, constitutes the MRF device 24. Specifically, the electromagnet has a yoke arranged to sandwich the disk-shaped rotor with a gap therebetween, and a coil attached to the yoke. When a current flows through the coil, a magnetic field is generated, which changes the viscosity (shear stress) of the magnetorheological fluid, thereby applying rotational resistance force to the rotating shaft.

[0031] The displacement portion 202 is a band-shaped member curved along a portion of the circumferential surface of the casing. The displacement portion 202 is formed of a flexible but highly rigid material. The base end of the displacement portion 202 is rotatably supported on the casing by a support shaft so that the tip of the displacement portion 202 can move toward and away from the casing. A binder 203 for binding the user's fingers is provided on the outer surface of the tip of the displacement portion 202. The binder 203 is a ring-shaped band-shaped member, such as a cloth tape. The displacement portion 202 may have a surface covered with a material that provides a variety of textures, such as silicone rubber or a fur-like material. The tip of the displacement portion 202 and the rotor's rotation shaft are connected by a link mechanism 204. The link mechanism 204 converts the movement of the displacement portion 202 toward and away from the casing into rotational movement of the rotation shaft and transmits it.

[0032] 1, a user holds the gripping body 201 with, for example, the thumb and middle finger, and inserts the index finger or other fingers into the binder 203 along the displacement portion 202. The user can move the displacement portion 202 by pushing the index finger in, or can extend the index finger to move the displacement portion 202 away from the gripping body 201.

[0033] The control unit 20 includes a control processor 20a such as a CPU or an MPU (Micro-Processing Unit), memories such as a ROM (Read Only Memory) 20b and a RAM (Random Access Memory) 20c, and an I / O 20d. The control unit 20 is, for example, a microcontroller. The control unit 20 controls each component based on a control program P202 stored in the ROM 20b to provide tactile sensations. The control program P202 may be a control program P202 stored in a storage medium (not shown) that is read by the control unit 20 and copied to a ROM 20b such as a flash memory. The control program P202 may also be a program downloaded from the server device 3 or another program server device and stored in an executable format.

[0034] The haptic data storage unit 21 is an auxiliary memory for the control unit 20, and stores haptic data, which is control data for the MRF device 24, in a rewritable manner.

[0035] The communication unit 22 is a communication module for short-range wireless communication, such as Bluetooth (registered trademark). The communication module includes a communication circuit. The control unit 20 transmits and receives data to and from the MR device 1 via the communication unit 22.

[0036] The control unit 20 is connected to the power supply unit 23, the MRF device 24, and the displacement sensor 25 via the I / O 20d, and exchanges signals with them.

[0037] The power supply unit 23 includes a rechargeable battery. When the power supply unit 23 is turned on, it supplies power to each component and the MRF device 24.

[0038] The displacement sensor 25 measures the position (angle) of the displacement portion 202 and outputs the position (angle) to the control portion 20. The displacement sensor 25 measures and outputs the displacement of the displacement portion 202 as an angle. The displacement sensor 25 may be configured with a plurality of sensors such as a gyro sensor and an acceleration sensor.

[0039] In the tactile presentation device 2 configured as shown in FIG. 5, when the displacement unit 202 is operated by a user, the displacement of the displacement unit 202 is transmitted to the rotational axis of the rotor of the MRF device 24 via the link mechanism 204. When the MRF device 24 is not operating, i.e., when the control current is zero, the rotational axis rotates freely, and the displacement unit 202 moves without resistance. On the other hand, when the MRF device 24 is operating and a control current is flowing, the viscosity (shear stress) of the magnetorheological fluid inside the MRF device 24 changes depending on the magnitude of the current flowing to the MRF device 24. The control unit 20 can change the resistance force against the displacement unit 202 and how it appears by continuously changing the magnitude of the current to the MRF device 24 or oscillating the current value at a predetermined frequency. In other words, the control unit 20 can present any tactile sensation by controlling the waveform and current value of the current flowing to the MRF device 24.

[0040] In this way, the tactile presentation device 2 can present a slippery tactile sensation by varying the resistance (current value) according to the amount of pressure applied to the displacement section 202, or a firm tactile sensation by increasing the resistance as the amount of pressure applied increases, or a crunchy tactile sensation by repeatedly varying the resistance or by repeatedly turning the resistance on and off using a square wave.

[0041] 6 is a block diagram showing the internal configuration of the server device 3. The server device 3 is a haptic data providing server capable of transmitting and receiving data to and from the MR device 1. In the following description, the server device 3 will be described as a single haptic data providing server, but it may also be configured such that multiple server computers are communicatively connected via a network N to perform distributed processing.

[0042] The server device 3 includes a server processing unit 30, a storage unit 31, and a server communication unit 32. The server processing unit 30 is a processor that uses a CPU and / or a GPU. The server processing unit 30 executes processing as a haptic data providing server based on a server program P3 stored in the storage unit 31.

[0043] The storage unit 31 is a non-volatile memory such as a hard disk or a flash memory. The storage unit 31 stores data referenced by the server processing unit 30. The storage unit 31 stores a server program P3. The server program P3 may be a server program P3 stored in a recording medium M3 that the server processing unit 30 reads and copies to the storage unit 31. The server program P3 may also be a program downloaded by the server processing unit 30 from another program server device via the server communication unit 32 and stored therein.

[0044] The storage unit 31 constitutes an object DB 311 and stores haptic data for a plurality of objects 4. The storage unit 31 manages the haptic data in association with a different object ID for each type of object 4.

[0045] The server communication unit 32 includes a communication circuit that realizes communication with the MR device 1 via the network N. The server processing unit 30 transmits and receives data to and from the MR device 1 via the server communication unit 32.

[0046] 7 is a flowchart showing an example of the processing procedure of the MR device 1 and the server device 3, and FIG. 8 is a schematic diagram showing mixed reality. Here, it is assumed that communication between the MR device 1 and the tactile presentation device 2 has been established.

[0047] The processing unit 10 of the MR device 1 captures an image in the user's field of view using the imaging unit 14 (step S111). The processing unit 10 inputs the captured image data into the learning model 110 to detect one or more objects 4 included in the captured image (step S112). In other words, one or more objects 4 in the real world within the user's field of view are extracted.

[0048] Next, the processing unit 10 determines whether or not a new object 4 has been detected in steps S111 and S112 (step S113). That is, the processing unit 10 confirms that the object 4 is not an object 4 that was previously acquired from the server device 3 and for which the storage unit 11 already stores haptic data.

[0049] It should be noted that haptic data does not exist for all objects 4, and it may be the case that haptic data cannot be acquired despite a request to the server device 3. In this case, the processing unit 10 may store the object 4 for which haptic data cannot be acquired in the storage unit 11. This can avoid unnecessary processing such as repeatedly requesting haptic data that cannot be acquired just because the haptic data is not available in the storage unit 11.

[0050] If it is determined that a new object 4 has not been detected (step S113: NO), the processing unit 10 advances the process to step S117.

[0051] If it is determined that a new object 4 has been detected (step S113: YES), the processing unit 10 sends a request to the server device 3 via the first communication unit 12 for haptic data of the newly detected object 4 (step S114). The request data includes an object ID indicating the newly detected object 4. In the first embodiment, the correspondence between the haptic data of an object and the object ID is assumed to be shared between the MR device 1 and the server device 3. The server device 3 stores the haptic data of an object in association with the object ID of the object. Meanwhile, the MR device 1 detects an object using the learning model 110 and obtains the object ID of the object. The object ID used by the server device 3 and the MR device 1 is shared and identifies the same object. Although the example of detecting an object on the MR device 1 side has been mainly described, the learning model may be provided on the server device 3 side. The MR device 1 transmits a captured image to the server device 3. The server device 3 receives the captured image transmitted from the MR device 1, inputs the received captured image into a learning model, thereby detecting the object, reads haptic data associated with the detected object from the storage unit 31, and transmits the data to the MR device 1.

[0052] The server processing unit 30 of the server device 3 receives the data request (step S131), and reads the haptic data of the requested object 4 from the object DB 311 in response to the data request (step S132).

[0053] Then, the server processing unit 30 transmits the read haptic data to the MR device 1 (step S133). If it is determined that there is no haptic data for the requested object 4, the server processing unit 30 transmits an error response to the MR device 1.

[0054] The processing unit 10 of the MR device 1 receives the haptic data (step S115) and stores the received haptic data in the storage unit 11 (step S116). If an error response is received from the server device 3, the processing unit 10 stores in the storage unit 11 a message indicating that no haptic data exists for the object 4 for which haptic data has been requested.

[0055] Next, the processing unit 10 measures the direction of the user's gaze using the eye tracking sensor 16 (step S117) and identifies the object 4 that the user is gazing at (step S118). That is, the processing unit 10 identifies the object 4 that the user is directing their gaze at.

[0056] Next, the processing unit 10 determines whether the object 4 that provides a tactile sensation will be changed (step S119). That is, it determines whether the object 4 that provides a tactile sensation will be changed and whether it is necessary to transmit tactile data to the tactile presentation device 2. Note that when transmitting tactile data to the tactile presentation device 2 for the first time, the processing unit 10 determines that the object 4 that provides a tactile sensation will be changed.

[0057] When it is determined that the object 4 to be tactilely presented will be changed (step S119: YES), the processing unit 10 transmits tactile data of the object 4 visually recognized by the user to the tactile presentation device 2 via the second communication unit 13 (step S120). The tactile presentation device 2 stores the tactile data transmitted from the MR device 1 and presents the tactile sensation of the object 4. In other words, the tactile sensation of the object 4 is reproduced.

[0058] When the processing of step S120 is completed or when it is determined that the object 4 will not be changed (step S119: NO), the processing unit 10 projects and displays a first virtual bounding box (first virtual object) 51 surrounding the object 4 that the user is viewing onto the mixed reality screen using the see-through display 15 (step S121), as shown in Fig. 8. That is, the first virtual bounding box 51 surrounding the object 4 that the user is directing his or her gaze is displayed. The first virtual bounding box 51 is a rectangular frame image that surrounds the object 4.

[0059] As shown in FIG. 8, the processing unit 10 may project and display a virtual pointer 50 representing the user's viewpoint.

[0060] 8, the processing unit 10 projects and displays a second virtual bounding box (second virtual object) 52 surrounding another object 4 that is not visually recognized by the user and for which haptic data is available, on the mixed reality screen using the see-through display 15 (step S122). The second virtual bounding box 52 is a rectangular frame image surrounding the other object 4. However, the first virtual bounding box 51 and the second virtual bounding box 52 are bounding boxes displayed in different modes. For example, they are bounding boxes with different colors or line types. The user can recognize that the haptic sensation of the object 4 surrounded by the first virtual bounding box 51 is being presented by the tactile presentation device 2.

[0061] For objects 4 within the user's field of view, such as those shown in FIG. 8, that do not have tactile data, a third virtual bounding box (not shown) that looks different from the first and second virtual bounding boxes 51, 52 may be displayed overlaid, or nothing may be displayed overlaid.

[0062] The processing unit 10 may project and display the recognition result of the object 4 enclosed by the first virtual bounding box 51, that is, the name of the object 4, its description, notes, etc., on the mixed reality screen using the see-through display 15. The user can recognize the type of object 4 that is recognized and tactilely presented by the MR device 1.

[0063] Thereafter, the MR device 1 repeatedly executes the processes of steps S111 to S122, thereby providing a tactile sensation for the object 4 that the user is looking at. If the user looks at a different object 4, a tactile sensation for that object 4 is provided.

[0064] As described above, the haptic presentation method, MR device 1, and computer program according to the first embodiment can construct a mixed reality and present the haptic sensation of a real object 4 that is visually recognized by the user. The user can experience the haptic sensation of the object 4 that is difficult to touch in reality.

[0065] Furthermore, it is possible to reproduce the tactile sensation of the object 4 that the user directs his / her gaze among the objects 4 included in the user's field of view. In other words, the user can change the object 4 to which the tactile sensation is presented simply by directing his / her gaze to a desired object 4.

[0066] Furthermore, the user can recognize the object 4 selected by the line of sight and for which a tactile sensation is presented, by using the first virtual bounding box 51.

[0067] Furthermore, the user can recognize the object 4 that allows the user to experience the tactile sensation by directing his or her gaze thereto.

[0068] The learning model 110 may be configured to recognize and detect the details of the object 4, in other words, the multiple object parts that make up the object 4. When the user approaches the object 4 and the details of the object 4 are captured, the learning model 110 may be configured to detect the object part that the user is viewing and present different tactile sensations depending on the object part.

[0069] It is also possible to provide a tactile sensation to objects 4 in television images, books, etc. For example, if a beer can is shown on television, it is possible to recognize the can and present the tactile sensation of the can to the user.

[0070] Furthermore, in the first embodiment, the tactile presentation device 2 using a magnetorheological fluid is exemplified, but the type of the tactile presentation device 2 is not particularly limited. The tactile presentation device 2 may be one that generates a force sensation in the user by a motor, a piezoelectric element, ultrasonic waves, or the like.

[0071] Furthermore, in this embodiment, an example of detecting object 4 using learning model 110 has been described, but it may also be configured to calculate image features on a rule basis and recognize object 4 contained in the captured image.

[0072] In addition, although an example has been described in which the user's line of sight is measured to identify the object 4 that the user is viewing, the object 4 that is in the center of the user's field of view may also be configured to be identified as the object 4 that the user is viewing.

[0073] (Embodiment 2) The tactile presentation system 200 in the second embodiment differs from the first embodiment in that the system is configured using a tablet terminal 6 instead of the MR device 1. The other configurations of the tactile presentation system 200 are similar to those of the tactile presentation system 100 according to the first embodiment, so similar parts are denoted by the same reference numerals and detailed descriptions thereof will be omitted.

[0074] FIG. 9 is a schematic diagram showing a tactile presentation system 200 according to the second embodiment, and FIG. 10 is a block diagram showing the internal configuration of a tablet terminal 6 according to the second embodiment. The tablet terminal 6 includes a processing unit 60, a storage unit 61, a first communication unit 62, a second communication unit 63, an imaging unit 64, and a display unit 65. The configurations of the processing unit 60, the storage unit 61, the first communication unit 62, and the second communication unit 63 are the same as those in the first embodiment. The contents of the computer program P61, the learning model 610, the sensory DB 61, and the recording medium M61 stored in the storage unit 61 are also the same as those in the first embodiment. The imaging unit 64 is an in-camera and can capture an image of the face of a user looking at the display unit 65. The hardware configuration of the imaging unit 64 is the same as that in the first embodiment. The display unit 65 is a display such as a liquid crystal display or an organic EL (Electro Luminescence) display.

[0075] FIG. 11 is a flowchart showing an example of the processing procedure of the tablet terminal 6 and the server device 3, and FIG. 12 is a schematic diagram showing the display screen of the tablet terminal 6. It is assumed that the tablet terminal 6 is displaying e-book content such as a novel, picture book, or manga. The processing unit 60 of the tablet terminal 6 acquires display data related to an image displayed on the display unit 65 (step S211). The display data includes image data, character data, HTML data, etc.

[0076] Next, the processing unit 60 detects object 4 based on the display data (step S212). When displaying text such as a novel, the processing unit 60 extracts the text of the page displayed on the display unit 65 and detects object 4 expressed by the text. When the text is a proper noun, the thing represented by the proper noun may be detected as object 4. Furthermore, adjectives or onomatopoeias or other expressions related to the sense of touch may also be detected as object 4. When the display data is image data of text, the text may be converted into text by OCR, and object 4 may be detected in the same manner. When the display data is image data including figures, illustrations, etc., object 4 may be detected by inputting the image data into the learning model 110.

[0077] Thereafter, the same processes as steps S113 to S116 in the first embodiment are executed (steps S213 to S216). Next, the processing unit 60 measures the user's line of sight by capturing an image of the user with the imaging unit 64 and tracing the user's line of sight (step S217). For example, based on the traced user's line of sight and the positional relationship between the imaging unit 64 and the display unit 65 provided in the tablet terminal 6, the processing unit 60 calculates the position of the user's viewpoint on the display unit 65. In the display unit 65, an image portion within a predetermined range including the user's viewpoint is recognized, and the object 4 is identified based on display data corresponding to the image portion (step S218).

[0078] The processing unit 60 determines whether the object 4 for providing a tactile sensation will be changed (step S219). If it is determined that the object 4 for providing a tactile sensation will be changed (step S219: YES), the processing unit 60 transmits the tactile data of the object 4 identified in step S218 to the tactile presentation device 2 via the second communication unit 63 (step S220). If the processing of step S220 is completed or if it is determined that the object 4 will not be changed (step S219: NO), the processing unit 60 displays a first virtual bounding box 51 that surrounds the object 4 that the user is viewing on the display unit 65, as shown in FIG. 12 (step S221). That is, the first virtual bounding box 51 that surrounds the object 4 that the user is looking at is displayed. The first virtual bounding box 51 is a rectangular frame image that surrounds the object 4.

[0079] 12, the processing unit 60 displays on the display unit 65 a second virtual bounding box 52 that surrounds another object 4 that is not visually recognized by the user and for which tactile data is available (step S222). The second virtual bounding box 52 is a rectangular frame image that surrounds the other object 4. However, the first virtual bounding box 51 and the second virtual bounding box 52 are bounding boxes displayed in different modes. For example, they are bounding boxes with different colors or line types. The user can recognize that the tactile sensation of the object 4 surrounded by the first virtual bounding box 51 is being presented by the tactile presentation device 2.

[0080] After completing the process of step S222, the processing unit 60 returns the process to step S211. Thereafter, the processing unit 60 repeatedly executes the processes of steps S211 to S222, thereby detecting the object 4 at which the user is looking and providing a tactile sensation.

[0081] The processing unit 60 may be configured to display the name of the detected object 4, its description, etc. on the display unit 65.

[0082] In this way, when text content such as a novel is displayed on the display unit 65, the tablet terminal 6 can present the tactile sensation of an object 4 displayed in the text at which the user is looking. For example, it is possible to present the tactile sensation of an object 4 expressed as a proper noun or the tactile sensation of an abstract object 4 expressed as an onomatopoeia.

[0083] When a comic is displayed on the display unit 65, the tablet terminal 6 analyzes the image of the frame that is attracting attention and can provide a tactile sensation appropriate to that frame. For example, if the frame is a scene of someone running on a sandy beach, the tactile sensation of sand can be provided through the tactile presentation device 2.

[0084] When an educational picture book is displayed on the display unit 65, the tablet terminal 6 identifies and analyzes the object 4 represented by the picture on which the gaze is focused, and provides the tactile sensation of the object 4. For example, if the picture is of thunder, the tactile sensation is provided as a high-speed vibration.

[0085] As described above, the tactile presentation method, tablet terminal 6, and computer program P1 according to the second embodiment can present the tactile sensation of the object 4 represented by the character or design at the location where the user is looking.

[0086] Although the example in which the tablet terminal 6 and the tactile presentation device 2 are separate entities has been described, the tactile presentation device 2 may be incorporated into the tablet terminal 6.

[0087] In addition, an example has been described in which all objects included in the display unit 65 are detected based on the display data, tactile data is obtained, and then objects within a predetermined range including the user's viewpoint are identified and tactile data is presented, but the configuration may also be such that tactile data is requested from the server device 3 and obtained each time an object is detected based on the display data within the predetermined range.

[0088] In this embodiment, in the case of text content such as novels or manga, a function for automatically reading the text content aloud can be added, and a tactile sensation corresponding to the words (object 4) being read aloud can be presented. The processing unit 60 displays a document corresponding to the text content on the display unit 65 and plays audio corresponding to the displayed document through a speaker (not shown). The playback speed is set based on a user's selection. In addition, the processing unit 60 displays, for example, a colored marker over the relevant portion of the document displayed on the display unit 65 so that the user can identify the portion that has been played and the portion that is currently being played. The portion where the colored marker is displayed can be assumed to be the portion that the user will visually recognize. The colored marker is an example of a display method for attracting the user's attention, and the present invention is not limited to the colored marker. In this example, the portion that is currently being played corresponds to the object that the user visually recognizes.

[0089] If the part being played corresponds to a word (object 4) to which a tactile sensation is to be imparted (corresponding to YES in step S119), tactile data corresponding to the word (object 4) is transmitted to the tactile presentation device 2 (corresponding to step S120). During generation of the word to which a tactile sensation is to be imparted, the color of the color marker may be a special color different from that of other words. This example can be applied not only to automatic reading of characters but also to automatic turning of pages. A tactile sensation is set for each page, and when the page moves to the next page, the tactile sensation set for the previous page is automatically provided through the tactile presentation device 2.

[0090] (Embodiment 3) FIG. 13 is a schematic diagram showing a tactile presentation system 300 according to the third embodiment. The tactile presentation system 300 in the third embodiment differs from the first or second embodiment in that the system is configured using an imaging device 7 instead of the MR device 1. The other configurations of the tactile presentation system 300 are the same as those of the tactile presentation system 100 according to the first embodiment or the tablet terminal 6 according to the second embodiment, and therefore, the same reference numerals are used for the same parts and detailed description thereof will be omitted.

[0091] The imaging device 7 according to the third embodiment includes a processing unit, a storage unit, a first communication unit, a second communication unit, an imaging unit 74, and a display unit 75, similar to those of the tablet terminal 6. However, the second communication unit is connected to the tactile presentation device 2 by wire.

[0092] The processing unit displays the captured image on the display unit 75 based on the image data captured by the imaging unit 74. The processing unit also detects objects included in the captured image by inputting the image data into a learning model. The processing unit requests and acquires tactile data of the object at the center of the captured image from the server device 3, and transmits the acquired tactile data to the tactile presentation device 2. The center of the captured image can be estimated as the location visually recognized by the user. In this example, the object at the center of the captured image corresponds to the object visually recognized by the user. The tactile presentation device 2 stores the tactile data transmitted from the MR device 1 and presents the tactile sensation of the object 4.

[0093] As described above, the tactile sensation presentation method, the imaging device 7, and the computer program according to the third embodiment can present the tactile sensation of the object 4 included in the captured image. The user can experience the tactile sensation of the object 4, which is difficult to touch in reality.

[0094] The means for solving the problems of the present disclosure are described below. (Appendix 1) An information processing device including a processing unit that executes processing related to tactile presentation using a tactile presentation device, The processing unit Identifying an object that is visually recognized by a user in real space; obtaining tactile data for presenting a tactile sensation of the identified object; The acquired tactile data is transmitted to a tactile presentation device. An information processing device that executes processing. (Appendix 2) The processing unit Imaging the user's field of view; Measure the user's gaze direction, Identifying the object that the user is viewing based on the captured image data and the measured gaze direction 10. The information processing device according to claim 1. (Appendix 3) The processing unit The operation of an augmented reality device including an imaging unit that captures an image of a user's field of view and an eye tracking sensor that measures the user's line of sight is controlled to capture an image and measure the line of sight. 10. The information processing device according to claim 1 or 2. (Appendix 4) The processing unit causing the augmented reality device to display a virtual object representing the identified object. 4. The information processing device according to claim 3. (Appendix 5) The processing unit When the captured image contains multiple objects, the haptic data causes the augmented reality device to display a virtual object representing a specified object. 5. The information processing device according to claim 3 or 4. (Appendix 6) The processing unit When a plurality of objects are included in the captured image corresponding to the field of view, a first virtual object representing an object in the user's line of sight is displayed on the augmented reality device, and a second virtual object representing another object is displayed on the augmented reality device. 6. An information processing device according to any one of Supplementary Note 3 to Supplementary Note 5. (Appendix 7) The processing unit Imaging the user's field of view; Based on the captured image data, an object at the center of the captured image is identified as the object visually recognized by the user. 7. An information processing device according to any one of Supplementary Note 1 to Supplementary Note 6. (Appendix 8) The processing unit Acquire display data relating to the image displayed on the display unit; Identifying an object viewed by a user based on the acquired display data 8. An information processing device according to any one of Supplementary Note 1 to Supplementary Note 7. (Appendix 9) The processing unit measuring a user's line of sight with respect to the screen of the display unit; Identifying an object viewed by the user based on display data corresponding to an image portion within a predetermined range including the user's viewpoint on the screen of the display unit. 9. The information processing device according to claim 8. [Explanation of symbols]

[0095] 1: MR device 2: Tactile presentation device 3: Server device 4: Object 10: Processing section 11: Storage section 12: First Communications Department 13: Second Communication Department 14: Imaging unit 15: See-through display 16: Eye tracking sensor 20: Control section 21: Tactile data storage unit 22: Communications Department 23: Power supply section 24: MRF device 25: Displacement sensor 30: Server processing section 31: Storage section 32: Server communication section 50: Virtual pointer 51: First virtual bounding box 52: Second virtual bounding box 110: Learning Model 6: Tablet device 7: Imaging device 111: Sensory DB M1: Recording medium P1: Computer Program

Claims

1. An information processing device including a processing unit that executes processing related to tactile presentation using a tactile presentation device, The processing unit Identifying an object that is visually recognized by a user in real space; obtaining tactile data for presenting a tactile sensation of the identified object; The acquired tactile data is transmitted to a tactile presentation device. An information processing device that executes processing.

2. The processing unit Imaging the user's field of view; Measure the user's gaze direction, Identifying the object that the user is viewing based on the captured image data and the measured gaze direction The information processing device according to claim 1 .

3. The processing unit The operation of an augmented reality device including an imaging unit that captures an image of a user's field of view and an eye tracking sensor that measures the user's line of sight is controlled to capture an image and measure the line of sight. The information processing device according to claim 2 .

4. The processing unit causing the augmented reality device to display a virtual object representing the identified object. The information processing device according to claim 3 .

5. The processing unit When the captured image contains multiple objects, the haptic data causes the augmented reality device to display a virtual object representing a specified object. The information processing device according to claim 3 .

6. The processing unit When a plurality of objects are included in the captured image corresponding to the field of view, a first virtual object representing an object in the user's line of sight is displayed on the augmented reality device, and a second virtual object representing another object is displayed on the augmented reality device. The information processing device according to any one of claims 3 to 5.

7. The processing unit Imaging the user's field of view; Based on the captured image data, an object at the center of the captured image is identified as the object visually recognized by the user. The information processing device according to claim 1 .

8. The processing unit Acquire display data relating to the image displayed on the display unit; Identifying an object viewed by a user based on the acquired display data The information processing device according to claim 1 .

9. The processing unit measuring a user's line of sight with respect to the screen of the display unit; Identifying an object viewed by the user based on display data corresponding to an image portion within a predetermined range including the user's viewpoint on the screen of the display unit. The information processing device according to claim 8 .

10. Identifying an object that is visually recognized by a user in real space; obtaining tactile data for presenting a tactile sensation of the identified object; The acquired tactile data is transmitted to a tactile presentation device, whereby a tactile sensation of the object visually recognized by the user is presented. Tactile presentation method.

11. A computer program for causing a computer to execute a process related to tactile presentation using a tactile presentation device, Identifying an object that is visually recognized by a user in real space; obtaining tactile data for presenting a tactile sensation of the identified object; The acquired tactile data is transmitted to a tactile presentation device. A computer program for causing the computer to execute a process.

Images