Information processing device and program

Abstract

To provide an information processing device and program capable of configuring a system capable of recognizing a shapes or the like of an object from a tactile sense.SOLUTION: In a system, an information processing device 1 includes an information acquisition unit 101, an image processing unit 103, a position processing unit 104, a visual information generation unit 105 and a tactile information generation unit 106. The information acquisition unit 101 acquires viewpoint information including information showing a visual field of a user, movement information showing movement of an irradiation object of an ultrasonic wave, and object information. The visual information generation unit 105 generates visual information obtained by overlapping a background image on an object image on the basis of a position calculated by the position processing unit 104, and outputting the generated visual information to a display device 26. The tactile information generation unit 106 generates tactile information for irradiating the user with an ultrasonic wave corresponding to the object on the basis of the position calculated by the position processing unit 104, the movement information, and the object information, and outputs the generated tactile information to an ultrasonic wave generation device 27.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to an information processing apparatus and a program.

Background Art

[0002] In recent years, the opportunities to use Augmented Reality (AR), Virtual Reality (VR), and Mixed Reality (MR) have been increasing. These technologies are for displaying images for human vision, but it is also desired to add the presentation of tactile sensations to them. For the presentation of tactile sensations, for example, those using ultrasonic waves have been proposed (for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] By the way, when presenting tactile information together with visual information, if the presented tactile sensation is simple enough to confirm the presence or absence of an object, it is difficult to use it to recognize the shape of the object from the tactile sensation or to operate the object.

[0005] In view of the above circumstances, the present invention aims to provide an information processing apparatus and a program that can configure a system capable of recognizing the shape of an object and the like from tactile sensations.

Means for Solving the Problems

[0006] According to one aspect of the present invention, an information processing device is provided. This information processing device comprises an information acquisition unit, an image processing unit, a position processing unit, a visual information generation unit, and a tactile information generation unit. The information acquisition unit is configured to acquire viewpoint information, motion information, and object information. The viewpoint information includes information indicating the user's field of view. The motion information is information indicating the motion of the object to be irradiated with ultrasound. The object information includes at least one of the object's coordinates, shape, surface condition, hardness, temperature, mass, and coefficient of friction. The image processing unit is configured to process object images. The object image is an image representing a virtual object and is updated based on the viewpoint information and motion information. The position processing unit is configured to calculate the position when superimposing a background image and an object image. The background image is an image corresponding to the user's field of view. The visual information generation unit is configured to generate visual information by superimposing the background image and the object image based on the position calculated by the position processing unit, and to output the generated visual information to a display device. The tactile information generation unit generates tactile information that irradiates the user with ultrasound appropriate to the object, based on the position calculated by the position processing unit, motion information, and object information, and is configured to output the generated tactile information to the ultrasound generator.

[0007] According to one aspect of the present invention, it becomes possible to recognize the shape of an object from the sense of touch. [Brief explanation of the drawing]

[0008] [Figure 1] This figure shows an example configuration of a system 10 including an information processing device 1 according to an embodiment of the present invention. [Figure 2] This is a diagram showing the configuration of the information processing device 1. [Figure 3] This is a block diagram showing the functional configuration of the information processing device 1. [Figure 4] This diagram shows an example configuration of a device including an operation information input device 3 and an ultrasonic generator 8. [Figure 5] Figure 4 shows an example of how to use the device. [Figure 6] This is a diagram showing an example of a user image. [Figure 7]This is a diagram showing an example of an object image. [Figure 8] This figure shows an example of an image displayed on the display device 7. [Figure 9] This diagram shows an example of an image that includes both permitted and prohibited objects. [Figure 10] This diagram shows an example of an image that includes both permitted and prohibited objects. [Figure 11] This diagram shows an example of the ultrasound irradiation position. [Figure 12] This diagram shows an example of the ultrasound irradiation position. [Figure 13] This diagram shows an example of a passive actuator. [Figure 14] This is a diagram illustrating the collaborative processing unit 109. [Modes for carrying out the invention]

[0009] Embodiments of the present invention will be described below with reference to the drawings. The various features shown in the embodiments below can be combined with each other.

[0010] Incidentally, the program for implementing the software appearing in this embodiment may be provided as a non-transitory computer-readable medium, or it may be provided so that it can be downloaded from an external server, or it may be provided so that the program is launched on an external computer and its functions are realized on a client terminal (so-called cloud computing).

[0011] Furthermore, in this embodiment, "part" may include, for example, hardware resources implemented by circuits in a broad sense, and the information processing of software that can be specifically realized by these hardware resources. In addition, various types of information are handled in this embodiment, and these types of information can be represented, for example, by the physical values ​​of signal values ​​representing voltage and current, the high or low values ​​of signal values ​​as a set of binary bits composed of 0s or 1s, or by quantum superposition (so-called qubits), and communication and calculations can be performed on circuits in a broad sense.

[0012] Furthermore, a circuit in a broad sense is a circuit realized by combining at least a suitable combination of circuits, circuits, processors, and memory. In other words, it includes application-specific integrated circuits (ASICs), programmable logic devices (for example, simple programmable logic devices (SPLDs), complex programmable logic devices (CPLDs), and field programmable gate arrays (FPGAs)), etc.

[0013] 1. Overall Structure Figure 1 shows an example of the configuration of a system 10 including an information processing device 1 according to an embodiment of the present invention. As shown in the figure, the system 10 is configured by connecting the information processing device 1 with a viewpoint information input device 21, an action information input device 22, an object information input device 23, a temperature information input device 24, an audio input device 25, a display device 26, an ultrasonic generator 27, an acoustic output device 28, and an airflow output device 29. These devices may be interconnected by communication means and may be distributed. Furthermore, the system 10 may use multiple equivalent systems, interconnected by communication means, and perform mutual interaction.

[0014] The viewpoint information input device 21 inputs viewpoint information indicating the user's field of view into the information processing device 1, and is, for example, a camera. This viewpoint information input device 21 can follow the user's movement and may change its posture and position, which may be realized, for example, by being integrated with an actuator.

[0015] The motion information input device 22 inputs, as motion information, the movement of the user or an object to be irradiated with ultrasonic waves, specifically, the movement of exposed parts of the skin such as the hand including the user's finger and the face, and the movement of an object that can be a passive actuator, into the information processing device 1, and is, for example, a sensor such as a camera. This motion information input device 22 can follow the user's movement and may change its posture and position, which may be realized, for example, by being integrated with an actuator.

[0016] The object information input device 23 inputs object information indicating an object to be virtually displayed into the information processing device 1, and is, for example, a server or the like that can communicate with the information processing device 1 via a network.

[0017] The temperature information input device 24 inputs temperature information indicating the temperature of the surface to be irradiated with ultrasonic waves, and is, for example, a thermal camera. This temperature information input device 24 can follow the user's movement and may change its posture and position, which may be realized, for example, by being integrated with an actuator. Note that the temperature information input device 24 can also be omitted.

[0018] The voice input device 25 inputs an instruction issued by the user by voice, and is, for example, a microphone. This voice input device 25 can follow the user's movement and may change its posture and position, which may be realized, for example, by being integrated with an actuator. Note that the voice input device 25 can also be omitted.

[0019] The display device 26 displays images such as AR, VR, and MR, and is, for example, a display. This display may be a stationary type, a goggle type worn by the user, or a portable type such as a smartphone or tablet. The display device 26 may follow the user's movements and change its posture and position, which may be realized, for example, by integrating it with an actuator. The images displayed by the display device 26 include videos and moving images.

[0020] The ultrasonic generator 27 irradiates ultrasound onto exposed parts of the user's skin (e.g., fingers, arms, or face) or objects that can be moved by ultrasound irradiation, and is, for example, a transducer array. This ultrasonic generator 27 may follow the user's movements and change its posture and position, which may be realized, for example, by integrating it with an actuator.

[0021] The sound output device 28 generates sound effects, etc., and is, for example, a speaker. This sound output device 28 may follow the user's movements and change its posture and position, and this may be realized, for example, by integrating it with an actuator. The sound output device 28 can also be omitted.

[0022] The airflow output device 29 generates airflow to give the user a sense of realism, and is, for example, a fan. This airflow output device 29 may follow the user's movements and change its posture and position, and this may be realized, for example, by integrating it with an actuator. It is also possible to use a fan or circulator that can be remotely controlled by a remote control as the airflow output device 29. The airflow output device 29 can also be omitted. Furthermore, it is possible to use the ultrasonic generator 27 as the airflow output device 29 and control the airflow by appropriately controlling the spatiotemporal distribution of the ultrasonic waves.

[0023] 2. Configuration of the information processing device Next, the configuration of the information processing device 1 will be described. Figure 2 shows the configuration of the information processing device 1. As shown in the figure, the information processing device 1 has a processing unit 11, a storage unit 12, a temporary storage unit 13, an external device connection unit 14, and a communication unit 15, and these components are electrically connected inside the information processing device 1 via a communication bus 16. The information processing device 1 also has a power supply unit 17.

[0024] The processing unit 11 is realized by, for example, a central processing unit (CPU) that operates according to a predetermined program stored in the memory unit 12, or by a programmable logic circuit configured according to a predetermined program stored in the memory unit 12, or by a combination thereof, and realizes various functions.

[0025] The memory unit 12 is a non-volatile storage medium that stores various types of information. This can be implemented using a storage device such as a hard disk drive (HDD) or a solid state drive (SSD). The memory unit 12 can also be located in a separate device that can communicate with the information processing device 1.

[0026] The temporary storage unit 13 is a volatile storage medium. This is implemented by memory such as Random Access Memory (RAM), and it stores information (arguments, arrays, etc.) that is temporarily needed when the processing unit 11 is operating.

[0027] The external device connection section 14 is a connection section conforming to standards such as Universal Serial Bus (USB) and High-Definition Multimedia Interface (HDMI®), and allows connection of input devices such as keyboards and display devices such as monitors. The viewpoint information input device 21, motion information input device 22, temperature information input device 24, voice input device 25, display device 26, ultrasonic generator 27, acoustic output device 28, airflow output device 29, etc., are connected to the external device connection section 14.

[0028] The communication unit 15 is a communication means conforming to, for example, a Local Area Network (LAN) standard, and enables communication between the information processing device 1 and a network such as the Local Area Network or the Internet via it. The object information input device 23, an information processing device similar to the information processing device 1, and other devices are connected to this network in a communicative manner. Furthermore, a viewpoint information input device 21, an action information input device 22, a temperature information input device 24, an audio input device 25, a display device 26, an ultrasonic generator 27, an acoustic output device 28, an airflow output device 29, etc., may also be connected.

[0029] The power supply unit 17 supplies electrical energy to operate the viewpoint information input device 21, motion information input device 22, temperature information input device 24, voice input device 25, display device 26, ultrasonic generator 27, acoustic output device 28, airflow output device 29, etc., and supplies electrical energy obtained from commercial power or a battery. If the device to which the electrical energy is supplied is connected by a wire via the external device connection unit 14, the electrical energy is supplied using the wire. Power supply may be provided by a wireless power supply system instead of wired power supply. Also, if the device to which the electrical energy is supplied is connected wirelessly, power supply will be provided by a wireless power supply system. Of course, if the viewpoint information input device 21, motion information input device 22, temperature information input device 24, voice input device 25, display device 26, ultrasonic generator 27, acoustic output device 28, airflow output device 29, etc. have batteries or are powered by commercial power, power supply from the power supply unit 17 to those devices is not necessary. The power supply unit 17 may also be integrated with all other devices, such as a battery or a wireless power supply system, or with any of them.

[0030] Furthermore, the information processing device 1 can utilize a personal computer or other type of computer, and it is also possible to configure the information processing device 1 using multiple computers.

[0031] 3. Functions of the Information Processing Device 1 Next, the functions of the information processing device 1 will be explained. The information processing device 1 generates information to provide the user with visual and tactile sensations, or tactile sensations only, based on the input information. It operates according to the program, thereby realizing the various functional units described later. This program is a program that makes the computer operate or function as an information processing device.

[0032] Figure 3 is a block diagram showing the functional configuration of the information processing device 1. As shown in the figure, the information processing device 1 comprises an information acquisition unit 101, an operation control unit 102, an image processing unit 103, a position processing unit 104, a visual information generation unit 105, a tactile information generation unit 106, an acoustic information generation unit 107, an airflow information generation unit 108, and a linkage processing unit 109.

[0033] The information acquisition unit 101 is configured to acquire viewpoint information, motion information, object information, audio information, and temperature information. The viewpoint information includes information indicating the user's field of view and is an image captured by a viewpoint information input device 21, for example, a camera. The motion information is information indicating the motion of the object to be irradiated with ultrasound, for example, the user's motion or the motion of an object, and is input from the motion information input device 22. The object information includes at least one of the object's coordinates, shape, surface condition, hardness, temperature, mass, and coefficient of friction, and preferably includes the surface condition of the object, for example, texture or material, and is input from the object information input device 23. The audio information is the voice uttered by the user and is input from the audio input device 25. However, audio information is not essential. The temperature information is information indicating the surface temperature of the object to be irradiated with ultrasound and is input from the temperature information input device 24, for example, a thermal camera. The temperature of the surface to which ultrasound is irradiated, such as the exposed skin of the user's fingers or face, or the surface of an object, changes due to the irradiation of ultrasound. Therefore, the irradiation position and intensity of the ultrasound can be determined from the temperature change. This temperature change can be measured using a thermal camera or estimated using an infrared camera. It is not necessary to continuously measure the target surface with a thermal camera or similar device; the target surface can be measured once, and the discrepancy between the output of the sensor that measures the position, the position where the thermal camera observes the temperature change, and the position where the ultrasonic transducer should be forming the ultrasound focal point can be corrected. Alternatively, instead of directly measuring the temperature of the target surface to which ultrasound is irradiated, a focal point can be formed on, for example, a nylon polymer mesh, and this can be photographed with a thermal camera. The position where the ultrasonic transducer should be forming the focal point can then be compared with the position where the thermal camera observes the temperature change to make corrections.

[0034] Here, an example of the configuration of the motion information input device 22 and the ultrasonic generator 27 will be described. Figure 4 is a diagram showing an example of the configuration of a device including the motion information input device 22 and the ultrasonic generator 27. The device shown in Figure 4 consists of a stage 1001, rollers 1002, contact pads 1003, and a ring-shaped substrate 1004. Multiple motion information input devices 22 and ultrasonic generators 27 are provided on both the stage 1001 and the ring-shaped substrate 1004. The stage 1001 is rotatable by the rollers 1002 and rotates when an arm or hand is placed on the contact pad 1003 and the arm or hand is moved. Note that the stage 1001, rollers 1002, and contact pads 1003 are optional. The device shown in Figure 4 is used in the state shown in Figure 5. Figure 5 is a diagram showing an example of the use of the device shown in Figure 4.

[0035] The operation control unit 102 is configured to control whether or not a user can operate the object. For example, the operation control unit 102 allows operation in response to a press of an authorized object and denies operation in response to an operation of a disallowed object. Authorized and disallowed objects are displayed as virtual objects, similar to the object, as will be described later. The operation control unit 102 may also control whether or not an operation is permitted in response to audio information acquired by the information acquisition unit 101. The control of whether or not an operation is permitted by the operation control unit 102 is used, for example, to limit which users can operate the object when multiple users are looking at the same object. This prevents conflicting operations, such as one user attempting to rotate the object to the right while another user attempts to rotate it to the left. Furthermore, if two users are cooperating and pulling on the object, the control can be configured to allow operation by both users. In addition, even when there is only one user, operation can be denied to prevent deformation of the object if it is easily deformable.

[0036] The image processing unit 103 is configured to process both background images and object images. The background image is an image corresponding to the user's field of view and is updated based on viewpoint information. The background image may be an image captured by the camera, which is the viewpoint information input device 21, or it may be an image that has been captured or generated in advance. The image processing unit 103 may also process user images.

[0037] The user image is used when the camera, which is the viewpoint information input device 21, captures an image other than the one captured by the camera, and is a pre-captured or generated image. It represents the user's hand, including their fingers, and is updated based on movement information. For example, the user image may look like the one shown in Figure 6. This user image is updated when the user moves their fingers, etc.

[0038] The object image is an image representing a virtual object and is updated based on viewpoint information and motion information. The object image may look like the one shown in Figure 7. This object image is updated based on motion information indicating the user's attempt to manipulate the object, and reflects the state after one of the following operations has been performed: movement, rotation, deformation, division, or merging. Of course, depending on the object, it may not be possible to perform operations such as division, and it is also possible to perform operations other than those shown here, such as deleting a part of the object.

[0039] The position processing unit 104 is configured to calculate the position when superimposing the background image processed by the image processing unit 103, the user image, and the object image. Of course, if a user image is not used, superimposing the user image is unnecessary. Furthermore, the object information may include environment-specific information for each of multiple environments. In this case, the position processing unit 104 calculates the position when superimposing the background image and the object image to correspond to the environment corresponding to the user's field of view, based on the environment-specific information and viewpoint information. As a result, differences may arise, such as the object being able to be superimposed upright on a flat surface, but not on a surface with a certain degree of inclination.

[0040] The visual information generation unit 105 generates visual information by overlaying the background image processed by the image processing unit 103, the user image, and the object image as needed, based on the position calculated by the position processing unit 104, and is configured to output the generated visual information to the display device 26. The image displayed on the display device 26 based on the visual information generated by the visual information generation unit 105 is, for example, like the one shown in Figure 8. In Figure 8, the background is shown as plain for simplification. This image is updated in response to changes in the user's viewpoint or user actions.

[0041] Furthermore, the visual information generation unit 105 can include in the visual information an authorized object corresponding to an authorized operation and a denied object corresponding to an unauthorized operation. In other words, the image displayed on the display device 26 can display both authorized and denied objects. This image may look like the one shown in Figures 9 and 10. Figures 9 and 10 are examples of images that include both authorized and denied objects. Figure 9 shows the image immediately after the authorized object 1101 is pressed, and Figure 10 shows the image immediately after the denied object 1102 is pressed. The image shown in Figure 10 is also the image seen by a user viewing the object from a position relative to the user viewing the image shown in Figure 9.

[0042] The tactile information generation unit 106 generates tactile information that irradiates the user with ultrasound appropriate to the object, based on the position calculated by the position processing unit 104, the operation information, and the object information, and is configured to output the generated tactile information to the ultrasound generator 27. Specifically, the tactile information generation unit 106 has a sensitivity pattern determination unit 106a and a drive signal determination unit 106b. The sensitivity pattern determination unit 106a determines a sensitivity pattern that indicates the type of touch to be presented to the user, which is determined by the object information or the application software described later. The type of touch is expressed by words such as "smooth" or "rough". The drive signal determination unit 106b generates a drive signal that determines the drive pattern of ultrasound to be irradiated to the user or an object that can be a passive actuator, based on the touch determined by the sensitivity pattern determination unit 106a, the position calculated by the position processing unit 104, and the operation information, and outputs it to the ultrasound generator 27.

[0043] The tactile information, or driving signal, has a temporal pattern in at least one of the following: the amplitude variation frequency of the ultrasound, its intensity, its phase, and its irradiation position. For example, as shown in Figures 11 and 12, there are multiple ultrasound irradiation positions, and these are changed over time. This temporal pattern is also used to control the tactile sensation of the presented stimulus. For example, by changing the temporal pattern at a high frequency, a stimulus accompanied by vibration is presented, while by changing it at a low frequency, a static stimulus (with little to no vibration) is presented.

[0044] Figures 11 and 12 show examples of ultrasound irradiation positions. In the example shown in Figure 11, ultrasound is irradiated onto area 1202 to allow the user's finger 1201 to feel surface 1301, while in the example shown in Figure 12, ultrasound is irradiated onto area 1203 to allow the finger 1201 to feel surface 1302. The tactile sensation of the presented stimulus can also be changed with these contact stimuli with surfaces. For example, by using the aforementioned stimuli that do not involve strong vibrations, the contact angle with the surface can be clearly presented.

[0045] Figure 13 shows an example of a passive actuator. The actuator 1401 shown in the figure is attached to the user's finger 1402 and can provide tactile sensations to the user by emitting ultrasound from the direction of arrow A in the figure. By changing the ultrasound irradiation pattern, it is possible to present vibrations or static forces. The actuator 1401 may also have markers or the like attached to the position where the ultrasound is emitted. Passive actuators are not limited to finger-mounted forms like actuator 1401, but can also be used for objects that come into contact with human skin, such as sunglasses and clothing, or objects that people grasp, such as cups and smartphone covers. In all of these cases, ultrasound is irradiated onto a passive mechanism. Furthermore, this ultrasound-driven passive mechanism may be used in combination with a small battery or electronic components.

[0046] The patterns included in tactile information are patterns corresponding to the shape of the object, or patterns corresponding to the surface condition or material of the object. By controlling the spatiotemporal distribution of the ultrasound waves presented, the characteristics of the object, such as its hardness, softness, surface texture, ease of movement (weight), and temperature, can be tactilely reproduced and presented to the user. Regarding hardness, for example, if the object is easily deformable, such as clay, the ultrasound waves can be initially irradiated only on a part of the user's finger, and then simultaneously on multiple parts of the finger, allowing the user to experience the sensation of their finger sinking into the clay. If the object is made of a hard material, the intensity of the ultrasound waves can be increased according to the force with which the user tries to push their finger in, preventing the user's finger from being pushed in. Regarding texture, for example, when a user lightly strokes the surface, the stimulation pattern applied to the finger can be changed over time according to the speed of the finger and the state of the surface (e.g., spatial wavelength), thereby presenting the tactile sensation of stroking the surface of a specific object. Regarding the sense of weight, for example, when reproducing a heavy object (an object that is difficult to move), this can be presented by delaying the movement of the stimulus position in time in relation to the movement of the fingertip that occurs when moving the object. This time delay in the movement of the stimulus position gives the user the experience that the object did not move (or was difficult to move) even though force was applied. Conversely, if the stimulus position follows without any delay, light objects can also be represented. Regarding the feeling of coldness of an object, this can be presented, for example, by spraying mist at the position where ultrasound is irradiated and allowing the mist to vaporize. Regarding the feeling of heat, this can be presented by attaching a structure that absorbs sound waves, such as a structure or mesh material with an air gap pattern sufficiently smaller than the wavelength of the ultrasound, to the position where the ultrasound is irradiated. In addition, tactile sensations are provided even to users who are not authorized to operate the device by the operation control unit 102. In addition, the patterns included in the tactile information do not necessarily include changes, and include cases where the intensity of the ultrasound and the irradiation position are constant. This is the pattern when a user touches a flat surface with their finger from the front.

[0047] Thus, tactile information is applied not only when a user touches an object, but also when an object touches the user. For example, if the object is a moving object or a living creature such as a virtual character, the tactile information described above can be presented not only to the fingers but also to exposed skin, including the face.

[0048] Furthermore, the tactile information generation unit 106 may correct the position and intensity of the ultrasonic irradiation based on the temperature information input from the temperature information input device 24. This utilizes the fact that the temperature of the user's fingers, etc., changes due to the irradiation of ultrasonic waves. Since the temperature of the user's fingers, etc., differs depending on whether the ultrasonic irradiation position is appropriate or inappropriate, the deviation in the irradiation position and intensity are detected and corrected. It may also be used to prevent pain sensation when ultrasonic waves enter gaps such as between fingers or in the palmar folds.

[0049] Incidentally, it is desirable that the haptic information generation unit 106 outputs haptic information within 50ms after the position of the user's hand or fingers is aligned with the position of the object, for example, within 10ms, 20ms, 30ms, 40ms, or any number in between. Furthermore, it is desirable that the user be able to be presented with a tactile sensation within these timeframes, that is, that the user can obtain a tactile sensation within 50ms after performing an action to touch the object.

[0050] Incidentally, tactile sensation reproduction does not necessarily require the use of ultrasonic mechanical stimulation alone; images and sounds may also be used as appropriate, or a combination of these may be employed. For this reason, the acoustic information generation unit 107 generates acoustic information that produces sound effects in response to the user's manipulation of the object, and activates the acoustic output device 28. This allows the user to perform operations more intuitively.

[0051] Similarly, the airflow information generation unit 108 generates airflow information indicating the state of the airflow generated by the airflow output device 29 or the ultrasonic generator 27. The airflow is used, for example, to present to the user the sensation of wind blowing when an object approaches or the sensation of breath being blown on a virtual character.

[0052] The linkage processing unit 109 is responsible for linking with other functions. Figure 14 is a diagram illustrating the linkage processing unit 109. As shown in the figure, the linkage processing unit 109 performs linkage processing with the function unit 120 and at least one of the other information processing devices 1. The function unit 120 is an independent function unit from the self-function unit 110, which includes the information acquisition unit 101, operation control unit 102, image processing unit 103, position processing unit 104, visual information generation unit 105, tactile information generation unit 106, acoustic information generation unit 107, airflow information generation unit 108, and linkage processing unit 109, and is realized by the operation of application software. This function unit 120 can replace at least a part of the information acquisition unit 101, image processing unit 103, position processing unit 104, visual information generation unit 105, and tactile information generation unit 106. This functional unit 120 is implemented, for example, by game application software. In this case, a controller (not shown) may be connected to the information processing device 1, and the connected controller can be used as a substitute for the operation information input device 22, etc. The cooperation processing unit 109 can also cooperate with multiple functional units implemented by multiple application software programs.

[0053] Furthermore, the other information processing devices 1 with which the linkage processing unit 109 interacts are the same as the information processing device 1, and can replace at least a part of the information acquisition unit 101, the image processing unit 103, the position processing unit 104, the visual information generation unit 105, and the tactile information generation unit 106. Of course, the other information processing devices 1 can also implement the functions realized by application software. It should also be noted that the linkage processing unit 109 can interact with multiple information processing devices 1.

[0054] 5. Others This system can be combined with existing actuators. For example, ultrasound could be emitted to a user holding a game controller, providing additional tactile sensations on top of those reproduced by the game controller.

[0055] Furthermore, the ultrasonic generator 27 may be covered with an object that can transmit sound waves, such as a mesh-like sheet. The surface of this sound-transmitting object may be colored as desired, thereby preventing any compromise in the design at the installation site.

[0056] The present invention may be provided in any of the following embodiments.

[0057] (1) An information processing device comprising an information acquisition unit, an image processing unit, a position processing unit, a visual information generation unit, and a tactile information generation unit, wherein the information acquisition unit is configured to acquire viewpoint information, motion information, and object information, the viewpoint information includes information indicating the user's field of view, the motion information is information indicating the motion of the object to be irradiated with ultrasound, the object information includes at least one of the object's coordinates, shape, surface condition, hardness, temperature, mass, and coefficient of friction, the image processing unit is configured to process an object image, the object image is an image representing a virtual object, and is updated based on the viewpoint information and the motion information, and the position processing An information processing device comprising: a position processing unit configured to calculate the position when superimposing a background image and an object image, wherein the background image is an image corresponding to the user's field of view; a visual information generation unit configured to generate visual information by superimposing the background image and the object image based on the position calculated by the position processing unit, and to output the generated visual information to a display device; and a tactile information generation unit configured to generate tactile information that causes the user to be exposed to ultrasonic waves corresponding to the object based on the position calculated by the position processing unit, the motion information, and the object information, and to output the generated tactile information to an ultrasonic generator.

[0058] (2) An information processing apparatus as described in (1) above, comprising a cooperation processing unit, wherein the cooperation processing unit performs cooperation processing between a functional unit and at least one of other information processing apparatuses, the functional unit is realized by the operation of application software and substitutes for at least a part of the information acquisition unit, the image processing unit, the position processing unit, the visual information generation unit, and the tactile information generation unit, and the other information processing apparatus substitutes for at least a part of the information acquisition unit, the image processing unit, the position processing unit, the visual information generation unit, and the tactile information generation unit.

[0059] (3) An information processing device according to (1) or (2) above, wherein the viewpoint information is an image captured by a camera, and the background image is the image captured.

[0060] (4) An information processing apparatus as described in (1) or (2) above, wherein the background image is an image that has been captured or generated in advance and is updated based on the viewpoint information, the image processing unit is configured to process the background image, the user image and the object image, the user image is an image representing the user's hand including the fingers and is updated based on the motion information, the position processing unit is configured to calculate the position when the background image, the user image and the object image are superimposed, and the visual information generation unit is configured to generate visual information by superimposing the background image, the user image and the object image based on the position calculated by the position processing unit and to output the generated visual information to a display device.

[0061] (5) An information processing device according to any one of (1) to (4) above, wherein the object information includes environment-dependent information corresponding to each of a plurality of environments, and the position processing unit calculates the position when superimposing the background image and the object image to correspond to the environment corresponding to the user's field of view, based on the environment-dependent information and the viewpoint information.

[0062] (6) An information processing device according to any one of (1) to (5) above, wherein the tactile information has a temporal pattern in at least one of the amplitude fluctuation frequency, intensity, phase, and irradiation position of the ultrasound.

[0063] (7) An information processing device as described in (6) above, wherein the pattern is a pattern corresponding to the shape of the object.

[0064] (8) An information processing device according to (6) or (7) above, wherein the pattern is a pattern corresponding to the surface condition or material of the object.

[0065] (9) An information processing device according to any one of (1) to (8) above, wherein the image processing unit updates the object image to a state in which one of the operations of moving, rotating, deforming, dividing, or combining the object has been performed on the object based on the operation information.

[0066] (10) An information processing device as described in (9) above, comprising an acoustic information generation unit, wherein the acoustic information generation unit is configured to generate acoustic information that generates sound effects corresponding to the operation.

[0067] (11) An information processing device according to (9) or (10) above, comprising an operation control unit, wherein the operation control unit is configured to control whether or not the operation can be performed.

[0068] (12) An information processing apparatus as described in (11) above, wherein the visual information generation unit includes in the visual information an authorized object corresponding to the authorization of the operation and a denied object corresponding to the denial of the operation, and the operation control unit authorizes the operation in response to the pressing of the authorized object and denies the operation in response to the operation of the denied object.

[0069] (13) An information processing device as described in (12) above, wherein the information acquisition unit acquires voice information, and the operation control unit controls whether or not to perform the operation according to the voice information.

[0070] (14) An information processing device according to any one of (1) to (13) above, wherein the information acquisition unit acquires temperature information, the temperature information is information indicating the temperature of the surface of the irradiated object, and the tactile information generation unit corrects the position in which the ultrasonic waves are irradiated based on the temperature information.

[0071] (15) In the information processing device described in (14) above, the information acquisition unit is an information processing device that acquires temperature information using a thermal camera or an infrared camera.

[0072] (16) An information processing device according to any one of (1) to (15) above, wherein the tactile information generation unit outputs the tactile information within 50 ms after the position of the user's hand or fingers is superimposed on the position of the object.

[0073] (17) A program that causes a computer to operate as an information processing device, which causes at least one computer to function as an information processing device as described in any one of (1) to (16) above. Of course, this is not always the case. [Explanation of Symbols]

[0074] 1: Information Processing Device 10: System 11: Processing Section 12: Storage section 13:Temporary storage 14: External device connection section 15: Communications Department 16: Communications bus 17: Power supply 21: Viewpoint information input device 22: Operation Information Input Device 23: Object Information Input Device 24: Temperature information input device 25: Voice input device 26:Display device 27: Ultrasonic generator 28: Acoustic output device 29: Airflow output device 101: Information acquisition department 102: Operation Control Unit 103: Image Processing Unit 104: Position Processing Unit 105: Visual Information Generation Unit 106: Tactile Information Generation Unit 106a: Sensory pattern determination unit 106b: Drive signal determination unit 107: Acoustic information generation section 108: Airflow Information Generation Unit 109: Interoperability Processing Unit 110: Own function section 120: Functional section 1001: Stage 1002: Laura 1003: Contact pad 1004: Ring-shaped substrate 1101: Allowed object 1102: Unauthorized object 1201 :Finger 1202 : Part 1203 : Part 1301 :face 1302 :face 1401: Actuator 1402 :Finger

Claims

1. An information processing device, It comprises an information acquisition unit, an image processing unit, a position processing unit, a visual information generation unit, and a tactile information generation unit. The aforementioned information acquisition unit is configured to acquire viewpoint information, motion information, and object information. The aforementioned viewpoint information includes information indicating the user's field of view. The aforementioned operation information is information indicating the operation of the object to which ultrasound is irradiated. The object information includes at least one of the object's coordinates, shape, surface condition, hardness, temperature, mass, and coefficient of friction. The aforementioned image processing unit is configured to process an image of an object, The aforementioned object image is an image representing a virtual object, and is updated based on the viewpoint information and the motion information. The position processing unit is configured to calculate the position when the background image and the object image are superimposed. The aforementioned background image is an image corresponding to the user's field of view. The visual information generation unit is configured to generate visual information by superimposing the background image and the object image based on the position calculated by the position processing unit, and to output the generated visual information to a display device. The tactile information generation unit generates tactile information that causes the user to be exposed to ultrasonic waves corresponding to the object, based on the position calculated by the position processing unit, the motion information, and the object information, and is configured to output the generated tactile information to an ultrasonic generator. The information acquisition unit acquires temperature information, The temperature information is information indicating the temperature of the surface of the object to be irradiated, The tactile information generation unit corrects the position where the ultrasonic waves are irradiated based on the temperature information. Information processing device.

2. In the information processing apparatus according to claim 1, Equipped with a collaborative processing unit, The aforementioned cooperation processing unit performs cooperation processing between the functional unit and at least one of other information processing devices. The aforementioned functional unit is realized by the operation of application software and replaces at least a part of the information acquisition unit, the image processing unit, the position processing unit, the visual information generation unit, and the tactile information generation unit. The other information processing device replaces at least a part of the information acquisition unit, the image processing unit, the position processing unit, the visual information generation unit, and the tactile information generation unit. Information processing device.

3. In the information processing apparatus according to claim 1, The aforementioned viewpoint information is an image captured by a camera, The background image is the captured image. Information processing device.

4. In the information processing apparatus according to claim 1, The aforementioned background image is an image that has been captured or generated in advance and is updated based on the viewpoint information. The image processing unit is configured to process the background image, the user image, and the object image. The user image is an image representing the user's hand, including their fingers, and is updated based on the motion information. The position processing unit is configured to calculate the position when the background image, the user image, and the object image are superimposed. The visual information generation unit is configured to generate visual information by superimposing the background image, the user image, and the object image based on the position calculated by the position processing unit, and to output the generated visual information to a display device. Information processing device.

5. In the information processing apparatus according to claim 1, The aforementioned object information includes environmental response information corresponding to each of the multiple environments, The position processing unit calculates the position for superimposing the background image and the object image to correspond to the environment according to the user's field of view, based on the environment-aware information and the viewpoint information. Information processing device.

6. In the information processing apparatus according to claim 1, The tactile information has a temporal pattern in at least one of the amplitude variation frequency, intensity, phase, and irradiation position of the ultrasound. Information processing device.

7. In the information processing apparatus described in claim 6, The aforementioned temporal pattern is a pattern in which at least one of the amplitude fluctuation frequency, intensity, phase, and irradiation position of the ultrasound changes over time according to the shape of the object. Information processing device.

8. In the information processing apparatus described in claim 6, The aforementioned temporal pattern is a pattern in which at least one of the amplitude fluctuation frequency, intensity, phase, and irradiation position of the ultrasound changes depending on the surface state or material of the object. Information processing device.

9. In the information processing apparatus according to claim 1, The image processing unit updates the object image to reflect the state in which one of the following operations—movement, rotation, deformation, division, or merging—has been performed on the object, based on the operation information. Information processing device.

10. In the information processing apparatus according to claim 9, Equipped with an acoustic information generation unit, The acoustic information generation unit is configured to generate acoustic information that generates sound effects corresponding to the operation. Information processing device.

11. In the information processing apparatus according to claim 9, Equipped with an operation control unit, The operation control unit is configured to control whether or not the operation is permitted. Information processing device.

12. In the information processing apparatus according to claim 11, The visual information generation unit includes in the visual information an authorized object corresponding to the authorization of the operation and a denied object corresponding to the denial of the operation. The operation control unit permits the operation in response to the pressing of the permitted object, and denies the operation in response to the operation of the disallowed object. Information processing device.

13. In the information processing apparatus according to claim 12, The information acquisition unit acquires audio information, The operation control unit controls whether or not the operation is permitted according to the voice information. Information processing device.

14. In the information processing apparatus according to claim 1, The information acquisition unit acquires the temperature information using a thermal camera or an infrared camera. Information processing device.

15. A program that makes a computer function as an information processing device, To enable at least one computer to function as an information processing device according to any one of claims 1 to 14. program.

Images