Pseudo-tactile presentation device, pseudo-tactile presentation method, and program

Abstract

On the basis of operation information representing an operation based on the movement of a body part and information representing a relationship of a visual change amount of a visual target with regards to an operation amount based on an operation, the present invention visually changes a visual target by a change amount according to the operation and outputs information for presenting the visual target that was visually changed. This relationship is determined according to at least an index representing the level of pseudo-haptics to be presented.

Description

[Technical Field]

[0001] The present invention relates to a technology for presenting a pseudo-tactile sensation. [Background technology]

[0002] By manipulating visual information based on information obtained from tactile input devices such as a mouse, keyboard, touch pen, or force feedback device, it is possible to provide the user with visual feedback in response to tactile input. By adding delays or fluctuations to the visual feedback, it is possible to give the user an illusionary tactile impression that is different from the actual one. This technology is called pseudo-haptic technology here.

[0003] There are also technologies that give users tactile impressions by tracking the user's physical and finger movements without touching them and manipulating visual information based on the tracking information, rather than using a tactile input device. For example, Non-Patent Document 1 discloses a touchless pseudo-haptic technology for a carousel interface that slides an image left and right based on the user's swipe gesture. This technology applies friction coefficients and magnetic forces to the carousel movement based on physical calculations, reproducing the natural movement and stopping of the carousel while providing the user with a pseudo-haptic sensation. Non-Patent Document 2 discloses that in a virtual reality environment, the perceived weight of a real object can be changed by manipulating the speed difference between a real object grasped in real space and visual feedback. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Gaucher, P., Argelaguet, F., Royan, J. & Lecuyer, A., "A novel 3D Carousel based on Pseudo-Haptic Feedback and Gestural Interaction for Virtual Showcasing", [online], 2013, 2013 IEEE Symposium on 3D User Interfaces (3DUI), 55-58, [searched on October 21, 2021], Internet <https: / / doi.org / 10.1109 / 3dui.2013.<6550197> [Non-Patent Document 2] Samad, M., Gatti, E., Hermes, A., Benko, H. & Parise, C., "Pseudo-Haptic Weight: Changing the Perceived Weight of Virtual Objects By Manipulating Control-Display Ratio", 2019, Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, 1-13, Internet <https: / / doi.org / 10.1145 / 3290605.33005....... [Summary of the Invention] [Problems to be Solved by the Invention]

[0005] However, there is no known technology that allows users to perceive pseudo-haptic sensations without performing complex physical calculations to simulate the real environment or using haptic sensations based on physical reactions to human movements. For example, in Non-Patent Document 2, although the weight of an object in a virtual environment is modulated by pseudo-haptic feedback, it uses the speed difference between the haptic sensation and visual feedback caused by the physical reaction received from the real object being held, and does not allow users to perceive pseudo-haptic sensations without using haptic sensations based on physical reactions to human movements.

[0006] The present invention has been made in consideration of these points, and aims to allow users to perceive pseudo-tactile sensations without performing complicated physical calculations to simulate a real environment or utilizing tactile sensations based on physical reactions to the movements of the human body. [Means for solving the problem]

[0007] Based on operation information representing an operation based on the movement of a body part and information representing the relationship between the amount of visual change of the visual object and the amount of operation based on the operation, the visual object is visually changed by an amount of change corresponding to the operation, and information for presenting the visually changed visual object is output, where this relationship is determined at least according to the degree of the pseudo-tactile sensation to be presented. [Effects of the Invention]

[0008] This allows users to perceive pseudo-tactile sensations without having to perform complicated physical calculations to simulate a real environment or utilize tactile sensations based on physical reactions to the movements of the human body. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram illustrating a pseudo-tactile presentation system according to an embodiment. [Figure 2] FIG. 2 is a conceptual diagram illustrating the pseudo-tactile presentation system of the embodiment. [Figure 3] FIG. 3 is a graph illustrating the relationship between the amount of operation and the amount of visual change of the visual object in the first embodiment. [Figure 4] 4A to 4C are diagrams illustrating the relationship between the amount of operation and the amount of visual change of the visual object in the first embodiment. [Figure 5] 5A to 5C are diagrams illustrating the relationship between the amount of operation and the amount of visual change of the visual object in the first embodiment. [Figure 6]Fig. 6A is a diagram for explaining the contents of an experiment demonstrating the effect of the first embodiment. Fig. 6B is a graph for illustrating the relationship between the vertical height of the fingers (the amount of operation based on the operation) and the amount of change in the diameter or phase of the visual object (the amount of visual change in the visual object). [Figure 7] Fig. 7 is a graph illustrating experimental results showing the effects of the first embodiment. Fig. 7 illustrates the relationship between the ratio (speed ratio) of the change in diameter of the visual target for visual stimulation (the visual change in the visual target) to the change in diameter of the visual target for reference (the visual change in the visual target), and the degree (evaluation score) of the sense of weight (pseudo-tactile sensation) perceived by the subject. [Figure 8] FIG. 8 is a diagram illustrating a visual object according to the second embodiment. [Figure 9] 9A to 9C are diagrams illustrating the relationship between the amount of operation and the amount of visual change of the visual object in the second embodiment. [Figure 10] 10A to 10C are diagrams illustrating the relationship between the amount of operation and the amount of visual change of the visual object in the second embodiment. [Figure 11] 11A to 11C are diagrams illustrating the relationship between the amount of visual change of the visual object and the amount of operation in the modified example of the second embodiment. [Figure 12] Fig. 12A is a diagram for explaining the contents of an experiment demonstrating the effect of the second embodiment. Fig. 12B is a graph illustrating the results of an experiment demonstrating the effect of the second embodiment. Fig. 12B illustrates the relationship between the ratio (speed ratio) of the phase change amount of the visual object for visual stimulation (visual change amount of the visual object) to the phase change amount of the visual object for reference (visual change amount of the visual object), and the degree (evaluation score) of the sense of weight (pseudo-tactile sensation) perceived by the subject. [Figure 13]Fig. 13 is a graph illustrating experimental results showing the effect of the modified example of the second embodiment. Fig. 13 illustrates the relationship between the ratio (speed ratio) of the phase change amount of the visual target for visual stimulation (the visual change amount of the visual target) to the phase change amount of the visual target for reference (the visual change amount of the visual target), and the degree (evaluation score) of the sense of weight (pseudo-tactile sensation) perceived by the subject. [Figure 14] FIG. 14 is a diagram illustrating a visual target according to the third embodiment. [Figure 15] Fig. 15A is a diagram for explaining the contents of an experiment demonstrating the effect of the third embodiment. Fig. 15B is a graph illustrating the results of an experiment demonstrating the effect of the third embodiment. Fig. 15B illustrates the relationship between the ratio (speed ratio) of the phase change amount of the visual object for visual stimulation (visual change amount of the visual object) to the phase change amount of the visual object for reference (visual change amount of the visual object), and the degree (evaluation score) of the sense of weight (pseudo-tactile sensation) perceived by the subject. [Figure 16] FIG. 16 is a block diagram illustrating the hardware configuration of the pseudo-tactile presentation device. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. [principle] First, the principle of each embodiment will be described. In each embodiment, the visual object is visually changed by an amount of change corresponding to the operation based on operation information representing an operation based on the movement of a body part and information representing the relationship between the amount of visual change of the visual object and the amount of operation based on the operation, and the visually changed visual object is presented to the user. However, the relationship between the amount of visual change of the visual object and the amount of operation based on the operation is determined at least according to the level of the pseudo-haptic sensation to be presented. This allows the user to perceive the pseudo-haptic sensation without performing complicated physical calculations to simulate a real environment or using haptics based on physical reactions to the movement of the human body. An example of a pseudo-haptic sensation that can be presented is a sense of weight. The sense of weight can also be rephrased as a sense of weight, a sense of resistance, or the like.

[0011] Specific examples of body parts include fingers, palms, arms, heads, facial parts, waists, trunks, and feet, but this does not limit the present invention, and operations may be based on the movement of any body part. The movement of a body part is, for example, the movement of a body part in a touchless environment. A touchless environment refers to an environment in which physical reactions to human body movements (i.e., reaction forces due to such reactions) are not used to present pseudo-haptics. An environment in which a human body does not grasp or wear a physical input device is a touchless environment. Furthermore, even if a human body grasps or wears a physical input device (e.g., a VR controller), it is also a touchless environment if the physical reactions from the input device to the human body movements are not used to present pseudo-haptics. The movement of a body part may be a movement that moves the position of the body part, a movement that rotates the body part, a movement that deforms the body part, or a combination of these. The position of a body part may be an absolute position or a relative position. An example of the former is a position in a coordinate system fixed to an external reference position (e.g., the ground or floor), while an example of the latter is a position relative to a visual object or a device that presents the visual object. Furthermore, the position of a body part may be a position represented in a three-dimensional coordinate system, a two-dimensional coordinate system, or a one-dimensional coordinate system. The action of rotating a body part may be an action of rotating the body part absolutely or a action of rotating it relatively. An example of the former is an action of rotating a body part relative to a coordinate system fixed to an external reference position, while an example of the latter is an action of rotating a body part relative to a visual object or a device that presents the visual object. The action of transforming a body part is an action of transforming a body part in accordance with the movement of muscles or joints. Examples of actions that transform a body part include pinching with fingers and clenching a fist. The action of a body part can be represented, for example, by the position, orientation, amount of movement, amount of rotation, movement speed, rotation speed, acceleration, angular acceleration, or at least a combination of these.

[0012] The operation based on the movement of a body part may be any operation performed based on the movement of a body part. For example, the operation based on the movement of a body part may be any operation in which the operation content and the operation amount based on the operation are determined based on the position, orientation, movement amount, rotation amount, movement speed, rotation speed, acceleration, angular acceleration of the body part, or at least a combination of these. For example, an operation using a body part performed in a touchless environment may be detected by a detection device such as a hand tracker, and the operation content and the operation amount based on the operation may be determined based on the position, orientation, movement amount, rotation amount, movement speed, rotation speed, acceleration, angular acceleration of the detected body part, or at least a combination of these. Furthermore, the operation information representing the operation based on the movement of a body part may be any information representing an operation based on the movement of a body part. For example, the operation information may be information representing the operation content, information representing the operation amount based on the operation, or information representing both the operation content and the operation amount based on the operation. The operation content represents the type of operation. Specific examples of the operation content include selection, start, continuation, and end of the operation. The operation amount based on the operation represents the amount of any type of operation. For example, the greater the movement of the body part, the greater the operation amount based on the operation. More specifically, for example, the greater the amount of movement of a body part, the greater the amount of operation based on the operation. Hereinafter, the amount of operation based on the operation will be referred to as the operation amount m. The operation amount m is, for example, an amount that is directly proportional to the amount of movement of the body part. The amount of movement of the body part may be the operation amount m itself, or a function value of the amount of movement of the body part (for example, a non-decreasing function value or a monotonically increasing function value) may be the operation amount m. In this case, the amount of movement of the body part may be the distance from the motion start position of the body part to the current position, or a specific directional component of the distance from the motion start position of the body part to the current position (for example, a vertical component, a vertically upward component, a vertically downward component, a horizontal component, a specific horizontal directional component, etc.), or the distance from the motion start position of the body part to the current position. Furthermore, the motion start position of the body part may be, for example, the position of the body part at the time when a state in which a predetermined start condition is not satisfied transitions to a state in which it is satisfied, or may be a predetermined position.Examples of the start condition include a body part starting to move or move (condition 1), a body part performing a specific triggering action (e.g., pinching with fingers or clenching a fist) (condition 2), a positional relationship of the body part with respect to a visual target or other positions satisfying a predetermined condition (condition 3), a device state satisfying a predetermined condition (condition 4), a device operation state satisfying a predetermined condition (condition 5), and an output from another processing unit (condition 6). Any combination of conditions 1 to 6 may be used as the start condition. Note that the operation amount m may be a value greater than or equal to 0, a positive value, a value less than or equal to 0, a negative value, or a value that can be positive, negative, or 0.

[0013] The visual object may be any object that is perceived visually and visually changes in accordance with an operation. For example, the visual object may be a two-dimensional image, a three-dimensional image, an image in a virtual space, a three-dimensional hologram, or a physical object in real space (e.g., a mechanically moving attraction device or an advertising billboard). The visual change of the visual object is not limited. For example, the size of the visual object may change, the shape of the visual object may change, the luminance may change, the color may change, the pattern may change, or any combination of these may change. Furthermore, at least one of the luminance, color, and pattern of the visual object may change without changing the size or shape of the visual object. At least one of the luminance, color, and pattern of the visual object may be uniform or non-uniform in the spatial domain. At least one of the luminance, color, and pattern of the visual object may be periodic or non-periodic in the spatial domain.

[0014] The visual change amount of a visual object refers to the amount of change from the initial state of a visual element of the visual object to its current state. The initial state of a visual element of a visual object may be predetermined, determined based on input information, or determined based on other information (e.g., the position of the presented visual object). The initial state of a visual element of a visual object may be the same regardless of the level of the presented pseudo-tactile sensation, or may differ depending on the level of the presented pseudo-tactile sensation. However, to clarify the differences in the level of the presented pseudo-tactile sensation, it is preferable to keep the initial state of the visual element of the visual object the same regardless of the level of the presented pseudo-tactile sensation. Hereinafter, the visual change amount of a visual object will be referred to as change amount c. Change amount c corresponds to the amount of change in at least one of the size, shape, brightness, color, or pattern of the visual object, and at least one of the size, shape, brightness, color, or pattern of the visual object changes by an amount corresponding to an operation based on the movement of a body part. Examples of the change amount of the size of a visual object include the change amount of the diameter, radius, area, or volume of the visual object. Examples of the change in shape of the visual object include the change in aspect ratio, the change in aspect ratio, and the change in flattening of the visual object. Examples of the change in luminance include the change in luminance, the change in lightness, and the change in phase in the spatial domain. Examples of the change in color include the change in pixel value, the change in color space, and the change in phase in the spatial domain. Examples of the change in pattern include the change in phase in the spatial domain, the change in pixel value, the change in pixel value, and the sum of the changes in pixel value. The change c may be a value greater than or equal to 0, a positive value, a value less than or equal to 0, a negative value, or a value that can be positive, negative, or 0.

[0015] The relationship between the amount of visual change c of the visual object and the amount of operation m based on the operation is determined according to the level of the pseudo-haptic sensation to be presented. This relationship is determined according to the level of the pseudo-haptic sensation to be presented. Hereinafter, this relationship will be referred to as relationship r. Only a single relationship r for the level of a single pseudo-haptic sensation may be determined, or multiple relationships r for multiple levels of pseudo-haptic sensations may be determined. Note that the level of the pseudo-haptic sensation to be presented may be determined in advance, based on input information, or based on other processing. Hereinafter, an index representing the level of the pseudo-haptic sensation to be presented will be referred to as index i. Examples of index i are values ​​(e.g., index, numerical value, vector, symbol) representing the magnitude (strength) of the pseudo-haptic sensation, such as a value representing the magnitude of the sense of weight, the magnitude of the sense of weight, or the magnitude of the sense of resistance. When multiple relationships r for multiple levels of pseudo-haptic sensations are determined, the relationship r will be different for different pseudo-haptic sensations to be presented. In other words, if the relationship when the index i is i1 (first index value) is expressed as relationship r(i1), and the relationship when the index i representing the level of the pseudo-tactile sensation to be presented is i2 (second index value), which is different from i1, is expressed as relationship r(i2), then the relationship r(i1) and the relationship r(i2) are different. The relationship r of the change amount c with respect to the operation amount m may be a linear relationship or a nonlinear relationship. An example of the relationship r is the ratio Δc / Δm of the change Δc in the visual change amount c of the visual object to the change Δm in the operation amount m. For example, Δm means the unit operation amount at the operation amount m. For example, if c is a function value f(m) of m, then an example of the ratio r = Δc / Δm is the first derivative (slope) dc / dm of c = f(m) with respect to m. Here, f(·) means a function with · as its domain.

[0016] If the ratio r=Δc / Δm when the index i is i1 (first index value) is expressed as the ratio r(i1) (first value), and the ratio r when the index i is i2 (second index value) different from i1 is expressed as the ratio r(i2) (second value), the ratio r(i1) (first value) and the ratio r(i2) (second value) are different from each other. For example, if the level of the pseudo-tactile sensation represented by the index i1 (first index value) (e.g., the magnitude of the pseudo-tactile sensation, such as the magnitude of the sense of weight, the magnitude of the sense of weight, the magnitude of the sense of resistance, etc.) is greater than the level of the pseudo-tactile sensation represented by the index i2 (second index value), the ratio r(i1) (first value) is smaller than the ratio r(i2) (second value). For example, the greater the level of the pseudo-tactile sensation to be presented, the smaller the ratio r. In other words, the smaller the ratio r, the greater the level of the pseudo-tactile sensation to be presented.

[0017] For the same index i, regardless of the magnitude of m (at least within a predetermined range of m), the ratio r = Δc / Δm may be constant, or the ratio r = Δc / Δm may vary depending on the magnitude of m. That is, depending on the operation, when the operation amount m changes from m1 (the first operation amount) to m2 (the second operation amount), and / or when the operation amount m changes from m2 (the second operation amount) to m1 (the first operation amount) (where m1 is different from m2), the ratio r (the third value) when the operation amount is m1 (the first operation amount) and the ratio r (the fourth value) when the operation amount is m2 (the second operation amount) may be equal to each other or may be different from each other. For example, for index i1, when the operation amount m is m1, the ratio r is r(i1,m1) (the third value), and when the operation amount m is the operation amount m2, the ratio r is r(i1,m2) (the fourth value), r(i1,m1) = r(i1,m2) may be true, or r(i1,m1) ≠ r(i1,m2) (that is, r(i1,m2) (the fourth value) is different from r(i1,m1) (the third value)) may be true. For example, the operation amount m1 (the first operation amount) may be larger than the operation amount m2 (the second operation amount), and r(i1,m1) (the third value) may be smaller than r(i1,m2) (the fourth value). In this case, the larger the operation amount m, the greater the degree of pseudo-haptic sensation. For example, the greater the operation amount m, the greater the degree of pseudo-haptic sensation. Conversely, for example, the operation amount m1 (the first operation amount) may be larger than the operation amount m2 (the second operation amount), and r(i1,m1) (the third value) may be larger than r(i1,m2) (the fourth value). In this case, the larger the operation amount m, the smaller the degree of pseudo-haptic sensation. For example, the greater the operation amount m, the smaller the degree of pseudo-haptic sensation. Or, for example, for operation amounts m1, m2, m3 that satisfy m1 < m2 < m3, when the ratio r for the operation amount m1 is r(i1,m1), the ratio r for the operation amount m2 is r(i1,m2), and the ratio r for the operation amount m3 is r(i1,m3), r(i1,m2) < r(i1,m) and r(i1,m2) < r(i1,m3) may be true. In this case, the degree of pseudo-haptic sensation presented when the operation amount is m2 is greater than the degree of pseudo-haptic sensation presented when the operation amounts are m1 and m3 before and after it. Conversely, r(i1,m2) > r(i1,m1) and r(i1,m2) > r(i1,m3) may be true.In this case, the degree of virtual tactile sensation presented at the time of the operation amount m2 is smaller than the degrees of virtual tactile sensation presented at the times of the operation amounts m1 and m3 before and after it.

[0018] In addition, according to the operation amount m, the index i representing the degree of virtual tactile sensation may be switched. For example, when the operation amount m changes from m1 (the first operation amount) to m2 (the second operation amount) according to the operation, and / or when the operation amount m changes from m2 (the second operation amount) to m1 (the first operation amount) (m1 is different from m2), the index i when the operation amount is m1 (the first operation amount) and the index i when the operation amount is m2 (the second operation amount) may be equal to each other or different from each other. Thus, by switching the index i according to the operation amount m, the degree of virtual tactile sensation may be changed. In this case, the ratio r(i) = Δc / Δm corresponding to each index i may be constant regardless of the magnitude of m, or the ratio r(i) = Δc / Δm may vary according to the magnitude of m (for example, as described above). For example, even if the ratio r(i) corresponding to each index i is constant regardless of the magnitude of m, by switching the index i according to the operation amount m, the degree of virtual tactile sensation presented according to the operation amount m can be changed. For example, when the operation amount m1 is larger than the operation amount m2 and the degree of virtual tactile sensation represented by the index i1 is larger than the degree of virtual tactile sensation represented by the index i2, it may be switched to the index i1 at the time of the operation amount m1 and switched to the index i2 at the time of the operation amount m2. Conversely, it may be switched to the index i2 at the time of the operation amount m1 and switched to the index i1 at the time of the operation amount m2. Or, for example, for the operation amounts m1, m2, m3 satisfying m1 < m2 < m3, when the degree of virtual tactile sensation represented by the index i1 is larger than the degree of virtual tactile sensation represented by the index i2 and the degree of virtual tactile sensation represented by the index i2 is larger than the degree of virtual tactile sensation represented by the index i3, it may be switched to the index i1 at the time of the operation amount m2, switched to the index i2 at the time of the operation amount m1, and switched to the index i3 at the time of the operation amount m3. Or, in this case, it may be switched to the index i3 at the time of the operation amount m2, switched to the index i2 at the time of the operation amount m1, and switched to the index i3 at the time of the operation amount m3.

[0019] [First embodiment] In the first embodiment, the body part is a user's finger, the visual object is a two-dimensional disk-shaped figure, the movement of the body part is a gesture of moving the visual object up and down, and the diameter of the visual object is changed by an amount corresponding to the operation based on this movement, causing the user to perceive a weight sensation as a pseudo-tactile sensation, but this is not intended to limit the present invention.

[0020] <Configuration> As illustrated in FIGS. 1 and 2, the pseudo-tactile sense presentation system 1 of this embodiment includes a pseudo-tactile sense presentation device 11, a detection device 12, and a visual object presentation device 13.

[0021] The pseudo-tactile presentation device 11 has an input unit 111, a storage unit 112, a motion information detection unit 113, an operation determination unit 115, a visual target update unit 116, and an output unit 117. Although not described further below, data obtained in each process is stored in the storage unit 112 and is read out and used as needed.

[0022] The detection device 12 is a device that detects the position of a body part 101 (for example, a finger) of the user 100 in a touchless environment and outputs the detection result. Examples of the detection device 12 include a hand tracker such as Leap motion (registered trademark), a motion capture, a touchless display, an acceleration sensor, a gyro sensor, etc., but these do not limit the present invention.

[0023] The visual object presentation device 13 is a device that visually presents a visual object 130 to the user 100. Examples of the visual object presentation device 13 include, but are not limited to, a liquid crystal display, a virtual reality headset, a video projector, and a stereoscopic hologram display.

[0024] <Pretreatment> As a preprocessing step, at least information p representing a relationship r between the amount of operation m based on the operation (operation based on the movement of the body part 101) and the visual change amount c of the visual object 130 is input to the input unit 111 and stored in the memory unit 112. Specific examples of the operation amount m, the visual object 130, the change amount c, and the relationship r have been described above. For simplicity's sake, this embodiment illustrates a case in which the visual object 130 is a two-dimensional disk-shaped figure, the visual change amount c of the visual object 130 is the change amount (positive value) in the diameter of the visual object 130, the operation amount m is the movement amount of the vertical component of the body part 101 (fingers) based on the movement of the body part 101 that moves the visual object 130 up and down, and the relationship r is the relationship between the amount of operation m and the change amount c of the diameter of the visual object 130. However, this does not limit the present invention. FIG. 3 illustrates an example of the relationship r between the amount of operation m and the change amount c. The horizontal axis of FIG. 3 illustrates the amount of manipulation m [cm] based on the movement of the body part 101. In the example of this embodiment, the amount of movement of the vertical component of the body part 101 (fingers) is the amount of manipulation m [cm]. The vertical axis of FIG. 3 illustrates the amount of visual change c [pixels] of the visual object 130. In the example of this embodiment, the amount of change in diameter of the visual object 130, which is a two-dimensional disk-shaped figure, is the amount of visual change c [pixels]. As described above, only a single relationship r may be defined for a single level of pseudo-tactile sensation, or multiple relationships r may be defined for multiple levels of pseudo-tactile sensation. In the example of FIG. 3, multiple relationships r are defined for multiple levels of pseudo-tactile sensation. That is, different relationships r are defined according to indices (Index 1, ..., Index 5) that represent different levels of pseudo-tactile sensation. The relationship r illustrated in FIG. 3 represents the ratio Δc / Δm of the change Δc in the visual change amount c of the visual object 130 to the change Δm in the amount of manipulation m. In the example of Fig. 3, the ratio Δc / Δm (slope) is constant regardless of the magnitude of m. That is, the relationship r of the change amount c to the manipulated variable m shown in Fig. 3 is a linear relationship. The information p may include a function representing the relationship r, a table representing the relationship r, or a parameter representing the relationship r.Furthermore, the information p may include information representing the initial (initial state) visual object 130 (e.g., information on the diameter of the initial visual object 130, information on the brightness, color, or pattern of the initial visual object 130, etc.). However, if the initial state of the visual object 130 (e.g., the diameter, brightness, color, or pattern of the initial visual object 130) is fixed (e.g., the initial value of the diameter is 0), the information p does not need to include information on the initial visual object 130 (e.g., information on the diameter of the initial visual object 130, information on the brightness, color, or pattern of the initial visual object 130, etc.). Furthermore, the information p may include information b representing the direction in which the diameter of the visual object 130 is to be changed (whether to increase or decrease the diameter) for a positive change amount c. For example, if the diameter is to be increased for a positive change amount c, b=b. + If the diameter is decreased with a positive change in c, then b = b - Here, b + is a positive real number, and b - is a negative real number. + The absolute value of and b - It is desirable that the absolute values ​​of are equal to each other, but they may be different from each other. + An example of is +1, and b + An example of b is -1. + may be a positive function value of m (for example, a monotonically increasing or decreasing function value), and b - may be a negative function value of m (for example, a monotonically increasing function value or a monotonically decreasing function value). However, this does not limit the present invention. Note that, if the direction in which the diameter of the visual object 130 changes for a positive change amount c is fixed, the information p does not need to include the information b.

[0025] <Operation> Next, an example of the operation of the pseudo-tactile presentation system 1 of this embodiment will be described. The visual object update unit 116 (FIG. 1) generates an initial visual object 130 and outputs information v representing the initial visual object 130. The initial visual object 130 exemplified in this embodiment is a two-dimensional disk-shaped figure with an initial diameter c1. This initial diameter value c1 may be a value included in the information p read from the storage unit 112, or may be a predetermined value. The brightness, color, and pattern of the initial visual object 130 may be values ​​included in the information p read from the storage unit 112, or may be predetermined values. The information v representing the initial visual object 130 is sent to the output unit 117 and output from the output unit 117 to the visual object presentation device 13. The visual object presentation device 13 presents (displays) the visual object 130 based on the sent information v (FIG. 2).

[0026] The user 100 moves a body part 101 (for example, a finger) in a touchless environment while looking at a visual object 130 presented by a visual object presentation device 13. The position of the body part 101 is detected by a detection device 12. Information d representing the position of the body part 101 detected by the detection device 12 is output to a motion information detection unit 113. The detection device 12 may detect the position of the body part 101 and output the information d, for example, continuously at predetermined time intervals, or may output the information d each time a movement of the body part 101 is detected. Each time the information d is input, the motion information detection unit 113 detects the movement of the body part 101 from the information d and outputs information a representing the movement of the body part 101. m The motion detection and information a of the body part 101 are output. m The output of information a representing the action is also performed continuously. m is information that represents at least one of the position, movement, and amount of movement m of the body part 101. In this embodiment, information a m10 shows an example in which the movement amount m of the body part 101 is information specifying the movement amount m of the body part 101. Specific examples of the movement amount m of the body part 101 have been described above. In this embodiment, an example is shown in which the movement amount is the vertical component of the distance from the movement start position of the body part 101 to the current position. Specific examples of the movement start position of this body part 101 have been described above. In this example, the movement start position is the position of the body part 101 at the time when the state transitions from an unsatisfied state to a satisfied state of the start condition, which is a combination of the above-described conditions 2 and 3. That is, in this example, the movement start position is the position of the body part 101 at the time when the body part 101 performs a specific trigger movement (condition 2) and the positional relationship of the body part 101 with respect to the presented visual object 130 and other positions transitions to a state in which a predetermined condition is satisfied (condition 3). The trigger movement may be any type, but in this embodiment, an example is shown in which a pinching movement with fingers is used as the trigger movement. Furthermore, the predetermined condition (condition 3) that the positional relationship satisfies may be any condition, but in the example of this embodiment, the predetermined condition is that the body part 101 (e.g., fingers) is more than a certain distance away from the presentation position of the visual object 130 (e.g., the display screen of the visual object presentation device 13). In this case, the motion information detection unit 113 measures the distance d1 between the thumb and index finger that are the body part 101 of the user 100 based on the information d, and further measures the distance d2 between the body part 101 (e.g., fingers) and the presentation position of the visual object 130 (e.g., the display screen of the visual object presentation device 13), and converts the distances d1 and d2 into information a that represents the motion of the body part 101. m =(d1,d2).

[0027] Information a m =(d1, d2) is input to the operation determination unit 115. The operation determination unit 115 receives the information a m Based on this, the operation determination unit 115 obtains and outputs operation information representing an operation based on the movement of the body part 101. Examples of operation information are as described above, but in this embodiment, an example will be described in which information representing an operation amount m is output as operation information. For example, first, the operation determination unit 115 determines whether the distance d1 is greater than a predetermined threshold d th1 Whether the distance d1 is less than the threshold d th1 The threshold dth1 is a positive real number that represents the distance. Here, the distance d1 is the threshold d th1 (or, if the distance d1 is greater than the threshold d th1 If the distance d1 is greater than or equal to the threshold value d2, the operation determining unit 115 determines that a pinching motion with the body part 101 (fingers in this example) is not being performed, i.e., a trigger motion is not being performed. th1 If the distance d1 is less than or equal to the threshold d th1 If the distance d2 is less than the predetermined threshold d th2 (or whether the distance d2 exceeds the threshold d th2 However, the threshold value d th2 is a positive real number that represents the distance. Here, the distance d2 is the threshold d th2 If the distance d2 is less than or equal to the threshold d th2 If the distance d2 is less than the threshold value d th2 (or, if the distance d2 is greater than the threshold d th2 or more), the operation determination unit 115 determines that the positional relationship of the body part 101 satisfies the predetermined condition (i.e., determines that condition 3 is satisfied). When the operation determination unit 115 determines that the trigger action has been performed and that the positional relationship of the body part 101 satisfies the predetermined condition (i.e., determines that both condition 2 and condition 3 are satisfied), the operation determination unit 115 obtains the vertical component of the distance from the action start position to the current position of the body part 101 as the movement amount based on distance d2, and sets this movement amount as the operation amount m. Note that the action start position is as described above. The action start position exemplified in this embodiment is the position of the body part 101 at the time when the state transitions from a state in which at least one of condition 2 or condition 3 is not satisfied to a state in which both condition 2 and condition 3 are satisfied. The operation determination unit 115 outputs information representing the operation amount m (operation information representing an operation based on the action of the body part) to the visual target update unit 116.

[0028] Information representing the operation amount m is input to the visual object update unit 116. The visual object update unit 116 references information p stored in the memory unit 112 and visually changes the visual object 130 by an amount of change corresponding to the operation, based on at least information representing the operation amount m (operation information representing an operation based on the movement of a body part) and information p (information representing the relationship between the visual change amount of the visual object and the operation amount based on the operation). For example, if the information p represents only a single relationship r for a single level of pseudo-tactile sensation, the visual object update unit 116 references this relationship r to obtain a change amount c corresponding to the input operation amount m, and visually changes the visual object 130 by the change amount c from its initial state. In this embodiment, the diameter of the visual object 130 is changed from the initial value c1 to c1 + b·c. That is, the visual object update unit 116 in this example generates a disk-shaped visual object 130 with a diameter c2 = c1 + b·c and outputs information v representing the visual object 130. b = {b + ,b -} may be information included in the information p or may be predetermined. On the other hand, for example, if the information p represents multiple relationships r for multiple degrees of pseudo-haptics, the visual object update unit 116 references the relationships r corresponding to the degrees of pseudo-haptics to be presented, obtains a change amount c corresponding to the input operation amount m, and visually changes the visual object 130 from its initial state by the change amount c. In the example of FIG. 3, the visual object update unit 116 references the relationship r corresponding to an index (in the example of FIG. 3, any one of Index 1, ..., Index 5) representing the degree of pseudo-haptics to be presented. As described above, the degree of pseudo-haptics to be presented may be predetermined, determined based on input information I input to the input unit 111, or determined based on other processing. In this case, the visual object update unit 116 also generates a disk-shaped visual object 130 with a diameter c2 = c1 + b·c and outputs information v representing the visual object 130.

[0029] 4A to 5C, the difference in change of the visual object 130 when the degree of the pseudo-tactile sensation to be presented differs will be illustrated. For example, when the index representing the degree of the pseudo-tactile sensation to be presented (for example, a sense of weight, a sense of weight, a sense of resistance) is Index 1 (FIG. 3), the visual object update unit 116 updates the diameter of the visual object 130 by the initial c1=c for the operation amount m, which is the vertical component of the distance from the motion start position vh1 of the body part 101 to the current position vh1. 1-1 From c2=c 1-2 =c 1-1 +b·c. For example, the visual object update unit 116 changes b=b + Then, the diameter of the visual object 130 is c1=c 1-1 From c2=c 1-2 (Figure 4A to Figure 4C), and b = b - Then, the diameter of the visual object 130 is c1=c 1-1 From c2=c 1-2 On the other hand, when the index representing the degree of the pseudo-tactile sensation to be presented is Index 2, the visual object update unit 116 reduces the diameter of the visual object 130 from the initial c1=c 2-1 From c2=c 2-2 =c 2-1 +b·c. For example, the visual object update unit 116 changes b=b + Then, the diameter of the visual object 130 is c1=c 2-1 From c2=c 2-2 (Figures 5A to 5C), and b = b - Then, the diameter of the visual object 130 is c1=c 2-1 From c2=c 2-2 Here, the degree of the pseudo-tactile sensation (for example, a sense of weight, a sense of weight, a sense of resistance) represented by the index Index 1 is greater than the degree of the pseudo-tactile sensation represented by the index Index 2. In this case, even though the operation amount m is the same, the change amount |c 1-2 -c 1-1 | is the change in diameter of the visual object 130 relative to the index Index 2 |c 2-2 -c 2-1| is smaller than |. In the example of this embodiment, the position of the body part 101 at the time when the state transitions from a state in which at least one of condition 2 or condition 3 is not satisfied to a state in which both condition 2 and condition 3 are satisfied is set as the movement start position. Therefore, the diameter of the visual object 130 is updated only when a trigger movement (pinching movement) is detected and it is detected that the body part 101 (for example, fingers) has moved away from the presentation position of the visual object 130 by a certain distance or more.

[0030] The information v representing the visual object 130 output from the visual object update unit 116 is sent to the output unit 117, which then outputs it to the visual object presentation device 13. The visual object presentation device 13 presents (displays) the visual object 130 based on the sent information v (FIG. 2). As a result, the visual object 130 presented to the user 100 changes by a change amount c according to the operation based on the movement of the body part 101, and a pseudo-tactile sensation (e.g., a sense of weight, a sense of weight, a sense of resistance) can be presented to the user 100. Furthermore, if multiple relationships r for multiple degrees of pseudo-tactile sensations are set, a selected degree of pseudo-tactile sensation (e.g., a selected magnitude of a sense of weight, a sense of weight, a sense of resistance) can be presented.

[0031] <Experimental Results> Next, the results of Experiment 1 for illustrating the effects of the first embodiment will be illustrated. As illustrated in FIG. 6A, in Experiment 1, a visual object presentation device 13 presented a visual object 130 (evaluation stimulus) to be evaluated, as well as a reference visual object 140 (standard stimulus). Both visual objects 130 and 140 were two-dimensional disk-shaped images, and their diameters varied based on the participant's finger (body part) movements. However, the visual objects 130 and 140 differed in the amount of diameter change relative to the amount of manipulation. That is, the diameter of visual object 140 changed by 20 pixels for every 1 cm change in the height (vertical component of the movement) of the finger. Regarding the visual object 130, the following cases were tested: Case 1, in which the diameter changed by 5 pixels for every 1 cm change in the height of the finger; Case 2, in which the diameter changed by 10 pixels; Case 3, in which the diameter changed by 20 pixels; Case 4, in which the diameter changed by 40 pixels; and Case 5, in which the diameter changed by 80 pixels for every 1 cm change in the height of the finger. That is, the ratio of the change in the diameter of the visual object 130 to the change in the diameter of the visual object 140 (speed ratio) for cases 1, 2, 3, 4, and 5 was 0.25, 0.5, 1.0, 2.0, and 4.0, respectively. Figure 6B shows the change in the diameter of the visual object 130 (pixels) relative to the height of the participant's finger (cm) at each speed ratio. In Experiment 1, the horizontal axis of Figure 6B represents the height of the participant's finger (cm), and the vertical axis represents the change in the diameter of the visual object 130 (pixels). For cases 1, 2, 3, 4, and 5, participants compared the visual object 130 with the visual object 140 and reported which of them felt heavier using a rating method. Specifically, participants reported their perception of the visual object 130 on a five-point scale, with five possible scores: 1: The left disk (visual object 140) is significantly heavier; 2: The left disk (visual object 140) is slightly heavier; 3: The left and right disks are the same weight; 4: The right disk (visual object 130) is slightly heavier; and 5: The right disk (visual object 130) is significantly heavier. Figure 7 shows the results of this experiment. The horizontal axis of Figure 7 represents the speed ratio, and the vertical axis represents the average weight evaluation score (from 1 to 5) for each participant. As illustrated in Figure 7, the smaller the speed ratio, the heavier the visual object 130 was perceived to be. In other words, the smaller the change in the diameter of the visual object relative to the amount of manipulation, the greater the perceived weight.

[0032] [Second embodiment] In the first embodiment, the visual object 130 is a two-dimensional disk-shaped figure, and the movement of the body part 101 is a gesture that moves the visual object 130 up and down. The diameter of the visual object 130 is changed by an amount corresponding to the operation based on this movement, causing the user to perceive a sense of weight, which is a pseudo-tactile sensation. That is, in the first embodiment, an example was shown in which the sense of weight is perceived by changing the size of the visual object 130 by an amount corresponding to the operation based on the movement. In the second embodiment, it is shown that the sense of weight can be perceived by changing at least one of the luminance, color, or pattern of the visual object by an amount corresponding to the operation based on the movement, even without changing the size or shape of the visual object. To demonstrate this, in this embodiment, a two-dimensional figure with concentric stripes of a fixed diameter is used as the visual object. In the second embodiment, the sense of weight is perceived by changing the phase in the spatial domain of the concentric stripes of the visual object (hereinafter simply referred to as "phase") by an amount corresponding to the operation based on the movement. The following description will focus on differences from the first embodiment, and the same reference numbers will be used to simplify the description of matters already described.

[0033] <Configuration> 1 and 2, the pseudo-tactile presentation system 2 of this embodiment includes a pseudo-tactile presentation device 21, a detection device 12, and a visual object presentation device 13. The pseudo-tactile presentation device 21 includes an input unit 111, a memory unit 112, a motion information detection unit 113, an operation determination unit 115, a visual object update unit 216, and an output unit 117.

[0034] <Pretreatment> The preprocessing of this embodiment is the same as that of the first embodiment, except that the visual target 130 is replaced with the visual target 230, a two-dimensional disk-shaped figure with concentric stripes of a fixed diameter (see, for example, FIG. 8 ), and the change in diameter c of the visual target 130 is replaced with the change in phase c of the concentric stripes of the visual target 230. In the example of FIG. 3 , one period of the concentric stripes of the visual target 230 in the spatial domain is set to 100 pixels. That is, the change in phase c [pixels] of the concentric stripes of the visual target 230 represents a phase change of 2π(c / 100) [rad]. FIG. 8 illustrates an example in which the change in luminance value with respect to the change in distance from the center O of the concentric stripes in the spatial domain is expressed as a sine wave (i.e., concentric stripes with sine wave luminance modulation). However, this does not limit the present invention, and any concentric stripes having a periodic stripe structure in which the change in luminance value with respect to the change in distance from the center O can be used. For example, the change in luminance value with respect to the change in distance from the center O may be expressed as a square wave or a triangular wave. Also, similar to the first embodiment, the information p may or may not include information on the initial visual target 230. Examples of the information on the initial visual target 230 include information on the phase of the concentric stripes of the initial visual target 230 (initial value of the phase), and information on the size, shape, luminance, color, and pattern of the initial visual target 230. Also, information b={b + ,b -} may be included.

[0035] <Operation> Next, an example of the operation of the pseudo-tactile sense presentation system 2 of this embodiment will be described. The visual object update unit 216 (FIG. 1) generates an initial visual object 230 and outputs information v representing the initial visual object 230. The initial visual object 230 exemplified in this embodiment has concentric stripes with an initial phase. The phase of the concentric stripes may be expressed in any manner. For convenience of explanation, in this embodiment, as exemplified in FIG. 8, a position P 1 from the center O of the concentric stripes where the luminance becomes a specific value (for example, 0 corresponding to black) is used. xThe phase is expressed by the distance x in the spatial domain to the initial visual object 230. However, this does not limit the present invention. According to this representation, the initial visual object 230 in this embodiment is a two-dimensional disk-shaped figure with concentric phase stripes, where the distance x has an initial value x1. The initial value x1 of the distance x (initial value of the phase) may be a value included in the information p read from the storage unit 112, or may be a predetermined value. Furthermore, the size, shape, brightness, color, and pattern of the initial visual object 230 may be values ​​included in the information p read from the storage unit 112, or may be predetermined values. Information v representing the initial visual object 230 is sent to the output unit 117 and output from the output unit 117 to the visual object presentation device 13. The visual object presentation device 13 presents (displays) the visual object 230 based on the sent information v (FIGS. 2 and 8).

[0036] The user 100 moves a body part 101 (for example, a finger) in a touchless environment while looking at a visual object 230 presented by the visual object presentation device 13. The position of the body part 101 is detected by the detection device 12. Information d representing the position of the body part 101 detected by the detection device 12 is output to the motion information detection unit 113. Every time information d is input, the motion information detection unit 113 detects the motion of the body part 101 from the information d and outputs information a representing the motion of the body part 101. m =(d1, d2). Otherwise, the operations of the detection device 12 and the motion information detection unit 113 are the same as those in the first embodiment.

[0037] Information a m =(d1, d2) is input to the operation determination unit 115, and the operation determination unit 115 determines the information a m Based on this, operation information representing an operation based on the movement of the body part 101 is obtained and output. As in the first embodiment, in this embodiment, an example will be described in which information representing the operation amount m is output as operation information. The operation of the operation determination unit 115 is the same as in the first embodiment.

[0038] Information representing the operation amount m is input to the visual object update unit 216. The visual object update unit 216 refers to information p stored in the storage unit 112, and visually changes the visual object 230 by an amount of change corresponding to the operation, based on at least information representing the operation amount m (operation information representing an operation based on the movement of a body part) and information p (information representing the relationship between the amount of visual change of the visual object and the operation amount based on the operation). This operation is the same as the operation of the visual object update unit 116 in the first embodiment, except that instead of changing the diameter of the visual object 130 from the initial value c1 to c2 = c1 + b·c, the phase of the concentric stripes of the visual object 230 is changed so that the distance x is changed from x1 to x2 = x1 + b·c. The amount of change c is positive, and b = b + In this case, changing the phase of the concentric stripes of the visual object 230 so that the distance x changes from x1 to x2 = x1 + b·c means moving the concentric stripes in the direction from the center O to the outside. + If b=b, the stripes are moved outward as the vertical position of the body part 101 (for example, fingers) increases, and the stripes are moved inward as the vertical position of the body part 101 (for example, fingers) decreases. - In this case, changing the phase of the concentric stripes of the visual object 230 so that the distance x changes from x1 to x2 = x1 + b·c means moving the concentric stripes from the outside toward the center O. That is, b = b - In this case, the stripes are moved inward as the height (vertical component of the amount of movement) of the body part 101 (for example, fingers) increases, and the stripes are moved outward as the height of the body part 101 (for example, fingers) decreases. Note that the diameter (size) and shape of the visual object 230 are not changed.

[0039] 9A to 11C are used to illustrate the difference in change of the visual object 230 when the degree of pseudo-tactile sensation to be presented differs. The horizontal axis in FIGS. 9B, 9C, 10B, 10C, 11B, and 11C represents the distance x, and the vertical axis represents the pixel value. For example, when the index representing the degree of pseudo-tactile sensation to be presented (for example, a sense of weight, a sense of weight, or a sense of resistance) is Index 1 (FIG. 3), the visual object update unit 216 updates the phase of the concentric stripes of the visual object 230 with respect to the operation amount m, which is the vertical component of the distance from the motion start position vh1 of the body part 101 to the current position vh1, so that the distance x is the initial x1=x 1-1 From x2=x 1-2 =x 1-1 For example, the visual object update unit 216 changes the value of b to b + If so, the concentric stripes of the visual object 230 are moved outward (FIGS. 9A to 9C), and b=b - On the other hand, if the index representing the degree of the pseudo-tactile sensation to be presented is Index 2, the visual object update unit 216 updates the phase of the concentric stripes of the visual object 230 for the same operation amount m such that the distance x is the initial x1 = x2. 2-1 From x2=x 2-2 =x 2-1 For example, the visual object update unit 216 changes the value of b to b + If so, the concentric stripes of the visual object 230 are moved outward (FIGS. 10A to 10C), and b=b - If so, the concentric stripes of the visual object 230 are moved inward (FIGS. 11A to 11C). Here, the degree of the pseudo-tactile sensation (for example, a sense of weight, a sense of weight, a sense of resistance) represented by the index Index 1 is greater than the degree of the pseudo-tactile sensation represented by the index Index 2. In this case, even though the operation amount m is the same, the amount of change in the phase of the visual object 230 with respect to the index Index 1 |x 1-2 -x 1-1 | is the phase change amount |x of the visual object 230 relative to the index Index 2 2-2 -x 2-1 Less than |

[0040] The information v representing the visual object 130 output from the visual object update unit 216 is sent to the output unit 117, which then outputs it to the visual object presentation device 13. The visual object presentation device 13 presents (displays) the visual object 230 based on the sent information v (FIGS. 2 and 8). As a result, the visual object 230 presented to the user 100 changes by a change amount c according to the operation based on the movement of the body part 101, and a pseudo-tactile sensation (e.g., a sense of weight, a sense of weight, a sense of resistance) can be presented to the user 100. Furthermore, if the information p represents multiple relationships r with respect to multiple degrees of pseudo-tactile sensation, a selected degree of pseudo-tactile sensation (e.g., a selected magnitude of a sense of weight, a sense of weight, a sense of resistance) can be presented.

[0041] <Experimental Results> Next, the results of Experiments 2 and 3 will be illustrated to show the effects of the second embodiment. Experiment 2: b = b +In this experiment, the stripes were moved outward as the vertical position of the body part 101 (e.g., fingers) increased, and the stripes were moved inward as the vertical position of the body part 101 (e.g., fingers) decreased. As illustrated in FIG. 12A, in Experiment 2, the visual object presentation device 13 presented a visual object 230 (evaluation stimulus) to be evaluated, as well as a reference visual object 240 (standard stimulus). Both the visual object 230 and the visual object 240 were two-dimensional disk-shaped images with concentric stripes, and their phases changed based on the movement of the participant's fingers. However, the visual object 230 and the visual object 240 differed in the amount of phase change relative to the amount of movement. That is, the position of the concentric stripes of the visual object 240 changed by 20 pixels for every 1 cm change in the height (vertical component of the movement) of the finger. In other words, if one period of the concentric stripes in the spatial domain is 100 pixels, the phase of the concentric stripes changes by 0.4π rad for every 1 cm change in the height of the finger. In contrast, for the visual target 230, Case 1, Case 2, Case 3, Case 4, Case 5, Case 6, Case 7, Case 8, Case 9, Case 10, Case 110, Case 120, Case 130, Case 140, Case 150, Case 160, Case 170, Case 180, Case 190, Case 200, Case 210, Case 220, Case 230, Case 240, Case 250, Case 260, Case 270, Case 280, Case 290, Case 310, Case 320, Case 330, Case 340, Case 350, Case 360, Case 370, Case 380, Case 39 ... That is, the ratio of the phase change of visual object 230 to the phase change of visual object 240 (speed ratio) for cases 1, 2, 3, 4, and 5 was 0.25, 0.5, 1.0, 2.0, and 4.0, respectively. Figure 6B shows the phase change [pixels] of the concentric stripes of visual object 130 relative to the height [cm] of the participant's finger at each speed ratio. In Experiment 2, the horizontal axis of Figure 6B represents the height [cm] of the participant's finger, and the vertical axis represents the phase change [pixels] of the concentric stripes of visual object 230. For cases 1, 2, 3, 4, and 5, participants compared visual object 230 and visual object 240 and reported which of them they perceived as heavier using a rating system.As in Experiment 1, the evaluation was conducted using a 5-point scale with evaluation scores ranging from 1 to 5. Figure 12B shows the results of Experiment 2. The horizontal axis of Figure 12B represents the speed ratio, and the vertical axis represents the average evaluation score (1 to 5) of the sense of heaviness for each participant. As illustrated in Figure 12B, it was found that the smaller the speed ratio, the heavier the visual object 230 felt. In other words, b = b. + In this case, the smaller the change in the phase of the concentric stripes of the visual object 230 relative to the amount of manipulation based on the operation, the greater the perceived sense of weight. Furthermore, this result shows that even if there is no change in the size or shape of the visual object, a sense of weight can be perceived by changing the phase of the concentric stripes of the visual object 230 according to the amount of manipulation based on the operation.

[0042] Experiment 3: b = b - In other words, the higher the vertical position of the body part 101 (for example, fingers), the more the stripes are moved inward, and the lower the vertical position of the body part 101 (for example, fingers), the more the stripes are moved outward. - The experimental method was the same as in Experiment 2, except for the above. Figure 13 shows the experimental results of Experiment 3. The horizontal axis of Figure 13 represents the speed ratio, and the vertical axis represents the average evaluation score (1 to 5) of the sense of weight for each participant. As shown in Figure 13, it was found that the visual object 230 felt heavier as the speed ratio became smaller. That is, b = b - In this case, too, the smaller the amount of change in the phase of the concentric stripes of the visual object 230 relative to the amount of operation based on the operation, the greater the perceived sense of weight. Furthermore, this result shows that regardless of the direction of change in the phase of the concentric stripes (direction of movement of the stripes), it is possible to perceive a sense of weight by changing the phase of the concentric stripes of the visual object 230 in accordance with the amount of operation based on the operation.

[0043] [Third embodiment] In the second embodiment, the sense of weight is perceived by changing the phase of the concentric stripes of the visual object by an amount corresponding to the operation based on the movement. However, this effect can be obtained not only by changing the phase of the concentric stripes, but also by changing the phase of other figures. To demonstrate this, in this embodiment, the visual object is a two-dimensional figure with one-dimensional stripes. The visual object in this embodiment also does not change in size or shape. The following description will focus on the differences from the first and second embodiments, and the same reference numbers will be used to simplify the description of items already described.

[0044] 1 and 2, the pseudo-tactile presentation system 3 of this embodiment includes a pseudo-tactile presentation device 31, a detection device 12, and a visual object presentation device 13. The pseudo-tactile presentation device 31 includes an input unit 111, a memory unit 112, a motion information detection unit 113, an operation determination unit 115, a visual object update unit 316, and an output unit 117.

[0045] <Pretreatment> The preprocessing of this embodiment is the same as that of the first embodiment, except that the visual object 130 is replaced with the visual object 330, which is a rectangular two-dimensional figure with one-dimensional stripes of fixed size (e.g., FIG. 14), and the change in diameter c of the visual object 130 is replaced with the change in phase c of the one-dimensional stripes of the visual object 330. Note that FIG. 14 illustrates an example in which the change in luminance value with respect to the change in distance from the edge E of the visual object 330 in the spatial domain is expressed as a sine wave (i.e., one-dimensional stripes with sinusoidal luminance modulation). However, this does not limit the present invention, and any one-dimensional stripes may be used as long as the change in luminance value with respect to the change in distance from the edge E has a periodic stripe structure. For example, the change in luminance value with respect to the change in distance from the edge E may be expressed as a square wave, a triangular wave, or the like. Furthermore, as in the first embodiment, the information p may or may not include information about the initial visual object 330. Examples of information about the initial visual target 330 include information about the phase of the one-dimensional stripes of the initial visual target 330 (initial value of the phase), and information about the size, shape, brightness, color, and pattern of the initial visual target 330. In addition, information b={b + ,b-} may be included.

[0046] <Operation> Next, an example of the operation of the pseudo-tactile sense presentation system 3 of this embodiment will be described. The visual object update unit 316 (FIG. 1) generates an initial visual object 330 and outputs information v representing the initial visual object 330. The initial visual object 330 exemplified in this embodiment has one-dimensional stripes with an initial phase. The phase of the one-dimensional stripes may be expressed in any manner. For convenience of explanation, in this embodiment, as exemplified in FIG. 14, a position P 1 from the edge E of the one-dimensional stripes where the luminance becomes a specific value (for example, 0 corresponding to black) is used. x The phase is expressed by the distance x in the spatial domain to the initial visual object 330. However, this does not limit the present invention. According to this expression, the initial visual object 330 in this embodiment is a rectangular two-dimensional figure having one-dimensional stripes of phase, with the distance x being an initial value x1. The initial value x1 of the distance x (initial value of the phase) may be a value included in the information p read from the storage unit 112, or may be a predetermined value. Furthermore, the size, shape, brightness, color, and pattern of the initial visual object 330 may be values ​​included in the information p read from the storage unit 112, or may be predetermined values. Information v representing the initial visual object 330 is sent to the output unit 117 and output from the output unit 117 to the visual object presentation device 13. The visual object presentation device 13 presents (displays) the visual object 330 based on the sent information v (FIGS. 2 and 14).

[0047] The user 100 moves a body part 101 (for example, a finger) in a touchless environment while looking at a visual object 330 presented by the visual object presentation device 13. The position of the body part 101 is detected by the detection device 12. Information d representing the position of the body part 101 detected by the detection device 12 is output to the motion information detection unit 113. Every time information d is input, the motion information detection unit 113 detects the motion of the body part 101 from the information d and outputs information a representing the motion of the body part 101. m =(d1, d2). Otherwise, the operations of the detection device 12 and the motion information detection unit 113 are the same as those in the first embodiment.

[0048] Information a m =(d1, d2) is input to the operation determination unit 115, and the operation determination unit 115 determines the information a m Based on this, operation information representing an operation based on the movement of the body part 101 is obtained and output. As in the first embodiment, in this embodiment, an example will be described in which information representing the operation amount m is output as operation information. The operation of the operation determination unit 115 is the same as in the first embodiment.

[0049] Information representing the operation amount m is input to the visual object update unit 316. The visual object update unit 316 refers to the information p stored in the memory unit 112, and visually changes the visual object 330 by an amount of change corresponding to the operation, based on at least the information representing the operation amount m (operation information representing an operation based on the movement of a body part) and the information p (information representing the relationship between the amount of visual change of the visual object and the operation amount based on the operation). This operation is the same as the operation of the visual object update unit 116 in the first embodiment, except that instead of changing the diameter of the visual object 130 from the initial value c1 to c2 = c1 + b·c, the phase of the one-dimensional stripes of the visual object 330 is changed so that the distance x is changed from x1 to x2 = x1 + b·c. The amount of change c is positive, and b = b + If the phase of the one-dimensional fringes of the visual object 330 is changed from x1 to x2 = x1 + b c, the fringes of the one-dimensional fringes are moved to the right from the edge E. That is, b = b + If the vertical position of the body part 101 (for example, a finger) is higher, the stripe is shifted to the right, and if the vertical position of the body part 101 (for example, a finger) is lower, the stripe is shifted to the left. Also, if the change amount c is positive and b=b - In this case, changing the phase of the one-dimensional fringes of the visual object 330 so that the distance x changes from x1 to x2 = x1 + b·c means moving the one-dimensional fringes from the right side toward the edge E (to the left). That is, b = b - In this case, the stripes are shifted to the left as the height (vertical component of the amount of movement) of the body part 101 (e.g., fingers) increases, and the stripes are shifted to the right as the height of the body part 101 (e.g., fingers) decreases. Note that the size and shape of the visual object 330 are not changed.

[0050] The information v representing the visual object 130 output from the visual object update unit 316 is sent to the output unit 117, which then outputs it to the visual object presentation device 13. The visual object presentation device 13 presents (displays) the visual object 330 based on the sent information v (FIGS. 2 and 14). As a result, the visual object 330 presented to the user 100 changes by an amount c in accordance with the operation based on the movement of the body part 101, and a pseudo-tactile sensation (e.g., a sense of weight, a sense of weight, a sense of resistance) can be presented to the user 100. Furthermore, if the information p represents multiple relationships r with respect to multiple degrees of pseudo-tactile sensation, a selected degree of pseudo-tactile sensation (e.g., a selected magnitude of a sense of weight, a sense of weight, a sense of resistance) can be presented.

[0051] <Experimental Results> Next, the results of Experiment 4 will be illustrated to demonstrate the effects of the third embodiment. As shown in FIG. 15A, in Experiment 4, the visual object presentation device 13 presented a visual object 330 (evaluation stimulus) to be evaluated, as well as a reference visual object 340 (standard stimulus). Both visual objects 330 and 340 were rectangular two-dimensional images with one-dimensional stripes, and their phases changed based on the participant's finger movements. However, the visual objects 330 and 340 differed in the amount of phase change relative to the amount of manipulation. That is, the position of the one-dimensional stripes of the visual object 340 changed by 20 pixels for every 1 cm change in the height of the finger (the vertical component of the movement). In other words, if one period of the one-dimensional stripes in the spatial domain is 100 pixels, the phase of the one-dimensional stripes changed by 0.4π rad for every 1 cm change in the height of the finger. In contrast, for visual target 330, the position of the one-dimensional stripes changed by 5 pixels for every 1 cm change in finger height: Case 1, Case 2, Case 3, Case 4, Case 5, Case 6, Case 7, Case 8, Case 9, Case 10, Case 20, Case 340, Case 40, and Case 580. In Experiment 4, the ratios of the phase of visual target 330 to the phase change of visual target 340 (speed ratios) for Cases 1, 2, 3, 4, and 5 were 0.25, 0.5, 1.0, 2.0, and 4.0, respectively (Figure 6B). As in Experiments 2 and 3, the horizontal axis of Figure 6B represents the height of the participant's finger (cm), and the vertical axis represents the phase change (pixels) of the one-dimensional stripes of visual target 330. For cases 1, 2, 3, 4, and 5, participants compared visual object 330 and visual object 340 and reported which of them felt heavier using a rating system. Similar to Experiment 1, the ratings were conducted on a five-point scale with ratings ranging from 1 to 5. Figure 15B shows the results of Experiment 4. The horizontal axis of Figure 15B represents the speed ratio, and the vertical axis represents the average weight rating score (1 to 5) for each participant. As illustrated in Figure 15B, it was found that, in this case as well, the smaller the speed ratio, the heavier the visual object 330 felt. In other words, the smaller the change in the phase of the one-dimensional stripes of visual object 330 relative to the amount of manipulation, the greater the perceived weight.Furthermore, these results show that regardless of the type of stripes or spatial structure, the sense of weight can be perceived by changing the phase of the stripes of the visual object 330 according to the amount of operation based on the operation.

[0052] [Other variations] In the first to third embodiments, an example was shown in which the user is made to perceive a sense of weight by changing the size and phase of a visual object by an amount corresponding to the operation, based on operation information representing an operation based on the movement of a body part and information representing the relationship between the amount of visual change in the visual object and the amount of operation based on the operation. However, as mentioned above, the present invention is not limited to this. For example, the shape of the visual object may be disk-shaped, elliptical, rectangular, other polygonal, or other shape. The visual change of the visual object is also not limited. For example, the way in which the size of the visual object changes is not limited to that of the first embodiment. The size may change while the visual object is deformed by an amount corresponding to the operation, or the size and shape of the visual object may change while at least one of the luminance, color, or pattern of the visual object changes. The phase of periodic stripes of the visual object may change while the size and shape of the visual object change by an amount corresponding to the operation. The visual object may be a two-dimensional image with a non-periodic pattern (e.g., a white noise image), and at least one of the size, shape, pattern, luminance, and color of the visual object may change by an amount corresponding to the operation. Furthermore, as described above, for the same index i, the relationship r (for example, r = Δc / Δm) may be constant, or the relationship r (for example, r = Δc / Δm) may vary depending on the magnitude of m. Furthermore, as described above, the visual object update unit 116, 216, 316 may switch the index i representing the degree of pseudo-tactile sensation depending on the operation amount m, refer to the relationship r corresponding to the switched index i, obtain the amount of change c corresponding to the input operation amount m, and visually change the visual object 130 with the obtained amount of change.

[0053] Furthermore, the various processes described above may not only be executed in chronological order as described, but may also be executed in parallel or individually depending on the processing capabilities of the devices executing the processes or as necessary. Needless to say, other modifications are possible within the scope of the present invention.

[0054] [Hardware configuration] The pseudo-tactile presentation devices 11, 21, and 31 in each embodiment are devices configured by a general-purpose or dedicated computer having a processor (hardware processor) such as a central processing unit (CPU) and memories such as random-access memory (RAM) and read-only memory (ROM) executing a predetermined program. That is, the pseudo-tactile presentation devices 11, 21, and 31 in each embodiment have, for example, a processing circuit configured to implement each of the components. This computer may have one processor and memory, or multiple processors and memories. This program may be installed on the computer or may be pre-recorded in a ROM or the like. Furthermore, some or all of the processing units may be configured using electronic circuits that independently realize processing functions, rather than electronic circuits that realize functional configurations by loading programs like a CPU. Furthermore, the electronic circuits constituting one device may include multiple CPUs.

[0055] FIG. 16 is a block diagram illustrating the hardware configuration of the pseudo-tactile presentation devices 11, 21, and 31 in each embodiment. As illustrated in FIG. 16, the pseudo-tactile presentation devices 11, 21, and 31 in this example include a central processing unit (CPU) 10a, an input unit 10b, an output unit 10c, a random access memory (RAM) 10d, a read-only memory (ROM) 10e, an auxiliary storage device 10f, and a bus 10g. The CPU 10a in this example includes a control unit 10aa, a calculation unit 10ab, and a register 10ac, and executes various calculation processes according to various programs loaded into the register 10ac. The input unit 10b is an input terminal for inputting data, a keyboard, a mouse, a touch panel, or the like. The output unit 10c is an output terminal for outputting data, a display, a LAN card controlled by the CPU 10a that has loaded a predetermined program, or the like. The RAM 10d is a static random access memory (SRAM), a dynamic random access memory (DRAM), or the like, and has a program area 10da where a predetermined program is stored and a data area 10db where various data are stored. The auxiliary storage device 10f is a hard disk, a magneto-optical disc (MO), a semiconductor memory, or the like, and has a program area 10fa where a predetermined program is stored and a data area 10fb where various data are stored. The bus 10g connects the CPU 10a, the input unit 10b, the output unit 10c, the RAM 10d, the ROM 10e, and the auxiliary storage device 10f so that information can be exchanged. The CPU 10a writes the program stored in the program area 10fa of the auxiliary storage device 10f to the program area 10da of the RAM 10d in accordance with the loaded OS (Operating System) program. Similarly, the CPU 10a writes various data stored in the data area 10fb of the auxiliary storage device 10f to the data area 10db of the RAM 10d. The addresses on the RAM 10d where the programs and data are written are stored in the register 10ac of the CPU 10a.The control unit 10aa of the CPU 10a sequentially reads out these addresses stored in the register 10ac, reads out programs and data from the areas on the RAM 10d indicated by the read addresses, causes the calculation unit 10ab to sequentially execute the calculations indicated by the programs, and stores the calculation results in the register 10ac. With this configuration, the functional configuration of the pseudo-tactile presentation devices 11, 21, and 31 is realized.

[0056] The above-mentioned program can be recorded on a computer-readable recording medium. Examples of computer-readable recording media include non-transitory recording media. Examples of such recording media include magnetic recording devices, optical disks, magneto-optical recording media, and semiconductor memories.

[0057] This program may be distributed, for example, by selling, transferring, or lending a portable recording medium, such as a DVD or CD-ROM, on which the program is recorded. Furthermore, the program may be distributed by storing the program in a storage device of a server computer and transferring the program from the server computer to other computers via a network. As described above, a computer that executes such a program may first temporarily store the program, recorded on a portable recording medium or transferred from the server computer, in its own storage device. Then, when executing a process, the computer reads the program stored in its own storage device and executes processing in accordance with the read program. Alternatively, the program may be executed by a computer that reads the program directly from a portable recording medium and executes processing in accordance with the program. Furthermore, the computer may execute processing in accordance with the received program each time a program is transferred from the server computer to the computer. Alternatively, the server computer may not transfer the program to the computer, but may instead execute the processing function simply by issuing an execution instruction and obtaining the results, thereby executing the processing described above through a so-called ASP (Application Service Provider) service. In this embodiment, the program includes information used for processing by an electronic computer that is equivalent to a program (such as data that is not a direct instruction to a computer but has properties that dictate computer processing).

[0058] In each embodiment, the device is configured by executing a predetermined program on a computer, but at least a part of the processing contents may be realized by hardware. [Explanation of symbols]

[0059] 11,21,31 Pseudo-tactile presentation device 116,216,316 Visual Object Update Unit

Claims

1. A visual object update unit that visually changes the visual object by an amount of change corresponding to the operation, based on operation information representing an operation based on a movement of a body part and information representing a relationship between an amount of visual change of the visual object and an amount of operation based on the operation; an output unit that outputs information for presenting the visually changed visual target, The relationship is determined at least in accordance with the degree of the simulated haptic sensation to be presented, the visual change amount of the visual object corresponds to a change amount of at least one of a size, a shape, a luminance, a color, and a pattern of the visual object; The visual object update unit changes at least one of the size, shape, brightness, color, and pattern of the visual object.

2. A visual object update unit that visually changes the visual object by an amount of change corresponding to the operation, based on operation information representing an operation based on a movement of a body part and information representing a relationship between an amount of visual change of the visual object and an amount of operation based on the operation; an output unit that outputs information for presenting the visually changed visual target, The relationship is determined at least in accordance with the degree of the simulated haptic sensation to be presented, the relationship being a ratio of a change in the change amount to a change in the manipulated variable, When the index representing the degree of the pseudo tactile sensation is a first index value, the ratio is a first value; A pseudo-tactile presentation device, wherein when the index is a second index value different from the first index value, the ratio is a second value different from the first value.

3. The pseudo-tactile presentation device according to claim 2, the degree of the pseudo haptic sensation represented by the first index value is greater than the degree of the pseudo haptic sensation represented by the second index value; A pseudo-tactile presentation device, wherein the first value is smaller than the second value.

4. The pseudo-tactile presentation device according to claim 2 or 3, In response to the operation, the operation amount changes from a first operation amount to a second operation amount and / or the operation amount changes from the second operation amount to the first operation amount; the first manipulated variable is different from the second manipulated variable; the ratio is a third value when the manipulated variable is the first manipulated variable; A pseudo-tactile presentation device, wherein the ratio when the operation amount is the second operation amount is a fourth value different from the third value.

5. The pseudo-tactile presentation device according to claim 4, the first operation amount is greater than the second operation amount, A pseudo-tactile presentation device, wherein the third value is smaller than the fourth value, or the third value is larger than the fourth value.

6. a visual object updating step of visually changing the visual object by an amount of change according to the operation, based on operation information representing an operation based on a movement of a body part and information representing a relationship between an amount of visual change of the visual object and an amount of operation based on the operation; and an output step of outputting information for presenting the visually changed visual target, The relationship is determined at least in accordance with the degree of the simulated haptic sensation to be presented, the visual change amount of the visual object corresponds to a change amount of at least one of a size, a shape, a luminance, a color, and a pattern of the visual object; A pseudo-tactile presentation method, wherein the visual object updating step changes at least one of the size, shape, brightness, color, and pattern of the visual object.

7. A visual object updating step of visually changing the visual object by an amount of change corresponding to the operation based on operation information representing an operation based on a movement of a body part and information representing a relationship between an amount of visual change of the visual object and an amount of operation based on the operation; and an output step of outputting information for presenting the visually changed visual target, The relationship is determined at least in accordance with the degree of the simulated haptic sensation to be presented, the relationship being a ratio of a change in the change amount to a change in the manipulated variable, When the index representing the degree of the pseudo tactile sensation is a first index value, the ratio is a first value; A pseudo-tactile presentation method, wherein when the index is a second index value different from the first index value, the ratio is a second value different from the first value.

8. A program for causing a computer to function as the pseudo-tactile presentation device according to any one of claims 1 to 5.

Images