Pseudo-force sense presentation device, pseudo-force sense presentation method, and program

Abstract

The present invention implements a pseudo-haptic effect without changing the trajectory and size of a cursor. This pseudo-haptic effect presentation device for this implementation comprises a cursor detection unit, a cursor update unit, and a video display unit. The cursor detection unit detects information on the position of the cursor being an image displayed on a video display unit for operation of the device by a user. The cursor update unit updates the cursor image while continuously modifying the cursor image on the basis of the position information. The video display unit presents the cursor, the image of which is updated, to the user according to the position information.

Description

[Technical Field]

[0001] The disclosed technology relates to a pseudo-force sensation presentation technology using optical illusions, for example, a technique for presenting image information to a user that creates an illusion of operating a cursor on a computer screen. [Background technology]

[0002] A technology similar to pseudo-force presentation technology is "pseudo-tactile technology." Pseudo-haptic technology provides the user with visual feedback in response to positional input by manipulating visual information (such as a cursor) based on positional information obtained from an input device (such as a mouse, keyboard, touchpad, touch pen, touch panel, gesture recognizer, or force feedback device). By adding delays or fluctuations to the visual feedback (such as a cursor) and changing its movement trajectory or size, the user can be given an illusionary haptic impression that is different from the actual one. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Costes et al., Touchy:A Visual Approach for Simulating Haptic Effects on Touchscreens, Frontiers in ICT, February 2019 / Vol.6 / Article 1<https: / / doi.org / 10.3389 / fict.2019.00001> Summary of the Invention [Problem to be solved by the invention]

[0004] Hereinafter, we will refer to the visual medium that is displayed on the video display unit (display, virtual reality headset, video projector screen, etc.) of a computing device (computer, smartphone, tablet, etc.) based on position information obtained from an input device to allow a user to operate the computing device as a "cursor." With conventional pseudo-haptic technology, it was necessary to change the cursor's movement path and size in response to user operations. However, changing the movement path and size of a cursor, which is used for computer operation, independently of the operation, posed a problem in that it interfered with the intended operation. [Means for solving the problem]

[0005] In order to solve the above problems, the pseudo-force presentation device according to the disclosed technology includes a cursor detection unit, a cursor update unit, and a video display unit. The cursor detection unit detects position information of a cursor, which is an image displayed on the video display unit for a user to operate the device. The cursor update unit continuously changes and updates the image of the cursor based on the position information. The video display unit presents the cursor with the updated image to the user in accordance with the position information. [Effects of the Invention]

[0006] According to the disclosed technology, it is possible to realize a device that presents a pseudo-force sensation to a user without changing the actual drawing position or drawing size of the cursor, and therefore without interfering with the user's actual cursor operation. [Brief explanation of the drawings]

[0007] [Figure 1] 5A to 5C are diagrams illustrating cursor operations according to the first embodiment. [Figure 2] 5A to 5C are diagrams illustrating changes in cursor patterns in the first embodiment. [Figure 3] 10A to 10C are diagrams illustrating a modification of the cursor operation of the first embodiment. [Figure 4] FIG. 1 is a functional block diagram of a pseudo-force sense presentation device according to a first embodiment. [Figure 5] FIG. 4 is a flowchart illustrating the operation of the pseudo force feedback device according to the first embodiment. [Figure 6] 10A to 10C are diagrams illustrating changes in cursor patterns in the second embodiment. [Figure 7] 13A to 13C are diagrams illustrating changes in cursor patterns in the third embodiment. [Figure 8] A diagram explaining cursor operation in Experiment 1. [Figure 9] A diagram illustrating the results of Experiment 1. [Figure 10] A diagram explaining cursor operation in Experiment 2. [Figure 11] A diagram illustrating the results of Experiment 2. [Figure 12] FIG. 2 is a diagram showing an example of the functional configuration of a computer. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, embodiments of the disclosed technology will be described in detail. Note that components having the same functions are assigned the same numbers, and duplicated descriptions will be omitted.

[0009] [Summary of the disclosed technology] The disclosed technology uses a phenomenon that creates an illusion of the visual position and size of the cursor displayed on the video display unit, allowing the user to perceive a pseudo-force without modifying the actual display position (movement trajectory) or size of the cursor. Specifically, by gradually shifting the cursor pattern displayed on the video display unit, the user is given the illusion of a force being applied in the direction opposite to the shift of the pattern, creating an optical illusion that makes the user feel as if the cursor itself is moving in the direction opposite to the shift of the pattern.

[0010] [First embodiment] <Description of the phenomenon the user sees> First, we will explain how pseudo-force sensations are presented on the display, using the example of a situation where a user operates a cursor using a mouse. Figure 1 shows an example of using a mouse 101 to operate a cursor 103 displayed on a display 102. As shown in Figure 1, the user operates the mouse so that the cursor moves horizontally within the display.

[0011] Using Figure 2, we will explain how the pattern displayed as the cursor changes. A base image 202 consisting of a random noise image is prepared, which is sufficiently larger than the size of the cursor 201. The horizontal and vertical directions of the display 102 correspond to the x-axis and y-axis of the base image 202. An image fragment cut out from a part of the base image 202 is pasted onto the cursor 201. 203 to 209 show how the image fragment displayed as the cursor changes as the cursor moves from left to right on the display (in the positive direction on the x-axis). In this example, the image fragment in the middle of the base image 202 in the y-axis direction is initially displayed as the cursor, but as the cursor moves along the x-axis, the image fragment at the top of the base image 202 gradually appears, and then the cursor pattern changes so that it gradually returns to displaying the original image fragment in the middle. In this case, the user sees the cursor moving horizontally, with its pattern (random noise pattern) first flowing downward and then flowing upward. This change in the cursor pattern gives the user the illusion that the cursor is moving in a convex trajectory, and the user perceives the force that would have been exerted if the cursor trajectory had actually changed.

[0012] In the above, we have taken the example of moving the cursor horizontally (parallel to the x-axis) on the display to make it easier to get an idea of ​​the pseudo-force sensation, but it should be clear that the cursor does not need to move parallel to the x-axis to present a pseudo-force sensation in the y-axis direction. When the cursor moves along a curve like the one in Figure 3, if you use the x-coordinate as a variable and change the random noise pattern displayed as the cursor in the same way as above, you can present a pseudo-force sensation in the y-axis direction. We will discuss later whether humans actually experience the above illusion.

[0013] <Pseudo-force presentation device and presentation procedure> FIG. 4 is a diagram showing the functional blocks of the pseudo-force sense presentation device, and FIG. 5 is a flowchart illustrating the operation of the pseudo-force sense presentation device. The pseudo-force sense presentation device 401 includes at least a cursor detection unit 402 , a cursor update unit 403 , and a video display unit 404 . The cursor detection unit 402 receives a signal from the input device 410, measures where the cursor operated by the user is located on the video display unit, or where it should be displayed, and detects the cursor position (step S501). In step S502, it is determined whether the cursor has moved, and if the cursor position has not changed, the process returns to step S501 (step S502). The cursor update unit 403 includes the base image 202 and a function f that defines the relationship between the cursor position and the image fragment to be extracted. If the cursor is circular, f may be a function that takes x as a variable and gives the center coordinate y of the image fragment (circle), for example. The cursor update unit 403 calculates a random noise image piece to be cut out from the basic image 202 based on x and f (step S503), and updates the cursor image (step S504). The updated cursor is presented to the user on the video display unit 404 (step S505), and the process returns to step S501.

[0014] In the above, the center of the circular area to be extracted is calculated using a function from the cursor position, but the method of associating the cursor position with the area of ​​the image fragment is not limited to this; for example, the cursor position and the area of ​​the image fragment to be extracted may be associated using a table. It is desirable that the change in the cursor pattern appears smooth and continuous to the human eye.

[0015] [Second embodiment] In the first embodiment, when the cursor is moved on the display, the random noise pattern is pasted while being shifted vertically (y direction) according to the x coordinate. In the second embodiment, the random noise pattern is shifted within the xy plane according to the x coordinate of the cursor. For example, as shown in Figure 6, the extracted image fragment is gradually shifted to the upper right region of the base image. At this time, the user has the illusion that the cursor is bending upward and to the right relative to its actual trajectory, and perceives a force pulling it upward and to the right. In the second embodiment, the cursor update unit 403 in Fig. 4 has a basic image 601 in Fig. 6 instead of the basic image 202 in Fig. 2. Also, the function for extracting image fragments is a function that inputs an x ​​coordinate and outputs an xy coordinate. Apart from the above, the device configuration and operation are the same as those in the first embodiment.

[0016] [Third embodiment] In the first and second embodiments, the cursor image is changed according to the x coordinate. In the third embodiment, the cursor image changes according to the x and y coordinates of the cursor. For example, a piece of image is cut out from a random noise pattern that is enlarged or reduced according to the cursor position and pasted onto the cursor. This allows the user to feel a force in the z direction (the front-to-back direction of the display), which is perpendicular to the x and y plane. In the third embodiment, the cursor update unit 403 in Fig. 4 is provided with the base image 701 in Fig. 7. The operation of extracting an image fragment involves obtaining the x and y coordinates of the cursor and enlarging or reducing the base image 701 according to the coordinates to extract the image fragment. Apart from the above, the device configuration and operation are the same as in the first embodiment. In addition, in the above third embodiment, the basic image was enlarged or reduced according to the x and y coordinates of the cursor to extract image fragments, but the extraction position within the basic image may be changed according to the x and y coordinates of the cursor, or image fragments may be extracted by combining the enlargement or reduction of the basic image with displacement within the basic image.

[0017] [Fourth embodiment] In the first to third embodiments, a basic image is prepared in advance and image fragments are extracted from the basic image according to the position of the cursor, but it is also possible to generate image fragments to be attached to the cursor sequentially as the cursor moves without preparing a basic image in advance. The images that are generated sequentially should be ones that appear smooth and continuous to the human eye when presented to the user as a changing cursor pattern. Examples of sequential images that can be generated sequentially include random noise images and repeating patterns (checkerboard patterns, tortoiseshell patterns, etc.). In the fourth embodiment, the cursor update unit in Fig. 4 does not have a base image, and instead of step S503 of calculating image fragments to be cut out from the base image, a step of newly generating image fragments based on cursor position information so that the changes in the cursor pattern presented to the user appear smooth and continuous to the human eye is executed. Apart from the above, the device configuration and operation are the same as in the first embodiment.

[0018] [Points to note when presenting to users] Experiments have shown that when the cursor is highly visible, the perception of pseudo-force sensations decreases (see below). Therefore, when presenting a cursor to the user, it is necessary to keep the following points in mind:

[0019] <Cursor shape border> It is preferable not to use a border around the cursor figure, and if one is used, it is preferable that the border has a brightness similar to the average brightness of the background image.

[0020] <Cursor pattern and background> It is desirable that the background and cursor shape of the cursor be similar, for example, that both the background and cursor shape be made up of random images.

[0021] <Pattern to display on cursor> The basic image used for the cursor pattern does not have to be a random noise pattern, and can be an image containing pictures or text. Also, the higher the contrast of the pattern within the cursor, the more clearly the pattern moves within the cursor, and the stronger the force felt.

[0022] [Verification of disclosed technology] <Experiment 1: Pseudo-force presentation in the y-axis direction> In Experiment 1, which verified the disclosed technology, a user used a mouse to move a cursor with a random noise pattern from the left edge (or right edge) of the screen to the right edge (or left edge) as shown in Figure 8, and evaluated how much unintended force they felt was being applied to the cursor in the upward or downward direction along the way. Specifically, the evaluation was done on a three-point scale: 1: no force was felt, 2: a weak force was felt, and 3: a strong force was felt. The pseudo-force sensation was presented when the cursor moved over a random noise background 701 that was simultaneously displayed on the display 102. When passing over the random noise background 701 in the center of the screen, the random noise pattern within the cursor slid in the y-axis direction with a maximum amplitude (noise drift value) of -7.0, -5.25, -3.5, -1.75, 0.0, 1.75, 3.5, 5.25, or 7.0 mm. For each noise drift value, 48 participants were given four pseudo-force presentation experiments and their responses were obtained. For each participant, the four responses were averaged, resulting in 48 samples for each noise drift value. The distribution of the evaluation results (box plot) is shown in Figure 9. The horizontal axis is the noise drift value, the vertical axis is the average score of each subject, the upper limit of the box is the 75th percentile, the lower limit is the 25th percentile, the center of the constriction is the median, and the dotted line is the average value. The participants indicated that the greater the absolute value of the shift in the random noise pattern, the greater the force they felt being applied to the cursor. The waist around the median of the plot represents the 95% confidence interval of the median, and the reason why the top of the plot appears to fold back at a noise drift value of 5.25 is because the median and the 75% point coincide and fall below the upper limit of the 95% confidence interval.

[0023] <Experiment 2: Relationship between cursor visibility and pseudo-force sensation> In Experiment 2, two conditions were prepared for the brightness of the circular frame surrounding the cursor: black and gray. As shown in 1001 and 1002 in Figure 10, a pseudo-force sensation was presented when the cursor with a gray and black border passed over the random noise background 701. Note that the diagonal lines surrounding the cursor in 1001 represent the "gray border." This is because it is known that the lower the contrast between the cursor and the background, i.e., the worse the visibility, the stronger the position illusion of a stationary cursor caused by random noise shift, and we believe that visibility is similarly important for the trajectory change illusion of a moving cursor. Visibility is worst when the circular frame surrounding the cursor is gray, which is the average luminance of the random noise in the background, and best when it is black or white. In addition to two conditions for the brightness of the circular frame, three conditions were prepared for the amount of shift of the random noise pattern within the cursor: 0, 3.5, and 7.0 mm. A total of six conditions were used, and the same experiment as Experiment 1 was conducted. For each condition (three noise drift conditions x two visibility conditions), 126 participants were given six pseudo-force presentations, and their responses were obtained using a three-point scale. 126 samples were obtained for each condition by averaging the six responses from each participant. Figure 11 shows the results of Experiment 2. The horizontal axis plots the amount of shift of the random noise pattern within the cursor, and the vertical axis plots the average rating for each individual. The gray and black bars represent the brightness of the circular frame surrounding the cursor, respectively. In the case of the poorly visible gray frame, participants perceived a greater force as the amount of shift in the random noise pattern within the cursor increased, as in Experiment 1. On the other hand, in the case of the highly visible black frame, participants did not perceive a greater force as the amount of shift in the random noise pattern within the cursor increased.

[0024] [Programs, recording media] The various processes described above can be implemented by loading a program that executes each step of the above method into the recording unit 2020 of the computer 2000 shown in Figure 12, and operating the control unit 2010, input unit 2030, output unit 2040, display unit 2050, etc.

[0025] The program describing the processing contents can be recorded on a computer-readable recording medium, which may be, for example, a magnetic recording device, an optical disk, a magneto-optical recording medium, a semiconductor memory, or any other suitable recording medium.

[0026] The program may be distributed, for example, by selling, transferring, lending, etc. a portable recording medium such as a DVD or CD-ROM on which the program is recorded. Furthermore, the program may be stored in a storage device of a server computer, and then transferred from the server computer to another computer via a network, thereby distributing the program.

[0027] A computer that executes such a program may first temporarily store the program recorded on a portable recording medium or transferred from a server computer in its own storage device. Then, when executing a process, the computer reads the program stored on its own recording medium and executes the process in accordance with the read program. Alternatively, the computer may read the program directly from a portable recording medium and execute the process in accordance with the program. Furthermore, the computer may execute the process in accordance with the received program each time a program is transferred from a server computer to the computer. Alternatively, the server computer may not transfer the program to the computer, but may execute the process through a so-called ASP (Application Service Provider) service, which realizes the processing function by issuing an execution instruction and obtaining the results. In this embodiment, the program includes information used for processing by a computer that is equivalent to a program (such as data that is not a direct instruction to the computer but has properties that define computer processing).

[0028] Furthermore, in this embodiment, the device is configured by executing a predetermined program on a computer, but at least a part of the processing contents may be realized by hardware.

Claims

1. A pseudo-force presentation device that presents a pseudo-force sensation to a user in a desired direction by displaying a part of an image on a cursor, which is a visual medium that operates in response to a user's operation, a cursor detection unit that detects position information of the cursor; a cursor update unit that updates an image within the cursor, which is an image displayed on the cursor, based on the position information; a video display unit that presents the cursor with the image inside the cursor updated to the user in accordance with the position information, the image within the cursor is a part of a basic image that is larger than the cursor, The cursor update unit shifts a position where the image within the cursor is cut out from the basic image in accordance with the pseudo force sense to be presented. Pseudo force sensation presentation device.

2. The pseudo-force presentation device according to claim 1, The cursor update unit generates an image that the user can recognize as being continuous with the image inside the cursor before updating, in accordance with the pseudo force sense to be presented, instead of cutting out a part of the basic image, and sets the image inside the cursor. Pseudo force sensation presentation device.

3. A pseudo-force presentation device that presents a pseudo-force sensation to a user in a desired direction by displaying a part of an image on a cursor, which is a visual medium that operates in response to a user's operation, a cursor detection unit that detects position information of the cursor; a cursor update unit that updates an image within the cursor, which is an image displayed on the cursor, based on the position information; a video display unit that presents the cursor with the image inside the cursor updated to the user in accordance with the position information, the image within the cursor is a part of a similar image of a base image larger than the cursor; The cursor update unit enlarges or reduces the basic image in accordance with the pseudo-force sense to be presented, and cuts out the image within the cursor. Pseudo force sensation presentation device.

4. A pseudo-force presentation device according to claim 3, The cursor update unit, instead of cutting out a part of a similar image of the basic image, generates an image that the user can recognize as an enlarged image within the cursor before updating, or an image that the user can recognize as a reduced image within the cursor before updating, in accordance with the pseudo force sense to be presented, and sets the image within the cursor. Pseudo force sensation presentation device.

5. A pseudo-force sensation presentation method for presenting a pseudo-force sensation to a user in a desired direction by displaying a part of an image on a cursor, which is a visual medium that operates in response to a user's operation, comprising: a cursor detection unit detecting position information of the cursor; a step of updating an image within the cursor, which is an image displayed on the cursor, by a cursor update unit based on the position information; a step of displaying the cursor with the image inside the cursor updated to the user in accordance with the position information by the video display unit; the image within the cursor is a part of a basic image that is larger than the cursor, The cursor update unit shifts a position where the image within the cursor is cut out from the basic image in accordance with the pseudo force sense to be presented. Pseudo force presentation method.

6. A program that causes a computer to function as the pseudo-force presentation device according to any one of claims 1 to 4.

Images