Information processing system, information processing method, and haptic presentation device

Abstract

An information processing system of the present disclosure includes: a grip detection section that detects a state of grip of a housing by a user holding the housing of a force sensation presentation device, the force sensation presentation device including a force sensation presentation section that presents a predetermined force sensation to the user by changing a center of gravity of the housing; and a control unit that controls the operation of the force sense presentation unit on the basis of the detected gripping state.

Description

Technical Field

[0001] This disclosure relates to information processing systems, information processing methods, and force perception devices. Background Technology

[0002] In related technologies, techniques for achieving realistic virtual reality (VR) experiences have become commonplace. As an existing technology related to such VR, a controller is known that enables users to perceive the weight of various tools in VR space (also known as force perception or force sensing).

[0003] Citation List

[0004] Patent documents

[0005] Patent Document 1: WO 2023 / 095475A Summary of the Invention

[0006] Technical issues

[0007] However, the aforementioned technologies have the problem of difficulty in specifying how the user should hold the controller. Therefore, they cannot provide appropriate force feedback based on how the user holds the controller.

[0008] Therefore, this disclosure provides an information processing system, information processing method, and force perception presentation device capable of presenting force perception based on the user's grip state.

[0009] Note that the above-described problems or objectives are merely one of many problems or objectives that can be solved or achieved through the various embodiments disclosed in this specification.

[0010] Solutions to the problem

[0011] The information processing system disclosed herein includes: a grip detection unit that detects the grip state of a user on the housing of a grip force sensing device, the force sensing device including a force sensing presentation unit that presents a predetermined force to the user by changing the center of gravity of the housing; and a control unit that controls the operation of the force sensing presentation unit based on the detected grip state. Attached Figure Description

[0012] Figure 1 This is a block diagram illustrating an example configuration of an information processing system according to an embodiment of the present disclosure.

[0013] Figure 2 This is a perspective view showing the appearance of the controller.

[0014] Figure 3 This is a schematic diagram showing the expansion of the pressure distribution sensor.

[0015] Figure 4This is a diagram illustrating an example of the attachment of a pressure distribution sensor in a controller.

[0016] Figure 5 This is a diagram showing the variable center of gravity of the controller.

[0017] Figure 6 It is a diagram showing an estimate of the position of the hand holding the controller.

[0018] Figure 7 This is a diagram illustrating the operational control of the variable center of gravity.

[0019] Figure 8 This is a diagram showing the calculation of the position of the variable part of the center of gravity.

[0020] Figure 9 This is a diagram illustrating an example of the operation of the variable part of the center of gravity.

[0021] Figure 10 This is a diagram illustrating an example of the operation of the variable center of gravity.

[0022] Figure 11 This is a diagram illustrating an example of the operation of a variable center of gravity.

[0023] Figure 12 This is a diagram illustrating an example of vibration in the vibrating part.

[0024] Figure 13 This is a diagram illustrating an example of torque control in a controller.

[0025] Figure 14 This is a diagram illustrating an example of torque control in a controller.

[0026] Figure 15 This is a diagram illustrating an example of a torque control feedback loop.

[0027] Figure 16 This is a diagram illustrating an example of the preparation of force intensity presented by the controller.

[0028] Figure 17 This is a diagram showing an estimate of the grip angle of the controller.

[0029] Figure 18 This is a graph showing an estimate of the controller's oscillation speed.

[0030] Figure 19 This is a graph showing the estimated end of the controller's swing.

[0031] Figure 20 This is a diagram illustrating an example of interaction based on the force used to grip the controller.

[0032] Figure 21This is a diagram illustrating an example of interaction based on the force used to grip the controller.

[0033] Figure 22 This is a diagram illustrating an example of interaction based on the position of the hand holding the controller.

[0034] Figure 23 This is a diagram illustrating an example of interaction based on the position of the hand holding the controller.

[0035] Figure 24 This is a diagram showing an example of its application to virtual musical instruments.

[0036] Figure 25 This is a diagram illustrating an example of button operation in a controller.

[0037] Figure 26 This is a diagram illustrating an example of button operation in a controller.

[0038] Figure 27 This is a diagram illustrating an example of a pressure detection method in a controller.

[0039] Figure 28 This is a diagram showing an example of operation buttons in a controller.

[0040] Figure 29 This is a flowchart illustrating an operational example of an information processing system according to an embodiment of the present disclosure.

[0041] Figure 30 This is a hardware configuration diagram showing an example of a computer. Detailed Implementation

[0042] In the following description, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Note that in this specification and the drawings, components having substantially the same functional configuration are indicated by the same reference numerals, and redundant descriptions are omitted.

[0043] Furthermore, in this specification and accompanying drawings, similar components in the embodiments can be distinguished from each other by adding at least one of different letters and numbers after the same reference numerals. However, when it is not necessary to specifically distinguish similar components from each other, the same reference numerals are given.

[0044] One or more embodiments (including examples, variations, and application examples) described below can be implemented independently. On the other hand, at least some of the various embodiments described below can be appropriately combined with at least some of the other embodiments and implemented. The various embodiments may include novel features that differ from each other. Therefore, the various embodiments can help solve different purposes or problems and can present different effects.

[0045] Figure 1This is a block diagram illustrating an example configuration of an information processing system according to an embodiment of the present disclosure.

[0046] like Figure 1 As shown, the information processing system 1 according to an embodiment of the present disclosure includes a controller 10, a control device 81, a camera 82, and a VR device (display device) 83. The control device 81 is connected to each of the controller 10, the camera 82, and the VR device 83 via wired or wireless means.

[0047] The control device 81 is implemented by a computer. The control device 81 stores an application (App) 812 in a storage unit (not shown), and a processor (not shown) can implement a processing unit by executing App 812.

[0048] Typically, an App 812 can be a game application. However, the type of App 812 is not limited to game applications. For example, an App 812 can be an application other than a game application.

[0049] App 812 moves forward based on information sent from controller 10, information sent from VR device 83, or images captured by camera 82. App 812 then controls the output of VR device 83 based on its own forward movement. For example, the output of VR device 83 may include a screen display from a display device or sound output from a speaker.

[0050] Here, the information sent from controller 10 may correspond to information used to control the execution of App 812. For example, the information sent from controller 10 may include information indicating that a button set in controller 10 has been pressed, or it may include information detected by a sensor set in controller 10. Similarly, the information sent from VR device 83 may include information indicating that a button set in VR device 83 has been pressed, or it may include information detected by a sensor set in VR device 83.

[0051] More specifically, App 812 constructs a VR space. Within this VR space, an avatar corresponding to the user exists. App 812 controls the avatar's movement within the VR space based on information sent from controller 10, information sent from VR device 83, or images captured by camera 82. At this time, App 812 provides images of the VR space to VR device 83, and the user can view the VR space through the display on VR device 83.

[0052] In addition, App 812 sends various control signals to controller 10. Furthermore, control device 81 stores software development kit (SDK) 813 in a storage unit (not shown). SDK 813 is a component required to execute App 812.

[0053] Note that, typically, it can be assumed that the control device 81 is a game console. However, the control device 81 is not limited to a game console. For example, the control device 81 could be a personal computer (PC), etc.

[0054] Camera 82 includes an image sensor and acquires images by imaging an imaging range using the image sensor. More specifically, camera 82 acquires multiple frames (i.e., motion images) by continuously imaging the imaging range in a time series using the image sensor. For example, controller 10 and VR device 83 can be included within the imaging range of camera 82.

[0055] Note that, typically, it is assumed that camera 82 is installed independently of other devices (e.g., controller 10, control device 81, VR device 83, etc.). However, camera 82 can be integrated with another device. For example, camera 82 can be located in VR device 83, or it can be located in control device 81.

[0056] VR device 83 performs output under the control of App 812. For example, VR device 83 includes a display device, and performs screen display through the display device under the control of App 812. Alternatively, VR device 83 includes a speaker, and performs sound output by the speaker under the control of App 812.

[0057] Note that VR device 83 is typically used by wearing it on the user's head. However, VR device 83 may not necessarily be worn on the user's head.

[0058] In addition, Figure 1 In the example shown, VR device 83 is in the form of being connected to control device 81. However, the form of VR device 83 is not limited. For example, VR device 83 could be in the form of a smartphone set up in an eye mask or glasses (Goggle). Alternatively, VR device 83 could be in the form of a headset that operates independently.

[0059] The controller 10 is an example of a force-sensing device that presents a predetermined force sensation to a user holding the casing. For example, the controller 10 serves as the casing for a virtual tool in a VR space. Examples of virtual tools include guns (e.g., shotguns), swords (e.g., katana), and wind instruments (e.g., flutes and saxophones).

[0060] Under the control of the control device 81, the controller 10 presents a predetermined force sensation to the user who has already grasped the shell, thereby providing the user with a realistic feeling as if the user is actually operating virtual tools (guns, swords, wind instruments, etc.) in VR space.

[0061] The controller 10 includes a pressure detection unit (pressure distribution sensor) 110, a microcomputer 120 (control unit), a center of gravity variable unit (movable part) 130, a vibration display unit 140, and a battery 150.

[0062] In addition to the above configuration, the controller 10 includes various sensors, such as a gravity sensor for detecting the attitude (device angle) of the controller 10 relative to the vertical direction, a position sensor using a Global Positioning System (GPS), and a temperature sensor. The gravity sensor in the controller 10 is an example of an angle detection unit.

[0063] The pressure detection unit 110 is a pressure sensor installed on the surface of the housing, and it detects the grip position and grip force when the user holds the housing.

[0064] Figure 2 This is a perspective view showing the appearance of the controller 10. (As shown) Figure 2 As shown, the housing E1 of the controller 10 is, for example, a cylindrical body. The shape of the housing E1 is not limited to a cylindrical shape. For example, the housing E1 of the controller 10 can take various shapes, such as a columnar body, or a hexagonal prism and a sphere.

[0065] Pressure distribution sensors 110a and 110b are attached to the surface of housing E1. As an example, on the surface of housing E1 from the center to both ends, pressure distribution sensor 110a is attached to the upper side of the example shown, while pressure distribution sensor 110b is attached to the lower side.

[0066] By attaching multiple pressure distribution sensors 110a and 110b to the surface of housing E1 in this manner, the grip position and grip force can be detected when the user holds housing E1 on the entire surface of housing E1.

[0067] Figure 3 This is a schematic diagram showing the unfolded pressure distribution sensors 110a and 110b. Figure 3 As shown, pressure distribution sensors 110a and 110b have, for example, 10 × 16 detection points and detect the pressure value at each detection point.

[0068] Pressure distribution sensors 110a and 110b can detect not only pressure values ​​but also shear stress, temperature, acceleration, etc. For example, shear stress can be detected by a strain sensor, temperature can be detected by a thermistor, and acceleration can be detected by an accelerometer.

[0069] Figure 4 This is a diagram illustrating an example of the attachment of a pressure distribution sensor in a controller. (See diagram for example.) Figure 4As shown in case a, the controller 10a has a vertically elongated shape. Therefore, the pressure distribution sensor attached to the surface of the housing E1 of the controller 10a can be divided into several parts, such as pressure distribution sensors 110c, 110d, and 110e. In this case, the wiring and flexible cables (not shown) of the pressure distribution sensors 110c, 110d, and 110e can be embedded in the housing E1.

[0070] Additionally, as shown in case b, the pressure distribution sensors 110c and 110e of controller 10b can be attached to parts that the user is likely to hold (hold), such as the two end sides.

[0071] Additionally, as shown in case c, the resolution of the pressure distribution sensor attached to the controller 10c can be varied depending on the location, such as high-resolution pressure distribution sensors 110f and 110h and a low-resolution pressure distribution sensor 110g. As an example, the high-resolution pressure distribution sensor 110f is attached to the part that the user is likely to hold (e.g., both ends), and the low-resolution pressure distribution sensor 110g is attached to the part that the user is unlikely to hold (e.g., the center side).

[0072] Furthermore, as shown in case d, in controller 10d, pressure distribution sensor 110i can be attached to only a portion of housing E1, rather than to the entire circumference of housing E1.

[0073] return Figure 1 The microcomputer 120 corresponds to a microcontroller unit (MCU) and may be configured with an integrated circuit in which a processor, memory (recording medium), input / output circuits, etc. are installed.

[0074] The functions of the microcomputer 120 can be implemented by the processor executing programs stored in memory. Note that if a microcomputer exists externally to the controller 10, the functions of the microcomputer 120 can be incorporated into that external microcomputer. For example, the functions of the microcomputer 120 can be incorporated into the control device 81.

[0075] The microcomputer 120 detects the user's grip state based on the detection information from the pressure detection unit 110. In other words, the microcomputer 120 is an example of a grip detection unit.

[0076] The microcomputer 120 controls the operation of the center-of-gravity variable unit 130 and the vibration display unit 140 based on the detected grip state. Note that the control device 81 may have the function of detecting the user's grip state based on the detection information of the pressure detection unit 110, and the function of controlling the operation of the center-of-gravity variable unit 130 and the vibration display unit 140.

[0077] Under the control of the microcomputer 120, the center of gravity variable unit 130 uses a motor or the like to move a counterweight set in the housing E1 to change the center of gravity of the controller 10 (making the center of gravity variable).

[0078] The vibration presentation unit 140 is a vibrator in the housing E1 that operates under the control of the microcomputer 120.

[0079] Figure 5 This is a diagram showing the variable center of gravity of controller 10. (See diagram below.) Figure 5 As shown, the outer casing E1 is provided with a variable center of gravity part 130 (movable part), which can be moved along the length of the cylinder in the outer casing E1 by means of a motor or the like.

[0080] In addition to the aforementioned motor, the movable part is also equipped with a position sensor, which detects the position of the movable part along the length of the cylinder within the housing E1, vibration display unit 140, etc. Furthermore, temperature sensors, inertial measurement unit (IMU) sensors, etc., can be mounted on the movable part.

[0081] The controller 10 can present various force sensations to the user via the hand H1 of the user holding the housing E1 by operating the movable parts (the center of gravity variable part 130 and the vibration presentation part 140).

[0082] For example, when the center of gravity variable part 130 is above the hand H1 holding the lower part of the housing E1, the user can feel as if the center of gravity of the controller 10 is above, and feel as if the user is holding a long virtual object (the shape is presented through force perception) in VR space.

[0083] In addition, by rapidly moving the center of gravity variable part 130, the user can feel as if the controller 10 has received an impact (the presentation of the impact).

[0084] In addition, by rapidly moving the vibration presentation unit 140 within the movable part, the user can feel as if an impact (vibration presentation) has been received by the controller 10.

[0085] Thus, the center of gravity variable part 130 and the vibration presentation part 140 are examples of force presentation parts, which present a predetermined force to the user holding the housing E1.

[0086] Back Figure 1 The battery 150 supplies power to each part of the controller 10. For example, the battery 150 supplies power to each of the pressure detection unit 110, the microcomputer 120, the center of gravity variable unit 130, and the vibration presentation unit 140.

[0087] Note that in Figure 1In the example shown, the vibration presentation part 140 is disposed outside the center of gravity variable part 130 (movable part), but it can also be included within the center of gravity variable part 130. In this case, the vibration presentation part 140 can be used as a counterweight in the center of gravity variable part 130.

[0088] Next, details will be described regarding the function of detecting the user's grip state based on the detection information from the pressure detection unit 110, and the function of controlling the operation of the center of gravity variable unit 130 and the vibration presentation unit 140 based on the detected grip state.

[0089] In this embodiment, the microcomputer 120 is shown performing the above-described functions, but needless to say, the control device 81 can perform the above-described functions via the microcomputer 120.

[0090] Figure 6 This is a diagram showing the estimated position of the hand H1 holding the controller 10. Figure 6 In the example, assume the user holds the lower part of the housing E1 with their hand H1. Therefore, in the pressure distribution sensor 110b attached to the lower part of the housing E1, pressure values ​​corresponding to the shape of the hand H1 are detected at the 10×16 detection points shown in the figure.

[0091] like Figure 6 As shown, the microcomputer 120 estimates the gripping position and shape of the hand H1 based on the pressure value (pressure distribution) sensed by the detection point of the pressure distribution sensor 110b.

[0092] For example, the microcomputer 120 performs the above estimation by using a machine learning learning model so that when the pressure value at each detection point of the pressure distribution sensor 110b is input, it outputs the grip position and shape of the hand H1.

[0093] In addition, the microcomputer 120 estimates whether the holding hand H1 is the right hand, the left hand, or both hands with a different pressure distribution than the hand H1 based on the pressure value (pressure distribution shape) sensed by the detection point of the pressure distribution sensor 110b.

[0094] For example, the microcomputer 120 performs the above estimation by using a machine learning learning model so that when the pressure value at each detection point of the pressure distribution sensor 110b is input, it outputs the holding hand H1 as one of the right hand, left hand, or both hands.

[0095] Furthermore, the microcomputer 120 can estimate the center G2 of the hand based on the pressure value (pressure distribution) sensed by the detection point of the pressure distribution sensor 110b.

[0096] As an example, the microcomputer 120 estimates the center of the location where the predetermined pressure value is detected as the center of the hand, G2. In this case, the microcomputer 120 can exclude values ​​whose location is significantly off-target.

[0097] In addition, the microcomputer 120 can detect whether the holding hand H1 is right-handed or left-handed based on whether the shape of the pressure distribution sensed by the detection point of the pressure distribution sensor 110b indicates the characteristic shape of holding with the right or left hand.

[0098] Furthermore, the microcomputer 120 can detect the hand H1 based on the context of the App 812. For example, when a virtual object corresponding to the controller 10 is held by the right hand in the VR space presented by the App 812, the microcomputer 120 assumes that the virtual object is held by the right hand and regards the center G2 of the hand as the center position of the right hand.

[0099] The microcomputer 120 controls the operation of the movable part (variable center of gravity part 130) based on the detected gripping position of the hand H1 (the center of the hand G2). For example, the microcomputer 120 changes the position of the variable center of gravity part 130 based on the detected center of the hand G2.

[0100] The detection of the user's hand H1's grip on the casing E1 can be performed not only by the microcomputer 120, but also by the control device 81, which has already acquired detection information from the controller 10. Furthermore, the control device 81 (which also includes the microcomputer 120) can detect the user's hand H1's grip on the casing E1 based on images captured by the camera 82. For example, the control device 81 can perform the detection using a machine learning model to output the user's grip state when an image captured by the camera 82 is input.

[0101] Figure 7 This is a diagram illustrating the operation control of the variable center of gravity unit 130. (See diagram below.) Figure 7 As shown in case C1, when the user holds the controller 10 with his hand H1, the user feels the torque (rotational torque) generated because the center of gravity G1 of the controller 10 is offset from the center G2 of the hand holding the controller H1.

[0102] Therefore, in order to prevent the user from feeling unnecessary rotational torque, the microcomputer 120 changes the position of the center of gravity variable part 130 based on the center of the hand G2. Specifically, the microcomputer 120 refers to table data, etc., indicating the position of the center of gravity G1 corresponding to the position of the center of gravity variable part 130, to control the position of the center of gravity variable part 130 to a predetermined center of gravity G1 relative to the center of the hand G2.

[0103] For example, as shown in cases C2 and C3, the microcomputer 120 controls the position of the center of gravity variable part 130 so that even if the gripping position of the hand H1 (the center of the hand G2) changes, the center of gravity G1 remains constant relative to the center of the hand G2.

[0104] Figure 8 This is a diagram showing the calculated position of the variable center of gravity 130. (See diagram below.) Figure 8 As shown, the position of the variable center of gravity 130 in the cylinder length direction of the controller 10 can be calculated by converting the position in the range of 1 to -1.

[0105] For example, the microcomputer 120 can set the position where the center of gravity G1 and the center of the hand G2 overlap to 0, consider "center of hand G2 < center of gravity G1" as positive, and "center of hand G2 > center of gravity G1" as negative. In the reference coordinate system (positive direction), the direction defined as "upward direction" when designing the controller 10 can be considered positive.

[0106] In addition, when the center of gravity variable part 130 is moved and then returned, the microcomputer 120 can return the center of gravity variable part 130 according to the gripping position of the gripping hand H1 (the center of the hand G2). Figure 9 This is a diagram illustrating an example of the operation of the variable center of gravity section 130.

[0107] like Figure 9 As shown, the controller 10 is treated as a virtual shotgun V1 in VR space. The microcomputer 120 is able to provide the user's hand H1 with a force (impact) corresponding to the firing of the virtual shotgun V1 by rapidly moving the center of gravity variable part 130.

[0108] At this time, the microcomputer 120 returns the position of the center of gravity variable part 130 so that the center of gravity G1 returns to the reference of the detected center of the hand G2. Thus, the controller 10 can provide force sensing so that the shape (rotational torque) of the virtual shotgun V1 does not change before and after the virtual shotgun V1 is fired.

[0109] Figure 10 This is a diagram illustrating an example of the operation of the variable center of gravity section 130. (See diagram for example.) Figure 10 As shown in cases C4 and C5, when the user holds the controller 10 with both hands H1 and H2, the microcomputer 120 can regard the center of the holding position of both hands as the origin (0).

[0110] Furthermore, the microcomputer 120 can control the position of the center of gravity variable part 130 based on the intensity (grip force) of the detected pressure value.

[0111] For example, as shown in case C6, the microcomputer 120 can change the origin position (the initial position of the center of gravity variable part 130) based on the intensity of the pressure value detected at the position of hand H1 and the intensity of the pressure value at the position of hand H2.

[0112] As an example, the microcomputer 120 can obtain the origin (0) based on the weighted average of the grip strength of hand H1 and the grip strength of hand H2 detected by the pressure detection unit 110.

[0113] Furthermore, the microcomputer 120 can control the vibration of the movable part (vibration presentation part 140) based on whether the detected hand H1 is the right hand, left hand, or both hands of hand H1 and hand H2. For example, the microcomputer 120 can reduce the vibration of a single hand (right hand or left hand) and increase the vibration of both hands. In addition, the microcomputer 120 can adjust the position of the center of gravity variable part 130 based on whether it is the right hand, left hand, or both hands.

[0114] Furthermore, the microcomputer 120 can control the vibration of the movable part (vibration presentation part 140) based on the detected gripping position of the hand H1 (the center G2 of the hand). Figure 11 This is a diagram illustrating an example of the operation of the variable center of gravity section 130. Figure 12 This is a diagram showing an example of vibration of the vibration presentation unit 140.

[0115] like Figure 11 As shown on the left, when the center of the hand G2 and the center of gravity G1 are close to each other, it is difficult to generate physical vibration, and the vibration is weakened when the center of gravity variable part 130 (vibration presentation part 140) is operated. Therefore, the microcomputer 120 moves the center of gravity variable part 130 based on the detected gripping position of the hand H1 (the center of the hand G2) so that the center of vibration (center of gravity G1) does not overlap with the center of the hand G2. Here, the microcomputer 120 can generate vibration by rapidly moving the center of gravity variable part 130 up and down, or by operating the vibration presentation part 140 provided in the movable part.

[0116] Furthermore, the microcomputer 120 can control the vibration to reproduce the vibration waveform based on the detected gripping position of the hand H1 (the center G2 of the hand). For example, as Figure 12 As shown, when the vibration display unit 140 vibrates away from the gripping position of the hand H1, the vibration is attenuated compared to when the vibration display unit vibrates near the gripping position. Taking this vibration attenuation into account, the microcomputer 120 controls the vibration to reproduce the vibration waveform.

[0117] For example, assuming the gripping position of hand H1 is o, the position of the movable part (vibration presentation part 140) is p, and the vibration attenuation characteristic is F(o, p), then the microcomputer 120 reproduces the vibration waveform obtained in advance by filtering the vibration to be reproduced using the inverse function of F. The microcomputer 120 can use the filter by taking into account the attenuation characteristic F(o, p, f) of each vibration frequency f.

[0118] In addition, the microcomputer 120 can perform position control (torque control) of the center of gravity variable unit 130 to present the expected torque magnitude to the user (force perception).

[0119] Specifically, the microcomputer 120 determines the position of the movable part (variable center of gravity part 130) based on the detected gripping position of the hand H1 (the center of the hand G2) and the posture of the controller 10 relative to the vertical direction (device angle).

[0120] Figure 13 and Figure 14 This is a diagram illustrating an example of torque control in controller 10. (See diagram for example.) Figure 13 As shown, assume the torque M to be presented is determined by the hand H1 holding the controller 10. The attitude (device angle) of the controller 10 relative to the vertical direction, detected by a gravity sensor or the like, is denoted by θ. The mass (device mass) of the controller 10 is m, and the acceleration due to gravity is g.

[0121] Based on the detected grip position o, device angle θ, device mass m, and gravitational acceleration g, the microcomputer 120 determines the center of gravity position p corresponding to the torque M to be presented as center of gravity position p = o + M / (m * g * sin(θ)).

[0122] Even when the user moves the controller 10 to various positions, the microcomputer 120 can present a constant torque M to the user by controlling the position of the center of gravity variable part 130 to place it at the aforementioned center of gravity position p.

[0123] Furthermore, the microcomputer 120 can obtain the torque to be applied to the hand H1 based on the pressure distribution of the gripping hand H1 detected by the pressure detection unit 110. For example... Figure 14 As shown, when the user grips the controller 10 with his hand H1, for example, an upward normal force N1 applied by the thumb of hand H1 and a downward normal force N2 applied by the little finger of hand H1 are applied to the controller 10.

[0124] These normal forces N1 and N2 in opposite directions generate a torque M around the center G2 of the hand. The microcomputer 120 obtains the torque M based on the integral of the pressure distribution (F) detected by the pressure detection unit 110.

[0125] Figure 15 This is a diagram illustrating an example of a torque control feedback loop. (See diagram for example.) Figure 15 As shown, the microcomputer 120 can perform torque control, wherein the torque M obtained from the pressure distribution of the hand H1 is put into the feedback loop L1.

[0126] In addition, the microcomputer 120 can estimate the size (area) of the hand H1 based on the range of pressure distribution where the pressure value detected by the pressure detection unit 110 is equal to or greater than a predetermined value.

[0127] The microcomputer 120 can control the operation of the movable parts (variable center of gravity part 130, vibration presentation part 140) based on the estimated size (area) of the hand H1. Specifically, the microcomputer 120 controls the intensity of the force sensation (e.g., impact and vibration) presented to the user through the operation of the movable parts.

[0128] Figure 16 This is a diagram illustrating an example of the preparation of force intensity presented by controller 10. (See diagram for example.) Figure 16 As shown, case C7 illustrates a wide range of pressure distribution 111 in the pressure detection unit 110 where the pressure value is a predetermined value or greater, and the hand H1 holding the controller 10 is large. On the other hand, case C8 illustrates a narrow range of pressure distribution 111 in the pressure detection unit 110 where the pressure value is a predetermined value or greater, and the hand H1 holding the controller 10 is small.

[0129] As shown in case C7, when the pressure distribution 111 is wide and the hand H1 holding the controller 10 is large, the microcomputer 120 adjusts the position of the movable part (vibration presentation part 140) (left end in the example shown) so that the center of gravity G1 is separated from the center G2 of the hand. Then, the microcomputer 120 vibrates the vibration presentation part 140 relatively strongly.

[0130] As shown in case C6, when the pressure distribution 111 is narrow and the hand H1 holding the controller 10 is short, the microcomputer 120 adjusts the position of the movable part (vibration presentation part 140) (near the center in the example shown) so that the center of gravity G1 is close to the center G2 of the hand. Then, the microcomputer 120 vibrates the vibration presentation part 140 relatively weakly.

[0131] In this way, the microcomputer 120 controls the intensity of the force sensation (e.g., impact and vibration) to be presented to the user based on the detected size of the hand H1. Thus, for example, for a person with a small hand H1, the force sensation can be adjusted to be small so that the person does not get tired.

[0132] In addition, the microcomputer 120 can estimate the angle (grip angle) of the controller 10 when the user holds the controller 10 based on the detection information of the pressure detection unit 110 when the movable part (center of gravity variable part 130, vibration presentation part 140) is operated.

[0133] Figure 17 This is a diagram showing an estimate of the grip angle of the controller 10. (See diagram below.) Figure 17 As shown, case C9 is when controller 10 is held vertically, and case C10 is when controller 10 is held horizontally.

[0134] As shown in case C9, when the controller 10 is kept vertical, even when the center of gravity variable part 130 is operated along the length of the cylinder, the movement is along the vertical direction, so no rotational torque is generated, and the pressure distribution 111 of the pressure detection part 110 does not change.

[0135] On the other hand, as shown in case C10, when the controller 10 is held horizontally, and the center of gravity variable part 130 is operated along the length of the cylinder, a rotational torque is generated corresponding to the tilt of the controller 10 relative to the vertical direction. Therefore, the force of the hand H1 holding the controller 10 changes in the opposite direction to the rotational torque, and the pressure distribution 111 of the pressure detection part 110 changes.

[0136] The microcomputer 120 estimates the angle of the controller 10 (grip angle) when the user holds the controller 10 based on the changes in the pressure distribution 111 generated in this way.

[0137] For example, the microcomputer 120 performs the above estimation by using a learning model that is machine-learned to output the grip angle when it receives input of a change in the pressure distribution 111 detected during a specified operation of the center of gravity variable section 130 in the direction of the cylinder length.

[0138] In addition, the microcomputer 120 can estimate the swing state of the controller 10 held by the user based on the detection information of the pressure detection unit 110 when the movable part (the center of gravity variable part 130 and the vibration presentation part 140) stops.

[0139] Figure 18 This is a graph showing an estimate of the swing speed of controller 10. Figure 19 This is a diagram showing the estimated end of the swing of controller 10.

[0140] like Figure 18 As shown, case C11 is when controller 10 slowly swings from a vertical position to a horizontal position, and case C12 is when controller 10 swings faster than in case C11.

[0141] The comparison between cases C11 and C12 clearly shows that the pressure distributions 111a and 111b when the controller 10 is held by hand H1 are significantly different depending on the speed at which the controller 10 swings.

[0142] For example, when the controller 10 swings downward to rotate laterally and stops, the pressure distributions 111a and 111b increase with the increase of the downward swing speed, and the pressure values ​​also increase.

[0143] The microcomputer 120 estimates the user's swing speed on the controller 10 based on the changes in the pressure distributions 111a and 111b generated in this way. For example, the microcomputer 120 performs the above estimation by using a learning model that is machine-learned to output a downward swing speed in response to changes in the input pressure distributions 111a and 111b.

[0144] like Figure 19 As shown, case C13 is the case where the controller 10 swings from vertical holding to horizontal holding midway, and case C14 is the case where the controller 10 stops in the horizontal holding state (swing end).

[0145] The comparison between cases C13 and C14 clearly shows that the pressure distributions 111a and 111b are significantly different when the controller 10 is in the middle of its swing and when the controller's swing ends.

[0146] For example, in case C13 when the controller 10 is in the middle of swinging, and in case C14 when the swinging of the controller ends, the force used to hold the controller 10 with hand H1 to stop the controller 10 increases, the pressure distributions 111a and 111b increase, and the pressure value also increases.

[0147] The microcomputer 120 estimates the swing state of the controller 10 (in the middle of the swing or at the end of the swing) based on the changes in the pressure distributions 111a and 111b generated in this way.

[0148] For example, the microcomputer 120 performs the above estimation by using a learning model, which is machine-learned to output the oscillation state in response to changes in the input pressure distributions 111a and 111b. The microcomputer 120 can also operate the vibration presentation unit 140 to present the vibration at a timing when the end of the oscillation is detected.

[0149] Furthermore, the microcomputer 120 can operate the movable parts (the variable center of gravity part 130 and the vibration presentation part 140) based on the detection information of the pressure detection unit 110 to present a predetermined force sensation (interaction).

[0150] For example, suppose a user holds the controller 10, which is an extended bundle of swords, in the VR space provided by App 812. In this case, the microcomputer 120 changes the position of the center of gravity variable unit 130 according to the user's grip force (the magnitude of the pressure value of the pressure detection unit 110).

[0151] Figure 20 This is a diagram illustrating an example of interaction based on the force used to grip the controller 10. Figure 20 In the process, the force exerted by the user holding the controller 10 with his hand H1 increases in the order of cases C15, C16, and C17.

[0152] like Figure 20 As shown, the microcomputer 120 moves the position of the center of gravity variable part 130 so that as the user's grip force (the magnitude of the pressure value of the pressure detection part 110) increases, the center of gravity G1 moves away from the center G2 of the hand.

[0153] As a result, the user can feel force, as if the rod-shaped sword extends as the force applied to gripping the controller 10 increases.

[0154] Furthermore, for example, suppose a user holds the controller 10 as a hose, from which water flows out in the VR space provided by the App 812.

[0155] In this case, the microcomputer 120 can present an interaction in which the water spray volume is changed by operating the center of gravity variable unit 130 according to the user's grip force (the magnitude of the pressure value of the pressure detection unit 110).

[0156] Figure 21 This is a diagram illustrating an example of interaction based on the force used to grip the controller 10. Figure 21 In this case, it is assumed that the force with which the user holds the controller 10 (virtual hose V2) with his hand H1 is greater in case C19 than in case C18.

[0157] like Figure 21 As shown, for example, when the grip force is increased, the microcomputer 120 operates to bring the position of the center of gravity variable part 130 closer to the position of the user's hand H1, and when the grip force is decreased, the microcomputer operates to move the position of the center of gravity variable part 130 away from the position of the user's hand H1.

[0158] In addition, when the gripping force increases, the microcomputer 120 can increase the vibration of the vibration display unit 140 and increase the vibration frequency, and when the gripping force decreases, it can decrease the vibration of the vibration display unit 140 and decrease the vibration frequency.

[0159] In this way, when the user grips the virtual hose V2 (controller 10) tightly, the user can feel a force as if water is being violently released from the hand, and when the user releases the force, the user can feel a force as if water is slowly flowing out.

[0160] Furthermore, for example, suppose a user holds a controller 10, which is a shotgun (virtual gun), in the VR space provided by App 812.

[0161] In this case, the microcomputer 120 can operate the center of gravity variable unit 130 according to the position of the user's hand.

[0162] Figure 22 This is a diagram illustrating an example of interaction based on the position of the hand holding the controller 10. (See diagram for example.) Figure 22 As shown, assume that the user holds the controller 10, which is a virtual shotgun V1, with hands H1 and H2.

[0163] like Figure 22 As shown, the microcomputer 120 operates the position of the center of gravity variable part 130 according to the sliding position of the hand H2.

[0164] For example, when the hand H2 is slid along the length of the barrel of the controller 10 to perform the pumping action of the virtual shotgun V1, the microcomputer 120 moves the center of gravity variable part 130 according to the position of the hand H2.

[0165] Thus, the controller 10 can present the user with a force sensation corresponding to the pumping action of the virtual shotgun V1. The microcomputer 120 can also vibrate the vibration presentation unit 140 when the hand H2 reaches a predetermined position to present the user with a force sensation corresponding to loading bullets into the virtual shotgun V1.

[0166] Furthermore, for example, suppose a user holds a controller 10, which functions as a virtual sword, in a VR space provided by App 812.

[0167] In this case, the microcomputer 120 can operate the center of gravity variable part 130 according to the position of the user's hand, thereby providing a force sensation for drawing the sword or sheathing it.

[0168] Figure 23 This is a diagram illustrating an example of interaction based on the position of the hand holding the controller 10. (See diagram for example.) Figure 22 As shown, assume that the user holds the controller 10, which is the virtual sword V3, with hands H1 and H2.

[0169] like Figure 23 As shown, when hand H2 slides on the surface of controller 10, microcomputer 120 maintains the position of center of gravity variable part 130 according to the position of hand H1 corresponding to the handle of virtual sword V3.

[0170] At this time, the microcomputer 120 obtains the speed of drawing the sword and sheathing the sword from the change in the sliding position of the hand H2, and vibrates the vibration presentation unit 140 according to the speed, thereby providing the user with the feeling of vibration corresponding to drawing the sword and sheathing the sword.

[0171] In the example shown, hand H1 holds the hilt of the virtual sword V3, but hand H2 can also hold the hilt of the virtual sword V3.

[0172] Furthermore, for example, suppose a user holds a controller 10, which functions as a virtual musical instrument, in a VR space provided by App 812.

[0173] In this case, the microcomputer 120 can detect the position of the user's fingers by the pressure detection unit 110 and assign the detected finger position to the operation of the instrument.

[0174] Figure 24 This is a diagram illustrating an example applied to Virtual Musical Instruments V4. (Example...) Figure 24 As shown, the user sets the controller 10 to a virtual musical instrument V4, such as a saxophone, and performs finger operations on the controller 10 as key operations on the virtual musical instrument V4.

[0175] The microcomputer 120 detects the user's finger movements from the pressure detection unit 110 and notifies the control device 81 of the detected finger movements. As a result, the control device 81 generates sound from the virtual musical instrument V4 in the VR space.

[0176] At this time, the microcomputer 120 changes the position of the center of gravity variable part 130 according to the weight of the virtual musical instrument V4 to be operated, or vibrates the vibration presentation part 140 according to the vibration (sound production) of the virtual musical instrument V4.

[0177] For example, the microcomputer 120 can move the position of the center of gravity variable part 130 downwards (in the direction of gravity) as the weight of the virtual musical instrument V4 increases. In addition, the microcomputer 120 can change the position of the center of gravity variable part 130 and the vibration frequency of the vibration presentation part 140 according to the level of the sound produced by the virtual musical instrument V4.

[0178] For example, when the sound level is low, the microcomputer 120 can move the center of gravity variable part 130 downward, and when the sound level is high, it can move the center of gravity variable part 130 upward.

[0179] Additionally, when the user temporarily moves the controller 10 according to the performance, the microcomputer 120 can remain in the position of the center of gravity variable part 130 without moving.

[0180] Furthermore, when the virtual instrument V4 is held horizontally, such as a violin or trumpet, the "downward direction" can be considered the direction away from the user.

[0181] Furthermore, in the VR space, the controller 10 can be assigned to the stick of a weapon or musical instrument, and the impact generated when the weapon or musical instrument collides with objects, drums, etc. in the VR space can be presented as force.

[0182] Specifically, the microcomputer 120 presents the information based on the notification from the control device 81 when it collides with an object or drum in the VR space, through the impact or vibration caused by the high-speed operation of the center of gravity variable part 130 or the vibration of the vibration presentation part 140.

[0183] At this time, the microcomputer 120 can change the intensity of the impact or vibration caused by the operation of the center of gravity variable part 130 and the vibration presentation part 140 according to the speed of the user swing controller 10.

[0184] Furthermore, the microcomputer 120 can change the intensity of the impact or vibration caused by the operation of the center of gravity variable part 130 and the vibration presentation part 140 according to the strength of the user's grip on the controller 10.

[0185] For example, the microcomputer 120 experiences increased impact and vibration as the swing speed increases, and also as the intensity of the user's grip on the controller 10 increases.

[0186] In addition to increasing the impact and vibration according to the strength of the user's grip on the controller 10, the microcomputer 120 can also change the vibration frequency. For example, the microcomputer 120 can increase the vibration frequency as the user's grip strength increases, and decrease the vibration frequency as the grip strength decreases.

[0187] In addition, when the virtual musical instrument V4 is a violin in the VR space, the controller 10 can be considered as the main body of the violin, and the wooden stick, pencil, etc. that are separated from the controller 10 can replace the violin bow.

[0188] At this time, the contact between the controller 10 and the wooden stick is detected by the pressure detection unit 110. The control device 81 emits sound from the virtual musical instrument V4 in the VR space and changes the sound according to the detection value (contact state). The microcomputer 120 can control the movable parts (variable center of gravity unit 130, vibration presentation unit 140) according to the sound generation and the contact position in the virtual musical instrument V4.

[0189] Additionally, when the saw is reproduced in VR space, the controller 10 can be regarded as the saw body, and the object to be cut can be replaced by an entity different from the controller 10 (such as a table).

[0190] At this time, in the VR space, the virtual object is placed in the same position as the physical object, and when the controller 10 comes into contact with the physical object, the virtual object also comes into contact with the saw. When the user attempts to cut the virtual object with the saw (controller 10), the microcomputer 120 vibrates the movable parts (variable center of gravity part 130, vibration presentation part 140) according to the material of the virtual object, and thus the feeling of cutting can be reproduced.

[0191] Note that the above example is just an example and can be used for interaction between controller 10 and physical entities in the absence of a VR space.

[0192] The hand twisting operation (sliding operation) detected by the pressure detection unit 110 can be assigned to the accelerator operation of a motorcycle or the operation of a pepper grinder or salt grinder in VR space.

[0193] At this time, when the torsional motion is performed, it is determined that the motorcycle's accelerator has rotated, and the microcomputer 120 causes the movable parts (the center of gravity variable part 130 and the vibration presentation part 140) to vibrate. As a result, the user can perceive the force of the accelerator operation.

[0194] Furthermore, when the pepper or salt grinder is in operation, as a video in VR space, the force of pepper or salt falling from the tip of the grinder and the force of the pepper or salt being rubbed can be represented by the vibration of movable parts (variable center of gravity part 130, vibration presentation part 140), etc.

[0195] In addition, operation buttons (operation commands) can be assigned to specific locations on the surface of the controller 10 pressure distribution sensor, and the microcomputer 120 can detect the pressing operation at that specific location (whether an operation command has been given).

[0196] Figure 25 and 26 This is a diagram illustrating an example of button operations in a controller. (Example) Figure 25 As shown, the virtual operation button 112 is arranged at a predetermined position of the pressure distribution sensor 110b of the controller 10 in the VR space.

[0197] The App 812 of the control device 81 obtains the detection information from the pressure distribution sensor 110b from the microcomputer 120 and determines whether a press operation has been performed at the location where the virtual operation button 112 is set. When a press operation of the virtual operation button 112 is detected, the App 812 generates an event corresponding to the virtual operation button 112 in the VR space. Note that the VR space can be an augmented reality (AR) space.

[0198] like Figure 26As shown, in application 812, controller 10 can be regarded as a virtual pistol V5 in VR space, and virtual operation button 112 can be arranged at the trigger position of virtual pistol V5.

[0199] App 812 obtains detection information from pressure distribution sensor 110b from microcomputer 120 and determines whether a sliding operation of the finger of hand H1 on virtual operation button 112 has been performed. When a sliding operation of virtual operation button 112 is detected (e.g., the sliding amount is equal to or greater than a threshold (the degree of trigger pull)), App 812 generates an event corresponding to virtual operation button 112 (firing of virtual pistol V5) in VR space.

[0200] Figure 27 This is a diagram illustrating an example of a pressure detection method in a controller. (See diagram for example.) Figure 27 As shown in the controller 10e, the pressure distribution sensor in the pressure detection unit 110 can be a capacitive touch panel 110j.

[0201] When the contact area of ​​the capacitive touchpad 110j held by the user can be detected, the contact area can be converted into a pressure value.

[0202] In addition, such as Figure 27 As shown in the controller 10f, the pressure distribution sensor in the pressure detection unit 110 can be a resistive film type touch panel 110k to 110n that is divided into multiple parts in the length direction of the cylinder.

[0203] Additionally, as shown in controller 10f, the pressure distribution sensor in pressure detection unit 110 can be a plurality of resistive film type touch panels 110o to 110q along the length of the cylinder.

[0204] Furthermore, as resistive film type touchpads 110k to 110q, pressure values ​​can be used to detect pressure values. Additionally, in the case of analog resistive film type, a large number of pressure sensors can be integrated to increase the number of pressure points that can be detected.

[0205] Furthermore, in the case of sensors such as touchpads, the sensor can be used solely as a contact position sensor, or a pressure sensor, such as a strain gauge, can be set separately from the contact position sensor, and the values ​​of both sensors can be used in combination. (The type of contact position sensor is not limited.)

[0206] Furthermore, in the controller 10, the operation button can be formed on the pressure distribution sensor of the pressure detection unit 110, and the pressing operation (pressure) of the operation button can be detected. Figure 28 This is a diagram showing an example of the operation buttons on the controller.

[0207] like Figure 28 As shown, the operation button on the pressure distribution sensor of the pressure detection unit 110 is clearly formed in a shape different from other parts on the surface of the housing E1.

[0208] Specifically, the operation button on the pressure distribution sensor of the pressure detection unit 110 can be a textured operation button 113a or an operation button 113b with protrusions.

[0209] In this way, operation buttons 113a and 113b can be clearly provided to the user.

[0210] Here, we will explain an example of how a user operates the controller 10 as a virtual shotgun V1 in a VR space. Figure 29 This is a flowchart illustrating an operational example of an information processing system according to an embodiment of the present disclosure.

[0211] like Figure 29 As shown, when processing begins, the microcomputer 120 waits for input from the pressure detection unit 110 (waiting for pressure detection) (S1). Next, it is assumed that the user presses the virtual button corresponding to the trigger in the controller 10 (S2).

[0212] When pressure is detected at the location of the virtual button in the pressure detection unit 110, the microcomputer 120 notifies the App 812 of the detection result. Then, in the App 812, the firing of the gun (virtual shotgun V1) is drawn in VR space (S3). Additionally, the microcomputer 120 drives the counterweight (center-of-gravity variable unit 130) to present the impact (force sensation) of firing to the user (S4).

[0213] Then, the microcomputer 120 waits for input from the pressure detection unit 110 (waiting for pressure detection) (S5). Next, in the controller 10, it is assumed that the user rubs the cylinder of the controller 10 (pumping action) (S6).

[0214] When a sliding operation corresponding to the pumping action is detected in the pressure detection unit 110, the microcomputer 120 notifies the App 812 of the detection result. Then, in the App 812, the pumping action is drawn in VR space based on the user's hand movement (S7).

[0215] Additionally, the microcomputer 120 drives the counterweight (variable center of gravity 130) in real time based on the user's hand movements and presents the user with force sensation corresponding to the pumping action (S8). Then, the microcomputer 120 terminates the processing by ending the actions related to the virtual shotgun V1 in the VR space (S9).

[0216] As described above, in the information processing system, the microcomputer 120 or control device 81 detects the user's gripping state on the housing E1 relative to a force-sensing device (controller 10), which includes a movable part (a variable center of gravity part 130 and a vibration-sensing part 140) that presents a predetermined force sensation to the user gripping the housing E1. The microcomputer 120 controls the operation of the force-sensing part based on the detected gripping state. Thus, the information processing system is able to present a force sensation corresponding to the user's gripping state.

[0217] Furthermore, the microcomputer 120 detects the user's grip position based on the pressure detection position of one of the multiple pressure sensors disposed on the surface of the housing E1. Thus, the information processing system is able to detect the user's grip position as the user's grip state.

[0218] Furthermore, the microcomputer 120, based on the detected user's grip position, displays the force corresponding to the grip position through the operation of the force sensor. Thus, the information processing system is able to display the force corresponding to the user's grip position.

[0219] Furthermore, the microcomputer 120, based on the detected change in the user's grip position, displays the force corresponding to the change in grip position through the operation of the force sensor. Thus, in the information processing system, the force corresponding to the change in the user's grip position can be displayed.

[0220] In addition, the microcomputer 120 detects the user's grip strength based on the values ​​detected by the pressure sensor. Thus, the information processing system can determine the user's grip strength as their grip state.

[0221] In addition, the microcomputer 120 detects the user's grip force based on the pressure detection range of multiple pressure sensors. Thus, the information processing system can determine the user's grip force as their grip state.

[0222] Furthermore, the microcomputer 120 presents a force sensation corresponding to the gripping force based on the detected gripping force of the user through the operation of the force sensation presentation unit. Thus, the information processing system can present a force sensation corresponding to the user's gripping force.

[0223] Furthermore, the microcomputer 120 detects the user's grip range based on a detection range where the pressure values ​​from multiple pressure sensors are equal to or greater than a predetermined value. Thus, the information processing system can detect the user's grip range as the user's grip state.

[0224] Furthermore, the microcomputer 120, based on the detected grip range of the user, presents the force sensation corresponding to the grip range through the operation of the force sensation presentation unit. Thus, the information processing system is able to present the force sensation corresponding to the user's grip range.

[0225] Furthermore, the microcomputer 120 or control device 81 detects whether the user is holding the casing E1 with their right hand, left hand, or both hands. Thus, the information processing system can detect whether the user is holding the casing E1 with their right hand, left hand, or both hands.

[0226] Furthermore, the microcomputer 120 or control device 81 presents a force sensation corresponding to the gripping force based on the detected gripping hand (right hand, left hand, or both hands) through the operation of the force sensation presentation unit. Thus, the information processing system can present a force sensation depending on whether the user is gripping the casing E1 with their right hand, left hand, or both hands.

[0227] Furthermore, the force perception unit presents force perception by changing the center of gravity G1 of the outer shell E1. In this way, the information processing system can present force perception corresponding to the change in the center of gravity G1.

[0228] Furthermore, the force sensing unit presents force through the vibrating housing E1. Thus, the information processing system can present force corresponding to the vibration.

[0229] In addition, the controller 10 also includes an angle detection unit that detects the angle of the housing E1. The microcomputer 120 controls the operation of the force sensor based on the detected housing angle. Thus, the information processing system can present a force sensation corresponding to the angle of the housing.

[0230] Furthermore, when the force sensing unit presents a predetermined force, the microcomputer 120 detects the angle of the casing based on the change in the detection information. Thus, the information processing system is able to detect the angle of the casing.

[0231] Furthermore, the microcomputer 120 detects a pressing operation at a predetermined location on the surface of the housing E1 based on the pressure detection position of one of a plurality of pressure sensors disposed on the surface of the housing E1. The microcomputer 120 determines whether an operation instruction has been assigned to the predetermined location based on whether a pressing operation is detected at the predetermined location. Thus, the information processing system can receive an operation instruction corresponding to a pressing operation on the surface of the housing E1 from the user.

[0232] Furthermore, the microcomputer 120 estimates the user's swing motion on the housing E1 based on the change in information detected when the force sensor stops operating. In this way, the information processing system can obtain the swing motion of the housing E1 when the user holds it.

[0233] Furthermore, the microcomputer 120 presents the force corresponding to the swinging motion of the housing E1 based on the estimated swinging motion of the housing E1 through the operation of the force presentation unit. Thus, the information processing system can present the force corresponding to the swinging motion of the housing E1.

[0234] Furthermore, the control device 81 (App 812) performs display control on the display device that presents the virtual space to the user based on the detected gripping state. Thus, the information processing system is able to perform a display in the virtual space corresponding to the user's gripping state.

[0235] (Example of hardware configuration)

[0236] For example, by having such Figure 30 The computer 1000 with the configuration shown implements information devices such as controller 10, control device 81 and VR device 83 in the information processing system according to the above embodiment. Figure 30 This is a hardware configuration diagram illustrating an example of a computer 1000. The computer 1000 includes a CPU 1100, RAM 1200, read-only memory (ROM) 1300, a solid-state drive (SSD) 1400, a communication interface 1500, and an input / output interface 1600. The various parts of the computer 1000 are connected via a bus 1050.

[0237] The CPU 1100 operates based on programs stored in ROM 1300 or SSD 1400 and controls each component. For example, the CPU 1100 loads programs stored in ROM 1300 or SSD 1400 into RAM 1200 and executes processing corresponding to various programs.

[0238] ROM 1300 stores boot programs such as the Basic Input / Output System (BIOS) and programs that depend on the hardware of computer 1000, which are executed by CPU 1100 when computer 1000 is activated.

[0239] SSD 1400 is a recording medium readable by computer 1000 that non-transitoryly records a program executed by CPU 1100, data used by the program, etc. Specifically, SSD 1400 is a recording medium that records a communication program according to this disclosure, which is an example of program data 1450. SSD 1400 can be another non-transitory recording medium such as a hard disk drive (HDD).

[0240] Communication interface 1500 is an interface for computer 1000 to connect to external network 1550 (e.g., the Internet). For example, CPU 1100 receives data from another device or sends data generated by CPU 1100 to another device via communication interface 1500.

[0241] Input / output interface 1600 is an interface for connecting input / output device 1650 and computer 1000. For example, CPU 1100 receives data from input devices such as touchpad, keyboard, mouse, microphone, or camera via input / output interface 1600. Additionally, CPU 1100 sends data to output devices such as monitor, speaker, or printer via input / output interface 1600. Furthermore, input / output interface 1600 can be used as a media interface for reading programs recorded on a predetermined recording medium (medium). This medium may be, for example, an optical recording medium such as a Digital Universal Disc (DVD) or Phase Change Rewritable Disc (PD), a magneto-optical recording medium such as a magneto-optical disc (MO), magnetic tape, magnetic recording media, semiconductor memory, etc.

[0242] For example, when computer 1000 is used as a controller 10, control device 81, and VR device 83 in an information processing system according to this embodiment, the CPU 1100 of computer 1000 implements various functional configurations by executing communication programs loaded on RAM 1200. Additionally, SSD 1400 stores programs and data according to this disclosure in RAM 1200. Note that CPU 1100 reads program data 1450 from SSD 1400 and executes the program data; however, as another example, these programs can be obtained from another device via external network 1550.

[0243] Note that this technology can also have the following configurations.

[0244] (1) An information processing system, comprising:

[0245] A grip detection unit detects the user's gripping state on the casing of a force-sensing device. The force-sensing device includes a force-sensing display unit that presents a predetermined force sensation to the user by changing the center of gravity of the casing.

[0246] The control unit controls the operation of the force sensor based on the detected grip state.

[0247] (2) The information processing system according to (1), wherein

[0248] The grip detection unit detects the user's grip position based on the pressure detection position of multiple pressure sensors set on the surface of the housing.

[0249] (3) The information processing system according to (2), wherein

[0250] Based on the detected grip position of the user, the control unit presents the force sensation corresponding to the grip position through the operation of the force sensation presentation unit.

[0251] (4) The information processing system according to (2), wherein

[0252] Based on the detected change in the user's grip position, the control unit presents the force sensation corresponding to the change in grip position through the operation of the force sensation presentation unit.

[0253] (5) The information processing system according to any one of (1) to (4), wherein

[0254] The grip detection unit detects the user's grip force based on the detection values ​​of pressure sensors installed on the surface of the housing.

[0255] (6) The information processing system according to any one of (1) to (5), wherein

[0256] The grip detection unit detects the user's grip force based on the pressure detection range of multiple pressure sensors set on the surface of the housing.

[0257] (7) The information processing system according to (5) or (6), wherein

[0258] The control unit presents a force sensation corresponding to the grip force by operating the force sensor, based on the detected grip force of the user.

[0259] (8) The information processing system according to any one of (1) to (7), wherein

[0260] The grip detection unit detects the user's grip range based on the detection range where the pressure value of one of the multiple pressure sensors disposed on the surface of the housing is equal to or greater than a predetermined value.

[0261] (9) The information processing system according to (8), wherein

[0262] Based on the detected grip range of the user, the control unit presents the force sensation corresponding to the user's grip range through the operation of the force sensation presentation unit.

[0263] (10) The information processing system according to any one of (1) to (9), wherein

[0264] The grip detection unit detects whether the user is holding the casing with their right hand, left hand, or both hands.

[0265] (11) The information processing system according to (10), wherein

[0266] The control unit presents a force sensation corresponding to the grip force by operating the force sensor based on the detected gripping hand of the user.

[0267] (12) The information processing system according to any one of (1) to (11), wherein

[0268] The force presentation unit presents force by vibrating the outer shell.

[0269] (13) The information processing system according to any one of (1) to (12) further includes an angle detection unit for detecting the angle of the housing.

[0270] The control unit controls the operation of the force sensing unit based on the detected angle of the outer shell.

[0271] (14) The information processing system according to (2), wherein

[0272] The grip detection unit detects a pressing operation at a predetermined position on the surface of the housing based on the pressure detection position of the plurality of pressure sensors, and

[0273] The control unit determines whether an operation instruction has been given to the predetermined position based on whether a press operation is detected at the predetermined position.

[0274] (15) The information processing system according to any one of (1) to (14), wherein

[0275] The control unit estimates the user's swing motion on the housing based on the change in the detection value of the pressure sensor set on the surface of the housing when the operation of the force sensor stops.

[0276] (16) The information processing system according to (15), wherein

[0277] The control unit presents the force corresponding to the swing motion by operating the force presentation unit based on the estimated swing motion of the housing.

[0278] (17) The information processing system according to any one of (1) to (16), wherein

[0279] The control unit performs display control of the display device that presents a virtual space to the user based on the detected grip state.

[0280] (18) An information processing method that causes a computer to perform processing for a force-sensing device, the force-sensing device including a force-sensing unit that presents a predetermined force to a user holding the housing by changing the center of gravity of the housing, the processing including:

[0281] Detects the user's grip on the casing; and

[0282] The operation of the force sensor is controlled based on the detected grip state.

[0283] (19) A force perception device, comprising:

[0284] The force perception unit presents a predetermined force perception to the user holding the casing by changing the center of gravity of the casing;

[0285] The detection unit detects the user's grip state based on detection information from pressure sensors mounted on the surface of the housing; and

[0286] The control unit controls the operation of the force sensor based on the detected grip state.

[0287] List of reference numerals

[0288] 10, 10a-10g controller

[0289] 81 Control Equipment

[0290] 82 camera

[0291] 83 VR devices

[0292] 110 Pressure Testing Department

[0293] 110a to 110e, 110i pressure distribution sensors

[0294] 110F, 110H high-resolution pressure distribution sensor

[0295] 110g low-resolution pressure distribution sensor

[0296] 110J capacitive touchpad

[0297] 110kΩ~110q resistive film touchpad

[0298] Pressure distribution at 111, 111a, and 111b

[0299] 112 Virtual operation buttons

[0300] Operation buttons 113, 113a, 113b

[0301] 120 microcomputer

[0302] 130 Variable Center of Gravity

[0303] 140 vibration presentation section

[0304] 150 batteries

[0305] 812 App

[0306] 813 SDK

[0307] 1000 computers

[0308] 1050 bus

[0309] 1100 CPU

[0310] 1200 RAM

[0311] 1300 Read-Only Memory

[0312] SSD 1400

[0313] 1450 program data

[0314] 1500 communication interface

[0315] 1550 external network

[0316] 1600 Input / Output Interface

[0317] 1650 Input / Output Devices

[0318] ad, C1-C19 cases

[0319] E1 casing

[0320] G1 center of gravity

[0321] G2's center

[0322] H1, H2 hands

[0323] L1 feedback loop

[0324] V1 Virtual Shotgun

[0325] V2 Virtual Hose

[0326] V3 Virtual Sword

[0327] V4 Virtual Musical Instruments

[0328] V5 Virtual Pistol

Claims

1. An information processing system, comprising: The grip detection unit detects the user's grip state on the housing of the grip force presentation device. The force presentation device includes a force presentation unit, which presents a predetermined force to the user by changing the center of gravity of the housing. as well as The control unit controls the operation of the force sensor based on the detected grip state.

2. The information processing system according to claim 1, wherein... The grip detection unit detects the user's grip position based on the pressure detection position of multiple pressure sensors set on the surface of the housing.

3. The information processing system according to claim 2, wherein... Based on the detected grip position of the user, the control unit presents the force sensation corresponding to the grip position through the operation of the force sensation presentation unit.

4. The information processing system according to claim 2, wherein... Based on the detected change in the user's grip position, the control unit presents the force sensation corresponding to the change in grip position through the operation of the force sensation presentation unit.

5. The information processing system according to claim 1, wherein... The grip detection unit detects the user's grip force based on the detection values ​​of pressure sensors installed on the surface of the housing.

6. The information processing system according to claim 1, wherein... The grip detection unit detects the user's grip force based on the pressure detection range of multiple pressure sensors set on the surface of the housing.

7. The information processing system according to claim 5, wherein... The control unit presents a force sensation corresponding to the grip force by operating the force sensor, based on the detected grip force of the user.

8. The information processing system according to claim 1, wherein The grip detection unit detects the user's grip range based on the detection range where the pressure value of one of the multiple pressure sensors disposed on the surface of the housing is equal to or greater than a predetermined value.

9. The information processing system according to claim 8, wherein Based on the detected grip range of the user, the control unit presents the force sensation corresponding to the user's grip range through the operation of the force sensation presentation unit.

10. The information processing system according to claim 1, wherein The grip detection unit detects whether the user is holding the casing with their right hand, left hand, or both hands.

11. The information processing system according to claim 10, wherein The control unit presents a force sensation corresponding to the grip force by operating the force sensor based on the detected gripping hand of the user.

12. The information processing system according to claim 1, wherein... The force presentation unit presents force by vibrating the outer shell.

13. The information processing system according to claim 1 further includes an angle detection unit for detecting the angle of the outer casing. in, The control unit controls the operation of the force sensing unit based on the detected angle of the housing.

14. The information processing system according to claim 2, wherein The grip detection unit detects a pressing operation at a predetermined position on the surface of the housing based on the pressure detection position of the plurality of pressure sensors, and The control unit determines whether an operation instruction has been given to the predetermined position based on whether a press operation is detected at the predetermined position.

15. The information processing system according to claim 1, wherein... The control unit estimates the user's swing motion on the housing based on the change in the detection value of the pressure sensor set on the surface of the housing when the operation of the force sensor stops.

16. The information processing system according to claim 15, wherein The control unit presents the force corresponding to the swing motion by operating the force presentation unit based on the estimated swing motion of the housing.

17. The information processing system according to claim 1, wherein The control unit performs display control of the display device that presents a virtual space to the user based on the detected grip state.

18. An information processing method that causes a computer to perform processing for a force-sensing device, the force-sensing device including a force-sensing unit that presents a predetermined force sensation to a user holding the housing by changing the center of gravity of the housing, the processing including: Detects the user's grip on the casing; as well as The operation of the force sensor is controlled based on the detected grip state.

19. A force perception device, comprising: The force perception unit presents a predetermined force perception to the user holding the casing by changing the center of gravity of the casing; The detection unit detects the user's grip state based on detection information from pressure sensors mounted on the surface of the housing; as well as The control unit controls the operation of the force sensor based on the detected grip state.

Images