Information processing device, information processing method, and information processing system

Abstract

This information processing method is performed using an information processing device provided with a processor. The processor acquires data of an image including a hand of a user. The processor uses the data of the image to identify whether the user is holding an object in the hand and performing a hand gesture operation with the hand holding the object. When the user holds the object in the hand and performs the hand gesture operation with the hand holding the object, the processor executes warning control for outputting a warning to the user, and safety ensuring control for ensuring the safety of the hand gesture operation with respect to the user.

Description

Information processing device, information processing method, and information processing system

[0001] The present invention relates to an information processing device, an information processing method, and an information processing system.

[0002] Head-mounted displays (sometimes referred to as HMDs), Augmented Reality (AR) glasses, and even information processing devices such as smartphones and personal computers use hand gesture operation functions that capture hand movements using a camera or the like and execute predetermined actions based on the captured information. For example, when a user operates an information processing device using hand gestures while holding an object in their hand, it can be convenient to perform operations quickly without letting go of the object. However, this can raise concerns about accidentally injuring themselves or other people or objects with the held object.

[0003] Here, Patent Document 1 states that "the wearable electronic device comprises a gesture detection unit that detects gestures within a detection area; an object detection unit that outputs a signal according to the presence or absence of an object within the detection area; an imaging unit that generates an image using an area including the detection area as an imaging area; a first control unit that detects the presence or absence of the object based on the output of the object detection unit, and when the presence of the object is detected, activates the imaging unit, which is in an inactive state, and causes the imaging unit to generate an image; and a second control unit that detects the presence or absence of an indicator for making a gesture based on the image captured by the imaging unit, and when the presence of the indicator is detected, causes the gesture detection unit to detect the gesture made by the indicator."

[0004] Japanese Patent Application Laid-Open No. 2016-218899

[0005] The above-mentioned document describes a technique for detecting the presence or absence of an indicator for making a gesture, such as a specific finger of a user's hand, that has entered a detection area, and detecting the presence of the finger serving as the indicator when the presence of the indicator is detected, thereby reducing false detection of gestures. However, the technique described in the above-mentioned document does not take into consideration or suggest, for example, the risk of injuring oneself or others with a held object, or the risk of damaging other objects with a held object. Therefore, there are concerns about the safe operation of hand gestures.

[0006] In view of the above, there is a need to provide a technique for ensuring the safety of hand gesture operations when using an information processing device.

[0007] According to a first aspect of the present invention, there is provided an information processing device as follows: The information processing device includes a processor, which executes warning control, which is control for outputting a warning to a user when the user performs a hand gesture operation with a hand holding an object, or safety control, which is control for ensuring the safety of the hand gesture operation for the user.

[0008] According to a second aspect of the present invention, there is provided an information processing method as follows. The information processing method is a method performed using an information processing device including a processor. The processor acquires image data including a user's hand. The processor uses the image data to identify whether the user is holding an object in his or her hand and performing a hand gesture operation with the hand holding the object. When the user is holding an object in his or her hand and performing a hand gesture operation with the hand holding the object, the processor executes warning control, which is control to output a warning to the user, and safety assurance control, which is control to ensure the safety of the hand gesture operation for the user.

[0009] According to a third aspect of the present invention, there is provided an information processing system as described below. The information processing system includes an information processing device and a device connected to the information processing device via a wired cable. When a user performs a hand gesture operation with a hand holding an object, the information processing device executes warning control, which is control for outputting a warning to the user, and safety control, which is control for ensuring the safety of the hand gesture operation for the user.

[0010] According to the present invention, a technique for ensuring the safety of hand gesture operations when using an information processing device is provided. Note that problems, configurations, and effects other than those described above will become apparent from the following description of the preferred embodiment of the invention.

[0011] 1 is a diagram for explaining an example of control of an information processing device during a hand gesture operation. FIG. 1 is a diagram for explaining an example of a hardware configuration of an information processing device. FIG. 2 is a diagram for explaining an example of a functional configuration of an information processing device. FIG. 3 is a diagram for explaining an example of control of an information processing device. FIG. 4 is a diagram for explaining an example of an object held in a hand making a gesture. FIG. 5 is a diagram for explaining an example of control of an information processing device taking into account the orientation of an object. FIG. 6 is a diagram for explaining an example of a usage environment of an information processing device. FIG. 7 is a diagram for explaining an example of control of an information processing device based on a special region. FIG. 8 is a diagram for explaining an example of control of an information processing device based on a special region. FIG. 9 is a diagram for explaining an example of a special region that can be set by a user. FIG. 10 is a diagram for explaining a case where the information processing device is a smartphone. FIG. 11 is a flowchart for explaining an example of an operation of an information processing device. FIG. 12 is a flowchart for explaining an example of an operation performed by an information processing device using a special region. FIG. 13 is a flowchart for explaining an example of an operation of an information processing device in a case where a wired-connected device is present. FIG. 14 is a diagram for explaining an example of an execution condition. FIG. 15 is a diagram for explaining an example of warning information. FIG. 16 is a diagram for explaining an example of data for safety assurance control 1. FIG. 17 is a diagram for explaining an example of data for safety assurance control 2. FIG. 18 is a diagram for explaining an example of data for safety assurance control 3.

[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The embodiment is an example for explaining the present invention, and for clarity of explanation, appropriate omissions and simplifications have been made. The present invention can be implemented in various other forms. Unless otherwise specified, each component may be singular or plural.

[0013] In order to facilitate understanding of the invention, the position, size, shape, range, etc. of each component shown in the drawings may not represent the actual position, size, shape, range, etc. Therefore, the present invention is not necessarily limited to the position, size, shape, range, etc. disclosed in the drawings.

[0014] Examples of various types of information may be described using expressions such as "table," "list," and "queue," but the various types of information may also be expressed using data structures other than these. For example, various types of information such as "XX table," "XX list," and "XX queue" may also be expressed as "XX information." When describing identification information, expressions such as "identification information," "identifier," "name," "ID," and "number" are used, but these are interchangeable.

[0015] When there are multiple components with the same or similar functions, they may be described using the same reference numeral with different subscripts. When there is no need to distinguish between these multiple components, the subscripts may be omitted.

[0016] In some embodiments, processing performed by executing a program will be described. Here, a computer executes the program using a processor (e.g., a CPU or a GPU) and performs processing defined by the program using storage resources (e.g., memory) and interface devices (e.g., communication ports). Therefore, the entity performing the processing by executing the program may be the processor. Similarly, the entity performing the processing by executing the program may be a controller, device, system, computer, or node having a processor. The entity performing the processing by executing the program may be any computing unit, and may include a dedicated circuit that performs specific processing. Here, the dedicated circuit may be, for example, an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), or a CPLD (Complex Programmable Logic Device).

[0017] A program may be installed on a computer from a program source. The program source may be, for example, a program distribution server or a computer-readable storage medium. When the program source is a program distribution server, the program distribution server may include a processor and storage resources for storing the program to be distributed, and the processor of the program distribution server may distribute the program to be distributed to other computers. In addition, in an embodiment, two or more programs may be realized as one program, or one program may be realized as two or more programs.

[0018] In the embodiment, a technology for ensuring the safety of hand gesture operations when using an information processing device will be described. First, an example of control of an information processing device using this technology will be described with reference to Fig. 1. When a user 11 performs a hand gesture operation while holding an object in their hand, the information processing device 10 (in this example, an HMD) performs control to issue a warning to the user 11. Furthermore, the information processing device 10 performs control to ensure the safety of the hand gesture operation.

[0019] As an example, the information processing device 10 displays one of the screens (I1, I2) shown in FIG. 1 , and the information processing device 10 displays warnings, such as a warning that there is danger, that hand gesture operations are prohibited, and that an input method other than hand gesture operations is encouraged. Furthermore, the information processing device 10 performs control (safety assurance control) to ensure the safety of hand gesture operations, such as prohibiting hand gesture operations, prohibiting a virtual reality (VR) display mode that displays virtual reality and switching to an augmented reality (AR) display mode that displays augmented reality, or generating and displaying a virtual object corresponding to a held object in the VR display mode. For example, if hand gesture operations are prohibited by the safety assurance control, the user 11 cannot operate virtual objects by hand gesture operations. In the example shown in the figure, when screen I1 is displayed, the user 11 can operate virtual objects by methods other than hand gesture operations, for example.

[0020] In this way, when the user 11 performs a hand gesture operation while holding an object in his / her hand, the information processing device 10 can alert the user 11 or sound an alarm about danger caused by the object held in his / her hand. Furthermore, the information processing device 10 can perform control to avoid or resolve the occurrence of a dangerous situation during the hand gesture operation.

[0021] Next, an example of the hardware configuration of an information processing device will be described with reference to Fig. 2A . As an example, the information processing device 10 is an HMD, and as shown in Fig. 2A , the information processing device 10 includes a left eye gaze sensor 21, a right eye gaze sensor 22, a camera 23, a distance measurement sensor 24, a positioning sensor 25, an acceleration sensor 26, a gyro sensor 27, and a geomagnetic sensor 28.

[0022] The information processing device 10 is configured with various sensors. Here, the various sensors may be configured to include the above-described components (21 to 28), as an example. However, as long as the information processing device 10 can execute appropriate processing, the various sensors may be configured to include, for example, other types of sensors. Furthermore, the various sensors may be configured, for example, with some components omitted from the above-described components (21 to 28).

[0023] The left eye gaze sensor 21 detects the gaze of the left eye of the user 11, and the right eye gaze sensor 22 detects the gaze of the right eye of the user 11. The left eye gaze sensor 21 and the right eye gaze sensor 22 are configured using, for example, appropriate cameras, and the information processing device 10 can track the gaze of the user 11 based on the detection results of the left eye gaze sensor 21 and the right eye gaze sensor 22.

[0024] The camera 23 is configured to capture images of the surroundings of the information processing device 10. For example, the camera 23 acquires video data by converting light input from a lens into an electrical signal using an electronic device such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) sensor. Note that the placement, angle of view, etc. of the camera 23 are not particularly limited as long as appropriate processing can be performed.

[0025] The ranging sensor 24 is a sensor used to measure the distance and angle to objects around the information processing device 10 and acquire distance data and object shape data. It irradiates light onto the object and receives the reflected light. When measuring the distance to an object, for example, a stereo camera may be used. Alternatively, the ranging sensor 24 may be a Light Detection and Ranging (LiDAR) sensor that irradiates an object with laser light such as infrared light and measures the scattered light that bounces back, a Time of Flight (TOF) sensor that measures distance by measuring the reflection time of pulsed light irradiated onto the subject for each pixel, or a millimeter-wave radar that emits millimeter-wave radio waves and captures the reflected waves.

[0026] An example of the positioning sensor 25 is a GPS (Global Positioning System). The information processing device 10 receives radio waves transmitted from GPS satellites and acquires location information (e.g., latitude and longitude) indicating the current location of the information processing device 10 based on the received radio waves. Alternatively, the camera 23 or the ranging sensor 24 may be used as the positioning sensor 25. The current location can be determined by analyzing images captured by a camera or three-dimensional point cloud data measured by a ranging sensor, and known techniques include VPS (Visual Positioning Service) and SLAM (Simultaneous Localization and Mapping). Alternatively, the positioning sensor 25 may be an acceleration sensor 26, a gyro sensor 27, or a geomagnetic sensor 28. The current position can be determined as a relative position from a reference position, and techniques such as PDR (Pedestrian Dead Reckoning) are known. Positioning may also be performed using wireless communication. The current position can be determined by detecting differences in the strength and arrival time of radio waves from multiple wireless communication access points and performing triangulation.

[0027] The acceleration sensor 26 is a sensor that detects the acceleration of the information processing device 10, and the information processing device 10 can detect, for example, tilt, movement, vibration, and impact with respect to the direction of gravity based on data acquired by the acceleration sensor 26. The gyro sensor 27 is a sensor that detects the angular velocity of the information processing device 10, and the information processing device 10 can detect its posture using the gyro sensor 27. Furthermore, when the information processing device 10 is an HMD, the information processing device 10 can detect the movement of the head of a user wearing the information processing device 10 using the acceleration sensor 26 and the gyro sensor 27.

[0028] The geomagnetic sensor 28 is a sensor that detects the magnetic force of the Earth and detects the orientation (direction) of the information processing device 10. When the information processing device 10 is an HMD, the information processing device 10 can detect head movement by capturing changes in the geomagnetic field in response to head movement using, for example, a three-axis geomagnetic sensor 28 that detects geomagnetism in the up-down direction in addition to the front-back and left-right directions.

[0029] The information processing device 10 also includes a microphone 31, a speaker 32, and a vibrator 33. The microphone 31 is a device for inputting sound, and for example, the voice of the user 11 is input to the microphone 31. The microphone 31 also receives, for example, ambient sound. The speaker 32 is a device for outputting sound, and for example, outputs sound processed by the information processing device 10. The vibrator 33 is a vibration device, and for example, vibrates in response to processing by the information processing device 10.

[0030] The information processing device 10 also includes a network communication interface (network communication I / F 41), a short-range wireless communication interface (short-range wireless communication I / F 42), and a device wired input / output interface (device wired input / output I / F 43). The network communication I / F 41 is an interface used for communication over a network, and the information processing device 10 connects to, for example, an external information server 51 via the network communication I / F 41. The short-range wireless communication I / F 42 is an interface used for short-range wireless communication, and the information processing device 10 connects to, for example, an operation device 52 via the short-range wireless communication I / F 42. The device wired input / output I / F 43 is an interface used for wired communication, and the information processing device 10 can be connected to devices such as the operation device 52 via the device wired input / output I / F 43.

[0031] Here, the information server 51 is a server that can acquire and hold data to be processed by the information processing device 10. The information processing device 10 may acquire the data held by the information server 51 from the information server 51 via the network communication I / F 41 and process the data.

[0032] The operation device 52 is an appropriate controller that the user 11 uses to operate the information processing device 10. The operation device 52 can be wirelessly connected to the information processing device 10 via the short-range wireless communication I / F 42. The operation device 52 can also be wiredly connected to the information processing device 10 via the device wired input / output I / F 43.

[0033] The information processing device 10 includes an operation input interface (operation input I / F 61 ), a display 62 , a processor 63 , and a memory 64 .

[0034] The operation input I / F 61 is configured to accept operation inputs from the user, and can be configured using, for example, a keyboard, key buttons, or a touchpad provided on the information processing device 10 .

[0035] The display 62 is configured to display an appropriate screen. When the information processing device 10 is an HMD, the display 62 is located in front of both eyes of the user 11 wearing the information processing device 10.

[0036] The processor 63 is a main body that executes predetermined processing and can be configured using, for example, a CPU (Central Processing Unit). However, as long as it can execute appropriate processing, it may be configured using, for example, other semiconductor devices. The processor 63 executes processing to display appropriate images, such as VR images and images in which virtual objects are superimposed on images of the outside world, on the display 62.

[0037] The memory 64 is configured to store programs and data used in predetermined processing, and includes, for example, a main memory device and storage. The main memory device can be a volatile storage medium (e.g., RAM (Random Access Memory)) that can read and write information at high speed, and is used as a storage area and a working area when the processor 63 processes data. The storage device can be, for example, a non-volatile storage device or non-volatile memory, and can store various types of data (information data 65), programs (such as an operating system 66 and application programs 67), content, etc.

[0038] The various components (21 to 28, 31 to 33, 41 to 43, 61 to 64) are connected via buses, and in this embodiment, these components are capable of inputting and outputting various signals and data to and from each other via the buses.

[0039] Next, an example of the functional configuration of the information processing device will be described with reference to Fig. 2B . As an example, the information processing device 10 is an HMD, and as shown in Fig. 2B , the information processing device 10 includes a virtual space display processing unit 71, a virtual object generation processing unit 72, a user state identification processing unit 73, a held object identification processing unit 74, a warning control processing unit 75, a hand gesture operation control processing unit 76, an input method control processing unit 77, a display mode switching processing unit 78, a surrounding environment identification processing unit 79, and a special area setting processing unit 80.

[0040] These components (71 to 80) can be realized using a processor 63 and a memory 64 (more specifically, a main storage device). The processor 63 stores appropriate data and programs in the memory 64 (more specifically, a main storage device) and performs processing. The information processing device 10 may store the programs used for processing in storage. The information processing device 10 may also obtain the programs used for processing from an external source such as an information server 51.

[0041] The virtual space display processing unit 71 performs processing to display the virtual space on the display 62. The virtual space display processing unit 71 performs processing using virtual space information 71'. Here, the virtual space information 71' is data for realizing the virtual space to be displayed on the display 62, such as data relating to an image of the virtual space. The virtual space information 71' may be held in storage of the information processing device 10, and the virtual space display processing unit 71 may acquire the virtual space information 71' from the storage during processing. Alternatively, the virtual space information may be held in the information server 51 or the like, and the virtual space display processing unit 71 may acquire the virtual space information 71' from an external device during processing.

[0042] The virtual object generation processing unit 72 generates a virtual object to be displayed on the display 62. The virtual object generation processing unit 72 also displays the generated virtual object on the display 62. Here, the data of the virtual object may be held in storage, and the virtual object generation processing unit 72 may acquire the data of the virtual object from the storage and perform processing. Alternatively, the data of the virtual object may be held in the information server 51 or the like, and the virtual object generation processing unit 72 may acquire the data of the virtual object from an external source during processing.

[0043] The user state identification processing unit 73 uses the user state information 73' to identify a hand gesture made by the user 11. The user state identification processing unit 73 also uses the user state information 73' to identify whether the user 11 is holding an object in their hand. The user state information 73' is data relating to an image of the hand area of ​​the user 11, and the user state identification processing unit 73 identifies the hand gesture made by the user 11 and whether the user 11 is holding an object, for example, based on image recognition. The user state information 73' may be a photographed image acquired using the camera 23. The user state information 73' may also be a distance image acquired using the distance measuring sensor 24 or the like.

[0044] The held object identification processing unit 74 uses the held object information 74' to identify the object held by the user 11. The held object information 74' is data relating to an image of the hand area of ​​the user 11, and the held object identification processing unit 74 identifies the object held by the user 11, for example, based on image recognition. Here, the held object identification processing unit 74 identifies the type, size, shape, orientation, etc. of the object. Note that the held object information 74' may be a photographed image acquired using the camera 23. Alternatively, the held object information 74' may be a distance image acquired using the ranging sensor 24 or the like.

[0045] The warning control processing unit 75 performs warning control using the warning information 75'. Here, the warning information 75' is data used for warning control, and the warning control processing unit 75 performs warning control under predetermined conditions based on the warning information 75'. An example of the warning information 75' will be described in detail later. The warning information 75' may be held in a storage of the information processing device 10, and the warning control processing unit 75 may acquire the warning information 75' from the storage during processing. Alternatively, the warning information 75' may be held in the information server 51 or the like, and the warning control processing unit 75 may acquire the warning information 75' from an external device during processing.

[0046] The hand gesture operation control processing unit 76 controls the execution of operation content based on the identified hand gesture. For example, data linking operation content to hand gesture types is pre-registered in the storage of the information processing device 10, and the hand gesture operation control processing unit 76 can refer to the registered data and execute operation content corresponding to the hand gesture performed by the user 11. However, if hand gesture operations are restricted due to safety assurance control or the like, the safety assurance control content takes priority. For example, if hand gesture operations are prohibited due to safety assurance control, the hand gesture operation control processing unit 76 controls so as not to accept input of hand gesture operations.

[0047] The input method control processing unit 77 controls the input method of the user 11 using input method information 77'. The input method information 77' is data related to the type of input method, and is stored in advance in the storage of the information processing device 10, for example. The input method information 77' includes, for example, an input method using the operation device 52, an input method using voice, an input method using gestures, and an input method using gaze. The input method control processing unit 77 then performs control to permit an input method selected by the user 11 from among the input methods included in the input method information 77'. However, when control such as changing or prohibiting the input method is performed due to safety control or the like, the content of the safety control takes priority.

[0048] The display mode switching processing unit 78 controls the display mode of the information processing device 10 using display mode information 78'. The display mode information 78' is data related to the display mode and, for example, is stored in advance in the storage of the information processing device 10. The display mode information 78' includes, for example, an AR display mode (AR display mode) and a VR display mode (VR display mode). The display mode switching processing unit 78 then performs control to display in a display mode selected by the user 11 from the display modes included in the display mode information 78'. However, when control such as changing the display mode is performed due to safety control or the like, the content of the safety control takes priority.

[0049] The information processing device 10 can be configured to perform video see-through or optical see-through display in the AR display mode. When performing video see-through display, the information processing device 10 displays captured images of virtual objects and objects in real space on the display 62. When performing optical see-through display, the information processing device 10 displays virtual objects on the display 62 in a state where the user 11 can view objects in real space in front of them. Furthermore, the information processing device 10 can be configured to display virtual objects on the entire screen of the display 62 in the VR display mode.

[0050] The surrounding environment identification processing unit 79 performs image recognition to identify objects (peripheral objects), people, and the like present in the surrounding environment. Here, the surrounding environment identification processing unit 79 may perform processing using a photographed image acquired using the camera 23. The surrounding environment identification processing unit 79 may also perform processing using a distance image acquired using the distance measuring sensor 24, etc. The surrounding environment identification processing unit 79 may also acquire distance data to the identified surrounding objects, etc. The surrounding environment identification processing unit 79 stores the processing result in the memory 64 as surrounding environment information 79'.

[0051] The specific area setting processing unit 80 performs processing related to the special area. For example, the specific area setting processing unit 80 stores data related to the settings made by the user 11 regarding the special area in storage as special area information 80'. The user 11 can set the special area using an appropriate configuration, and the special area is an area used for control, which will be described later. For example, the user 11 can set an area that follows the user's hand or an area that follows the user's body as the special area.

[0052] Next, an example of control of the information processing device will be described with reference to Fig. 3. Fig. 3 is a diagram for explaining an example of control of the information processing device. One of a screen I3 based on the VR display mode and a screen I4 based on the AR display mode is displayed on the screen of the display 62 of the information processing device 10.

[0053] Here, when the user 11 makes a hand gesture with the hand holding an object while the screen I3 based on the VR display mode is displayed, the information processing device 10 identifies that the hand gesture is being made with the hand holding the object, displays a message indicating that the VR display mode is prohibited, and further performs control to switch the VR display mode to the AR display mode.

[0054] In this way, it is identified that the safety of the hand gesture operation for the user 11 is insufficient, and the information processing device 10 executes warning control to call attention to the hand gesture operation and safety ensuring control to ensure the safety of the hand gesture operation. As a result, when the user 11 performs a hand gesture operation while holding an object in his or her hand, for example, it is possible to prevent the hand gesture operation from being performed in a state where the user 11 cannot see the object held in his or her hand, and it is possible to ensure the safety of the hand gesture operation for the user.

[0055] Note that any appropriate response may be implemented from the viewpoint of the safety of the user 11, and any appropriate changes may be made. Instead of the above-described response, the information processing device 10 may, for example, generate and display a virtual object corresponding to the held object while remaining in the VR display mode. In the AR display mode, if the held object is visible through the display 62, the display of the virtual object for this object may be omitted.

[0056] Next, an example of control of the information processing device according to an object held by a user will be described with reference to Fig. 4. Figs. 4A and 4B are diagrams showing an example of an object held in the hand of a user performing a gesture. When the user 11 performs a hand gesture operation while holding an object in his or her hand, the information processing device 10 identifies the object held by the user 11 and performs control according to the type or shape of the identified object.

[0057] Here, when the information processing device 10 determines that the object does not have a sharp part (a pointed part in this embodiment) like the object 91 shown in Fig. 4A and is therefore low in danger, the information processing device 10 performs control so as not to issue a warning to alert the user about the hand gesture operation (warning control) or to perform control to ensure the safety of the hand gesture operation (safety ensuring control). On the other hand, when the information processing device 10 determines that the object has a sharp part 93 like the object 92 shown in Fig. 4B and is therefore high in danger, the information processing device 10 appropriately performs warning control and safety ensuring control.

[0058] In this way, the information processing device 10, for example, determines the danger of an object held by the user 11 in his / her hand, and if the danger is low, does not issue a warning to the user about the hand gesture operation or control to ensure the safety of the hand gesture operation. This allows the information processing device 10 to apply the user warning control and control to ensure the user's safety only when these controls are warranted. Even when the user 11 holds an object in his / her hand, the user 11 can perform the hand gesture operation while ensuring safety. Furthermore, depending on the degree of danger of the object held by the user 11, the information processing device 10 can execute the user warning control and control to ensure the user's safety. This allows the user to selectively use each control according to their preferred settings. The information processing device 10 can identify the type, size, shape, etc. of an object based on image recognition and determine the danger of the object. Examples of low-risk objects include a controller for operating the information processing device 10 (e.g., an HMD) and an object without a sharp portion 93.

[0059] Next, an example of control of the information processing device according to the orientation of an object will be described with reference to Fig. 5A. As an example, the information processing device 10 identifies the orientation of the object held by the user 11 in addition to the type and shape of the object held by the user 11 based on image recognition, and performs control according to the identification result.

[0060] 5A , an object 92 held in the hand of the user 11 has a sharp portion 93 formed thereon, and the sharp portion 93 faces the user. Therefore, when the information processing device 10 (such as an HMD) identifies that the object held in the hand of the user 11 is the object 92 having the sharp portion 93 and that the sharp portion 93 faces in a direction toward the user 11, it is considered that the object held by the user 11 is highly likely to cause harm to the user 11, and therefore the information processing device 10 issues a warning against the hand gesture operation and appropriately executes control to ensure the safety of the hand gesture operation.

[0061] In this example, the information processing device 10 determines the danger by identifying the orientation of the object held by the user 11 in addition to the type and shape of the object held by the user 11. The information processing device 10 can then avoid or resolve the occurrence of a dangerous situation in the hand gesture operation, for example, by alerting the user 11 to the hand gesture operation or sounding an alarm. Furthermore, the number of times or the form of the warning may be changed depending on the degree of danger of the object held by the user 11. This allows for more effective warning to the user 11.

[0062] 5B and 5C, an example of control of the information processing device according to the surrounding environment will be described. As an example, the information processing device 10 identifies the type and shape of an object held by the user 11, as well as the orientation of the object held by the user 11 and the environmental condition around the user's hand, based on image recognition, and performs control according to the identification results.

[0063] 5B , object 92 held by user 11 in his / her hand has a sharp portion 93, and sharp portion 93 of object 92 held by user 11 in his / her hand is close to a peripheral object 94 and faces in the direction of the peripheral object 94. Therefore, when information processing device 10 (HMD, etc.) identifies that object 92 held by user 11 in his / her hand has a sharp portion 93 and that sharp portion 93 is close to a peripheral object 94 or the like (peripheral object 94, person, etc.) and faces in the direction of the peripheral object 94 or the like, it is considered that there is a high possibility that the held object will cause harm to the peripheral object 94 or the like, and therefore information processing device 10 issues a warning against the hand gesture operation and appropriately executes control to ensure the safety of the hand gesture operation.

[0064] In this example, the information processing device 10 determines danger by identifying the type and shape of the object held by the user 11, the orientation of the object held by the user 11, the environmental conditions around the user's hand, etc. Then, the information processing device 10 can, for example, issue a warning or sound an alarm regarding the hand gesture operation, and can also avoid or eliminate the occurrence of a dangerous situation in the hand gesture operation. Furthermore, the safety of not only the user 11 but also those around them can be ensured.

[0065] In the example of FIG. 5C , an object 92 held by the user 11 in his / her hand has a sharp portion 93 formed thereon, and the sharp portion 93 faces in a direction different from the direction toward the user 11. Furthermore, there are no surrounding objects 94 or the like near the sharp portion 93. Therefore, in such a state, the held object is unlikely to harm the user 11, surrounding people, objects, or the like, and therefore the information processing device 10 (e.g., HMD) may be configured not to issue a warning or perform control to ensure the safety of hand gesture operations. Note that, when the object held by the user 11 in his / her hand is in an environmental state away from surrounding objects 94 or the like, the object is considered unlikely to cause harm, and therefore the information processing device 10 may be configured to perform similar control.

[0066] In this example, the information processing device 10 identifies the type, shape, and the like of the object held by the user 11, as well as the orientation of the object held by the user 11 and the environmental conditions around the user 11's hand. If the risk is low, the information processing device 10, for example, does not issue a warning about a hand gesture operation or perform control to ensure the safety of the hand gesture operation. Therefore, the information processing device 10 can apply warning control to the user and the surroundings and control to ensure safety only when these controls are warranted, and even when the user 11 holds an object in his or her hand, the user 11 can perform a hand gesture operation while ensuring safety. Furthermore, depending on the degree of risk of the object held by the user 11, it is possible to perform warning control to the user and the like and control to ensure the safety of the user and the like, and it is possible to use each control method according to the user's preferred settings.

[0067] 6A and 6B, an example of control of the information processing device based on the special region will be described. A special region that covers and follows the vicinity of the hand of the user 11 is set by the user 11, and the information processing device 10 performs control depending on whether an object held in the hand is contained within the special region.

[0068] 6A , when the object held by the user 11 is contained within the special area 95, the information processing device 10 (HMD, etc.) does not issue a warning against the hand gesture operation and does not perform control to ensure the safety of the hand gesture operation. On the other hand, when at least a part of the held object is outside the special area 95, as shown in FIG. 6B , the information processing device 10 (HMD, etc.) appropriately issues a warning against the hand gesture operation and performs control to ensure the safety of the hand gesture operation.

[0069] In this example, a special area 95 is set that covers and follows the vicinity of the hand of the user 11, and a hand gesture operation can be performed while holding an object within the set special area 95. This reduces the possibility that the object held by the user 11 will harm the user 11, surrounding people, objects, etc. while the user 11 is performing a hand gesture operation. Furthermore, because the information processing device 10 only needs to recognize the state near the hand of the user 11, the area that the information processing device 10 must recognize can be made smaller compared to when the information processing device 10 is required to recognize a wider area around the user 11.

[0070] 6C , a special area 96 may be set that follows the user 11 while covering an area within a predetermined distance range far from the user 11. As described above, the information processing device 10 (such as an HMD) does not issue a warning against a hand gesture operation or perform control to ensure the safety of the hand gesture operation when the object held by the user 11 is contained within the special area 96. On the other hand, the information processing device 10 appropriately issues a warning against a hand gesture operation and performs control to ensure the safety of the hand gesture operation when at least a part of the held object is outside the special area 96.

[0071] In this case, as in the above case, hand gesture operations can be performed while holding an object within the set special area 96, thereby reducing the possibility that the user 11 will harm the user 11, surrounding people, objects, etc. with the object they are holding while performing a hand gesture operation. Also, in the example of Fig. 6C , the width W1 of the special area in the direction away from the user 11 (horizontally, for example, in a 360-degree horizontal direction) is small, and the width W2 in the height direction is large. This forces the user 11 to hold the object in a safer position and orientation, ensuring the safety of the hand gesture operation, and allows the user 11 to perform hand gesture operations while holding the object within the set special area 96.

[0072] Next, an example of control when a user holds an object connected to an information processing device by wire and performs a hand gesture operation will be described with reference to FIG.

[0073] As shown in FIG. 7 , a device 97 (object) different from the information processing device 10 (e.g., an HMD) is connected to the information processing device 10 via a wired cable 98, and the user 11 holds the device 97 connected via the wired cable 98. Here, when the user 11 performs a hand gesture operation with the hand holding the device 97, the information processing device 10 (e.g., an HMD) performs control to prohibit the VR display mode and switch to the AR display mode, as an example. Alternatively, the information processing device 10 (e.g., an HMD) displays at least a portion of the wired cable 98 as a virtual object. Note that the information processing device 10 (e.g., an HMD) may display the device 97, the hand of the user 11 holding the device 97, or the like as a virtual object. In the AR display mode, if at least a portion of the wired cable 98 is visible through the display 62, the display of this virtual object may be omitted. Similarly, if the device 97 and the hand of the user 11 holding the device 97 are visible through the display 62, the display of this virtual object may be omitted.

[0074] In this example, the user 11 can easily recognize the wired cable 98, which may interfere with the hand gesture operation, and can perform the hand gesture operation conveniently. Furthermore, by recognizing the wired cable 98, the user 11 can perform the hand gesture operation without interfering with the wired cable 98. Therefore, safety in the hand gesture operation is ensured. Examples of other devices 97 different from the information processing device 10 include a controller (operation device 52), a remote control, a charger, earphones, a monitor, and a personal computer (PC). The information processing device 10 can be wiredly connected to the device 97 using the device wired input / output I / F 43 and the wired cable 98.

[0075] Next, an example of control of an information processing device other than an HMD will be described with reference to FIG. 8 . FIG. 8 illustrates a case where the information processing device is a smartphone. The smartphone 101 is held, for example, by the user 11 in the hand on the side not used to perform the hand gesture operation. The smartphone 101 analyzes the image and distance measurement data based on the acquired image and distance measurement data. As described above, the smartphone 101 then determines whether the user 11 is performing the hand gesture operation while holding an object. The smartphone 101 also identifies the object held by the user 11. Based on the result, the smartphone 101 displays a warning on the display 102 and appropriately executes safety control. When the user 11 performs a hand gesture operation using the smartphone 101, it is conceivable that the user 11 will perform the hand gesture operation very close to the smartphone 101. In this case, a warning display and safety control are useful to prevent the smartphone 101 from being damaged by the object held by the user 11.

[0076] The information processing device 10 may be an HMD, a smartphone 101, or an AR glass, a smart glass, a personal computer, a tablet, or the like. The information processing device 10, such as the smartphone 101, the AR glass, the smart glass, the personal computer, or the tablet, may be appropriately configured using, for example, the same configuration as that shown in FIGS. 2A and 2B . For example, when the information processing device 10 is the smartphone 101, the camera 23 may be appropriately configured using an in-camera 23a on the front side of the smartphone 101 and an out-camera on the back side. Furthermore, the smartphone 101 may have a ranging sensor on the outside (back side), or may have a ranging sensor 24a on the inside (front side). When the information processing device 10 is a personal computer, for example, the acceleration sensor 26, the gyro sensor 27, and the like may be omitted.

[0077] Next, an example of the operation of the information processing device will be specifically described with reference to Fig. 9A. Fig. 9A is a flowchart for describing an example of the operation of the information processing device.

[0078] The information processing device 10 generates a virtual object to be used for a hand gesture operation in the virtual object generation processing unit 72, and displays the virtual object on the display of the information processing device 10 (S901).

[0079] The information processing device 10 analyzes the image and distance measurement data using the image of the hand area of ​​the user 11 and the data related to distance measurement in the user state identification processing unit 73, and identifies whether the user 11 is making a hand gesture (S902). If it is identified that the user 11 is making a hand gesture, the process proceeds to S903 (S902-YES).

[0080] The information processing device 10 refers to the execution conditions for warning control and the execution conditions for safety assurance, which will be described in detail later, and determines whether the execution conditions for warning control and safety assurance control are met (S903). If at least one of the execution conditions is met, the process proceeds to S904 (S903-YES). If not, the process proceeds to S909 (S903-NO).

[0081] The information processing device 10 performs warning control using the warning information 75', which will be described in detail later (S904). If it is determined in S903 that the conditions for executing warning control are not met, warning control is not executed.

[0082] The information processing device 10 determines whether to perform safety ensuring control based on the result of S903 (S905). If it is determined in S903 that the execution conditions for safety ensuring control are met, the process proceeds to S906 (S905-YES). If not, the process proceeds to S909 (S905-NO). In addition, in S905, the user may be allowed to select whether to perform safety ensuring control. This allows the user 11 to appropriately select whether to further enhance the safety of hand gesture operations.

[0083] The information processing device 10 determines whether display is being performed in VR display mode (S906). If display based on VR display mode is being performed, the process proceeds to S908 (S906-YES). If not, the process proceeds to S907 (S906-NO).

[0084] In S907, the information processing device 10 executes safety ensuring control based on safety ensuring control 1, which will be described in detail later, and the process proceeds to S910. In S908, the information processing device 10 executes safety ensuring control based on safety ensuring control 2, which will be described in detail later, and the process proceeds to S910. Note that the safety ensuring control ends when it is detected that the execution conditions for the safety ensuring control are no longer satisfied.

[0085] In S909, the information processing apparatus 10 controls the hand gesture operation control processing unit 76 to execute the operation content based on the hand gesture executed by the user 11. Then, the process proceeds to S910.

[0086] When the user 11 has finished the hand gesture operation, the process ends. If it is not detected that the execution conditions for the safety assurance control are no longer satisfied before the hand gesture operation ends, the safety assurance control ends together with the hand gesture operation when the hand gesture operation ends. If the hand gesture operation is not finished, the process from S902 to S909 is repeated (S910-NO). In this case, it is determined in S902 whether or not a new hand gesture operation should be performed.

[0087] Next, an example of an operation performed by the information processing device using the special region will be specifically described with reference to Fig. 9B. Fig. 9B is a flowchart for describing an example of an operation performed by the information processing device using the special region.

[0088] As in S901 above, the information processing apparatus 10 generates a virtual object to be used for a hand gesture operation in the virtual object generation processing unit 72, and displays the generated virtual object on the display of the information processing apparatus 10 (S1001).

[0089] As in S902 above, the information processing device 10 identifies the hand gesture of the user 11 (S1002), and if it is identified that the user 11 is making a hand gesture, the process proceeds to S1003 (S1002-YES).

[0090] The information processing device 10 notifies the user 11 via the special area setting processing unit 80, prompting the user 11 to set the special areas (95, 96). If the user 11 sets the special areas (95, 96) (YES in S1003), the information processing device 10 stores the user 11's settings for the special areas (95, 96) in storage as special area information 80' via the special area setting processing unit 80, and the process proceeds to S1004. If the special areas (95, 96) are not set, the process proceeds to S1011 (NO in S1003). Alternatively, the same processes as those in S903 to S909 described above may be performed, and if the user 11 does not end the hand gesture operation, the process in S1002 may be performed again. In this case, it is determined in S1002 whether a new hand gesture operation will be performed. Alternatively, the notification prompting the user 11 to set the special area may be omitted, and the same processes as those in S903 to S909 described above may be performed.

[0091] The information processing device 10 uses the held object identification processing unit 74 to identify the object held by the hand performing the hand gesture operation, and determines whether at least a part of the object is outside the special area (95, 96) (S1004).

[0092] If it is determined that the held object has left the special area (95, 96), the process proceeds to S1005 (S1004-YES). Then, the information processing device 10 performs the same processes as S903 to S908 described above (S1005 to S1010). On the other hand, if it is determined that the held object has not left the special area (95, 96) (S1004-NO), the information processing device 10 performs the same process as S909 described above (S1011).

[0093] Then, when the user 11 ends the hand gesture operation, the processing ends. Note that if it is not detected that the execution conditions for the safety ensuring control are no longer satisfied before the hand gesture operation ends, the safety ensuring control ends together with the hand gesture operation when the hand gesture operation ends. If the hand gesture operation is not ended, the processing from S1002 to S1011 is repeated (S1012). In this case, it is determined in S1002 whether or not a new hand gesture operation should be performed.

[0094] Next, an example of the operation of the information processing device when a wired device is connected will be specifically described. Fig. 9C is a flowchart for explaining an example of the operation of the information processing device when a wired device is connected.

[0095] As in the above (S901, S1001), the information processing apparatus 10 causes the virtual object generation processing unit 72 to generate a virtual object to be operated with a hand gesture, and displays the generated virtual object on the display (S1101).

[0096] As in the above (S902, S1002), the information processing device 10 identifies the hand gesture of the user 11 (S1102), and if it is identified that the user 11 is making a hand gesture, the process proceeds to S1103 (S1102-YES).

[0097] The information processing apparatus 10 checks whether the device 97 is connected to the device wired input I / F 43 (S1103), and if the device 97 is confirmed to be connected via a wire, the process proceeds to S1104 (S1103-YES).If not, the process proceeds to S1109 (S1103-NO).

[0098] The information processing device 10 performs warning control using warning information 75', which will be described in detail later (S1104).

[0099] The information processing device 10 determines whether to perform safety ensuring control (S1105). Here, the information processing device 10 refers to the execution conditions for safety ensuring, which will be described in detail later, and determines whether the execution conditions for safety ensuring control are met. If it is determined that the execution conditions for safety ensuring control are met, the process proceeds to S1106 (YES in S1105). If not, the process proceeds to S1109 (NO in S1105). Alternatively, in S1105, the user may be allowed to select whether to perform safety ensuring control. This allows the user 11 to appropriately select whether to further enhance the safety of hand gesture operations.

[0100] As in the above (S906, S1008), the information processing device 10 determines whether display is being performed in VR display mode (S1106). If display based on VR display mode is being performed, the process proceeds to S1108 (S1106-YES). If not, the process proceeds to S1107 (S1106-NO).

[0101] As in the above (S907, S1009), in S1107, the information processing device 10 executes safety ensuring control based on safety ensuring control 1, which will be described in detail later, and the process proceeds to S1110. On the other hand, in S1108, the information processing device 10 executes safety ensuring control based on safety ensuring control 3, which will be described in detail later, and the process proceeds to S1110. Note that the safety ensuring control ends when it is detected that the execution conditions for the safety ensuring control are no longer satisfied.

[0102] As in the above (S909, S1011), in S1109, the information processing device 10 controls the hand gesture operation control processing unit 76 to execute operation content based on the hand gesture performed by the user 11. Then, the process proceeds to S1110.

[0103] When the user 11 has finished the hand gesture operation, the process ends. If it is not detected that the execution conditions for the safety assurance control are no longer satisfied before the hand gesture operation ends, the safety assurance control ends together with the hand gesture operation when the hand gesture operation ends. If the hand gesture operation is not to be ended, the process of S1102 to S1109 is repeated (S1110). In this case, it is determined in S1102 whether or not a new hand gesture operation should be performed.

[0104] Next, an example of the execution condition for warning control and the execution condition for ensuring safety will be described with reference to FIG. 9D . In this embodiment, the execution condition C1 for warning control and the execution condition C2 for ensuring safety are the same. As described above, these execution conditions (C1, C2) are used in S903, S1005, etc.

[0105] As shown in FIG. 9D , the execution conditions (C1, C2) include, for example, the following: the user 11 is holding an object in the hand making the hand gesture (condition 1); the object held by the user 11 is an object other than an operating device (condition 2); the object held by the user 11 in the hand making the hand gesture has a sharp portion (condition 3); the sharp portion of the object held by the user 11 is facing the user 11 (condition 4); the sharp portion of the object held by the user 11 is facing a surrounding object within a predetermined distance (condition 5); and the sharp portion of the object held by the user 11 is facing a nearby person within a predetermined distance (condition 6). However, the execution conditions (C1, C2) shown in FIG. 9D are merely examples and may be changed as appropriate. For example, other conditions may be added. Furthermore, the execution conditions (C1, C2) may be set so that warning control and safety control are not executed when the object held by the user 11 has a sharp portion and the sharp portion is not facing the user 11, the surrounding object, or the nearby person.

[0106] The data on the warning control execution condition C1 and the data on the safety assurance execution condition C2 are stored in advance in, for example, a storage device and are used as appropriate in the processing of the information processing device 10. The information processing device 10 can acquire the data on these execution conditions (C1, C2) based on an appropriate method. The information processing device 10 may acquire the data on the execution conditions (C1, C2) from, for example, a connected storage device or an information server 51. The information processing device 10 may also acquire the data on the execution conditions (C1, C2) and update the contents of the execution conditions (C1, C2) that it stores.

[0107] In the process of determining whether the execution conditions (C1, C2) are met (e.g., S903, S1005), the information processing device 10 can perform the process based on data acquired by the user state identification processing unit 73, the held object identification processing unit 74, the surrounding environment identification processing unit 79, etc. When at least one condition among the execution conditions C1 for warning control is detected (i.e., when at least one condition among the execution conditions C1 for warning control is met), the information processing device 10 determines to perform the warning control. Furthermore, when at least one condition among the execution conditions C2 for safety assurance is detected (i.e., when at least one condition among the execution conditions C2 for safety assurance is met), the information processing device 10 determines to perform the safety assurance control. However, if the execution conditions (C1, C2) are set so that the warning control and the safety assurance control are not executed when the object held by the user 11 has a sharp part and the sharp part is not directed toward the user 11, surrounding objects, or surrounding people, the information processing device 10 does not execute the warning control and the safety assurance control when this setting is met.

[0108] Next, the warning information will be described with reference to Fig. 9E. The warning information 75' is data summarizing what type of warning control will be performed. As described above, the warning information 75' is used in, for example, S904, S1006, and S1104.

[0109] 9E , the warning information 75′ includes, for example, a notification that hand gesture manipulation is dangerous (control 1), a notification that hand gesture manipulation is prohibited (control 2), a notification that prompts input using an input method other than hand gesture manipulation (control 3), a notification that an input mode will be switched to an input mode other than hand gesture manipulation (control 4), a notification that VR display mode will be prohibited (control 5), a notification that a display mode will be switched to AR display mode (control 6), a notification that a virtual object corresponding to an object held in the hand of the user 11 making the hand gesture will be generated and displayed (control 7), a notification that a virtual object corresponding to the connected wired cable 98 will be generated and displayed (control 8), a notification that prompts the user 11 to drop the object held in the hand of the user 11 making the hand gesture (control 9), a notification that prompts the user 11 to change the position of the object held in the hand of the user 11 making the hand gesture (control 10), a notification that prompts the user 11 to change the orientation of the object held in the hand of the user 11 making the hand gesture (control 11), and a notification that prompts the user 11 to disconnect the connected wired cable 98 (control 12). However, the warning information 75' shown in Fig. 9E is merely an example and may be changed as appropriate. For example, other controls may be added.

[0110] The warning information 75' is stored in advance in, for example, a storage device and is used as appropriate in the processing of the information processing device 10. The information processing device 10 can acquire the warning information 75' based on an appropriate method. The information processing device 10 may acquire the warning information 75' from, for example, a connected storage device, the information server 51, etc. Furthermore, the information processing device 10 may acquire the warning information 75' and update the content of the stored warning information 75'.

[0111] In the process of performing warning control (e.g., S904, S1006, S1104), the information processing device 10 causes the warning control processing unit 75 to execute at least one warning control included in the warning information 75′. Here, the warning control related to the wired connection (e.g., control 8, control 12) among the warning information 75′ is executed in S1104. Note that, as an example of the manner of notification in the warning control, a display indicating the fact may be given. However, notification may also be given in other manners, such as by voice through a speaker or by vibration through a vibrator.

[0112] Next, an example of safety assurance control will be described with reference to Figures 9F to 9H. Figures 9F to 9H are diagrams showing an example of data used for safety assurance control. This data summarizes what type of safety assurance control will be performed.

[0113] First, safety ensuring control 1 will be described with reference to Fig. 9F. Safety ensuring control 1 is the control performed in the above steps (S907, S1009, S1107).

[0114] As shown in Fig. 9F, data 81'a used in safety ensuring control 1 includes, for example, prohibiting hand gesture operation (control 1), transitioning to an input mode using another input method (control 2), and prohibiting VR display mode (control 3). However, the data 81'a shown in Fig. 9F is merely an example and may be changed as appropriate. For example, other controls may be added.

[0115] The data 81'a used in the safety ensuring control 1 is, for example, stored in advance in a storage device and used as appropriate in the processing of the information processing device 10. The information processing device 10 can acquire this data 81'a based on an appropriate method. The information processing device 10 may acquire the data 81'a used in the safety ensuring control 1 from, for example, a connected storage device, an information server 51, or the like. Furthermore, the information processing device 10 may acquire the data 81'a used in the safety ensuring control 1 and update the content of the stored data 81'a.

[0116] When executing safety assurance control 1 (e.g., S907, S1009, S1107), the information processing device 10 executes at least one safety assurance control 1 included in the data 81'a used for safety assurance control 1 using the hand gesture operation control processing unit 76, the input method control processing unit 77, the display mode switching processing unit 78, etc.

[0117] Next, safety ensuring control 2 will be described with reference to Fig. 9G. Safety ensuring control 2 is the control performed in the above steps (S908, S1010).

[0118] 9G , data 81′b used for safety ensuring control 2 includes, for example, prohibiting hand gesture operation (control 1), transitioning to an input mode for another input method (control 2), prohibiting VR display mode (control 3), transitioning to AR display mode (control 4), and generating and displaying a virtual object corresponding to an object held in the hand of the user 11 performing the hand gesture (control 5). However, the data 81′b shown in FIG. 9G is merely an example and may be changed as appropriate. For example, other controls may be added.

[0119] The data 81'b used in the safety ensuring control 2 is, for example, stored in advance in a storage device and used as appropriate in the processing of the information processing device 10. The information processing device 10 can acquire this data 81'b based on an appropriate method. The information processing device 10 may acquire the data 81'b used in the safety ensuring control 2 from, for example, a connected storage device, an information server 51, or the like. Furthermore, the information processing device 10 may acquire the data 81'b used in the safety ensuring control 2 and update the content of the stored data 81'b.

[0120] When executing safety control 2 (e.g., S908, S1010), the information processing device 10 executes at least one safety control 2 included in data 81'b used for safety control 2 using the hand gesture operation control processing unit 76, the input method control processing unit 77, the display mode switching processing unit 78, the virtual object generation processing unit 72, etc.

[0121] Next, safety ensuring control 3 will be described with reference to Fig. 9H. Safety ensuring control 3 is the control performed in S1108 above.

[0122] 9H , data 81′c used in safety assurance control 3 includes, for example, prohibiting hand gesture operations (control 1), transitioning to an input mode using another input method (control 2), prohibiting VR display mode (control 3), transitioning to AR display mode (control 4), generating and displaying a virtual object corresponding to an object held in the hand of the user 11 performing the hand gesture (control 5), and generating and displaying a virtual object corresponding to the connected wired cable 98 (control 6). However, the data 81′c shown in FIG. 9H is merely an example and may be changed as appropriate. For example, other controls may be added.

[0123] The data 81'c used in the safety ensuring control 3 is, for example, stored in advance in a storage device and used as appropriate in the processing of the information processing device 10. The information processing device 10 can acquire this data 81'c based on an appropriate method. The information processing device 10 may acquire the data 81'c used in the safety ensuring control 3 from, for example, a connected storage device or an information server 51. Furthermore, the information processing device 10 may acquire the data 81'c used in the safety ensuring control 3 and update the content of the stored data 81'c.

[0124] When executing safety assurance control 3 (e.g., S1108), the information processing device 10 executes at least one safety assurance control 3 included in data 81'c used for safety assurance control 3 using the hand gesture operation control processing unit 76, the input method control processing unit 77, the display mode switching processing unit 78, the virtual object generation processing unit 72, etc.

[0125] The information processing device 10 may notify the user of the details of the safety control to be executed. For example, the information processing device 10 may display the details of the safety control to be executed on a display. Note that the information processing device 10 may notify the user in other ways (for example, by using sound or vibration).

[0126] According to the embodiment, for example, a technique is provided that allows the user 11 to easily perform hand gesture operations while ensuring safety in the hand gesture operations.

[0127] Although the embodiments have been described above, the present invention is not limited to the above-described embodiments and includes various modifications and equivalent configurations within the spirit of the appended claims. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, for example, other configurations may be added, deleted, or replaced with part of the configuration of the embodiment.

[0128] The display format of the warning display and the details of the safety control to be executed is not particularly limited as long as the user 11 can understand the details. The information processing device 10 may display using messages, characters, symbols, figures, etc. For example, the information processing device 10 may display a message such as "Hand gesture operations are dangerous!" on the display as a notification (control 1) that hand gesture operations are dangerous. The information processing device 10 may also display a message such as "Please input using a method other than hand gesture operations!" on the display as a notification (control 3) that encourages input using an input method other than hand gesture operations. The information processing device 10 may also display a message such as "Hand gesture operations are prohibited!" on the display as a notification of control to prohibit hand gesture operations (safety control). The information processing device 10 may also display a message such as "VR display mode is prohibited!" on the display as a notification of control to prohibit VR display mode (safety control).

[0129] The information processing device 10 can be configured so that various operations (such as operations of virtual objects and operations related to various settings of the information processing device 10) can be performed by hand gesture operations. The information processing device 10 may then display a warning and perform safety control based on the hand gesture operations.

[0130] Furthermore, the above-mentioned configurations, functions, processing units, processing means, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. Furthermore, the above-mentioned configurations, functions, etc. may be implemented in software by a processor interpreting and executing a program that implements each function. The processor may include transistors and other circuits and may be considered circuitry or processing circuitry. Information such as programs, tables, and files that implement each function may be stored in memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD, or may be stored in a device on a communication network. Furthermore, the control lines and information lines shown are those considered necessary for explanation, and do not necessarily represent all control lines and information lines in the product. In reality, it may be assumed that almost all components are interconnected.

[0131] 10 Information processing device 71 Virtual space display processing unit 72 Virtual object generation processing unit 73 User state identification processing unit 74 Held object identification processing unit 75 Warning control processing unit 76 Hand gesture operation control processing unit 77 Input method control processing unit 78 Display mode switching processing unit 79 Surrounding environment identification processing unit 80 Special area setting processing unit

Claims

1. An information processing apparatus comprising a processor, wherein the processor performs warning control, which is control to output a warning to the user when the user performs a hand gesture operation with a hand holding an object, or safety assurance control, which is control to ensure the safety of the hand gesture operation for the user.

2. The information processing apparatus according to claim 1, wherein the processor performs the warning control and the safety assurance control on the condition that the object held by the user is an object other than the operating device of the information processing apparatus.

3. The information processing apparatus according to claim 1, wherein the processor performs the warning control and the safety assurance control on the condition that the object held by the user has a sharp part.

4. The information processing apparatus according to claim 1, wherein the processor performs the warning control and the safety assurance control on the condition that the object held by the user has a sharp part and the sharp part is facing the user, a surrounding object within a predetermined distance, or a surrounding person within a predetermined distance.

5. The information processing apparatus according to claim 1, wherein the processor does not perform the warning control and the safety assurance control when the object held by the user does not have a sharp part.

6. The information processing apparatus according to claim 1, wherein the processor performs control to prohibit a hand gesture operation as the safety assurance control.

7. The information processing apparatus according to claim 1, wherein the processor performs control to prohibit a VR display mode for displaying virtual reality as the safety assurance control.

8. The information processing apparatus according to claim 1, wherein the information processing apparatus performs control to prohibit a VR display mode for displaying virtual reality and switch to an AR display mode for displaying augmented reality as the safety assurance control.

9. The information processing apparatus according to claim 1, wherein the information processing apparatus generates a virtual object corresponding to an object held by the user as the security control, and executes control to display the virtual object in a VR display mode for displaying virtual reality. The information processing apparatus is characterized by this.

10. The information processing apparatus according to claim 1, wherein the processor acquires special area information regarding data of a special area that follows while covering the user's hand, and does not execute the warning control and the security control when an object held by the user is included in the special area. The information processing apparatus is characterized by this.

11. The information processing apparatus according to claim 1, wherein the processor acquires special area information regarding data of a special area that follows while covering the user's hand, and executes the warning control and the security control when at least a part of the object held by the user protrudes from the special area. The information processing apparatus is characterized by this.

12. The information processing apparatus according to claim 1, wherein the processor acquires special area information regarding data of a special area that follows the user while covering an area within a predetermined distance range far from the user, and does not execute the warning control and the security control when an object held by the user is included in the special area. The information processing apparatus is characterized by this.

13. The information processing apparatus according to claim 1, wherein the processor acquires special area information regarding data of a special area that follows the user while covering an area within a predetermined distance range far from the user, and executes the warning control and the security control when at least a part of the object held by the user protrudes from the special area. The information processing apparatus is characterized by this.

14. An information processing method performed using an information processing apparatus including a processor, the processor acquiring data of an image including a user's hand, the processor using the data of the image to identify whether the user is holding an object in the hand and performing a hand gesture operation with the hand holding the object, warning control which is control for outputting a warning to the user when the user is holding an object in the hand and performing a hand gesture operation with the hand holding the object, and safety ensuring control which is control for ensuring the safety of the hand gesture operation to the user, the information processing method being characterized by executing the warning control and the safety ensuring control.

15. An information processing system including an information processing apparatus and a device connected to the information processing apparatus via a wired cable, the information processing apparatus executing warning control which is control for outputting a warning to the user when the user performs a hand gesture operation with the hand holding an object, and safety ensuring control which is control for ensuring the safety of the hand gesture operation to the user.

Images