Control device, control method, and program
By introducing a selection and control unit into the information processing device, allowing users to select a control mode based on the imaging unit or physical contact, the problem that the user cannot control the HMD when the input device is not used is solved, and user-friendly control experience and power consumption are achieved reduce.
Patent Information
- Application Number
- CN202280094112.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-09-30
- Filing Date
- 2022-12-21
- Publication Date
- 2024-11-15
AI Technical Summary
The existing information processing devices cannot control electronic devices, such as head-mounted displays (HMDs) based on the user's desired control method without using the input device, resulting in the inability to select an appropriate application or control method.
By introducing a selection unit and a control unit in the information processing device, the user is allowed to select a first control mode (based on the user's specific position state determined by the imaging unit) or a second control mode (based on the physical contact operation of the user with the specific operating component) so as to Control electronic devices according to the user's intentions.
The control method of electronic devices is realized according to the user's desired control method, which improves the user's convenience and control accuracy, and reduces the power consumption of the information processing equipment.
Smart Images

Figure CN118974684A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a control device, a control method and a program. Background Art
[0002] Mixed reality (MR) technology and virtual reality (VR) technology are known, which allow users to use head mounted displays (HMD) to experience a space different from the real space. In these technologies, it has been considered that the user performs various controls on the HMD while wearing the HMD.
[0003] Patent document 1 discloses an information processing device that recognizes an input device and an operating medium (user's hand) that operates the input device, and selects an application based on the recognition result. For example, in the case where a keyboard (input device) and a user's hand are recognized, the information processing device starts a text editor application that allows text to be input by operating the keyboard. On the other hand, in the case where a controller (input device) and a hand are recognized, the information processing device starts an application to perform CG adjustment by operating the controller.
[0004]
Citation List
[0005] [Patent Literature]
[0006] Patent document 1: Japanese Patent Application Publication No. 2017-059062. Summary of the invention
[0007] Problem that the invention aims to solve
[0008] The information processing device according to Patent Document 1 selects an application to be started when both the input device and the operation medium are recognized. This means that if a user controls an electronic device such as an HMD without using an input device, the information processing device cannot select an appropriate application, for example. Therefore, in some cases, the electronic device may not be controlled according to the control method desired by the user.
[0009] In view of the above circumstances, an object of the present invention is to provide a technology that allows an electronic device to be controlled according to a control method desired by a user.
[0010] Solutions for solving problems
[0011] One aspect of the present invention is a control device for controlling an electronic device, comprising: a selection unit configured to select any one of a first control mode and a second control mode as a control mode for controlling the electronic device; and a control unit configured to: 1) control the electronic device according to a state of a specific part of a user determined based on imaging by an imaging unit when the selection unit selects the first control mode, and 2) control the electronic device according to an operation of the user involving physical contact with a specific operating component when the selection unit selects the second control mode.
[0012] Another aspect of the present invention is a method for controlling an electronic device, comprising: a selection step of selecting any one of a first control mode and a second control mode as a control mode for controlling the electronic device; and a control step: 1) in a case where the first control mode is selected in the selection step, controlling the electronic device according to a state of a specific part of the user determined based on imaging by a camera unit; and 2) in a case where the second control mode is selected in the selection step, controlling the electronic device according to an operation of the user involving physical contact with a specific operating component.
[0013] Effects of the Invention
[0014] According to the present invention, an electronic device can be controlled according to a control method desired by a user. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] Description of Reference Numerals
[0016] Figure 1 It is a diagram for describing the information processing system according to Embodiment 1.
[0017] Figure 2 This is an internal block diagram of the HMD and the like according to the first embodiment.
[0018] Figure 3 is a diagram for describing a ray control mode according to Embodiment 1.
[0019] Figure 4 is a diagram for describing the tracking control mode according to the first embodiment.
[0020] Figure 5 This is a flowchart of the mode selection process according to the first embodiment.
[0021] Figure 6 This is a flowchart of the mode selection process according to the second embodiment.
[0022] Figure 7 is a diagram for describing a method for selecting a control mode according to Embodiment 2.
[0023] Figure 8 This is a flowchart of the mode selection process according to the third embodiment.
[0024] Fig. 9 This is a flowchart of the mode selection process according to the fourth embodiment.
[0025] Fig.10 This is a flowchart of the mode selection process according to the fifth embodiment.
[0026] Fig.11 A and Fig.11 B is a conceptual diagram for describing a preparatory action according to the fifth embodiment. DETAILED DESCRIPTION
[0027] Each embodiment will now be described with reference to the accompanying drawings. The same or similar constituent elements, parts or processing steps shown in the figures will be represented by the same reference numerals, and repeated descriptions will be omitted. In each figure, some constituent elements, parts and processes will be omitted.
[0028] <Example 1>
[0029] Will refer to Figure 1 An information processing system 1 according to Embodiment 1 is described. The information processing system 1 includes an HMD 100 , an image processing apparatus 110 , and a controller 120 .
[0030] The HMD 100 is a head mounted display device (electronic device) worn on the user's head. A composite image composed of a captured image of a range in front of the user captured by the HMD 100 and contents such as computer graphics (CG) in a form according to the posture of the HMD 100 is displayed on the HMD 100.
[0031] The image processing device 110 is a control device (electronic device) for controlling the HMD 100. For example, the image processing device 110 is a smartphone, a tablet terminal, or a personal computer (PC). The image processing device 110 is connected to the HMD 100 via a cable or wirelessly. The image processing device 110 generates a composite image by synthesizing a captured image with CG, and transmits the composite image to the HMD 100. Each configuration of the image processing device 110 may be included in the HMD 100.
[0032] The controller 120 is a device for performing various controls on the HMD 100. If the image processing device 110 is in a specific control mode and the user operates the controller 120, the HMD 100 is controlled according to the user's operation. For example, the controller 120 has a ring shape so as to be worn on the user's finger, such as Figure 1As shown. If the controller 120 can be worn on the user's finger like this, the user can freely move their hand while holding the controller 120. The controller 120 includes a button surrounding an optical touch pad (hereinafter referred to as "OTP"). The controller 120 wirelessly communicates with the image processing device 110 via Bluetooth.
[0033] For example, the user causes the HMD 100 to display a menu including a pointer by pressing the button of the OTP for a long time. Then, by touching the OTP with a finger and sliding the finger in any direction, the user can set the pointer on the desired item. Then, by pressing the button of the OTP, the user can perform a confirmation operation to confirm the selection of this item. Here, the shape of the controller 120 is a ring, but the shape is not limited thereto. For example, the shape of the controller 120 can be a glove, which can be worn on the hand. Therefore, the controller 120 can be held by hand or worn on the user's hand so that the user can easily use it.
[0034] The controller 120 may include any operating component instead of the OTP as long as the user can operate the component through physical contact. For example, the controller 120 may include a touch pad, a touch panel, a cross key, a joystick, or a track pad device instead of the OTP.
[0035] (Internal configuration of HMD)
[0036] Will refer to Figure 2 The internal configuration of the HMD 100 is described. The HMD 100 includes an HMD control unit 201 , an imaging unit 202 , an image display unit 203 , and a posture sensor unit 204 .
[0037] The HMD control unit 201 controls each constituent element of the HMD 100. When the HMD control unit 201 acquires a synthesized image (an image generated by synthesizing a captured image of the space in front of the user captured by the camera unit 202) with CG from the image processing device 110, the HMD control unit 201 displays the synthesized image on the image display unit 203. Therefore, the user wearing the HMD 100 can view the synthesized image displayed on the image display unit 203. The user can experience various mixed realities generated by integrating CG into the real space.
[0038] The camera unit 202 includes two cameras (camera devices). In order to capture an image of a space similar to the space that the user normally views, the two cameras are arranged near the positions of the left and right eyes of the user when the HMD 100 is worn. The images of the subject (the range in front of the user) captured by the two cameras are output to the image processing device 110. The two cameras of the camera unit 202 can obtain information related to the distance from the two cameras to the subject as distance information based on the distance measurement of the stereo camera.
[0039] The image display unit 203 displays the synthesized image. The image display unit 203 includes, for example, a liquid crystal panel, an organic EL panel, etc. In a state where the user is wearing the HMD 100, the organic EL panels are respectively provided in front of the user's eyes.
[0040] The posture sensor unit 204 detects the posture (and position) of the HMD 100. Then, the posture sensor unit 204 detects (acquires) the posture of the user (the user wearing the HMD 100) corresponding to the posture (and position) of the HMD 100. The posture sensor unit 204 includes an inertial measurement unit (IMU). The posture sensor unit 204 outputs information about the user's posture (posture information) to the image processing device 110.
[0041] (Internal Configuration of Image Processing Device)
[0042] Will refer to Figure 2 The internal configuration of the image processing apparatus 110 is described. The image processing apparatus 110 includes a control unit 211 , a content database 212 , and a communication unit 213 .
[0043] The control unit 211 receives an image (captured image) acquired by the imaging unit 202 and posture information acquired by the posture sensor unit 204 from the HMD 100. The control unit 211 performs image processing on the captured image so as to eliminate aberrations in the optical system of the imaging unit 202 and the optical system of the image display unit 203. Then, the control unit 211 synthesizes the captured image with an arbitrary CG, and thereby generates a synthesized image. The control unit 211 sends the synthesized image to the HMD control unit 201 of the HMD 100.
[0044] Furthermore, the control unit 211 controls the position, direction, and size of CG in the synthesized image based on the information (distance information and posture information) acquired by the HMD 100. For example, in a case where a virtual object indicated by CG is located near a specific object existing in the real space in the space generated by the synthesized image, the control unit 211 increases the size of the virtual object (CG) as the distance between the specific object and the camera unit 202 becomes shorter. By controlling the position, direction, and size of CG like this, the control unit 211 can generate a synthesized image as if an object of CG that is not located in the real space is actually located in the real space.
[0045] The content database 212 is a storage unit for storing information such as CG, etc. The control unit 211 can select a CG read from the content database 212 (that is, a CG used to generate a synthesized image).
[0046] (Internal configuration of the controller)
[0047] Will refer to Figure 2 The internal configuration of the controller 120 is described. The controller 120 includes a controller control unit 221 , an operation unit 222 , and a communication unit 223 .
[0048] The controller control unit 221 controls each constituent element of the controller 120 .
[0049] The operation unit 222 includes buttons surrounding the OTP, and information that a finger has pressed or slid on the OTP in the operation unit 222 is transmitted to the image processing apparatus 110 via the communication unit 223 .
[0050] For example, the user can move a pointer displayed on the HMD 100 to a desired position by sliding his finger on the OTP. In addition, the user can instruct a specific process to the HMD 100 or the image processing device 110 by pressing a button of the OTP. Therefore, the user can control the HMD 100 by a combination of sliding a finger on the OTP and pressing a button of the OTP. For example, the user causes the HMD 100 to display a menu by pressing the button of the OTP for a long time, sets the pointer to a desired position, and then presses the OTP button, so that the position can be selected.
[0051] The communication unit 223 performs wireless communication with the image processing device 110 (communication unit 213 ).
[0052] (Control Mode)
[0053] In the present embodiment, the image processing device 110 can realize control of the HMD 100 (for example, control of selecting a CG or moving the position of a CG in a composite image displayed on the HMD 100 ) through the following three modes (control methods).
[0054] As described above, the first control mode is a control mode in which the HMD 100 is controlled according to the operation of the OTP (an operation involving physical contact of the user) by the controller 120 (hereinafter referred to as "OTP control mode"). If the image processing device 110 is set to the OTP control mode, the control unit 211 controls the HMD 100 according to the user's operation of the OTP.
[0055] The second control mode is a control mode in which the HMD 100 is controlled using a virtual ray corresponding to the direction of the user's hand (hereinafter referred to as "ray control mode"). If the image processing device 110 is set to the ray control mode, the control unit 211 controls the HMD 100 according to the position of the virtual ray extending along the extension line of the direction of the user's hand.
[0056] Specifically, in the ray control mode, the control unit 211 determines (recognizes) the direction of the user's hand based on the color and shape of the user's hand captured in the captured image. Figure 3 As shown, the control unit 211 displays the CG of the ray extending in the direction pointed by the index finger on the image display unit 203. Then, the control unit 211 displays (moves) the pointer to the position where the ray is directed. Therefore, the user can change the position and direction of the ray by changing the direction of the hand.
[0057] In the ray control mode, if the state of directing the ray to a specific display item continues for more than a predetermined time, for example, a process of determining the selection of a specific item indicated by the ray (determination process) is performed. For example, if a menu panel (a panel displaying a menu) is always displayed at a corner of the screen of the image display unit 203, and determination processing is performed while the ray is directed to the panel, the menu is displayed.
[0058] For example, in the ray control mode, the control unit 211 displays the composite images of the CGs of the panels 301 to 303 respectively corresponding to the three options as a menu on the composite image, such as Figure 3 Here, if the state of the rays being directed to one panel 303 continues for more than a predetermined time, the control unit 211 performs a process corresponding to the panel 303 to which the rays are directed.
[0059] An inertial measurement unit (IMU) provided in the controller 120 worn on the user's hand may be used to determine the direction in which the user is pointing. Specifically, based on the information acquired by the IMU (information about the inclination and acceleration of the controller 120) and the information about the finger direction determined in the image, the control unit 211 may improve the matching rate between the direction in which the user is pointing and the extension direction of the ray.
[0060] The third control mode is a control mode in which the HMD 100 is controlled using hand tracking corresponding to the position of the user's hand (hereinafter referred to as "tracking control mode"). If the image processing device 110 is set to the tracking control mode, the control unit 211 determines (recognizes) the position of the hand based on the direction, color, and shape of the user's hand captured in the captured image. Then, the control unit 211 controls the HMD 100 according to the position of the hand in the space indicated by the synthesized image (specifically, the distance between the hand and the CG).
[0061] For example, Figure 4 As shown, the control unit 211 displays CGs of panels 301 to 303 corresponding to the three options, respectively, on the composite image, and calculates the distance between the three panels 301 to 303 and the fingertip of the index finger of the hand in the space indicated by the composite image. If the state in which the distance between a panel among the three panels 301 to 303 and the fingertip is equal to or less than the predetermined distance continues for more than a predetermined time, the control unit 211 determines that the panel is selected. Then, the control unit 211 performs determination processing corresponding to the selected panel. In the tracking control mode, as in the ray control mode, a menu panel (a panel for displaying a menu) is displayed on the corner of the screen of the image display unit 203.
[0062] Each of the three modes has its own advantages and disadvantages. For example, in a control mode in which control corresponding to the state of the hand (the state of the hand determined based on the captured image) is performed (in the case of the ray control mode and the tracking control mode), the user can intuitively perform quick control. However, in these control modes, since the hand must be included in the image capturing range, the hand must always be kept at a certain height. Therefore, the ray control mode and the tracking control mode are not suitable for long-term control. On the other hand, in the OTP control mode, the user can control the HMD 100 with their hands lowered, but the control speed is not that fast. In other words, in the OTP mode, stable control by the user can be expected even if the control time is extended for a long time.
[0063] The ray control mode that can control the HMD 100 according to the ray is convenient when selecting a display item (panel) far from the hand. On the other hand, the tracking control mode is convenient when selecting a display item (panel) close to the hand.
[0064] As described above, the three control modes are suitable for different use scenarios, so if the control mode for controlling the HMD 100 is appropriately switched (selected) among these control modes, user convenience can be improved.
[0065] (Mode selection processing)
[0066] Will refer to Figure 5The flowchart in describes a mode selection process of selecting a control mode to be used for controlling the HMD 100 among three control modes (operation modes). When the user starts using the HMD 100, the process of this flowchart starts. Figure 5 The flowchart in FIG. 1 is implemented by the control unit 211 (eg, a processor) executing a program.
[0067] In step S501, the control unit 211 determines whether the user's hand is captured within a predetermined range of the captured image (the angle of view of the camera of the imaging unit 202: the imaging range). If the user's hand is captured within the predetermined range, the processing proceeds to step S502. If the user's hand is not captured within the predetermined range, the processing proceeds to step S503. The predetermined range here is, for example, the entire range of the captured image (the imaging range). However, in the case where the hand is only partially captured at the edge of the captured image, it may become difficult to determine the shape of the hand, etc. Therefore, the predetermined range may be a range slightly inside the captured image, within which the hand can be fully captured. The predetermined range may also be, for example, a range fixed with reference to the position of the user's face.
[0068] In step S502, the control unit 211 calculates the distance from the imaging unit 202 (HMD 100) to the user's hand by measuring the distance using the stereo camera of the two cameras of the imaging unit 202. Then, the control unit 211 determines whether the distance from the imaging unit 202 to the user's hand is longer than the threshold value Th (whether the user's hand is farther from the HMD 100 than the threshold value Th).
[0069] Generally, if the user wants to control using rays, the user tends to move his hand forward to point to a target. In other words, the position of the user's hand tends to become relatively far away from the user. Therefore, in the case where the distance from the camera unit 202 to the user's hand is greater than the threshold value Th, the processing proceeds to step S504 to select the ray control mode, because the user is more likely to be pointing his finger at a far position away from the user. On the other hand, in the case where the distance from the camera unit 202 to the user's hand is equal to or less than the threshold value Th, the processing proceeds to step S505 to select the tracking control mode, because the user is more likely to be selecting the position by moving his hand to a position close to the user.
[0070] The range of a user's hand reaching varies from person to person. Therefore, the control unit 211 can set the threshold Th in step S502 according to the length of the user's arm input in advance, so that the threshold Th is set to a value suitable for the individual. In this case, as the user's arm length is longer, the control unit 211 increases the value of the threshold Th. For example, the control unit 211 multiplies the user's arm length by a predetermined value (e.g., 0.8) and sets the value as the threshold Th.
[0071] The HMD 100 may include (use) a time of flight (ToF) sensor to measure the distance from the camera unit 202 to the user's hand. In addition, the control unit 211 may estimate the distance from the camera unit 202 to the user's hand based on the size of the hand captured in the captured image.
[0072] In step S503 , the control unit 211 selects the OTP control mode (control mode through OTP using the controller 120 ) as the control mode for controlling the HMD 100 .
[0073] In step S504 , the control unit 211 selects the ray control mode as the control mode for controlling the HMD 100 .
[0074] In step S505 , the control unit 211 selects the tracking control mode as the control mode for controlling the HMD 100 .
[0075] In step S506, the control unit 211 enables the control mode selected in steps S503 to S505. In other words, when the user performs an action (operation) corresponding to the selected control mode, the control unit 211 controls the HMD 100 (display on the image display unit 203) according to the action (operation). In addition, the control unit 211 disables control modes other than the selected control mode. For example, if the OTP control mode is selected, the control unit 211 controls the HMD 100 according to the operation on the OTP, and does not control the HMD 100 according to the shape or position of the user's hand. The control unit 211 may display a different user interface (UI) on the image display unit 203 according to the enabled control model. The control unit 211 may display a display item (e.g., a text indicating the name of the control mode) for identifying the enabled control mode on the image display unit 203.
[0076] In step S507, the control unit 211 determines whether to perform a process of ending the operation of the HMD 100. For example, if the power button of the HMD 100 is pressed, the process of ending the operation of the HMD 100 is performed. If it is determined that the process of ending the operation of the HMD 100 is not performed, the process returns to step S501. If it is determined that the process of ending the operation of the HMD 100 is performed, the process of the flowchart ends, and the operation of the HMD 100 ends.
[0077] The state of the user's hand (whether the user's hand is captured in a predetermined range of the image, and the distance from the camera unit 202 to the user's hand) often indicates the user's intention of how the user wants to control the HMD 100. Therefore, according to Embodiment 1, a control mode that conforms to the user's intention can be selected, and the HMD 100 can be controlled in the control mode that conforms to the user's intention. As a result, the user convenience of the HMD 100 control can be improved.
[0078] Furthermore, by disabling the unselected control mode, the processing required by the image processing device 110 can be reduced, and thus a reduction in power consumption of the image processing device 110 can be expected. For example, in the case where the OTP control mode is disabled, the image processing device 110 can be controlled not to communicate with the controller 120.
[0079] Figure 5 The processing of each step in the flowchart is performed by the above-mentioned control unit 211 , but the processing in each step may also be performed by the HMD control unit 201 of the HMD 100 .
[0080] If it is determined in step S501 that the user's hand is captured in the predetermined range, the processing in step S504 or S505 may be performed without performing the processing in step S502. For example, if it is determined in step S501 that the user's hand is captured in the predetermined range, the control unit 211 may select a control mode preset by the user from the ray control mode and the tracking control mode.
[0081] <Example 2>
[0082] Will refer to Figure 6 The information processing system 1 that selects (switches) the control mode of the HMD 100 according to Embodiment 2 is described. In Embodiment 2, the mode selection process is different from that in Embodiment 1, and therefore, the mode selection process will be described below.
[0083] Figure 6 The processing in the flowchart in Figure 5 The flowchart in FIG. 1 is different only in steps S501 and S502, and therefore, only the processing in steps S601 to S603 performed instead of steps S501 and S502 will be described below.
[0084] In step S601, the control unit 211 determines whether the user's hand is in contact with the OTP of the controller 120. If it is determined that the user's hand is in contact with the OTP, this may be because the user is preparing to operate the OTP, so the process proceeds to step S503. If it is determined that the user's hand is not in contact with the OTP, the process proceeds to step S602.
[0085] In step S602, the control unit 211 determines whether the user's hand is captured in a predetermined range (angle of view of the camera) of the captured image. If it is determined that the user's hand is not captured in the predetermined range, the process returns to step S601 because control that requires the hand to be captured (control using a ray or hand tracking method) cannot be performed. If it is determined that the user's hand is captured in the predetermined range, the process proceeds to step S603.
[0086] In step S603, the control unit 211 determines whether the angle θ between the direction pointed by the user's finger and the horizontal direction (horizontal plane) is greater than a threshold value θth (see Figure 7 ). If the angle θ is less than the threshold θth, the process proceeds to step S504 to select the ray control mode. If the angle θ is equal to or greater than the threshold θth, the process proceeds to step S505 to select the tracking control mode. In other words, Figure 7 As shown, in the case of angle θ=θ1, the process proceeds to step S504 because the angle θ is smaller than the threshold value θth. In the case of angle θ=θ2, the process proceeds to step S505 because the angle θ is equal to or larger than the threshold value θth.
[0087] Generally, when a user wishes to control using ray tracking, the user moves his hand forward to point to a target, such as a selection item. Therefore, his finger tends to point forward in a direction close to the horizontal direction. In other words, when a user wishes to control using the ray control mode, the angle θ tends to be small. On the other hand, when a user wishes to control using hand tracking, the direction in which his finger points tends to be relatively closer to the user's upper vertical direction than when using ray control, such as Figure 4 In other words, in situations where the user wishes to use hand tracking for control, the angle θ tends to be large.
[0088] Therefore, if the control mode is selected according to the size of the angle θ, the control mode desired by the user can be appropriately selected. This enables the control unit 211 to control the HMD 100 using the control method desired by the user.
[0089] In the above description, the control unit 211 switches the control mode by comparing the angle θ between the horizontal direction and the direction in which the user's finger is pointing with the threshold θth. However, the control unit 211 may change the threshold θth according to the conditions. For example, the posture sensor unit 204 may detect the degree to which the user bends forward / backward (the inclination of the user's upper body), so the control unit 211 may determine the threshold θth based on the inclination of the user's upper body each time the processing in step S602 is performed. For example, the more the user bends forward, the more the control unit 211 lowers the threshold θth.
[0090] The state of the user's hand (whether the user's hand is in contact with the OTP, and the angle between the direction in which the user's finger is pointing and the horizontal direction) often indicates the user's intention of how the user wants to control the HMD 100. Therefore, according to Embodiment 2, the information processing system 1 can more appropriately select the control mode desired by the user by determining whether the user is in contact with the OTP, and the angle between the direction in which the user's finger is pointing and the horizontal direction.
[0091] In the case where it is determined in step S601 that the user's hand is not in contact with the OTP, the processing may proceed to step S603 instead of step S602. In addition, in step S601, the control unit 211 may determine whether the user's hand is open (at least one finger of the user's hand is not bent) or closed. Whether the user's hand is open may be determined by analyzing the captured image and / or detecting whether the hand is in contact with the periphery of the ring of the controller 120. If it is determined that the user's hand is closed (all fingers of the user's hand are bent), the processing proceeds to step S503 because control using rays or tracking is unlikely. If it is determined that the user's hand is open (at least one finger of the user's hand is not bent), the processing proceeds to step S602. "Hand open" may be interpreted as "index finger not bent", and "hand closed" may be interpreted as "index finger bent".
[0092] <Example 3>
[0093] Will refer to Figure 8 The information processing system 1 for selecting (switching) the control mode of the HMD 100 according to Embodiment 3 is described. In Embodiment 3, the mode selection process is different from Embodiments 1 and 2, so the following will refer to Figure 8 Describe the mode selection process.
[0094] Figure 8 The processing in the flowchart in Figure 5 The flowchart in FIG. 8 is different only in steps S805 and S806, and therefore, only the processing in steps S805 and S806 will be described.
[0095] In Embodiments 1 and 2, the information processing system 1 selects one of the OTP control mode, the ray control mode, and the tracking control mode to control the HMD 100. On the other hand, in Embodiment 3, the information processing system 1 selects a plurality of control modes as control modes for controlling the HMD 100.
[0096] In step S805, the control unit 211 selects the ray control mode, and at the same time, the OTP control mode is selected just like in step S504. Thus, in step S506, the control of the operation of the HMD 100 using the ray and the control of the operation of the HMD 100 using the OTP are enabled at the same time. For example, if the control of the pointer is enabled using the ray and the OTP at the same time, the pointer is pointed to a rough position by the ray using the left hand, and the position can be finely adjusted by the ring-type controller 120 worn on the finger of the right hand.
[0097] In step S806, the control unit 211 selects the tracking control mode and the OTP control mode at the same time. Thus, in step S506, the operation control of the HMD 100 using hand tracking and the control of the operation of the HMD 100 using OTP are simultaneously enabled.
[0098] Therefore, in Embodiment 3, if the ray control mode or the tracking control mode is selected, the information processing system 1 selects the OTP control mode at the same time. In other words, in step S805, a control mode in which control using rays and control using OTP are simultaneously enabled is selected, and in step S806, a control mode in which control using hand tracking and control using OTP are simultaneously enabled is selected. In Embodiment 3, the OTP control mode is always enabled, so the information processing system 1 can always control the HMD 100 according to the operation on the OTP.
[0099] As an alternative to the ray control mode or the tracking control mode, a control mode for controlling the HMD 100 according to a combination of bent fingers (or unbent fingers) among the five fingers of the hand in the captured image may be used. Furthermore, as an alternative to the ray control mode and the tracking control mode for controlling the HMD 100 according to the state of the hand, a control mode for controlling the HMD 100 according to the state of another specific part of the user may be used. For example, a control mode for controlling the HMD 100 according to the position and movement of the forearm of the user may be used. In the case of using a control mode for controlling the HMD 100 according to the position and movement of the forearm of the user, for example, if the camera unit 202 captures the forearm (if the forearm is captured in the captured image), the control unit 211 selects the control mode. On the other hand, if the camera unit 202 does not capture the forearm, the control unit 211 selects the OTP control mode.
[0100] In addition, a control mode for controlling the HMD 100 according to the number of blinks in a predetermined time may also be adopted. For example, in the case of using the control mode for controlling the HMD 100 according to the number of blinks in a predetermined time, if both eyes of the user are captured by an image capturing unit for capturing eye images (an image capturing unit different from the image capturing unit 202), the control unit 211 selects the control mode. On the other hand, if neither eye of the user is captured by the image capturing unit, the control unit 211 selects the OTP control mode.
[0101] <Example 4>
[0102] The information processing system 1 for selecting (switching) the control mode of the HMD 100 according to Embodiment 4 will be described. The controller 120 here may include a plurality of buttons (not shown). The control mode linked to each button can be selected by the user pressing the button. For example, the controller 120 includes three buttons, and the image processing device 110 selects one of the OTP control mode, the ray control mode, and the tracking control mode according to the pressed button. In Embodiment 4, the mode selection process is different from Embodiments 1 to 3, so the following will refer to the embodiment 4. Fig. 9 Describe the mode selection process.
[0103] Fig. 9 The processing in the flowchart in Figure 6 The flowchart in FIG. 1 is different only in steps S601 to S603. Therefore, only the processing in steps S901 to S903 performed instead of steps S601 to S603 will be described below.
[0104] In step S901, the control unit 211 determines whether the OTP button of the controller 120 is pressed. If it is determined that the OTP button is pressed, the process proceeds to step S503 because the user is likely to use OTP for operation. If it is determined that the OTP button is not pressed, the process proceeds to step S902.
[0105] In step S902, the control unit 211 determines whether a tracking control button (not shown) of the controller 120 is pressed. If it is determined that the tracking control button is pressed, the process proceeds to step S505. If it is determined that the tracking control button is not pressed, the process proceeds to step S903.
[0106] In step S903, the control unit 211 determines whether a ray control button (not shown) of the controller 120 is pressed. If it is determined that the ray control button is pressed, the process proceeds to step S504. If it is determined that the ray control button is not pressed, the process returns to step S901 because it is difficult to select an appropriate control mode.
[0107] In Embodiment 4, after the user selects a control mode by pressing a certain button once, the control unit 211 maintains the selected control mode until another button is pressed. For example, once the OTP button is pressed and the OTP control mode is selected, the OTP control mode is not canceled even if the OTP button is released thereafter. Here, an example of selecting the OTP control mode by pressing the OTP button is described, but the OTP control mode may also be selected by the user's finger touching the OTP.
[0108] In the example described in Embodiment 4, the controller 120 includes buttons associated with each mode, respectively, but the number of buttons for selecting a mode may be one. In this case, each time the button is pressed, the control mode to be selected may be changed. In addition, the controller 120 may include two buttons: a button for changing the control mode in a forward direction (in a specific order) each time the button is pressed; and a button for changing the control mode in a reverse direction each time the button is pressed.
[0109] <Example 5>
[0110] An information processing system 1 for selecting (switching) a control mode of the HMD 100 according to Embodiment 5 will be described. In Embodiment 5, the mode selection process is different from Embodiments 1 to 4, so the following will refer to Fig.10 Describe the mode selection process.
[0111] Fig.10 The processing in the flowchart in Figure 5 The flowchart in FIG. 1 is different only in steps S501 to S504. Therefore, only the processing in steps S1001 to S1005 and S805 performed in place of steps S501 to S504 will be described. The description of the processing in step S805 will be omitted because it is different from the processing in step S805. Figure 8 The processing is the same as in step S805.
[0112] In Embodiment 5, the information processing system 1 selects one of two types as a control mode for controlling the HMD 100: a tracking control mode; and a mode using both the ray control mode and the OTP mode.
[0113] In step S1001, the control unit 211 determines whether the tracking control mode is selected as the control mode at the current time (current point). If it is determined that the tracking control mode is selected, the process proceeds to step S1005. If it is determined that the tracking control mode is not selected, the process proceeds to step S1002.
[0114] In step S1002, assuming that the ray control mode and the OTP control mode are selected, the control unit 211 determines whether the user is currently operating the controller 120. For example, if the OTP is currently operating, the OTP button is pressed, or a button (not shown) of the controller 120 is pressed, the control unit 211 can determine that the controller is currently operating. If it is determined that the controller is currently operating, the process returns to step S1001 because it is not desired to change the control mode (maintaining the current control mode is desired). If it is determined that the controller is not currently being operated, the process proceeds to step S1003 because changing the control mode may not cause a problem.
[0115] In step S1003, the control unit 211 determines whether the amount of change in the posture of the camera unit 202 within a predetermined time is equal to or greater than a predetermined value (whether the posture of the camera unit 202 has changed by a predetermined value or more) to prevent an unexpected change in the control mode due to a change in the user's posture. For this determination, the control unit 211 calculates the amount of change in the posture information acquired from the posture sensor unit 204 within a predetermined time, and compares the amount of change with a predetermined value. For example, it is assumed that A0 is the posture information at a moment before a predetermined time, A1 is the posture information at the current moment, and T is a predetermined value. In this case, A1-A0 is the amount of change in the posture within a predetermined time, and the above determination can be made by comparing the amount of change in the posture with T. The posture information may be angle information in a polar coordinate system centered on the HMD 100, or three-dimensional position information in an orthogonal coordinate system. In the above example, a simple subtraction is used to calculate the amount of change, but the Euclidean distance may also be calculated. If it is determined that the amount of change in the posture within the predetermined time is equal to or greater than the predetermined value, the process returns to step S1001 because the user is likely to have no intention of changing the control mode. If it is determined that the amount of posture change within the predetermined time is smaller than the predetermined value, the process proceeds to step S1004.
[0116] In step S1003, as a method for determining whether the user's posture changes greatly, an example of comparison with a predetermined value is described. However, the control unit 211 may determine whether the user's posture changes greatly by inputting the user's posture information and / or an image that can determine the user's posture into a model that has been pre-trained by machine learning. In other words, in step S1003, any method may be used if a predetermined change in the posture of the user (camera unit 202) can be determined.
[0117] In step S1004, the control unit 211 as a preparatory operation determination unit determines whether a preparatory action for the tracking operation is performed. Fig.11 When the hand shape shown in A is recognized as a preparatory movement for attempting to touch the CG object, the control unit 211 determines that a preparatory movement for a tracking operation has been performed. Fig.11 A is an example of a still image, but the control unit 211 can determine whether a preparatory action is performed through a series of moving images within a predetermined time. For the determination algorithm, an algorithm for determining matching / non-matching with a pre-registered preparatory action can be used, or a model pre-trained by machine learning can be used. The pre-registered preparatory action does not always have to be a natural action of the user, and may also be a predetermined action registered in advance, such as Fig.11 B. However, in this case, the control mode is changed by the preparatory action registered here, so it is necessary to prevent the user from changing the control mode unintentionally.
[0118] Therefore, you should register Fig.11 A can be used to determine the action of attempting to perform a tracking operation, or it should be registered as shown in Fig.11 B is an intentional action that is easily distinguished from other actions (less likely to be misidentified). If it is determined that a preparatory action for the tracking operation is performed, the process proceeds to step S505. If it is determined that a preparatory action for the tracking operation is not performed, the process returns to step S1001.
[0119] In step S1005, the control unit 211 determines whether a preparatory action for a control operation has been performed. Specifically, when the user's finger contacts the OTP contained in the controller 120, when the OTP button is pressed, or when a button other than the OTP (not shown) is pressed, it is determined that a preparatory action for a control operation has been performed. If it is determined that a preparatory action for a controller operation has been performed, the processing proceeds to step S805. If it is determined that a preparatory action for a controller operation has not been performed, the processing returns to step S1001. In the example described here, the control unit 211 determines whether a preparatory action for a controller operation has been performed, but it may also be determined whether a preparatory action for a ray operation has been performed.
[0120] Although the left hand and the right hand are not distinguished in the above description, the control unit 211 can serve as a left-right determination unit to determine whether the hand is the left hand or the right hand. Then, using the determination result, the control unit 211 can independently select (set) a control mode for each of the left hand and the right hand. In order to determine the left hand or the right hand, the left hand and the right hand can be registered in advance, and the left hand or the right hand can be determined using image recognition. In addition, in order to determine the left hand or the right hand, the controller can also register information on which of the left hand and the right hand the controller is worn on in advance, and the left hand or the right hand can be determined based on the information.
[0121] In this case, the control unit 211 determines whether to process the left hand or the right hand before step S1001. For example, the control unit 211 may first process the dominant hand of the user. Fig.10The processing content in each step in the flowchart in is interpreted as processing on the determined processing target hand. The control unit 211 determines whether the processing of both the left hand and the right hand is completed after step S506. If the processing is not completed, the processing returns to step S1001. If the processing is completed, the processing proceeds to step S507. In the case of returning to step S1001, the control unit 211 switches the processing target to the unprocessed hand of the left hand and the right hand. This prevents the processing in step S1001 from being repeated only on one of the left hand and the right hand. Needless to say, the control unit 211 can perform the processing of the left hand and the processing of the right hand in parallel by time division. In this case, the control unit 211 processes the left hand and the right hand independently. Fig.10 The process in .
[0122] In the above description, the phrase "if A is equal to or greater than B, the processing proceeds to step S1, and if A is less than (lower than) B, the processing proceeds to step S2" can be interpreted as "if A is greater than (higher than) B, the processing proceeds to step S1, and if A is equal to or less than B, the processing proceeds to step S2". In addition, "if A is greater than (higher than) B, the processing proceeds to step S1, and if A is equal to or less than B, the processing proceeds to step S2" can be interpreted as "if A is equal to or greater than B, the processing proceeds to step S1, and if A is less than (lower than) B, the processing proceeds to step S2". In other words, as long as no inconsistency arises, "equal to or greater than A" can be interpreted as "equal to or greater than (higher than; longer than; more than) A", or can be interpreted as "greater than (higher than; longer than; more than) A". In addition, "equal to or less than A" can be interpreted as "A or less than (lower than; shorter than; less than) A", or can be interpreted as "less than (lower than; shorter than; less than) A". In addition, “greater than (higher than; longer than; more than) A” can be interpreted as “equal to or greater than A”, and “less than (lower than; shorter than; less than) A” can be interpreted as “equal to or less than A”.
[0123] Although the present invention has been described with reference to its preferred embodiments, the present invention is not limited to these specific embodiments but includes various forms within the scope not departing from the spirit of the present invention. The above-mentioned embodiments may be partially combined with each other as required.
[0124] Each functional unit of each of the above-mentioned embodiments (each modification) may or may not be independent hardware. The functions of two or more functional units may be implemented by general hardware. Each of the multiple functions of a functional unit may be implemented by independent hardware respectively. Two or more functions of a functional unit may be implemented by general hardware. Each functional unit may or may not be implemented by hardware such as ASIC, FPGA and DSP. For example, the device may include a processor and a memory (storage medium) storing a control program, and the functions of at least a part of the functional units included in the device may be implemented by the processor reading the control program from the memory and executing the program.
[0125] (Other embodiments)
[0126] The present invention can be implemented by providing a program that implements one or more functions of the above-mentioned embodiments to a system or device via a network or storage medium, and one or more processors of a computer in the system or device read and execute the processing of the program. The present invention can also be implemented by a circuit (such as an ASIC) that implements one or more functions.
[0127] The present invention is not limited to the above-described embodiments, but may be changed and modified in various ways without departing from the spirit and scope of the present invention. Therefore, the following claims are attached to disclose the scope of the present invention.
[0128] This application claims the benefit of Japanese Patent Application No. 2022-049042 filed on March 24, 2022 and Japanese Patent Application No. 2022-158875 filed on September 30, 2022, which are hereby incorporated by reference herein in their entirety.
[0129] [Description of Reference Numerals]
[0130] 1 Information Processing System
[0131] 100 HMD
[0132] 110 Image processing device
[0133] 120 Controller
[0134] 202 Camera Unit
[0135] 211 Control Unit
Claims
1. A control device for controlling an electronic device, comprising: a selection unit configured to select any one of a first control mode and a second control mode as a control mode for controlling the electronic device; as well as A control unit configured to: 1) control the electronic device according to the state of a specific part of the user determined based on the imaging of the imaging unit when the selection unit selects the first control mode, and 2) control the electronic device according to the user's operation involving physical contact with a specific operating component when the selection unit selects the second control mode.
2. The control device according to claim 1, wherein: The selection unit selects one of the first control mode and the second control mode as the control mode based on the state of the specific part of the user.
3. The control device according to claim 1, wherein: The selection unit selects any one of the first control mode and the second control mode as the control mode based on an operation on an operation member.
4. The control device according to claim 2, further comprising: a determining unit configured to determine whether the specific part is included in a predetermined range, Here, when it is determined that the specific part is included in the predetermined range, the selection unit selects the first control mode, and when it is determined that the specific part is not included in the predetermined range, the selection unit selects the second control mode.
5. The control device according to claim 4, wherein: The predetermined range is the imaging range of the imaging unit.
6. The control device according to any one of claims 1 to 5, wherein: When the selection unit selects the first control mode, the control unit controls the electronic device according to the state of the specific part determined based on the imaging, and also controls the electronic device according to an operation involving physical contact of the user with the specific operating component.
7. The control device according to claim 2, wherein: The selection unit selects the second control mode when the user is in contact with the specific operation member, and selects the first control mode when the user is not in contact with the specific operation member.
8. The control device according to any one of claims 1 to 7, wherein: The specific part is a hand.
9. The control device according to claim 8, wherein: The first control mode is any one of the third control mode and the fourth control mode, and The control unit: 1) controls the electronic device according to the position on the extension line of the direction pointed by the user's hand when the selection unit selects the third control mode, and 2) controls the electronic device according to the position of the user's hand when the selection unit selects the fourth control mode.
10. The control device according to claim 9, wherein: When the first control mode is selected, In a case where the distance between the position of the hand of the user and the imaging unit is longer than a first threshold, the selection unit selects the third control mode as the first control mode, and The selection unit selects the fourth control mode as the first control mode when the distance between the position of the hand of the user and the imaging unit is shorter than the first threshold.
11. The control device according to claim 10, wherein: The first threshold is a value based on the arm length of the user.
12. The control device according to claim 9, wherein: When the first control mode is selected, In a case where the angle between the direction pointed by the hand of the user and the horizontal direction is smaller than a second threshold, the selection unit selects the third control mode as the first control mode, and In a case where an angle between the direction pointed by the hand of the user and the horizontal direction is greater than the second threshold, the selection unit selects the fourth control mode as the first control mode.
13. The control device according to any one of claims 1 to 12, wherein: The specific operating member is any one of an optical touch pad, a touch pad, a touch panel, a cross key, a joystick, and a touch pad device.
14. The control device according to any one of claims 1 to 13, wherein: The specific operating components are included in the controller, and The controller can be held by the user's hand or can be worn on the user's hand.
15. The control device according to claim 14, wherein: The controller can be worn on a finger of the user.
16. The control device according to any one of claims 1 to 15, further comprising a left-right determination unit configured to determine the left hand or the right hand of the user, in, The selection unit can select a control mode for each of the left hand and the right hand of the user independently using the determination result of the left and right determination unit.
17. The control device according to any one of claims 1 to 16, wherein: The selection unit maintains a current control mode when a predetermined change in the posture of the imaging unit is detected.
18. The control device according to any one of claims 1 to 17, further comprising a preparatory action determination unit configured to determine whether the state of the user's hand is a state of a preparatory action for performing an operation, in, The selection unit selects a control mode based on a determination result of the preparatory movement determination unit.
19. A method for controlling an electronic device, comprising: A selection step for selecting either a first control mode or a second control mode as a control mode for controlling the electronic device; as well as, A control step for 1) controlling the electronic device according to the state of a specific part of the user determined based on the image captured by the image capturing unit when the first control mode is selected in the selection step; and 2) in a case where the second control mode is selected in the selecting step, controlling the electronic device according to the user's operation involving physical contact with a specific operating member.
20. A program for causing a computer to function as each unit of the control device according to any one of claims 1 to 18.
Citation Information
Patent Citations
Sheet material carrying device
JP2022049042A
Payment control device, payment control system, payment control method, and application program
JP2022158875A
