Information processing device, information processing method, and program

JP2024025272A5Active Publication Date: 2025-08-14CANON KK

View PDF 0 Cites 0 Cited by

Patent Information

Application Number: JP2022128583
Authority / Receiving Office: JP · JP
Patent Type: Applications
Current Assignee / Owner
Filing Date: 2022-08-12
Publication Date: 2025-08-14
Estimated Expiration: 2042-08-12

AI Technical Summary

Technical Problem

Existing technologies do not allow for convenient control of multiple display items indicating positions, such as a mouse cursor and selection frames, in mixed reality and virtual reality environments.

Method used

An information processing device and method that switches between two control modes: the ray control mode, which uses inertial information of a controller to display a ray indicating a position, and the OTP control mode, which uses operations on a specific operation member to display a mouse cursor or other items, allowing seamless transitions based on the controller's position and user interactions.

Benefits of technology

Enables convenient control of multiple display items by ensuring precise positioning and reducing the likelihood of losing sight of the cursor or selection frames, enhancing user experience in mixed and virtual reality applications.

✦ Generated by Eureka AI based on patent content.

Smart Images

Figure 00000000_0000_ABST

Patent Text Reader

Abstract

To provide a technology for controlling a plurality of display items having high convenience to a user and showing a position.SOLUTION: An information processing device includes first display control means for controlling a first display item showing a position based on inertial information of a controller so as to be displayed in a display area of display means in a first control mode, second display control means for controlling a second display item showing a position corresponding to a first operation to a specific operation member in a first area in the display area to be displayed in the display area in a second control mode, and switching means for performing switching from the first control mode to the second control mode when a specific operation is performed in the case that the position showing the first display item is included in a second area in the display area in the first control mode.SELECTED DRAWING: Figure 3

Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] Mixed reality (MR) technology and virtual reality (VR) technology are known that use a head mounted display (HMD) to make a user feel a space different from the real space. In such technology, it is being considered that a user can perform various controls on a head mounted display (HMD) while wearing the HMD. Patent Document 1 discloses an information processing device that recognizes an input device and an operating medium (user's hand) that operates the input device, and switches applications depending on the recognition result.

[0003] Meanwhile, technology is being considered that improves user operability by controlling a display item that indicates a position. Patent Document 2 discloses a technology that improves operability by moving the mouse cursor to a specified position when a dialog window is displayed, regardless of the position of the mouse cursor (display item) that was displayed immediately before. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2017-059062 A [Patent Document 2] Japanese Patent Application Publication No. 8-152972 Summary of the Invention [Problem to be solved by the invention]

[0005] In addition, from the viewpoint of convenience, a user may wish to switch between multiple display items (e.g., a mouse cursor and a selection frame) that indicate a position. However, the above-mentioned technology does not assume the use of multiple display items, and therefore it is not possible to control multiple display items that indicate a position in a highly convenient manner for a user.

[0006] Therefore, an object of the present invention is to provide a technique for controlling a plurality of display items indicating a position, which is highly convenient for a user. [Means for solving the problem]

[0007] One aspect of the present invention is a method for producing a composition comprising the steps of: a first display control means for controlling, in a first control mode, a first display item indicating a position based on inertia information of the controller to be displayed in a display area of the display means; a second display control means for controlling, in a second control mode, to display in the display area a second display item indicating a position in a first area of the display area corresponding to a first operation performed on a specific operating member; a switching means for switching from the first control mode to the second control mode when a specific operation is performed in a case where a position indicated by the first display item is included in a second area of the display area in the first control mode; The information processing device is characterized by having:

[0008] One aspect of the present invention is a method for producing a composition comprising the steps of: a first display control step of controlling, in a first control mode, a first display item indicating a position based on inertia information of the controller to be displayed in a display area of the display means; a second display control step of controlling, in a second control mode, to display in the display area a second display item indicating a position in a first area of the display area corresponding to a first operation performed on a specific operating member; a switching step of switching from the first control mode to the second control mode when a specific operation is performed in a case where a position indicated by the first display item is included in a second area of the display area in the first control mode; The information processing method is characterized by having the following features. Effect of the Invention

[0009] According to the present invention, it is possible to control a plurality of display items indicating a position in a highly convenient manner for a user. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating an information processing system according to a first embodiment. [Diagram 2] 1 is a diagram showing the internal configuration of an HMD and the like according to a first embodiment. [Diagram 3] 4 is a flowchart of a mode switching process according to the first embodiment. [Figure 4] FIG. 2 is a diagram for explaining a display in a ray control mode according to the first embodiment. [Diagram 5] FIG. 4 is a diagram for explaining a display in an OTP control mode according to the first embodiment. [Figure 6] 4 is a flowchart of a mode switching process according to the first embodiment. [Figure 7] FIG. 13 is a diagram illustrating a display in a ray control mode according to the first modified example. [Figure 8] FIG. 13 is a diagram for explaining a display in an OTP control mode according to the first modified example. [Figure 9] FIG. 11 is a diagram illustrating a display in a ray control mode according to Modification 2. [Figure 10] FIG. 11 is a diagram for explaining a display in an OTP control mode according to Modification 2. [Figure 11] FIG. 13 is a diagram for explaining a display in an OTP control mode according to Modification 3. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings, taking an HMD as an example. Note that an HMD has a display range of, for example, 360 degrees of field of view. Therefore, a user can feel that the display range of an HMD is much larger than that of a conventional PC display. Therefore, while an HMD has the advantage of being able to display multiple items at the same time, a display item (UI) that indicates (specifies) a position, such as a mouse cursor or pointer, is easily lost if it moves out of the user's line of sight.

[0012] <Embodiment 1> An information processing system 1 according to the first embodiment will be described with reference to Fig. 1. The information processing system 1 includes an HMD 100, an image processing device 110, and a controller 120.

[0013] The HMD 100 is a head-mounted display device (electronic device) that can be worn on the head of a user. The HMD 100 displays a composite image that combines a captured image of the area in front of the user with content such as CG (computer graphics) in a form that corresponds to the posture of the HMD 100.

[0014] The image processing device 110 is a control device (information processing device; electronic device) that controls the HMD 100. The image processing device 110 is, for example, a smartphone, a tablet terminal, or a PC (personal computer). The image processing device 110 is connected to the HMD 100 wirelessly or by wire. The image processing device 110 generates a composite image by combining a captured image and CG, and transmits the composite image to the HMD 100. Note that each component of the image processing device 110 may be included in the HMD 100.

[0015] The controller 120 is a device for performing various controls of the HMD 100. If the device 110 is in a specific control mode, when a user operates the controller 120, the HMD 100 is controlled in response to the user's operation. The controller 120 is, for example, ring-shaped so that it can be worn on a user's finger, as shown in FIG. 1. If the controller 120 can be worn on a user's finger, the user can move his or her hand freely while holding the controller 120. The controller 120 also has a button with a built-in optical track pad (hereinafter referred to as "OTP") that can detect the amount of planar movement. The controller 120 communicates wirelessly with the image processing device 110 by Bluetooth (registered trademark).

[0016] For example, the user presses and holds the button of the OTP to display a menu including a pointer on the HMD 100. After that, the user can place a finger on the OTP and rub it in any direction to move the pointer to a desired item. Then, the user can perform a decision action to decide the selection of the item by pressing the button of the OTP. Note that the shape of the controller 120 is a ring type, but is not limited to this. For example, the shape of the controller 120 may be a shape that can be worn on the hand, such as a glove type. In this way, the controller 120 may be in a form that can be held by the user's hand or worn on the hand, so that the user can easily use it.

[0017] (About the internal structure of the HMD) The internal configuration of the HMD 100 will be described with reference to Fig. 2. The HMD 100 includes an HMD control unit 201, an imaging unit 202, an image display unit 203, and a posture sensor unit 204.

[0018] The HMD control unit 201 controls each component of the HMD 100. When the HMD control unit 201 acquires a composite image (an image obtained by combining a captured image of the space in front of the user captured by the imaging unit 202 with CG) from the image processing device 110, the HMD control unit 201 displays the composite image on the image display unit 203. Therefore, by wearing the HMD 100, the user can view the composite image displayed on the image display unit 203. The user can experience various mixed realities, such as CG blended into real space.

[0019] The imaging unit 202 includes two cameras (imaging devices). The two cameras are disposed near the positions of the left and right eyes of the user when wearing the HMD 100 in order to capture an image of a space similar to the space the user normally sees. Images (captured images) captured by the two cameras of a subject (the range in front of the user) are output to the image processing device 110. In addition, the two cameras in the imaging unit 202 can acquire information on the distance from the two cameras to the subject as distance information by measuring distances with a stereo camera.

[0020] The image display unit 203 displays the composite image. The image display unit 203 has, for example, a liquid crystal panel or an organic EL panel. When the user is wearing the HMD 100, an organic EL panel is disposed in front of each eye of the user.

[0021] It is also possible to use a device using a semi-transmissive half mirror for the image display unit 203. In this case, for example, the image display unit 203 may display an image by using a technology generally called AR so that CG appears to be directly superimposed on the real space seen through the half mirror. Also, the image display unit 203 may display an image of a completely virtual space by using a technology generally called VR without using a captured image.

[0022] The attitude sensor unit 204 detects the attitude and position of the HMD 100. The attitude sensor unit 204 has an inertial measurement unit (IMU) that acquires inertial information. The attitude sensor unit 204 detects the attitude and position of the HMD 100. The information is output to the image processing device 110.

[0023] (Internal configuration of image processing device) The internal configuration of the image processing device 110 will be described with reference to Fig. 2. The image processing device 110 includes a control unit 211, a content DB 212, and a communication unit 213.

[0024] The control unit 211 (display control unit) receives an image (captured image) acquired by the imaging unit 202 and orientation information acquired by the orientation sensor unit 204 from the HMD 100. The control unit 211 performs image processing on the captured image to cancel aberrations in the optical system of the imaging unit 202 and the optical system of the image display unit 203. Then, the control unit 211 composites the captured image with any CG to generate a composite image. The control unit 211 transmits the composite image to the HMD control unit 201 in the HMD 100.

[0025] The control unit 211 controls the position, orientation, and size of the CG in the composite image based on the information (distance information and orientation information) acquired by the HMD 100. For example, when placing a virtual object represented by the CG near a specific object existing in real space in the space represented by the composite image, the control unit 211 makes the virtual object (CG) larger as the distance between the specific object and the imaging unit 202 becomes closer. By controlling the position, orientation, and size of the CG in this manner, the control unit 211 can generate a composite image in which a CG object not placed in real space appears as if it were placed in real space.

[0026] The content DB 212 is a storage unit that stores information such as CG, etc. The control unit 211 can switch the CG read from the content DB 212 (that is, the CG used to generate a composite image).

[0027] (Internal structure of the controller) The internal configuration of controller 120 will be described with reference to Fig. 2. Controller 120 has controller control section 221, operation section 222, communication section 223, and attitude sensor section 224. Here, it is not necessary to mount a plurality of light emitting diodes (such as large sensors) on controller 120. Therefore, controller 120 can be made compact.

[0028] The controller control unit 221 controls each component of the controller 120 .

[0029] The operation unit 222 includes a button with an OTP (specific operation member) built in. Information (operation information) regarding pressing or sliding a finger on the OTP on the operation unit 222 is transmitted to the image processing device 110 via the communication unit 223.

[0030] For example, the user can move the pointer displayed on the HMD 100 to a specified position by sliding a finger on the OTP. Also, the user can instruct the HMD 100 and the image processing device 110 to perform a specific process by pressing a button on the OTP. In this way, the user can control the HMD 100 by a combination of sliding a finger on the OTP and pressing a button. More specifically, the user can control the HMD 100 to display a menu by pressing and holding the OTP button, for example, and select a desired location by aligning the pointer with the desired location and pressing the OTP button.

[0031] The operation unit 222 may include any operation member instead of the OTP as long as the user can perform operations by physical contact. For example, the operation unit 222 may include any of a touch pad, a touch panel, a cross key, a joystick, and a track pad device instead of the OTP.

[0032] The communication unit 223 performs wireless communication with the image processing device 110 (the communication unit 213).

[0033] The attitude sensor unit 224 has an IMU (inertial measurement unit; inertial sensor) that can detect inertial information (spatial movement and rotational movement) of the controller 120 as the user's movement. Note that the attitude sensor unit 224 may be a device that does not impede the miniaturization of the controller 120 and can detect information on spatial movement (positional displacement, velocity, acceleration) and / or rotational movement (amount of rotation, rotational velocity, angular velocity) as the inertial information.

[0034] (For each control mode) Various processes according to the control modes according to the first embodiment will be described with reference to FIGS.

[0035] 4 is a diagram for explaining a display in the ray control mode (first control mode) according to the first embodiment. The ray control mode is a control mode for controlling a ray (a virtual light beam) according to inertial information (at least one of a position and an attitude) of the controller 120. Here, the ray is a display item that indicates a position designated by the user in the display area of the HMD 100.

[0036] 4 is a schematic diagram showing a display area (composite image) in the image display unit 203 of the HMD 100. The HMD 100 displays the scenery around the HMD 100 (scenery captured by the imaging unit 202). The background includes the user's hand 402, as shown in FIG. 4. The HMD 100 can composite a screen similar to a desktop screen of a general PC with a background showing the scenery. This screen is called a "wide area desktop."

[0037] By displaying the "wide area desktop" on the HMD 100, the user can perform various tasks using the HMD 100 without using a large PC monitor connected to the image processing device 110 (such as a PC). In addition, since the display area of the HMD 100 is expanded to fill the user's field of vision, it is possible to display images with a resolution that is difficult to achieve on a real PC monitor, and to display multiple images side by side at the same time.

[0038] In the following description, it is assumed that a "wide area desktop" is displayed in the display area of the HMD 100, and that the user is performing multiple tasks at the same time. In addition, in the HMD 100, multiple windows 401 are displayed as if floating in space. Specifically, as shown in FIG. 4, a window 401A, a window 401B, and a window 401C are displayed side by side. For example, the user can operate an Internet browser in the window 401A and display search results. The user can create a document in the window 401B. The user can play a video or play music by operating a music player in the window 401C.

[0039] The controller 120 is attached to a finger (index finger) of the user's hand 402. A ray 403 (light beam) extending from the index finger in the direction pointed by the index finger is displayed as CG. When the controller 120 moves, the attitude sensor unit 224 acquires new inertial information, and the inertial information is sent from the controller control unit 221 to the control unit 211. Then, the control unit 211 controls the display of the ray based on the inertial information.

[0040] FIG. 5 is a diagram for explaining a display in the OTP control mode (second control mode) according to the first embodiment. The OTP control mode is a mode in which operations for OTP (hereinafter referred to as "OTP operations") are performed. This is a control mode in which the mouse cursor is controlled according to the user's input (e.g., the position of the mouse cursor). Here, the mouse cursor is a display item that indicates a position or the like designated by the user in a specific area (the area of each window) of the display area of the HMD 100.

[0041] FIG. 5 shows the display area (composite image) in the image display unit 203, similar to FIG. 4. A "wide area desktop" is displayed on the HMD 100. Here, an example will be described in which the user precisely controls the position of the mouse cursor 501 in the window 401C. In the window 401C, a music player is displayed, and the user can play music. The user selects a button icon in the music player to play or fast forward a song. The user can also select a song they want to listen to by selecting a song from a song list.

[0042] When music is played using window 401C while window 401A and window 401B are displayed, window 401C is displayed in a part of the display area of image display unit 203. In addition, the area in which button icons and selection items are displayed is further limited within the area of window 401C. For this reason, when moving mouse cursor 501 in the area in which button icons and selection items are displayed, the user needs to specify the position in detail. Therefore, mouse cursor 501 moves two-dimensionally within window 401C in response to, for example, an operation of tracing the surface of the OTP of controller 120 (hereinafter referred to as a "slide operation"). This allows the user to specify the position of mouse cursor 501 in detail.

[0043] The ray is controlled by changing the position or attitude of the controller 120. Therefore, when the position of the tip of the ray is moved significantly, there is a high possibility that the ray can be moved as intended by the user, but when the position of the tip of the ray is moved only a small distance, the position of the tip of the ray may deviate from the desired position. Furthermore, when the position of the tip of the ray is close to the hand, the position can be controlled finely to a certain extent, but when the length of the ray is long and a window is displayed at a position far from the hand, it is difficult to control the position finely. Therefore, when specifying a position finely within the window 401C, it is easier for the user to specify the position in the OTP control mode than in the ray control mode.

[0044] (Regarding mode switching process) The process of switching from the ray control mode to the OTP control mode will be described with reference to the flowchart of Fig. 3. At the start of the flowchart of Fig. 3, it is assumed that a composite image as shown in Fig. 4 is displayed. Also, at the start of the flowchart of Fig. 3, it is assumed that the control mode is set to the ray control mode, and a ray is displayed in the display area of the image display unit 203.

[0045] In step S1001, the control unit 211 acquires inertial information (information on the attitude and position of the controller 120) detected by the attitude sensor unit 224 (IMU) of the controller 120.

[0046] In step S1002, the control unit 211 moves the ray based on the inertial information. Specifically, the control unit 211 determines the position of the tip pointed to by the index finger of the user based on the inertial information, and moves the position of the tip of the ray to the position pointed to by the index finger of the user.

[0047] In step S1003, the control unit 211 determines whether or not the user has performed an OTP operation. If it is determined that the OTP operation has been performed, the process proceeds to step S1004. If it is determined that the OTP operation has not been performed, the process returns to step S1001.

[0048] The OTP operation may be not only the sliding operation of the OTP, but also a tapping operation such as tapping the surface of the OTP, an operation of pressing a push button that is integrated with the OTP, etc. In other words, the OTP operation may be any operation using the OTP. Note that if an operation for moving the mouse cursor (i.e., a sliding operation, etc.) is performed as an OTP operation in the OTP control mode, it is particularly convenient because there is continuity of the operation from the ray control mode.

[0049] In step S1004, control unit 211 determines whether or not the tip of the ray is located in the switching region. In the example of Fig. 4, the switching region is windows 401A to 401C in the display region of image display unit 203. If it is determined that the tip of the ray is located in the switching region, the process proceeds to step S1005. If it is determined that the tip of the ray is not located in the switching region, the process proceeds to step S1006.

[0050] In step S1005, control unit 211 switches the control mode to the OTP control mode. Then, control unit 211 displays the mouse cursor at the position of the tip of the ray and hides the ray. This prevents the user from losing sight of the mouse cursor when switching the control mode, since the position indicated by the display item that specifies the position (the tip of the ray and the mouse cursor) does not change.

[0051] Note that the control unit 211 may display the mouse cursor at a predetermined position (predetermined position) instead of displaying the mouse cursor at the tip of the ray. For example, when a music player is started, the control unit 211 displays the mouse cursor on the play button, and when the next operation is clear, displays the mouse cursor at a position according to the next operation. This may improve the user's operability. Also, when resuming document creation and inputting characters from the previous position, it may be more convenient to display the mouse cursor at the same position as the previous time, rather than at the tip of the ray. Note that the optimal position of the mouse cursor differs depending on the content of the operation performed by the user and the situation. The control unit 211 may display the mouse cursor at a different position depending on the situation.

[0052] In step S1006, the control unit 211 ignores the OTP operation and continues the ray control mode (does not switch the control mode). By performing the process of step S1006, it is possible to prevent unnecessary switching of the control mode in cases where the user unintentionally touches the OTP.

[0053] Also, in the flowchart of FIG. 3, when the tip of the ray is located in the switching area and an OTP operation is performed, the ray control mode is switched to the OTP control mode. However, when the tip of the ray is located in the switching area, after an operation for intentionally selecting the switching area (window) is performed, if an OTP operation different from this operation is further performed, the ray control mode may be switched to the OTP control mode. By adding an operation for intentionally selecting the switching area, the possibility of switching to the OTP control mode against the user's intention can be reduced. The "operation for intentionally selecting the switching area" may be an operation for shaking the controller 120 in the ray control mode (an operation for making the controller 120 perform a tapping motion) or an operation for touching the OTP. Also, the "operation for intentionally selecting the switching area" may be an operation different from the operation for controlling the ray and the OTP operation.

[0054] Next, the process of switching from the OTP control mode to the ray control mode will be described with reference to the flowchart in Fig. 6. The mouse cursor basically points only to a position within one window 401. For this reason, even if the user wants to select another window 401A when the mouse cursor is displayed on window 401C in the OTP control mode, the mouse cursor will not reach window 401A. Therefore, the user must first It is necessary to select window 401A after switching the display item from the mouse cursor to a ray. In addition, since there is a high possibility that the distance between each of the windows 401 is relatively long, there is also an advantage that it is more convenient for the user to specify the position using a ray. Note that the display areas of windows 401A to 401C are displayed on a two-dimensional plane in a three-dimensional space, and therefore the depth is fixed.

[0055] In step S2001, the control unit 211 judges whether or not information on a slide operation for an OTP has been acquired. If it is judged that information on a slide operation has been acquired, in step S2002, the control unit 211 moves the mouse cursor according to the information on the slide operation. If it is judged that information on a slide operation has not been acquired, the process proceeds to step S2003.

[0056] In step S2003, the control unit 211 determines whether or not an operation (switching operation) for switching the control mode from the OTP control mode to the ray control mode has been performed. If it is determined that a switching operation has been performed, the process proceeds to step S2004. If it is determined that a switching operation has not been performed, the process proceeds to step S2005.

[0057] The switching operation is, for example, an operation of moving the controller 120 or the user's hand (changing the position or posture of the controller 120 or the user's hand) so as to satisfy a specific condition. The switching operation is, for example, an operation of quickly and vigorously changing the direction of the controller 120 as if twisting the wrist, or an operation of continuously shaking the controller 120. Alternatively, the switching operation may be a hand gesture that changes the hand into a specific shape. In other words, the switching operation may be an operation different from the OTP operation (such as a slide operation) and is not performed in the OTP control mode.

[0058] In step S2004, control unit 211 switches the control mode to the ray control mode and displays the ray so that the tip of the ray is located at the position indicated by the mouse cursor. Then, control unit 211 makes the mouse cursor invisible. In step S2005, control unit 211 ignores the switching operation and continues the OTP control mode (does not switch the control mode).

[0059] As described above, according to the first embodiment, in the ray control mode, when an OTP operation is performed while the tip of the ray is located inside the window (switching area), the ray control mode is switched to the OTP control mode, which is a control mode in which a position is specified by the OTP operation. Then, the user can control the mouse cursor by the OTP operation inside the window to specify a fine position. Therefore, according to the first embodiment, it is highly convenient for the user, and multiple display items (ray and mouse cursor) indicating the position can be controlled.

[0060] In the above description, the orientation sensor unit 224 has an example including an IMU, but it is sufficient if it can acquire information for specifying a position (such as information on a specific position or orientation). For example, the orientation sensor unit 224 may include a camera (a camera mounted on the HMD 100) that determines the origin position and an acceleration sensor. Alternatively, the orientation sensor unit 224 may include only a camera, and detect the orientation of the controller 120 by the camera.

[0061] The switching area may be a specific area (menu area, OK button area, or pop-up area) in window 401. Also, instead of the ray, a pointer displayed at a position corresponding to the tip of the ray may be used.

[0062] <Variation 1> Hereinafter, the HMD 100 according to the first modification will be described with reference to Fig. 7 and Fig. 8. In the first modification, an example will be described in which a thick frame indicating a selected icon is used instead of a mouse cursor.

[0063] 7 shows the display area of the HMD 100 in the ray control mode. In the display area, a window 401A, a window 401B, and a window 401C are displayed. In the first modification, the switching area is also the windows 401A to 401C. In the area of the window 401C, a plurality of icons 701 are arranged in a row, as on the screen of a smartphone or tablet. When one icon 701 is selected from the plurality of icons 701, the application indicated by the selected icon 701 is started. Note that the window 401C may be a menu screen (a screen on which functions can be selected or set) on which text information is displayed in a row instead of the icons 701.

[0064] Here, as shown in FIG. 7, when an OTP operation is performed after the tip of the ray is moved to be positioned in window 401C, the ray control mode is switched to the OTP control mode, and a composite image as shown in FIG. 8 is displayed.

[0065] 8 shows a state in which the icon 701 located at the tip of the ray is highlighted by a thick frame 702 (display item). Instead of being surrounded by the thick frame 702, the icon 701 may be highlighted (hovered) by being displayed as if it is floating forward (or may be displayed so as to be distinguishable from other icons 701). The interval between the icons 701 is narrow, and it may be difficult to accurately select one icon 701 by the ray. Therefore, if the tip of the ray is located in the window 401C, the control unit 211 switches the control mode to the OTP control mode in response to the OTP operation. Then, the control unit 211 highlights the icon 701 by displaying the thick frame 702.

[0066] Furthermore, in the OTP control mode, the control unit 211 moves the thick frame 702 indicating the icon 701 to be selected to the adjacent icon 701 every time an OTP operation is performed (for example, every time a specific OTP button is pressed). This allows the user to accurately select the intended icon 701 even if the size of the icon 701 is small. Even if the size of the icon 701 is large, the user can move the thick frame 702 to the adjacent icon 701 with one operation, so that the thick frame 702 can be moved to the intended icon 701 in one go.

[0067] <Variation 2> Hereinafter, an HMD 100 according to Modification 2 will be described with reference to Fig. 9 and Fig. 10. In Modification 2, an example will be described in which CG is displayed instead of a window and an icon is used instead of a mouse cursor.

[0068] Fig. 9 shows the display area of the HMD 100 (image display unit 203) in the ray control mode. Three rectangular parallelepiped CG801 are displayed in the display area of the image display unit 203. Here, in the second modification, the switching area is a "CG" instead of the "window" according to the first embodiment. In Fig. 9, one of the rectangular parallelepiped CG801 is shown by a ray. Then, when an OTP operation is performed in this state, the control mode is switched to the OTP mode, and a composite image as shown in Fig. 10 is displayed.

[0069] In the OTP control mode, the user can enlarge or reduce the CG 801 by performing an OTP operation. For example, the control unit 211 enlarges the CG 801 when a rightward sliding operation is performed, and reduces the CG 801 when a leftward sliding operation is performed. do.

[0070] According to this, it is possible to switch the control mode and enlarge the CG by the OTP operation, and since the CG801 is shown by a ray, it is possible to continuously control the size of the CG801. Also, since the size of the CG801 located at the tip of the ray changes, the user does not need to move his / her line of sight and does not lose sight of the target of the operation.

[0071] In the OTP control mode, the control unit 211 displays an arrow-shaped icon 802 indicating the size of the CG 801. However, the control unit 211 may handle the edge line of the CG 801 (the vertical line indicated by the icon 802) or the side of the CG 801 as a display item (a display item that specifies a position) without displaying the arrow-shaped icon 802. In this case, the control unit 211 may display the edge line of the CG 801 in a display form different from that of the other lines. For example, the control unit 211 may cause the edge line of the CG 801 to blink or to be displayed in a color different from that of the other lines.

[0072] <Modification 3> Hereinafter, the HMD 100 according to the modified example 3 will be described with reference to Fig. 11. In the ray control mode, the display area of the HMD 100 displays a ray and CG801 as shown in Fig. 9, similar to the modified example 2. When an OTP operation is performed in a state in which the tip of the ray is located on CG801 as shown in Fig. 9, the control mode switches to the OTP control mode, and a composite image as shown in Fig. 11 is displayed.

[0073] Fig. 11 shows the display area of the HMD 100 in the OTP control mode. In Fig. 11, the color and pattern of the surface of a CG 801 are controlled in response to an OTP operation. A color palette 901 showing candidate patterns and colors is displayed adjacent to the CG 801. When a pattern or color is selected by an OTP operation, the control unit 211 reflects the selection in the display of the CG 801. In the third modification, when an OTP operation is performed, the control unit 211 moves a square frame 902 showing the pattern to be selected in the color palette 901 (the area where the color palette 901 is displayed) instead of the mouse cursor.

[0074] Also, in the above, "If A is equal to or greater than B, proceed to step S1, and if A is smaller (lower) than B, proceed to step S2" may be read as "If A is greater (higher) than B, proceed to step S1, and if A is equal to or less than B, proceed to step S2." Conversely, "If A is greater (higher) than B, proceed to step S1, and if A is equal to or less than B, proceed to step S2" may be read as "If A is greater (higher) than B, proceed to step S1, and if A is smaller (lower) than B, proceed to step S2." For this reason, unless a contradiction occurs, the expression "equal to or greater than A" may be read as "A or greater (high; long; many)," or may be read as "greater than A (high; long; many)." On the other hand, the expression "equal to or less than A" may be read as "A or smaller (low; short; few)," or may be read as "smaller than A (low; short; few)." Furthermore, "bigger (higher; longer; more) than A" may be read as "A or greater," and "smaller (lower; shorter; fewer) than A" may be read as "A or less."

[0075] Although the present invention has been described in detail based on the preferred embodiments, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Parts of the above-described embodiments may be combined as appropriate.

[0076] Each functional unit in each of the above embodiments (variations) may or may not be an individual piece of hardware. Each of the functions of one functional unit may be realized by individual hardware. Two or more functions of one functional unit may be realized by common hardware. Furthermore, each functional unit may or may not be realized by hardware such as an ASIC, FPGA, or DSP. For example, the device may have a processor and a memory (storage medium) in which a control program is stored. Then, the functions of at least some of the functional units of the device may be realized by the processor reading and executing the control program from the memory.

[0077] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) for implementing one or more of the functions.

[0078] The disclosure of the above embodiments includes the following configurations, methods, and programs. [Configuration 1] a first display control means for controlling, in a first control mode, a first display item indicating a position based on inertia information of the controller to be displayed in a display area of the display means; a second display control means for controlling, in a second control mode, to display in the display area a second display item indicating a position in a first area of the display area corresponding to a first operation performed on a specific operating member; a switching means for switching from the first control mode to the second control mode when a specific operation is performed in a case where a position indicated by the first display item is included in a second area of the display area in the first control mode; 13. An information processing device comprising: [Configuration 2] when the control mode is switched from the first control mode to the second control mode, the second display control means displays the second display item in the display area so as to indicate a position indicated by the first display item. 2. The information processing device according to configuration 1. [Configuration 3] when the control mode is switched from the first control mode to the second control mode, the second display control means displays the second display item in the display area so as to indicate a predetermined position in the first area. 2. The information processing device according to configuration 1. [Configuration 4] The first operation is an operation of tracing the surface of the specific operation member. 4. The information processing device according to any one of configurations 1 to 3. [Configuration 5] The specific operation is an operation on the specific operating member. 5. The information processing device according to any one of configurations 1 to 4. [Configuration 6] the switching means, in the first control mode, does not switch from the first control mode to the second control mode even if the specific operation is performed when a position indicated by the first display item is not included in the second area; 6. The information processing device according to any one of configurations 1 to 5. [Configuration 7] The switching means is In the first control mode, when the position indicated by the first display item is included in the second area, the specific operation is performed after an operation of selecting the second area. and switching from the first control mode to the second control mode; In the first control mode, when a position indicated by the first display item is included in the second area, the control mode is not switched from the first control mode to the second control mode even if the specific operation is performed unless an operation of selecting the second area is performed. 7. The information processing device according to any one of configurations 1 to 6. [Configuration 8] the switching means switches from the second control mode to the first control mode when a second operation different from both the first operation and the specific operation is performed in the second control mode. 8. The information processing device according to any one of configurations 1 to 7. [Configuration 9] The second operation is an operation of moving the controller or the user's hand so as to satisfy a specific condition. 9. The information processing device according to configuration 8. [Configuration 10] the first display item is a ray; The position indicated by the first display item is the position of the tip of the ray. 10. The information processing device according to any one of configurations 1 to 9. [method] a first display control step of controlling, in a first control mode, a first display item indicating a position based on inertia information of the controller to be displayed in a display area of the display means; a second display control step of controlling, in a second control mode, to display a second display item in a first area of the display area, the second display item indicating a position in the display area corresponding to a first operation of a specific operation member; a switching step of switching from the first control mode to the second control mode when a specific operation is performed in a case where a position indicated by the first display item is included in a second area of the display area in the first control mode; 13. An information processing method comprising: [program] A program for causing a computer to function as each of the means of the information processing device according to any one of configurations 1 to 10. [Explanation of symbols]

[0079] 110: information processing device, 211: control unit, 100: HMD, 203: image display unit

Claims

1. a first display control means for controlling, in the first control mode, to display a first display item indicating a position based on inertial information of the controller in a display area of the display means; a second display control means for controlling, in a second control mode, a second display item indicating a position in a first area of the display area according to a first operation performed on a specific operating member, to be displayed in the display area; a switching means for switching from the first control mode to the second control mode when a specific operation is performed in a case where a position indicated by the first display item is included in a second area of the display area in the first control mode; An information processing device comprising:

2. when the control mode is switched from the first control mode to the second control mode, the second display control means displays the second display item in the display area so as to indicate a position indicated by the first display item.

2. The information processing apparatus according to claim 1, wherein:

3. when the control mode is switched from the first control mode to the second control mode, the second display control means displays the second display item in the display area so as to indicate a predetermined position in the first area.

2. The information processing apparatus according to claim 1, wherein:

4. the first operation is an operation of tracing the surface of the specific operation member; 4. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

5. The specific operation is an operation on the specific operating member.