Information processing device, information processing system, information processing program, and information processing method

Abstract

To provide an information processing device that can improve user operability even when operating in three-dimensional space. [Solution] The information processing device includes an acquisition means for acquiring operations on virtual objects arranged in space, and a control means for controlling the virtual object to execute a first process corresponding to the first operation when a first operation is acquired by the acquisition means, wherein the control means controls the acquisition means to execute a second process, which is different from the first process, regarding the rearrangement of the virtual object in space, even if a first operation is acquired by the acquisition means, provided that predetermined conditions are met.

Description

[[Technical Field]]

[0001] The present disclosure relates to an information processing apparatus. [[Background Art]]

[0002] Typical input operations when using an HMD (Head Mounted Display) include an operation combining line of sight and hand gesture, and ray operation using a controller. There are also methods such as direct touch, where the user directly touches a CG (Computer Graphics) object with a hand recognized through hand tracking.

[0003] When an operation target such as a virtual object is small, there is a risk that an operation not intended by the user may be performed. To address such a problem, Patent Document 1 describes a method for preventing erroneous operations when an operation target object is displayed in a small size on a tablet terminal, and executing an operation matching the user's intention by performing processing according to the display size of the object. [[Prior Art Literature]] [[Patent Literature]]

[0004] [[Patent Document 1]] Japanese Unexamined Patent Application Publication No. 2012-104095 [[Summary of Invention]] [[Problem to be Solved by the Invention]]

[0005] However, the conventional technology disclosed in the above-mentioned patent document assumes a tablet terminal, and does not consider input operations in a three-dimensional space like that of an HMD.

[0006] Accordingly, the present disclosure has been made in view of these points, and aims to improve user operability even during operations in a three-dimensional space. [[Means for Solving the Problem]]

[0007] One aspect of the present disclosure is an information processing apparatus comprising: acquisition means for acquiring operations on virtual objects arranged in space; and control means for controlling the virtual object to perform a first process corresponding to the first operation when a first operation is acquired by the acquisition means, wherein the control means controls the acquisition means to perform a second process relating to the rearrangement of the virtual object in space, which is different from the first process, even when a first operation is acquired by the acquisition means, provided that predetermined conditions are met. [Effects of the Invention]

[0008] According to this disclosure, user operability can be improved even when operating in three-dimensional space. [Brief explanation of the drawing]

[0009] [Figure 1] This diagram illustrates the internal configuration of an information processing device according to the first embodiment of this disclosure. [Figure 2] This diagram illustrates how a CG object appears to the user when it is properly positioned (appropriate size, location, and orientation). [Figure 3] This diagram illustrates how a CG object appears to a user when it is small and of an inappropriate size. [Figure 4] This diagram illustrates how a CG object appears to a user when it is positioned higher than the user's eye level. [Figure 5] This diagram illustrates how a CG object appears to the user when another CG object is placed in front of it as an obstacle. [Figure 6] This diagram illustrates how a CG object appears to a user when it is positioned diagonally to the user in the yaw direction. [Figure 7]This diagram shows a top-down view of the space in Figure 2. [Figure 8] This is a diagram showing the space as viewed from above in Figure 6. [Figure 9] This diagram illustrates how a CG object appears to the user when it is positioned diagonally in the pitch direction relative to the user. [Figure 10] This diagram shows CG object 1000 equipped with UI components for a browser. [Figure 11] This flowchart illustrates the process in the first embodiment for determining whether to rearrange the CG object or to perform processing in response to an input operation. [Figure 12] This flowchart illustrates the process in the first embodiment for determining whether to rearrange the CG object or to perform processing in response to an input operation. [Figure 13] This is a flowchart illustrating the process of rearranging CG objects in the second embodiment. [Figure 14] This flowchart illustrates the process in the third embodiment for determining whether to rearrange the CG object or to perform processing in response to an input operation. [Figure 15] This is a flowchart illustrating the process of rearranging CG objects in the third embodiment. [Figure 16] This flowchart shows the process in the fourth embodiment for determining whether to rearrange the CG object or to perform processing in response to an input operation. [Figure 17] This flowchart shows the process for determining whether or not to return the CG object in the fifth embodiment to its placement state before rearrangement. [Figure 18] This diagram illustrates a scene where two users are simultaneously viewing the same CG object. [Figure 19] This diagram shows a scene where a rearranged CG object is being viewed. [Figure 20]FIG. 1 is a diagram illustrating a scene where a non-rearranged CG object is being viewed. [Figure 21] FIG. 2 is a diagram illustrating a scene where a rearranged CG object is being viewed. [Figure 22] FIG. 3 is a diagram illustrating a scene where the arrangement of a CG object has been changed. [Figure 23] FIG. 4 is a diagram illustrating a scene where a CG object after arrangement change is being viewed. [Figure 24] FIG. 5 is a diagram illustrating a scene where the arrangement of a CG object is changed when an input operation is completed. [Figure 25] FIG. 6 is a diagram illustrating a scene where two users simultaneously view a common CG object after completion of an input operation. [Figure 26] FIG. 7 is a flowchart of overall processing in the sixth embodiment. [Figure 27] FIG. 8 is a diagram of a CG object provided with UI parts as a browser. [Figure 28] FIG. 9 is a diagram illustrating a CG object to which an effect is applied to the entire thereof. [Figure 29] FIG. 10 is a diagram illustrating a CG object to which an effect is applied to a part thereof. DESCRIPTION OF EMBODIMENTS

[0010] Hereinafter, each embodiment will be described with reference to the drawings. The same or equivalent components, members, and processes shown in the respective drawings are denoted by the same reference signs, and overlapping descriptions are omitted as appropriate. In addition, some of the components, members, and processes are omitted in the respective drawings. The following embodiments do not limit the present disclosure, and not all combinations of features described in the present embodiment are necessarily essential to the solution of the present disclosure. The configuration of the embodiments can be appropriately modified or changed depending on the specifications of the apparatus to which the present disclosure is applied and various conditions (usage conditions, usage environment, etc.). In the following embodiments, identical configurations will be described with the same reference signs.

[0011] (First Embodiment) This embodiment assumes a use case in which a user wears an HMD and uses a virtual window application such as a browser. Furthermore, the CG object (virtual object) is described assuming a window application with browser functionality that the user can operate. A window application with browser functionality is merely an example, and the CG objects dealt with in this disclosure are not limited to browsers or window applications.

[0012] In this embodiment, when a user inputs an operation on a CG object, if the situation is prone to errors, the CG object is rearranged to reduce the likelihood of errors. Furthermore, if the situation is not prone to errors, the process is switched to execute a corresponding action based on the input, thereby preventing errors and ensuring that operations are performed in accordance with the user's intentions.

[0013] <Configuration of the information processing device> The configuration of the information processing device according to the first embodiment will be described below with reference to Figure 1.

[0014] Figure 1 is a block diagram of the information processing device in this embodiment. In this embodiment, the information processing device A100 is assumed to be an HMD (Head-Mounted Display). In the first embodiment, an HMD is assumed as an example of a device constituting the information processing system, but this disclosure is not limited to these embodiments. For example, it may be a smartphone or tablet terminal equipped with a camera, or other devices such as a personal computer (PC) or digital camera. Also, while an HMD is assumed as an example of a head-mounted display device with an integrated information processing device, the information processing device is not limited to this. The information processing device connected to the head-mounted display device by wire or wireless may be a smartphone, tablet terminal, or PC. In this case, some of the processing of the HMD described later may be performed by the information processing device connected to the head-mounted display device by wire or wireless.

[0015] The information processing device A100 consists of a control unit A101, a storage unit A102, a memory A103, an input unit A104, an output unit A105, a sensor A106, and a communication unit A107.

[0016] The control unit A101 controls each part of the information processing device A100. The control unit A101 executes programs stored in the memory unit A102, utilizes memory A103 as a work area, and controls the processing unit of the entire information processing device A100. The control unit A101 has at least one CPU (Central Processing Unit) that executes programs stored in A102, and at least one other circuit. The control unit A101 may be composed of, for example, one or more processors such as CPUs or GPUs. Alternatively, instead of the control unit A101 controlling the entire device, multiple hardware components may share the processing to control the entire device.

[0017] The memory unit A102 is an electrically erasable and recordable non-volatile storage medium, such as an SSD (Solid State Drive) or flash memory, and stores programs executed by the control unit A101, as well as databases, user settings, and other data.

[0018] Memory A103 is, for example, RAM (Random Access Memory) and is used as a buffer memory for temporarily holding various data, or as a control area for the control unit A101, etc.

[0019] The input unit A104 is used to input instructions to the information processing device A100. The input unit A104 includes, for example, a power button to instruct the ON / OFF of the information processing device A100, and operation buttons to instruct screen transitions. The input unit A104 does not necessarily have to be built into the information processing device A100. Input may be performed via a controller or the like, through the communication unit A107 described later. In addition to buttons, instructions to the information processing device A100 may also be input via at least one of the following: operation input by gestures recognized from images, voice input, and eye-tracking input.

[0020] The output unit A105 may include, for example, a GUI (Graphical User Interface) display for interactive operation, a light-emitting device such as an LED (Light Emitting Diode), or a sound output device such as a speaker. The output unit A105 may also be configured to output via the communication unit A107, which will be described later.

[0021] Sensor A106 includes, for example, an image sensor for capturing surrounding images, and sensors such as LiDAR (Light Detection and Ranging) and ToF (Time of Flight) for measuring surrounding conditions. Furthermore, sensor A106 includes sensors such as an IMU (Inertial Measurement Unit) for attitude and position measurement, and a geomagnetic sensor. Note that sensor A106 does not necessarily need to be built into the information processing unit A100.

[0022] The communication unit A107 includes, for example, a serial bus and a parallel bus for connecting to other devices. Furthermore, the communication unit A107 includes a NIC (Network Interface Card) with a built-in connector (RJ45) for connecting to Ethernet and communication ICs, and a communication unit for wireless connection with controllers and external devices.

[0023] Although the terminal configuration was described assuming an HMD as the information processing device A100, the aforementioned terminal configuration is merely an example, and the information processing device A100 does not have to be an HMD, and the terminal configuration may also be different.

[0024] <Method for evaluating the likelihood of errors occurring> The following describes a method for evaluating the likelihood of errors occurring when input operations are performed on CG objects in this embodiment. The reason for evaluating the likelihood of errors is that it is one factor in determining whether to rearrange the CG object or perform processing corresponding to the input operation when an input operation is performed on the CG object.

[0025] Factors to consider when evaluating the likelihood of errors include the size of the CG object, the orientation of the CG object, and the distance and positional relationship between the user and the CG object.

[0026] <Method for evaluating the likelihood of errors: Size of CG objects> The following describes a method for evaluating the likelihood of errors occurring in relation to the size of CG objects, with reference to Figures 2 and 3.

[0027] Figure 2 illustrates how a CG object appears to a user when it is properly positioned (appropriate size, location, and orientation). When a CG object is displayed at the appropriate size, the UI elements on the CG object are also displayed at the appropriate size, allowing the user to accurately perform their intended operations using eye-tracking input, ray manipulation, etc. Figure 3 illustrates how a CG object appears to a user when it is too small and inappropriately sized. As shown in Figure 3, when a CG object is too small, the UI elements displayed on the CG object also become small, making it difficult for the user to accurately perform their intended operations using eye-tracking input, ray manipulation, etc., without making mistakes.

[0028] As one indicator, some HMD manufacturers provide design guidelines regarding UI display suitable for their HMDs. If the UI deviates significantly from these guidelines, it can be evaluated as a state where input operations are difficult and errors are likely to occur. For example, if the design guide states that the display size of buttons must be 60pt or larger, you should check whether the size of the buttons on the CG object is 60pt or larger than the display size specified in the design guide. If the displayed buttons are less than 60pt, the CG object can be evaluated (determined) as being in a state where errors are likely to occur.

[0029] The method described above for evaluating the likelihood of errors based on the size of CG objects is merely one example, and other methods for evaluating the likelihood of errors may also be used.

[0030] <Method for evaluating the likelihood of errors: Position of CG objects> The following describes a method for evaluating the likelihood of errors occurring from the perspective of the position of CG objects, with reference to Figures 4 and 5.

[0031] Several methods can be considered for evaluating the likelihood of errors occurring in terms of the position of CG objects, but in this embodiment, we will describe three methods that seem to be of particular importance.

[0032] The first method involves evaluating the likelihood of user error based on the distance between the user and the CG object. Since the apparent size of the CG object decreases as the distance between the user and the object increases, this method can be used to evaluate the likelihood of user error in the same way as evaluating it based on the size of the CG object.

[0033] The second method involves evaluating the likelihood of errors occurring based on the height of the CG object. Figure 4 shows how a CG object appears to the user when it is positioned higher than the user's eye level.

[0034] Generally, when a CG object is displayed at the user's eye level, the user can view the object in a comfortable posture. In contrast, as shown in Figure 4, when a CG object is displayed at a position significantly higher than eye level, the user needs to operate it with their head tilted upwards. The more awkward the posture, the more likely errors are to occur. To evaluate whether a CG object is prone to errors, for example, a threshold for head angle can be set at 15 degrees up or down. If the user needs to tilt their head beyond this value, the CG object can be evaluated as being prone to errors.

[0035] Finally, the third method is to evaluate the likelihood of errors occurring based on whether there are obstacles between the user and the CG object. Figure 5 shows how a CG object appears to the user when another CG object is placed in front of it as an obstacle.

[0036] When there is an obstacle between the user and the CG object, part or all of the CG object may be hidden by the obstacle and become invisible, as shown in Figure 5. When the user attempts to perform an input operation on the CG object, they must avoid the obstacle, making it easy for errors to occur. To evaluate whether a CG object is prone to errors, for example, if the CG object is hidden beyond a threshold percentage that is acceptable, then the CG object can be evaluated as being prone to errors. For example, such a threshold could be set to 25%. Furthermore, if other CG objects are placed in the area that the user is interacting with, even if the CG object is not significantly hidden, it can be determined that there is a high possibility of user errors. Therefore, it may be possible to evaluate (determine) whether the UI for input operations provided on the CG object is hidden by other CG objects, or whether the UIs of multiple CG objects are closer than a predetermined distance from each other.

[0037] The method described above for evaluating the likelihood of errors based on the position of CG objects is subject to change as the user's position changes, so it is necessary to consider the user's position when calculating. Furthermore, these methods are merely examples, and other methods may be used to evaluate the likelihood of errors.

[0038] <Method for evaluating the likelihood of errors: Orientation of CG objects> The following describes a method for evaluating the likelihood of errors occurring in terms of the orientation of CG objects, with reference to Figures 6, 7, 8, and 9.

[0039] Figure 6 shows how a CG object appears to the user when it is positioned diagonally in the yaw direction relative to the user. In other words, it shows a state where the CG object is not facing directly towards the user, with the right side of the CG object being closer to the user and the left side being further away. Figure 7 shows a top-down view of the space in Figure 2, and Figure 8 shows a top-down view of the space in Figure 6. Figure 9 shows how a CG object appears to the user when it is positioned diagonally in the pitch direction relative to the user. In other words, it shows a state where the CG object is facing directly towards the user, but is positioned so that it appears to be tilted towards the user.

[0040] Here, with reference to Figures 7 and 8, we will explain a method for evaluating the likelihood of errors occurring in CG objects from the perspective of their yaw orientation.

[0041] R700, shown in Figure 7, is the angle formed by the user's position, the center of CG object 200, and its left edge.

[0042] R800, shown in Figure 8, is the angle formed by the user's position, the center of the CG object 600, and its left edge.

[0043] Comparing R700 and R800, CG object 200 is positioned to face directly in front of the user, while CG object 600 is positioned diagonally from the user's perspective, resulting in a smaller angle for R800. Assuming the user moves the indicated position by a predetermined angle to indicate a specific location, the amount of movement on the CG object is greater in Figure 8 compared to Figure 7. Therefore, when selecting UI elements on a CG object using eye-tracking or ray manipulation, the CG object placement in Figure 8 results in a larger movement on the CG object for the same angle than in Figure 7, making it more prone to errors. For example, when precise adjustments are needed, fine-tuning can be difficult, potentially leading to errors. Additionally, compared to when the CG object is positioned directly in front, the diagonal placement of the CG object relative to the user reduces the area of ​​buttons and other controls available to the user, potentially increasing the risk of errors. The CG object 600 in Figure 8 is positioned by rotating the CG object 200 in Figure 7 by 45 degrees. However, as the angle approaches 90 degrees, the amount of movement relative to the angle increases, which means that errors are more likely to occur.

[0044] Furthermore, while Figure 6 shows the device rotated in the yaw direction so that it is positioned diagonally to the user, similarly, when the device is rotated in the pitch direction so that it is positioned diagonally to the user, as shown in Figure 9, the closer the angle approaches 90 degrees, the more likely it is to cause errors.

[0045] To evaluate whether a CG object is in a state prone to erroneous operations, for example, the angle thresholds of the CG object are set to 15 degrees respectively in the roll direction and the pitch direction (where the angle at which the CG object faces the user is defined as 0 degrees). Then, if the CG object is arranged at an angle exceeding the threshold, it can be evaluated that the CG object is in a state prone to erroneous operations. It should be noted that the same evaluation may be adopted when the angle exceeds the threshold, or it may be evaluated that the larger the difference from the threshold when the angle exceeds the threshold, the more likely an erroneous operation is to occur.

[0046] Since the orientation of a CG object also changes when the posture (position or orientation) of the user observing it changes, it is necessary to calculate the orientation of the CG object from the user's perspective.

[0047] The above-mentioned method for evaluating the likelihood of an erroneous operation from the perspective of the orientation of the CG object is merely an example, and a different method for evaluating the likelihood of an erroneous operation may be used.

[0048] <Method for Determining Whether to Relocate CG Object or Execute Processing According to Input Operation> Hereinafter, the method for determining whether to relocate the CG object or execute processing according to an input operation in the present embodiment will be described.

[0049] When an input operation is performed on a CG object, it is possible to determine whether to relocate the CG object or execute processing according to the input operation based on whether the CG object is in a state prone to erroneous operations.

[0050] In the present embodiment, when an input operation is performed on a CG object, whether the CG object is in a state where an erroneous operation is likely to occur is evaluated from the plurality of viewpoints described above, such as the size and orientation of the CG object, and the distance and position between the user and the CG object. Then, when it is evaluated that the CG object is in a state where an erroneous operation is likely to occur from any viewpoint, the CG object is repositioned. When it is not evaluated that the CG object is in a state where an erroneous operation is likely to occur from any viewpoint, the input operation performed on the CG object is executed as it is.

[0051] The method of determining whether to reposition the CG object or execute processing according to an input operation described above is merely an example, and a method of determining whether to reposition the CG object or execute processing according to an input operation may be different.

[0052] <Method for Repositioning CG Object> A method for repositioning a CG object in the present embodiment will be described below.

[0053] When it is determined that the CG object is to be repositioned by the above-described method of determining whether to reposition the CG object or execute processing according to an input operation, in the present embodiment, the CG object is repositioned as follows.

[0054] When it is determined that repositioning of the CG object is necessary, the CG object is repositioned so that erroneous operations are less likely to occur. It is expected that, when an input operation is performed again after repositioning the CG object, the CG object has been repositioned such that repositioning is not necessary and processing according to the input operation can be executed.

[0055] As with CG object 200 in FIG. 2, the rearranged CG object is expected to be displayed in an appropriate size and arranged at an appropriate position and orientation based on the user's posture (position / orientation). Here, the appropriate size refers to, for example, a size that reduces the possibility of an erroneous operation different from the operation assumed by the user. The size that reduces the possibility of an erroneous operation may be determined based on, for example, an indicator such as a pointer or a ray that indicates the position indicated by the user. If a button is smaller than the size of the pointer, an erroneous operation is likely to occur, and it is determined that the size is not appropriate. Further, when a button is sufficiently larger than the size of the pointer, it can be determined that the size is appropriate. For example, when a user performs a selection operation or a confirmation operation via a button on a controller, if the button is displayed large enough to make it difficult for an erroneous operation to occur even if the position of the controller's pointer is shifted due to the button operation, it can be determined that the size is appropriate.

[0056] In the present embodiment, rearrangement is performed such that a CG object is placed at a distance reachable by the user, at a position close to the front as viewed from the user that is not blocked by an obstacle, at the same height as the user's line of sight, oriented frontward toward the user, and displayed at the original size. Since the reachable distance for a user varies depending on the length of the user's arm, it is ideal to use a value tuned for each user, but in the present embodiment, this distance is uniformly set to 50 cm. Even when the size of a CG object is set to the original size, depending on the configuration of the CG object, the size may not be optimal, but the following description is based on the premise that the CG object will have an appropriate size when displayed at the original size in the present embodiment.

[0057] The method for rearranging a CG object described above is merely an example, and different methods for rearranging a CG object may be used.

[0058] <Method for executing processing according to input operation on CG object> Hereinafter, with reference to FIG. 10, a method for executing processing according to an input operation on a CG object in the present embodiment will be described.

[0059] If, as described above, the method for determining whether to rearrange the CG object or perform processing in response to the input operation determines that processing should be performed on the CG object in response to the input operation, then in this embodiment, the processing is performed as follows.

[0060] Figure 10 shows a CG object 1000 equipped with UI components for a browser. CG object 1000 is composed of multiple UI components, with 1001 being a reload button and 1002 being a back button.

[0061] For example, if an input operation (tap operation) is performed on the reload button 1001 of CG object 1000, CG object 1000 will perform a page reload process. If an input operation (tap operation) is performed on the back button 1002 of CG object 1000, CG object 1000 will display the page that was displayed previously.

[0062] In other words, executing a process on a CG object in response to an input operation means that the behavior implemented in the CG object is being performed as usual.

[0063] The detailed operation of the information processing device in this embodiment will be described below with reference to Figures 11 and 12.

[0064] Figures 11 and 12 are flowcharts of the information processing device A100 in this embodiment. The flowchart shown in Figure 11 is initiated when a system or application that controls the display of CG objects is launched.

[0065] In step S1100, the control unit A101 determines whether to terminate processing based on the state of each part of the information processing device A100. For example, if a termination operation is performed via the input unit A104, the control unit A101 terminates the processing in this flowchart. If it determines to continue processing, the control unit A101 proceeds to step S1101.

[0066] In step S1101, the control unit A101 determines whether an input operation has been performed on any CG object via the input unit A104. If the control unit A101 determines that an input operation has been performed on a CG object, it proceeds to step S1102. If the control unit A101 does not determine that an input operation has been performed on a CG object, it proceeds to step S1100.

[0067] In step S1102, the control unit A101 acquires placement information of the CG object for which an input operation was performed in step S1101. The placement information shall include at least one of the following: size, position, and orientation.

[0068] In step S1103, the control unit A101 acquires the user's posture information. User posture refers to the orientation of the user's head, and since the information processing unit A100, as an HMD, is worn on the user's head, it is sufficient to acquire the posture information of the information processing unit A100 as the user's posture information. The posture information shall include at least information on position and orientation. In cases where an external camera is installed in the space, such as in an outside-in method, the user's own posture information may be acquired. In this case, SLAM or the like may be used to detect the direction the user is facing forward from the orientation of predetermined parts of the user's body, such as the torso and legs, and acquire this as the user's posture.

[0069] In step S1104, the control unit A101 evaluates the likelihood of erroneous operation of the CG object that was operated on in step S1101, and determines whether to reposition the CG object or execute processing corresponding to the input operation. The method for determining whether to reposition the CG object or execute processing corresponding to the input operation is as described above. Using the CG object placement information obtained in step S1102 and the user's posture information obtained in step S1103, the control unit A101 evaluates the likelihood of erroneous operation and uses the results to determine whether to reposition the CG object. If the control unit A101 determines to reposition the CG object, it proceeds to step S1105. Alternatively, if the control unit A101 determines to execute processing corresponding to the input operation, it proceeds to step S1106. The processing in this step corresponds to the flow in Figure 12, and the details will be described later.

[0070] In step S1105, the control unit A101 rearranges the CG object that was input in step S1101. The rearrangement of the CG object is as described above, and the rearrangement is performed by determining the placement information (size, position, and orientation) of the rearranged CG object using the user's posture information value obtained in step S1103.

[0071] In step S1106, the control unit A101 executes processing corresponding to the input operation performed in step S1101 on the CG object. The method for executing processing corresponding to the input operation on the CG object is as described above.

[0072] <Method for determining whether or not to rearrange> Figure 12 is a flowchart showing the details of the process by which the information processing device A100 determines whether to rearrange the CG object or to perform processing in response to an input operation in this embodiment. The processes from steps S1200 to S1205 in this flowchart correspond to the process in step S1104 in Figure 11.

[0073] In step S1200, the control unit A101 evaluates the likelihood of erroneous operation of the CG object in terms of size, based on the placement information of the CG object acquired in step S1102 and the user's posture information acquired in step S1103. The method for evaluating the likelihood of erroneous operation of the CG object in terms of size is as described above. The control unit A101 stores the evaluation results in memory A103 and refers to them in step S1203, which will be described later.

[0074] In step S1201, the control unit A101 evaluates the likelihood of erroneous operation of the CG object from a positional perspective, based on the placement information of the CG object acquired in step S1102 and the user's posture information acquired in step S1103. The method for evaluating the likelihood of erroneous operation of the CG object from a positional perspective is as described above. The control unit A101 stores the evaluation results in memory A103 and refers to them in step S1203, which will be described later.

[0075] In step S1202, the control unit A101 evaluates the likelihood of erroneous operation of the CG object in terms of orientation, based on the placement information of the CG object acquired in step S1102 and the user's posture information acquired in step S1103. The method for evaluating the likelihood of erroneous operation of the CG object in terms of orientation is as described above. The control unit A101 stores the evaluation results in memory A103 and refers to them in step S1203, which will be described later.

[0076] In step S1203, the control unit A101 obtains the values ​​for the likelihood of errors evaluated in each process from step S1200 to step S1202 via memory A103, and determines whether to rearrange the CG object based on these values. If the control unit A101 determines that the CG object is likely to be prone to errors from any perspective, it proceeds to step S1204 to rearrange the CG object. If the control unit A101 determines that the CG object is not likely to be prone to errors from any perspective, it proceeds to step S1205.

[0077] In step S1204, the control unit A101 determines that it will perform the process of rearranging the CG object. That is, it determines that it will proceed to step S1105 in the flow shown in Figure 11.

[0078] In step S1205, the control unit A101 determines that it will perform processing according to the input operation. That is, it determines that it will proceed to step S1106 in the flow shown in Figure 11.

[0079] As described above, when a user inputs an operation on a CG object, the information processing device A100 of this embodiment rearranges the CG object to reduce the likelihood of errors if the situation is prone to errors. If the situation is not prone to errors, it switches the processing to execute the corresponding operation.

[0080] This prevents errors in input operations in a three-dimensional space, such as with an HMD, and allows for operations that match the user's intentions.

[0081] (Second embodiment) In the first embodiment, when rearranging the CG object, it was rearranged so that it was of an appropriate size, position, and orientation in front of the user.

[0082] However, if the only factor that could cause errors is size, then when rearranging the CG object, only the size needs to be changed. Similarly, if the only factors that could cause errors are position and orientation, then when rearranging the CG object, only the position and orientation need to be changed, respectively.

[0083] By rearranging CG objects, it is expected that changing only the elements that could potentially cause errors, rather than changing all elements such as size, position, and orientation, will result in less overall change before and after the rearrangement of the CG object, thus improving the user experience.

[0084] Therefore, in this embodiment, we will describe the operation of the information processing device A100 when there is only one factor that may cause an error when an input operation is performed on a CG object.

[0085] Since this embodiment shares many commonalities with the first embodiment, the explanation will focus on the parts unique to this embodiment.

[0086] The method for rearranging CG objects in this embodiment will be described below with reference to Figure 13.

[0087] In this embodiment, when rearranging CG objects, the rearrangement process is switched based on the evaluation results of the likelihood of errors, which were used to determine whether to rearrange the objects. If it is evaluated that errors are likely to occur only from a specific perspective, the CG object is modified according to that perspective; otherwise, the CG object is rearranged in the same way as in the first embodiment.

[0088] Figure 13 is a flowchart showing the details of the process for rearranging CG objects in the information processing device A100 in this embodiment. Each process in this flowchart is performed in step S1105 of Figure 11.

[0089] In step S1300, the control unit A101 refers to memory A103 to confirm whether it was evaluated in step S1200 as being prone to erroneous operation only in terms of size. If the control unit A101 confirms that it was evaluated as being prone to erroneous operation only in terms of size, it proceeds to step S1301. If the control unit A101 determines otherwise, it proceeds to step S1302.

[0090] In step S1301, the control unit A101 changes the size of the CG object. The size of the CG object should be changed to, for example, the same size, so that errors are not likely to occur in terms of size when an input operation is performed again.

[0091] In step S1302, the control unit A101 refers to memory A103 to confirm whether it was evaluated in step S1201 that errors are likely to occur only in terms of position. If the control unit A101 confirms that errors are likely to occur only in terms of position, it proceeds to step S1303. If the control unit A101 does not determine that errors are likely to occur only in terms of position, it proceeds to step S1304.

[0092] In step S1303, the control unit A101 changes the position of the CG object. The position of the CG object should be changed to a position that is within reach of the user, close to directly in front of the user, not obstructed by obstacles, and at eye level, so that errors in position are not likely to occur when input operations are performed again. The distance within reach of the user will vary depending on the length of the user's arm, so it would be ideal to use a value tuned for each user, but in this embodiment, it is uniformly set to 50 cm, as in the first embodiment.

[0093] In step S1304, the control unit A101 refers to memory A103 to confirm whether it was evaluated in step S1202 that errors are likely to occur only in terms of orientation. If the control unit A101 confirms that errors are likely to occur only in terms of orientation, it proceeds to step S1305. If the control unit A101 determines otherwise, it proceeds to step S1306.

[0094] In step S1305, the control unit A101 changes the orientation of the CG object. The orientation of the CG object should be changed so that it faces the user, for example, so that it is not easily misinterpreted in terms of orientation when an input operation is performed again.

[0095] In step S1306, the control unit A101 changes the size, position, and orientation of the CG object. The CG object is rearranged in the same manner as the rearrangement method of the first embodiment so that errors are not likely to occur in any respect when an input operation is performed on the CG object again. For example, the CG object should be placed within reach of the user, in a position that is close to directly in front of the user and not obstructed by any obstacles, at the same height as the user's line of sight, facing the user directly, and displayed at its original size.

[0096] As described above, when an input operation is performed on a CG object, if there is only one element that may cause an error, the information processing device A100 of this embodiment rearranges the CG object to change only that element.

[0097] In the first embodiment described above, all elements of the CG object—size, position, and orientation—were changed when the CG object was rearranged. In this embodiment, the amount of change before and after rearranging the CG object is reduced, and therefore, a better user experience is expected compared to the first embodiment.

[0098] (Third embodiment) In the second embodiment, when rearranging CG objects, only the elements that could potentially cause errors were modified. However, depending on the type of CG object and its placement conditions, it may be better to resolve the issue by modifying elements other than those that could potentially cause errors.

[0099] For example, if the size of a CG object is fixed, it is undesirable for the user if the size of the CG object changes when it is rearranged. In such cases, it is better to rearrange the CG object to change its position rather than its size.

[0100] Similarly, when the position and orientation of a CG object are fixed, it is undesirable for the user that the position and orientation are changed by rearrangement of the CG object. Even in such a case, rearrangement may be performed to change other parameters such as the size and position of each CG object.

[0101] Therefore, in the present embodiment, the operation of an information processing apparatus A100 that switches the rearrangement method of CG objects according to the type and arrangement conditions of CG objects will be described, taking as an example the case where the size of a CG object is fixed.

[0102] Note that since the present embodiment has many parts in common with the first embodiment and the second embodiment, description will focus on parts specific to the present embodiment.

[0103] <Method for evaluating susceptibility to erroneous operation from the viewpoint of the position of a CG object> Hereinafter, a method for evaluating susceptibility to erroneous operation from the viewpoint of the position of a CG object in the present embodiment will be described.

[0104] The basic idea of the method for evaluating susceptibility to erroneous operation from the viewpoint of the position of a CG object in the present embodiment is the same as that of the first embodiment.

[0105] However, as a difference from the first embodiment, there is a condition that the size of the CG object is fixed. Therefore, when evaluating the susceptibility to erroneous operation based on the distance between the user and the CG object, it is necessary to also take into account the fixed size of the CG object.

[0106] In the first embodiment, when it is determined that an erroneous operation is likely to occur from the perspective of the size of the CG object, the CG object is changed to an appropriate size through rearrangement. Therefore, when evaluating whether an erroneous operation is likely to occur from the perspective of the position of the CG object, the evaluation may be performed on the premise that the size of the CG object is appropriate. In contrast, in the present embodiment, since the CG object is not necessarily fixed at an appropriate size, the evaluation is performed after calculating the apparent size of the CG object from the distance between the user and the CG object and the size of the CG object.

[0107] For example, if a design guide specifies that the display size of a button must be 60pt or larger, to evaluate whether the state is prone to erroneous operation, it is sufficient to check whether the apparent size of the button on the CG object is equal to or larger than the display size specified in the design guide. If the displayed button is smaller than 60pt, it can be evaluated that the CG object is in a state where erroneous operations are likely to occur.

[0108] The aforementioned method for evaluating whether an erroneous operation is likely to occur from the perspective of the position of a CG object is merely an example, and a different method for evaluating the likelihood of an erroneous operation may be used.

[0109] <Flow for determining whether to rearrange a CG object or execute processing in accordance with an input operation> Hereinafter, a method for determining whether to rearrange a CG object or execute processing in accordance with an input operation in the present embodiment will be described with reference to FIG. 14.

[0110] FIG. 14 is a flowchart showing details of processing for determining whether to rearrange a CG object in the information processing apparatus A100 according to the present embodiment. The processing of this flowchart corresponds to the processing of step S1104 in FIG. 11.

[0111] In step S1400, the control unit A101 refers to the memory A103 and determines whether the size of the CG object is fixed. If the control unit A101 determines that the size of the CG object is fixed, it proceeds to step S1401. If the control unit A101 does not determine that the size of the CG object is fixed, it proceeds to step S1200.

[0112] In step S1401, the control unit A101 evaluates the likelihood of an erroneous operation from the perspective of position in consideration of the size of the CG object, based on the placement information of the CG object acquired in step S1102 and the value of the user's posture information acquired in step S1103. The method for evaluating the likelihood of an erroneous operation from the perspective of position in consideration of the size of the CG object is as described above. The control unit A101 stores the evaluation result in the memory A103 and refers to it in step S1203.

[0113] Note that the processing from step S1200 to step S1205 is the same as the content described above with reference to FIG. 12, and thus a description thereof will be omitted.

[0114] As described above, in the present flow, if the size of the operated CG object is fixed, the likelihood of an erroneous operation is evaluated from viewpoints other than the size, and if the size of the CG object is not fixed, the likelihood of an erroneous operation is evaluated from viewpoints including the size.

[0115] <Method for changing position of CG object> Hereinafter, the method for changing the position of a CG object according to the present embodiment will be described.

[0116] In the CG object rearrangement methods according to the first embodiment and the second embodiment, the position of the CG object is changed such that the distance from the user to the CG object is 50 cm at the height of the user's line of sight in front of the user. However, in the present embodiment, the position of the CG object is changed not to a uniform 50 cm, but to a distance that takes into consideration the size of the CG object.

[0117] In this embodiment, the case where the size of the CG object is at the original magnification is used as a reference. When the size of the CG object is the original magnification, the position of the CG object is changed such that the distance between the user and the CG object is 50 cm in the same manner as in the first embodiment. In other cases, the position of the CG object is changed such that the apparent size thereof is the same as that of the reference case of the original magnification CG object.

[0118] When the size of the CG object is reduced to one half, the distance between the user and the CG object is also reduced to one half, so that the apparent size of the CG object does not change due to perspective. For example, when the size of the CG object is one half of the original magnification, the position of the CG object is changed such that the distance between the user and the CG object changes from 50 cm to 25 cm. By this means, it is possible to change the position of the CG object such that the apparent size thereof is the same as that of the reference case of the original magnification CG object.

[0119] Even when the CG object has another size, the target position to which the CG object is to be changed can be obtained by using the same idea based on perspective.

[0120] The method for changing the position of the CG object described above is merely an example, and different methods for changing the position of the CG object may be used.

[0121] <Method for Relocating CG Objects> Hereinafter, a method for relocating CG objects in the present embodiment will be described with reference to FIG. 15.

[0122] In the CG object relocation processing of this embodiment, the processing is switched depending on whether the size of the CG object is fixed. When changing the position of the CG object, if the size of the CG object is fixed, the position is changed in consideration of the size of the CG object. Otherwise, the position of the CG object is changed by the same method as in the second embodiment.

[0123] Figure 15 is a flowchart showing the details of the process for rearranging CG objects in the information processing device A100 in this embodiment. Each process in this flowchart is performed in step S1105 of Figure 11.

[0124] Steps S1300 and S1301 are the same as those shown in Figure 13 above, so their explanation is omitted.

[0125] In step S1302, the control unit A101 refers to memory A103 to confirm whether it was evaluated in step S1201 that errors are likely to occur only in terms of position. If the control unit A101 confirms that errors are likely to occur only in terms of position, it proceeds to step S1500. If the control unit A101 does not determine that errors are likely to occur only in terms of position, it proceeds to step S1304.

[0126] In step S1500, the control unit A101 refers to memory A103 and determines whether the size of the CG object is fixed. If the control unit A101 determines that the size of the CG object is fixed, it proceeds to step S1501. If the control unit A101 does not determine that the size of the CG object is fixed, it proceeds to step S1303.

[0127] In step S1501, the control unit A101 changes the position of the CG object, taking into account the size of the CG object, based on the placement information of the CG object acquired in step S1102 and the user's posture information acquired in step S1103. The method for changing the position of the CG object, taking into account the size of the CG object, is as described above.

[0128] Step S1303 is the same as in Figure 13 above, so the explanation is omitted.

[0129] In step S1304, the control unit A101 refers to memory A103 and determines whether it was evaluated in step S1202 that errors are likely to occur only in terms of orientation. If the control unit A101 determines that errors are likely to occur only in terms of orientation, it proceeds to step S1305. If the control unit A101 does not determine that errors are likely to occur only in terms of orientation, it proceeds to step S1502.

[0130] Step S1305 is the same as in Figure 13 above, so the explanation is omitted.

[0131] In step S1502, the control unit A101 refers to memory A103 and determines whether the size of the CG object is fixed. If the control unit A101 determines that the size of the CG object is fixed, it proceeds to step S1503. If the control unit A101 does not determine that the size of the CG object is fixed, it proceeds to step S1306.

[0132] In step S1503, the control unit A101 changes the position of the CG object, taking into account the orientation and size of the CG object. Here, the position of the CG object is changed based on the placement information of the CG object obtained in step S1102 and the user's posture information obtained in step S1103. The method for changing the orientation of the CG object is the same as in the first and second embodiments, and the method for changing the position of the CG object, taking into account the size of the CG object, is as described above.

[0133] Step S1306 is the same as in Figure 13 above, so the explanation is omitted.

[0134] As described above, in this embodiment, when the size of a CG object is fixed depending on the type and placement conditions of the CG object, the information processing device A100 rearranges the CG object to change its position while taking the size of the CG object into consideration.

[0135] This allows CG objects, when their size is fixed, to be repositioned to a state where accidental operation is less likely to occur without changing the size of the CG object itself.

[0136] In this embodiment, the case where the size of the CG object is fixed was described as an example. However, if the position of the CG object is fixed, the size of the CG object can be changed by rearranging it while considering the distance between the user and the CG object. Similarly, if the orientation of the CG object is fixed, the position of the CG object can be changed by rearranging it while considering its orientation.

[0137] (Fourth embodiment) In the first, second, and third embodiments, when an input operation was performed on a CG object, the system evaluated whether the CG object was in a state prone to errors from several perspectives. Based on this evaluation, it was determined whether to rearrange the CG object or to perform processing corresponding to the input operation.

[0138] However, in addition to the likelihood of accidental operation, the decision to rearrange a CG object or perform an action in response to an input operation may also be based on whether the user is within touching distance of the CG object.

[0139] When the user and the CG object are more than a certain distance apart, the user cannot directly touch the CG object even by reaching out, making direct touch input impossible. This is not a problem when using eye-tracking or ray manipulation, but when direct touch operation is desired, it is undesirable for the user to have direct touch excluded from their input options, leading to a decrease in usability.

[0140] Therefore, in this embodiment, we will describe a case in which we consider whether the user may perform a direct touch input operation on the CG object when determining whether to rearrange the CG object or perform processing in response to an input operation.

[0141] Since this embodiment shares many common parts with the first, second, and third embodiments, this description will focus on the parts unique to this embodiment.

[0142] The following describes how to determine whether to rearrange the CG object in this embodiment or to perform processing in response to an input operation, with reference to Figure 16.

[0143] Figure 16 is a flowchart showing the details of the process for determining whether to rearrange the CG objects of the information processing device A100 in this embodiment. Each process in this flowchart is performed in step S1104 of Figure 11.

[0144] Steps S1200 through S1202, S1204, and S1205 are the same as those shown in Figure 12 above, so their explanation is omitted.

[0145] In step S1203, the control unit A101 obtains the values ​​for the likelihood of errors evaluated in each process from step S1200 to step S1202 via memory A103, and determines whether to rearrange the CG object based on these values. If the control unit A101 determines that the CG object is likely to be prone to errors from any perspective, it proceeds to step S1204 to rearrange the CG object. If the control unit A101 determines that the CG object is not likely to be prone to errors from any perspective, it proceeds to step S1600.

[0146] In step S1600, the control unit A101 checks whether direct touch input is enabled and whether the CG object is out of the user's reach. To check whether direct touch input is enabled, the control unit A102 can be consulted to confirm that the setting is enabled. To determine whether the CG object is out of the user's reach, the distance between the user and the CG object can be calculated and then checked to see if it is within the user's reach. For example, the distance between the user and the CG object can be calculated based on the CG object placement information obtained in step S1102 and the user's posture information obtained in step S1103. Alternatively, the distance between the HMD worn by the user and the CG object can be calculated. If the control unit A101 determines that direct touch input is enabled and the CG object is out of the user's reach, it proceeds to step S1105 to reposition the CG object. If the control unit A101 determines otherwise, it proceeds to step S1205 to execute processing corresponding to the input operation on the CG object.

[0147] As described above, when the information processing device A100 of this embodiment determines whether to rearrange a CG object or execute processing in response to an input operation, it considers whether the user may perform an input operation on the CG object by direct touch.

[0148] This allows CG objects to be directly manipulated by repositioning them regardless of the likelihood of accidental operation, even if they are out of the user's reach. As a result, after the CG object is repositioned, direct touch will always be included in the input options, improving usability.

[0149] (Fifth embodiment) In the first, second, third, and fourth embodiments, CG objects are rearranged when there is a high possibility of an erroneous operation occurring during manipulation of the CG object or when direct touch cannot be performed.

[0150] The purpose of rearranging CG objects is to prevent erroneous operations during input operations and expand the options for input operations, so there is no need to keep the CG objects arranged there after the input operation is completed. Since the state before the rearrangement of the CG object is considered to be the ideal arrangement set by the user, it is considered that user experience will be improved if the CG object is returned to the arrangement state before the rearrangement after the input operation on the CG object is completed.

[0151] Therefore, in the present embodiment, the operation of the information processing apparatus A100 that returns a CG object to the arrangement state before rearrangement at the timing when the input operation on the CG object is completed will be described.

[0152] Note that since the present embodiment has many parts in common with the first, second, third, and fourth embodiments, the description will focus on parts specific to the present embodiment.

[0153] <Method for Determining Whether Input Operation on CG Object is Completed> Hereinafter, a method for determining whether an input operation on a CG object in the present embodiment is completed will be described.

[0154] In the present embodiment, when no operation is performed for a predetermined period after an input operation is performed on a CG object, it is determined that the input operation on the CG object is completed.

[0155] For example, if a period of 5 seconds is set and no input operation is performed on the CG object for 5 seconds or more, it is determined that the input operation is completed at that time. If an input operation is performed within 5 seconds, it is determined that the input operation is completed at the timing when 5 seconds have elapsed since the last input operation was performed.

[0156] The method described above for determining whether an input operation on a CG object has been completed is merely an example, and the method for determining whether an input operation on a CG object has been completed may be different. For example, when the user performs an operation via a gesture, this may be triggered when a predetermined period has elapsed while the user has lowered their hand. Further, when the user performs an operation via a gesture, this may be triggered when a predetermined period has elapsed after the user's hand moves outside the range of a captured image. Furthermore, the timing to start counting the predetermined period may be the timing at which a gesture operation is received, the timing at which the end of a gesture operation is detected, or the timing at which processing corresponding to the gesture operation is completed.

[0157] <Method for determining whether to return a CG object to its arrangement state before rearrangement, and timing for returning a CG object to its arrangement state before rearrangement> Hereinafter, a method for determining whether to return a CG object to its arrangement state before rearrangement and the timing for returning a CG object to its arrangement state before rearrangement in the present embodiment will be described.

[0158] In the present embodiment, it is determined whether to return a CG object to its arrangement state before rearrangement based on whether the arrangement of the CG object has been changed by the user during the period from when the CG object is rearranged until the input operation is completed.

[0159] If the arrangement of the CG object has been changed by the user during the period from when the CG object is rearranged until the input operation is completed, it is determined that the CG object will not be returned to the arrangement state before rearrangement. That is, it is determined that the rearranged position is to be maintained.

[0160] If the arrangement of the CG object has not been changed by the user during the period from when the CG object is rearranged until the input operation is completed, it is determined that the CG object will be returned to the arrangement state before rearrangement. That is, it is determined that the rearranged position is not to be maintained.

[0161] The timing for returning the CG object to the arrangement state before re-arrangement is the timing when the input operation on the CG object is completed. By returning the CG object to the arrangement before re-arrangement at the timing when the input operation on the CG object is completed in this way, the possibility of an input operation being an erroneous operation can be reduced, and furthermore, the CG object can be prevented from being arranged at a position unintended by the user.

[0162] It should be noted that the method for determining whether to return the CG object to the arrangement state before re-arrangement and the timing for returning the CG object to the arrangement state before re-arrangement described above are merely examples, and these determination methods and timings may be different.

[0163] <Flow for determining whether to return a CG object to the arrangement state before re-arrangement> Hereinafter, with reference to Fig. 17, the detailed operation of the information processing apparatus in the present embodiment will be described.

[0164] Fig. 17 is a flowchart of the information processing apparatus A100 in the present embodiment. The flowchart shown in Fig. 17 is started when a startup operation of a system or application that performs display control of CG objects is performed. It should be noted that step S1100, step S1102, step S1103, and step S1105 are the same as those in Fig. 11 described above, so descriptions thereof will be omitted.

[0165] In step S1101, the control unit A101 determines whether an input operation has been performed on any CG object via the input unit A104. If the control unit A101 determines that an input operation has been performed on the CG object, the process proceeds to step S1700. If the control unit A101 does not determine that an input operation has been performed on the CG object, the process proceeds to step S1704.

[0166] In step S1700, the control unit A101 stores the time when the input operation is performed on the CG object in the memory A103, and the process proceeds to step S1102.

[0167] In step S1104, the control unit A101 evaluates the likelihood of erroneous operation of the CG object that was input in step S1101 and determines whether to reposition the CG object or execute processing corresponding to the input operation. The method for determining whether to reposition the CG object or execute processing corresponding to the input operation is as described above. Using the CG object placement information obtained in step S1102 and the user's posture information obtained in step S1103, the control unit A101 evaluates the likelihood of erroneous operation and uses the result to determine whether to reposition the CG object. If the control unit A101 determines to reposition the CG object, it proceeds to step S1701. If the control unit A101 determines to execute processing corresponding to the input operation, it proceeds to step S1106.

[0168] In step S1701, before rearranging the CG objects in step S1105, the control unit A101 saves the current arrangement information before rearranging to memory A103, and then proceeds to step S1105.

[0169] In step S1702, the control unit A101 determines whether the placement of the CG object has been changed by the processing corresponding to the input operation in step S1106. If the control unit A101 determines that the placement of the CG object has been changed, it proceeds to step S1703. If the control unit A101 does not determine that the placement of the CG object has been changed, it proceeds to step S1100.

[0170] In step S1703, the control unit A101 deletes the placement information of the CG object before its repositioning, which was saved in memory A103 in step S1701, and proceeds to step S1100. The control unit A101 checks memory A103, and if the placement information of the CG object has already been deleted or has not been saved, it does nothing and proceeds to step S1100.

[0171] In step S1704, the control unit A101 obtains the time of the last input operation performed on the CG object stored in memory A103 in step S1700, and determines whether the input operation on the CG object has finished. The method for determining whether the input operation on the CG object has finished is as described above. If the control unit A101 determines that the input operation on the CG object has finished, it proceeds to step S1705. If the control unit A101 does not determine that the input operation on the CG object has finished, it proceeds to step S1100.

[0172] In step S1705, the control unit A101 checks whether the placement information of the CG object before its rearrangement, which was saved in memory A103 in step S1701, is retained. If the control unit A101 determines that the placement information of the CG object is retained, it proceeds to step S1706. If the control unit A101 determines that the placement information of the CG object is not retained, or that the placement information of the CG object has been deleted, it proceeds to step S1100.

[0173] In step S1706, the control unit A101 retrieves the placement information of the CG objects before rearrangement, which was saved in memory A103 in step S1701, modifies the placement of the CG objects, and then proceeds to step S1707.

[0174] In step S1707, the control unit A101 deletes the placement information of the CG object before its repositioning, which was saved in memory A103 in step S1701, and proceeds to step S1100. The control unit A101 checks memory A103, and if the placement information of the CG object has already been deleted or has not been saved, it does nothing and proceeds to step S1100.

[0175] As described above, the information processing device A100 of this embodiment performs a process to return the CG object to its placement state before rearrangement when the input operation on the CG object is completed.

[0176] This improves the user experience by reducing the likelihood of errors and offering a wider range of input options when performing input operations on CG objects. After the user has finished inputting to the CG object, it returns to the ideal positioning they set for themselves.

[0177] (Sixth embodiment) In the first, second, third, fourth, and fifth embodiments, the description was based on the premise that the user viewing the CG object is a single user wearing an HMD.

[0178] However, some HMD systems offer a sharing function that allows multiple people to view CG objects simultaneously.

[0179] When using such sharing features, if someone performs an input operation on a CG object and the CG object is rearranged, the user who performed the input operation will be less likely to make mistakes and will have more input options. However, for users who did not perform an input operation, the CG object will be rearranged automatically, making it difficult to view, which could worsen the user experience.

[0180] Therefore, in this embodiment, when multiple people are viewing a CG object simultaneously, the user performing the input operation is shown the CG object rearranged as needed, while other users are shown the CG object in its original position. The operation of this information processing device A100 will be described below.

[0181] Since this embodiment shares many common features with the first, second, third, fourth, and fifth embodiments, this description will focus on the aspects unique to this embodiment.

[0182] The following explanation describes how CG objects appear to each user, referring to Figures 18, 19, and 20.

[0183] Figure 18 shows a scene in which two users (U1801 and U1802) wearing HMDs are simultaneously viewing a common CG object 1800. In this scene, both users are able to see the CG object 1800, albeit from an oblique angle.

[0184] In this explanation, we assume a scenario where user U1801 performs input operations on CG object 1800, while user U1802 only views the CG object.

[0185] Figure 19 shows a scene where user U1801 performs an input operation, resulting in the rearrangement of CG objects. CG object 1900 is the rearranged version of CG object 1800.

[0186] Figure 20 illustrates a scenario in which, when a CG object is rearranged for user U1801, CG object 1800 is displayed to user U1802, who is not manipulating the CG object.

[0187] As shown in Figure 19, the repositioned CG object 1900 is displayed to user U1801, who has performed an input operation on the CG object. However, if the repositioned CG object 1900 is displayed to user U1802 in the same way, the visibility of the CG object will be reduced because user U1802 is directly to the side of the CG object. Therefore, in a scenario like Figure 19, the CG object is positioned as shown in Figure 20 below to prevent a decrease in the visibility of the CG object for user U1802. Alternatively, when user U1801 finishes the input operation, the CG object may return to the state shown in Figure 18, and the same positioned CG object 1800 may be displayed again to both user U1801 and user U1802.

[0188] The aforementioned descriptions of how CG objects appear to each user are merely examples, and the number and placement of users, the placement of CG objects, and other conditions may vary.

[0189] <A flow that determines whether to return the CG object to its previous placement state or change it to a different placement state when the input operation is completed.> The following describes how to determine whether to return the CG object to its previous placement state or change it to a different placement state when the input operation in this embodiment is completed.

[0190] In the fifth embodiment, since there was only one user, it was assumed that if a CG object was rearranged by an input operation, and the user did not further change its position, the CG object would be returned to its position before rearrangement when the input operation was completed.

[0191] However, in cases where multiple people are viewing a CG object, as in this embodiment, there is a possibility that multiple users may simultaneously perform input operations on the CG object.

[0192] Therefore, in this embodiment, if the placement of a CG object is changed by another user during an input operation, the CG object is not returned to its original placement state at the end of the input operation, but rather changed to the placement determined by the other user.

[0193] The method described above for determining whether to return the CG object to its previous placement state or change it to a different placement state at the end of the input operation is merely an example, and other methods may be used.

[0194] <Specific examples of actions taken when an input operation is completed> The operation at the end of the input operation in this embodiment will be described below with specific examples, referring to Figures 18, 19, 20, 21, 22, 23, 24, and 25.

[0195] Figure 18 illustrates a scenario where users U1801 and U1802 are viewing the same CG object 1800. Figure 19 illustrates a scenario where user U1801 performs an input operation, which triggers the rearrangement of the CG object. Figure 20 illustrates a scenario where, at the moment user U1801 performs an input operation and the CG object is rearranged, CG object 1800 is displayed to user U1802, who is not interacting with the CG object.

[0196] Figure 21 is a diagram illustrating a scenario where the repositioned CG object 2100 is displayed when user U1802 performs an operation to change the placement of the CG object. Figure 22 is a diagram illustrating a scenario where user U1802 performs an operation to change the placement of CG object 2100 to that of CG object 2200. Figure 23 is a diagram illustrating a scenario where the CG object 2200, whose placement has been changed by user U1802's operation in Figure 22, is displayed.

[0197] Figure 24 also illustrates a scenario where, after user U1801 completes their input operation, the placement of CG object 1900 is changed to that of CG object 2200.

[0198] Figure 25 also illustrates a scenario in which users U1801 and U1802 are viewing the same rearranged CG object 2200.

[0199] The following describes the processing flow from Figure 18 to Figure 25. First, users U1801 and U1802 are viewing the same CG object 1800 (Figure 18), and user U1801 performs an input operation. At this time, user U1801 is shown the repositioned CG object 1900 (Figure 19), while user U1802 continues to see the unrepositioned CG object 1800 (Figure 20). Next, user U1802 performs an operation to change the position of CG object 1800, and CG object 1800 is repositioned to the position of CG object 2100 (Figure 21). Subsequently, user U1802 performs an operation to position CG object 2100 to the position of CG object 2200 (Figure 22), and user U1802 is shown the screen with CG object 2200 positioned (Figure 23). At this point, assuming that user U1801 has not yet completed the input operation, user U1801 continues to see the repositioned CG object 1900 (Figure 19). Finally, once user U1801 completes the input operation, CG object 1900 is repositioned to the location of CG object 2200 for user U1801 (Figure 24). As a result, ultimately, both user U1801 and user U1802 will be viewing the same repositioned CG object 2200 (Figure 25).

[0200] Thus, if another user changes the placement of a CG object while the user is performing input operations on the CG object, the CG object will not revert to its previous state when the user finishes inputting, but will instead be changed to the new placement made by the other user.

[0201] The actions described above upon completion of an input operation are merely examples, and the actions upon completion of an input operation may differ.

[0202] Furthermore, the system may notify users that another user is manipulating a CG object, that the CG object has been rearranged on another user's screen, or that the CG object is positioned differently on another user's screen. Alternatively, the system may notify users that the CG object will be rearranged by another user's actions, or the position it will be moved to after being rearranged. This allows users to know in advance whether another user is currently manipulating the object, or whether rearrangement will occur later due to another user's actions.

[0203] <Flowchart for rearranging CG objects when multiple users are viewing the same object> The detailed operation of the information processing device in this embodiment will be described below with reference to Figure 26.

[0204] Figure 26 is a flowchart of the information processing device A100 in this embodiment. The flowchart shown in Figure 26 is initiated when a system or application that displays CG objects to multiple users is launched.

[0205] Steps S1101 to S1106 and S1700 to S1707 are the same as those shown in Figure 17 above, so their explanation is omitted.

[0206] In step S1100, the control unit A101 determines whether to terminate processing based on the state of each part of the information processing device A100. For example, if a termination operation is performed via the input unit A104, the control unit A101 terminates the processing in this flowchart. If it determines to continue processing, the control unit A101 proceeds to step S2600.

[0207] In step S2600, the control unit A101 determines via the communication unit A107 whether the placement of the CG object has been changed by another user. If the control unit A101 determines that the placement of the CG object has been changed by another user, it proceeds to step S2601. If the control unit A101 determines that the placement of the CG object has not been changed by another user, it proceeds to step S1101.

[0208] In step S2601, the control unit A101 obtains new placement information of CG objects modified by another user via the communication unit A107.

[0209] In step S2602, the control unit A101 obtains the last time an input operation was performed on the CG object stored in memory A103 in step S1700, and determines whether an input operation is currently being performed on the CG object. If the control unit A101 determines that an input operation is currently being performed on the CG object, it proceeds to step S2603. If the control unit A101 determines that an input operation is not currently being performed on the CG object, it proceeds to step S2604.

[0210] In step S2603, the control unit A101 saves the new placement information of the CG object acquired in step S2601 to memory A103. At this time, the control unit A101 saves the data to the same location where the placement information was saved in step S1701, and if placement information has already been saved by step S1701 or step S2603, it overwrites that data.

[0211] In step S2604, the control unit A101 changes the placement of the CG objects using the new placement information of the CG objects acquired in step S2601.

[0212] The process in step S1706 is the same as the behavior described in the fifth embodiment. However, if the placement information of the CG object saved in memory A103 in step S1701 has been overwritten in step S2603, the CG object will not return to its state before repositioning, but will be changed to a new state modified by another user.

[0213] As described above, when multiple people are simultaneously viewing a CG object, the information processing device A100 of this embodiment displays the rearranged CG object to the user performing the input operation as needed. It also displays the original arrangement of the CG object to users who are not performing the input operation.

[0214] This prevents a deterioration of the user experience when multiple people are viewing a CG object simultaneously. For example, if someone performs an input operation on a CG object, it will not cause other users to have difficulty viewing the object.

[0215] (Seventh Embodiment) In the first, second, third, fourth, fifth, and sixth embodiments, the CG object was rearranged as needed when an input operation was performed on the CG object. Whether the CG object is rearranged or a corresponding process is executed when an input operation is performed on the CG object is as described in each embodiment. However, it can be difficult for the user to determine whether the CG object will be rearranged or a corresponding process will be executed until they actually perform an input operation on the CG object. If the user does not know which process will be executed when they perform an input operation, and the executed action differs from the user's expectation, the user experience will be poor.

[0216] Therefore, in this embodiment, when a user attempts to perform an input operation on a CG object, the display method of the CG object is switched to inform the user whether to rearrange the CG object or execute a process corresponding to the input operation. The operation of such an information processing device A100 will be described below.

[0217] Since this embodiment shares many common parts with the first, second, third, fourth, fifth, and sixth embodiments, this description will focus on the parts unique to this embodiment.

[0218] The effects displayed on the CG objects in this embodiment will be described below with reference to Figures 27, 28, and 29.

[0219] Figure 27 shows a CG object 2700 equipped with UI components that function as a browser. The CG object 2700 is composed of multiple UI components that function as a browser, and the pointer 2701 is an indicator that shows where the user is pointing within the browser. The pointer is displayed at the position where the user's gaze is directed when using eye-tracking input, at the position where the hand or controller is pointed when using ray control, or at the position where the finger is about to touch when using direct touch.

[0220] In this embodiment, when an input operation is attempted on a CG object, the effect displayed on the CG object is switched depending on whether the CG object is rearranged or processing corresponding to the input operation is executed.

[0221] When an input operation is performed on a CG object and the CG object is to be repositioned, the effect is applied to the entire CG object 2700 as shown in Figure 28. In Figure 28, the effect is applied to the entire CG object, and the user can be informed that the CG object will be repositioned when an input operation (tap operation) is performed in this state. In other words, when the effect is applied to the entire CG object 2700 as shown in Figure 28, the operation to reposition the CG object is performed instead of the original input operation.

[0222] When an input operation is performed on a CG object, if processing is to be executed in response to the input operation, an effect is applied to a part of the UI on the CG object, as shown in Figure 29. In Figure 29, an effect is applied to button 2900, which is pointed to by pointer 2701. Performing an input operation (tap operation) in this state informs the user that button 2900 is being pressed. In other words, when an effect is applied to a part of the UI on a CG object, as shown in Figure 29, the original input operation is performed.

[0223] The effects displayed on the aforementioned CG objects are merely examples, and the method of communicating the actions performed by input operations to the user may differ.

[0224] As described above, when a user attempts to perform an input operation on a CG object, the information processing device A100 of this embodiment switches the effect displayed on the CG object depending on whether it rearranges the CG object or executes processing corresponding to the input operation.

[0225] This allows users to understand whether the intended input operation or a repositioning process will be executed when they perform an input operation, and since the actions performed match the user's perception, it prevents a deterioration of the user experience.

[0226] While preferred embodiments of the present disclosure have been described above, the present disclosure is not limited to these embodiments, and various modifications and changes are possible within the scope of its essence.

[0227] (Other embodiments) Furthermore, this disclosure can also be realized by performing the following process: that is, supplying software (program) that realizes the functions of the above-described embodiment to a system or device via a network or various storage media, and having the computer (or control unit or MPU, etc.) of that system or device read and execute the program code. In this case, the program and the storage media storing the program constitute this disclosure.

[0228] While the present disclosure has been described in detail above based on its preferred embodiments, the present disclosure is not limited to these specific embodiments, and various forms that do not depart from the gist of the disclosure are also included. Some of the embodiments described above may be combined as appropriate.

[0229] Furthermore, each functional unit in each of the above embodiments (each modified example) may or may not be individual hardware. The functions of two or more functional units may be implemented by common hardware. Each of the multiple functions of a single functional unit may be implemented by individual hardware. Two or more functions of a single functional unit may be implemented by common hardware. In addition, each functional unit may or may not be implemented by hardware such as an ASIC, FPGA, or DSP. For example, the device may have a processor and a memory (storage medium) in which a control program is stored. The functions of at least some of the functional units of the device may be implemented by the processor reading and executing the control program from the memory.

[0230] This disclosure can also be implemented by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be implemented by a circuit (e.g., an ASIC) that implements one or more functions.

[0231] Furthermore, in each of the examples described above, "processor" refers to a processor in a broad sense, including general-purpose processors (e.g., CPUs) and specialized processors (e.g., GPUs, ASICs, FPGAs, and programmable logic devices, etc.).

[0232] This embodiment includes the following configurations, methods, and programs.

[0233] [Configuration 1] An information processing device connected to or integrated with a display device, A means for obtaining operations on virtual objects placed in space, The system includes a control means that, when a first operation is acquired by the acquisition means, controls the virtual object to execute a first process corresponding to the first operation, The control means, when certain conditions are met, controls the acquisition means to execute a second process, which is different from the first process, relating to the rearrangement of the virtual object in the space, even if the first operation is acquired by the acquisition means. An information processing device characterized by the following:

[0234] [Configuration 2] If the situation is such that the virtual object is prone to being misoperated, then the predetermined conditions are met. The information processing device according to configuration 1, characterized by the above.

[0235] [Configuration 3] The system further includes a determination means for determining whether or not the virtual object is in a situation where it is likely to be misoperated, If the evaluation means determines that the situation is prone to erroneous operation of the virtual object, then the predetermined conditions are met. An information processing device according to configuration 1 or 2, characterized by the above.

[0236] [Structure 4] The aforementioned predetermined conditions are predetermined conditions relating to at least one of the size, position, and orientation of the virtual object. An information processing device according to any one of configurations 1 to 3.

[0237] [Composition 5] The aforementioned virtual object is a virtual window, If the virtual window is not facing the display device, the predetermined conditions are met. An information processing apparatus according to any one of configurations 1 to 4, characterized by the above.

[0238] [Composition 6] If the size of the virtual object that can be manipulated by the user is smaller than the threshold, then the predetermined condition is met. An information processing device according to any one of configurations 1 to 5, characterized by the above.

[0239] [Composition 7] If at least a portion of the virtual object on which the first operation is performed is hidden by another virtual object, then the predetermined conditions are met. An information processing device according to any one of configurations 1 to 6.

[0240] [Structure 8] If the predetermined conditions are met, the control means, as the second process, changes at least one of the size, position, and orientation of the virtual object. An information processing device according to any one of configurations 1 to 7, characterized by the above.

[0241] [Composition 9] If at least one of the elements of the virtual object, including its size, position, and orientation, is fixed, the fixed element of the virtual object's size, position, and orientation will not be rearranged. An information processing device according to any one of configurations 1 to 8.

[0242] [Configuration 10] The control means controls the execution of the second process so that it becomes a predetermined size. An information processing apparatus according to any one of configurations 1 to 9, characterized by the above.

[0243] [Composition 11] The aforementioned predetermined conditions are conditions relating to the distance to the virtual object, The control means, when the predetermined conditions are met, executes the second process of repositioning the virtual object to a distance from which the user can directly manipulate the virtual object. An information processing apparatus according to any one of configurations 1 to 10, characterized by the above.

[0244] [Composition 12] The system further includes display control means for controlling the display of the virtual object on the display device, The display control means controls the display of the virtual object to differ depending on whether a predetermined condition is met or not. An information processing apparatus according to any one of configurations 1 to 11, characterized by the features described herein.

[0245] [Composition 13] The system further includes a notification means for notifying that, if the aforementioned predetermined conditions are met, the first operation will not be performed even if the first process is carried out. An information processing device according to any one of configurations 1 to 12, characterized by the above.

[0246] [Composition 14] The control means controls the virtual object that has undergone the second process to return to the state it was in before the second process was performed if, after the first process has been performed, no operation has been acquired by the acquisition means for a predetermined period of time. An information processing device according to any one of configurations 1 to 13, characterized by the above.

[0247] [Composition 15] When the virtual object is viewed simultaneously by multiple users, including a first user and a second user, even if the first operation performed by the first user executes a second process relating to the rearrangement of the virtual object in the space, and the virtual object displayed to the first user is rearranged, the virtual object displayed to the second user, who did not perform the first operation, will not be rearranged. An information processing apparatus according to any one of configurations 1 to 14, characterized by the features described herein.

[0248] [Composition 16] When the virtual object is viewed simultaneously by multiple users, including the first user and the second user, The control means executes the second process in response to the first operation by the first user, and while the state of the virtual object displayed to the first user is being rearranged, if the state of the virtual object is changed by the second user, the control means controls the virtual object displayed to the first user to be changed to the state of the virtual object changed by the second user after the first process in response to the first operation by the first user is executed. An information processing apparatus according to any one of configurations 1 to 15, characterized by the above.

[0249] [method] A retrieval step to obtain operations on virtual objects placed in space, a control step of controlling, when a first operation is acquired by said acquisition means, execution of a first process corresponding to said first operation on said virtual object; in said control step, when a predetermined condition is satisfied, control is performed to execute a second process relating to relocation of said virtual object in said space, which is different from said first process, even when the first operation is acquired by said acquisition means An information processing method characterized by the above.

[0250] [Program] A program for causing a computer to function as each means of the information processing apparatus according to any one of 1 to 16 above.

[0251] [System] an acquisition device that acquires an operation on a virtual object arranged in a space; a control device that controls, when a first operation is acquired by said acquisition means, execution of a first process corresponding to said first operation on said virtual object; said control device controls, when a predetermined condition is satisfied, to execute a second process relating to relocation of said virtual object in said space, which is different from said first process, even when the first operation is acquired by said acquisition means An information processing system characterized by the above.

Claims

1. An information processing device connected to or integrated with a display device, A means for obtaining operations on virtual objects placed in space, The system includes a control means that, when a first operation is acquired by the acquisition means, controls the virtual object to execute a first process corresponding to the first operation, The control means, when certain conditions are met, controls the acquisition means to execute a second process, which is different from the first process, relating to the rearrangement of the virtual object in the space, even if the first operation is acquired by the acquisition means. An information processing device characterized by the following:

2. If the situation is such that the virtual object is prone to being misoperated, then the predetermined conditions are met. The information processing apparatus according to feature 1.

3. The system further includes a determination means for determining whether or not the virtual object is in a situation where it is likely to be misoperated, If the evaluation means determines that the situation is prone to erroneous operation of the virtual object, then the predetermined conditions are met. The information processing apparatus according to feature 1.

4. The aforementioned predetermined conditions are predetermined conditions relating to at least one of the size, position, and orientation of the virtual object. The information processing apparatus according to feature 1.

5. The aforementioned virtual object is a virtual window, If the virtual window is not facing the display device, the predetermined conditions are met. The information processing apparatus according to feature 1.

6. If the size of the virtual object that can be manipulated by the user is smaller than the threshold, then the predetermined condition is met. The information processing apparatus according to feature 1.

7. If at least a portion of the virtual object on which the first operation is performed is hidden by another virtual object, then the predetermined conditions are met. The information processing apparatus according to feature 1.

8. If the predetermined conditions are met, the control means, as the second process, changes at least one of the size, position, and orientation of the virtual object. The information processing apparatus according to feature 1.

9. If at least one of the elements of the virtual object, including its size, position, and orientation, is fixed, the fixed element of the virtual object's size, position, and orientation will not be rearranged. The information processing apparatus according to feature 1.

10. The control means controls the execution of the second process so that it becomes a predetermined size. The information processing apparatus according to feature 1.

11. The aforementioned predetermined conditions are conditions relating to the distance to the virtual object, The control means, when the predetermined conditions are met, executes the second process of repositioning the virtual object to a distance from which the user can directly manipulate the virtual object. The information processing apparatus according to feature 1.

12. The system further includes display control means for controlling the display of the virtual object on the display device, The display control means controls the display of the virtual object to differ depending on whether a predetermined condition is met or not. The information processing apparatus according to feature 1.

13. The system further includes a notification means for notifying that, if the aforementioned predetermined conditions are met, the first operation will not be performed even if the first process is carried out. The information processing apparatus according to feature 1.

14. The control means controls the virtual object that has undergone the second process to return to the state it was in before the second process was performed if, after the first process has been performed, no operation has been acquired by the acquisition means for a predetermined period of time. The information processing apparatus according to feature 1.

15. When the virtual object is viewed simultaneously by multiple users, including a first user and a second user, even if the first operation performed by the first user executes a second process relating to the rearrangement of the virtual object in the space, and the virtual object displayed to the first user is rearranged, the virtual object displayed to the second user, who did not perform the first operation, will not be rearranged. The information processing apparatus according to feature 1.

16. When the virtual object is viewed simultaneously by multiple users, including the first user and the second user, The control means executes the second process in response to the first operation by the first user, and while the state of the virtual object displayed to the first user is being rearranged, if the state of the virtual object is changed by the second user, the control means controls the virtual object displayed to the first user to be changed to the state of the virtual object changed by the second user after the first process in response to the first operation by the first user is executed. The information processing apparatus according to feature 1.

17. A retrieval step to obtain operations on virtual objects placed in space, The acquisition means includes a control step that controls the virtual object to perform a first process corresponding to the first operation when the first operation is acquired, In the control step, if predetermined conditions are met, the acquisition means controls the execution of a second process, which is different from the first process, relating to the rearrangement of the virtual object in the space, even if the first operation is acquired. An information processing method characterized by the following:

18. A program for causing a computer to function as each of the means of the information processing apparatus described in claim 1.

19. An acquisition device that acquires operations on virtual objects placed in space, The system includes a control device that, when a first operation is acquired by the acquisition means, controls the virtual object to execute a first process corresponding to the first operation, The control device, when certain conditions are met, controls the acquisition means to execute a second process, which is different from the first process, relating to the rearrangement of the virtual object in the space, even if the first operation is acquired. An information processing system characterized by the following:

Images