Image processing device, method for controlling the image processing device, and program

Abstract

To provide an image processing apparatus, a method for controlling an image processing apparatus, and a program that can increase a possibility of making a transition of a three-dimensional virtual object after operation related to editing into a state intended by an operator.SOLUTION: A head-mounted display 100, for which operation related to editing a three-dimensional virtual object 201 is performed by an operator, comprises: first display means 302 that displays a first path 209 after the operation; second display means 303 that displays a second path 210 different from the first path 209 after the operation; first selection means 304 that allows the operator to select any coordinates on the first path 209; second selection means 305 that allows the operator to select any coordinates on the second path 210; and determination means 307 that determines a state of the three-dimensional virtual object 201 on the basis of the coordinates that the first selection means 304 or the second selection means 305 has allowed the user to select.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, a control method for an image processing apparatus, and a program.

Background Art

[0002] In recent years, technologies for providing an operator with an experience in an environment that combines reality and virtuality, such as augmented reality (AR) and mixed reality (MR), have been developing. Such technologies can show an operator a composite world in which virtual objects are superimposed on a video image of the real world, for example, by a head-mounted display (hereinafter referred to as "HMD") worn on the head. Further, in mixed reality (MR), an operator can edit a three-dimensional virtual object in a sense as if moving, enlarging / shrinking, rotating, etc. the three-dimensional virtual object with his / her own hand by the movement of the hand captured by various sensors. However, it is generally difficult to edit a three-dimensional virtual object as the operator desires. This is because training is required for the operator to move his / her hand precisely, and the accuracy of the sensors also affects the editing result. Therefore, Patent Document 1 discloses a technique for displaying, as candidates for an editing state to transition to after cancellation when an operator desires to cancel an edit, intermediate states of a history based on the operation trajectory of the operator.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, even when displaying intermediate states in the history based on the operator's operation trajectory, there was a problem that the displayed candidates were not those intended by the operator. To explain in more detail, consider hand interaction as one method of operation in 3D space, where the operator's hand is recognized by image recognition and the 3D virtual object is edited based on the hand's movements. In editing using hand interaction, as mentioned above, the operator needs training to move their hand precisely, and depending on the accuracy of the sensor, it may faithfully reproduce even the tremors of the operator's hand, or conversely, it may not faithfully reproduce the operator's hand trajectory. In such situations, even if the state of the 3D virtual object is displayed retrospectively after editing operations, it may not display the candidates intended by the operator.

[0005] The present invention has been made in view of the above-mentioned problems. The present invention aims to provide an image processing apparatus, a control method for the image processing apparatus, and a program that can increase the possibility of transitioning a three-dimensional virtual object after editing operations to a state intended by the operator. [Means for solving the problem]

[0006] To achieve the above objective, the present invention provides an image processing apparatus that allows an operator to perform operations related to editing a three-dimensional virtual object, comprising: a first display means for displaying a first path after the operation; a second display means for displaying a second path different from the first path after the operation; a first selection means for enabling the operator to select any coordinate on the first path; a second selection means for enabling the operator to select any coordinate on the second path; and a determination means for determining the state of the three-dimensional virtual object based on the coordinate selected by the operator using the first selection means or the second selection means. [Effects of the Invention]

[0007] According to the present invention, it is possible to increase the likelihood of transitioning a three-dimensional virtual object to the state intended by the operator after editing operations. [Brief explanation of the drawing]

[0008] [Figure 1] This diagram shows the configuration of the HMD100. [Figure 2] This figure shows an example of an image displayed by the HMD100 in the first embodiment. [Figure 3] This diagram shows the software logic configuration of the HMD100. [Figure 4] This figure shows an example of the route information holding means 301 in the first embodiment. [Figure 5] This is a flowchart for realizing the first display means 302 in the first embodiment. [Figure 6] This is a flowchart for implementing the first selection method 304. [Figure 7] This is a flowchart for implementing the image control means 306. [Figure 8] This is a flowchart for implementing decision-making means 307. [Figure 9] This figure shows an example of an image displayed by the HMD100 in the second embodiment. [Figure 10] This figure shows an example of the route information holding means 301 in the second embodiment. [Figure 11] This is a flowchart for realizing the first display means 302 in the second embodiment. [Modes for carrying out the invention]

[0009] The embodiments of the present invention will now be described in detail with reference to the drawings. However, the configurations described in the following embodiments are merely illustrative, and the scope of the present invention is not limited by the configurations described in each embodiment. For example, each part constituting the present invention can be replaced with any configuration that can perform a similar function. In addition, any additional components may be added. Furthermore, any two or more configurations (features) from each embodiment can be combined. Moreover, in the accompanying drawings, the same or similar configurations are given the same reference numeral, and redundant descriptions are omitted.

[0010] <First Embodiment> The first embodiment will be described below with reference to Figures 1 to 7. Figure 1 shows the configuration of HMD100 as an example of an image processing device to which the present invention can be applied. HMD100 includes a CPU101, ROM102, RAM103, I / F104, and bus105. CPU101, ROM102, RAM103, and I / F104 are each connected by bus105. CPU101 is the processor, ROM102 is read-only memory, RAM103 is random access memory, and I / F104 is an external interface. The operation of HMD100 is realized by executing programs recorded in ROM102 and RAM103.

[0011] RAM103 is also used as work memory to store temporary data for processing performed by CPU101. I / F104 is an interface for communication with the outside world. I / F104 receives data such as real-world image data and operation information, which will be described later. I / F104 also outputs image data displayed on HMD100. Although only one CPU101 is shown in Figure 1, HMD100 may be implemented with multiple processors. Furthermore, HMD100 may include supplementary components such as a graphics processing unit (GPU).

[0012] Furthermore, although Figure 1 only shows RAM 103 as a configuration for holding temporary work memory, secondary and tertiary storage areas in the HMD 100 may be provided on the same or different media. Other media include, for example, hard disk drives (HDDs) and solid-state drives (SSDs). Also, the configuration of the bus 105 is not limited to the configuration shown in Figure 1, and may be a multi-stage configuration connecting each component. Moreover, the image processing device to which the present invention can be applied is not limited to the HMD 100 having all of the components shown in Figure 1. For example, a device connected to the HMD 100 by wire or wirelessly, having some or all of the components shown in Figure 1, may be provided separately from the HMD 100.

[0013] Figure 2 shows an example of an image displayed by the HMD 100 in the first embodiment. In Figure 2, the operator is performing operations related to editing the 3D virtual object 201 using hand interaction. Figure 2(a) shows the state immediately before the editing of the 3D virtual object 201 is decided. Trajectory 202 shows the trajectory of the operator's hand during hand interaction when editing the 3D virtual object 201. In trajectory 202, the point of contact with the 3D virtual object 201 is the operation start point 203 (starting point of the trajectory), and the tip opposite to the operation start point 203 is the operation end point 204 (end point of the trajectory). The operator starts editing the 3D virtual object 201 by selecting the operation start point 203 with their hand. Subsequently, the operator's hand moves according to trajectory 202, reaching the operation end point 204. Operation virtual object 205 shows the shape of the 3D virtual object 201 when the operator's hand movement reaches the operation end point 204. Therefore, in the editing shown in Figure 2, the size of the virtual object 205 during operation, that is, the size of the 3D virtual object 201, is changed according to the operator's hand movements.

[0014] When the operator determines to edit the three-dimensional virtual object 201, the state shown in Fig. 2(a) transitions to the state shown in Fig. 2(b). Fig. 2(b) is a diagram showing the state when the operator determines to edit the three-dimensional virtual object 201. As shown in Fig. 2(b), when the operator determines to edit the three-dimensional virtual object 201, the post-editing virtual object 206 is displayed. The post-editing virtual object 206 is displayed in the size of the three-dimensional virtual object 201 when the operator determines the edit. If the size of the post-editing virtual object 206 does not match the size intended by the operator, the operator cancels the edit of the three-dimensional virtual object 201. Then, the state shown in Fig. 2(b) transitions to the state shown in Fig. 2(c).

[0015] Fig. 2(c) is a diagram showing the state immediately after the operator cancels the edit of the three-dimensional virtual object 201. As shown in Fig. 2(c), immediately after the operator cancels the edit of the three-dimensional virtual object 201, the post-cancellation virtual object 207, the pre-cancellation virtual object 208, the first path 209, and the second path 210 are displayed. The post-cancellation virtual object 207 is displayed in the size of the three-dimensional virtual object 201 immediately before the operator performs the edit, that is, in the size of the three-dimensional virtual object 201 shown in Fig. 2(a). The pre-cancellation virtual object 208 is displayed in the size of the three-dimensional virtual object 201 immediately before the operator cancels the edit, that is, in the size of the post-editing virtual object 206 shown in Fig. 2(b). The first path 209 is the locus 202 shown in Fig. 2(a) and is displayed in thick line.

[0016] The second path 210 is a straight line connecting the operation start point 203 and the operation end point 204 of the first path 209, that is, the locus 202 shown in Fig. 2(a), and is displayed in thick line. Therefore, the second path 210 is different from the first path 209 and shows the shortest distance from the operation start point 203 to the operation end point 204 of the locus 202 shown in Fig. 2(a). In the state where the edit of the three-dimensional virtual object 201 is cancelled, the operator can change the size of the post-cancellation virtual object 207 by moving the hand on the first path 209 or the second path 210.

[0017] If, according to the movement of the operator's hand, the size of the virtual object 207 after cancellation, which has been changed accordingly, contains the size intended by the operator, the operator selects with the hand the coordinates on the first path 209 or the second path 210 that will be the size intended by the operator, and determines to cancel the editing. That is, the operator determines to cancel the editing of the three-dimensional virtual object 201 by selecting with the hand any coordinate on the first path 209 or the second path 210. On the other hand, if, according to the movement of the operator's hand, the size of the virtual object 207 after cancellation, which has been changed accordingly, does not contain the size intended by the operator, the operator ends the cancellation of the editing of the three-dimensional virtual object 201 and performs the editing of the three-dimensional virtual object 201 again. Note that when, according to the movement of the operator's hand, the size of the virtual object 207 after cancellation, which has been changed accordingly, does not contain the size intended by the operator, the operator may not end the cancellation of the editing of the three-dimensional virtual object 201 as described above, but may perform another method. For example, the operator may select with the hand the endpoint of the first path 209 or the second path 210.

[0018] Also, the method of realizing the operation of starting the editing of the three-dimensional virtual object 201, the operation of determining the editing, the operation of canceling the editing, the operation of selecting coordinates (the operation of determining the cancellation of the editing), and the operation of ending the cancellation of the editing does not limit the scope of the present invention. As an example of the method of realizing each operation, a gesture by the operator's hand can be considered. In this case, forms of gestures recognizable by sensors such as pinching and grasping with the operator's hand or by image processing are assigned to the respective operations. Also, each operation may be realized by a gesture by a part other than the operator's hand or by a physical input means. Examples of physical input means include dedicated buttons for operations that the hardware worn on the operator's hand or the like has. Also, for the operation of selecting coordinates and the operation of determining the cancellation of the editing of the three-dimensional virtual object 201, gestures of the same form may be assigned, or gestures of different forms may be assigned.

[0019] Figure 3 shows the software logic configuration of the HMD100. The path information holding means 301 holds trajectory information included in the operation information input from the I / F104. Operation information is information about an operation performed by the operator in a single instance. Trajectory information is information showing the operator's hand movements in an operation performed by the operator in a single instance. Specifically, the trajectory information is the information showing the trajectory 202 in Figure 2(a), that is, the information showing the first path 209 in Figure 2(b). In this way, if the path information holding means 301 holds the trajectory information included in the operation information, it becomes possible to undo editing, specifically, to undo editing of the 3D virtual object 201.

[0020] Furthermore, the path information holding means 301 may hold information indicating the operator's hand movements during operations performed by the operator multiple times by holding trajectory information contained in each of the multiple operation information. Also, the definition of "once" may differ for operations performed by the operator at one time. For example, the operation period corresponding to "once" may be determined by the operator's gesture. Alternatively, the period sandwiched between periods in which the operation is stopped for a certain amount of time or longer may be considered the operation period corresponding to "once". In this case, the determination of whether the operation has been stopped for a certain amount of time or longer may be performed using a threshold for the amount of movement.

[0021] The first display means 302 displays the trajectory information held in the trajectory information holding means 301 as a trajectory when the operator performs an operation to cancel editing. Here, the trajectory information held in the trajectory information holding means 301 is included in the operation information, which is information about an operation performed by the operator at one time, as described above. Therefore, the first display means 302 displays the trajectory of the operator's hand movements in an operation performed by the operator at one time. Specifically, the first display means 302 displays the first trajectory 209 shown in Figure 2(c) above. Note that the trajectory information displayed by the first display means 302 is not limited to what is included in one operation information, but may be included in multiple operation information, for example. In this case, the first display means 302 displays the trajectory of the operator's hand movements for each operation performed by the operator several times using multiple lines.

[0022] The second display means 303 displays a straight line connecting the start and end points of the trajectory information held in the trajectory information holding means 301 as a path when the operator performs an operation to cancel editing. Here, the trajectory information held in the trajectory information holding means 301 is included in the operation information, which is information about an operation performed by the operator at one time, as described above. Therefore, the second display means 303 displays a single straight line connecting the start and end points of the hand movements performed by the operator in a single operation as a path. Specifically, the second display means 303 displays the second path 210 shown in Figure 2(c) above. However, the trajectory information handled by the second display means 303 is not limited to that included in one operation information, but may be included in multiple operation information, for example. In this case, the second display means 303 may display a single straight line connecting the start point of the operator's hand movements in the first operation and the end point of the operator's hand movements in the last operation as a path, among operations performed by the operator several times. Furthermore, the second display means 303 may display multiple straight lines as paths, connecting the starting and ending points of the operator's hand movements in each operation.

[0023] The first selection means 304 allows the operator to select any coordinate by selecting an arbitrary position on the path displayed by the first display means 302. The second selection means 305 allows the operator to select any coordinate by selecting an arbitrary position on the path displayed by the second display means 303. In other words, the operator can select any coordinate from the path displayed by the first display means 302 and the path displayed by the second display means 303 using either the first selection means 304 or the second selection means 305. Specifically, the operator can select any coordinate from the first path 209 and the second path 210 shown in Figure 2(c).

[0024] The image control means 306 modifies the processing of the image displayed on the HMD 100 until the operator selects a coordinate. This modification switches the level of detail of the polygon display, the placement of the light source, and whether or not post-processing is required, depending on the operator's hand movement speed, thereby ensuring the visual effect in the 3D space of the HMD 100 without affecting the processing speed of editing the 3D virtual object 201. Note that the trigger for the image processing modification is not limited to the operator's hand movement speed, but may also be, for example, the operator's hand position, posture, gestures, or other actions of the operator. Furthermore, the image processing modification is not limited to those described above, but may also involve switching or not performing calculations commonly used in image processing, such as calculating depth information, extracting feature points, or detecting planes.

[0025] When the operator selects any coordinates, the determination means 307 determines the state to which the object being edited will transition as a result of the undo editing operation, based on the coordinates selected by the operator. Specifically, after the operator performs an operation to undo editing of the 3D virtual object 201, the determination means 307 determines the size to which the 3D virtual object 201 will transition, based on the coordinates selected by the operator. Furthermore, the determination means 307 reverts the image processing that the image control means 306 had modified.

[0026] Figure 4 shows an example of the path information holding means 301 in the first embodiment. As shown in Figure 4, the path information holding means 301 holds trajectory information in the form of continuous three-dimensional coordinate information. As described above, the trajectory information is information that shows the movement of the operator's hand during an operation performed by the operator in a single step. The granularity of the trajectory information is not limited to a specific granularity; for example, it may be the smallest granularity that can be represented as a three-dimensional space, or it may be a granularity that depends on the device or sensor. Furthermore, the granularity of the trajectory information may be changed each time trajectory information is acquired, or it may be changed during the acquisition of trajectory information. In addition, the trajectory information may include three-dimensional vector information.

[0027] Next, the processing flow performed by the HMD100 will be explained using the flowcharts in Figures 5 to 8. The flowcharts in Figures 5 to 8 (control method for the image processing device) are realized by loading the program recorded in ROM 102 into RAM 103 and executing it on CPU 101 (computer). This is also true for the flowchart in Figure 11, which will be discussed later.

[0028] Figure 5 is a flowchart for realizing the first display means 302 in the first embodiment (first display step). The flowchart in Figure 5 starts when the operator performs an operation to cancel editing, as described in the explanation of the software logic configuration in Figure 3 above. In step S501, the CPU 101 draws a trajectory on the three-dimensional space of the HMD 100 based on the trajectory information held by the path information holding means 301. Specifically, the CPU 101 displays the first path 209 shown in Figure 2(c) above. After that, the flowchart in Figure 5 ends.

[0029] Furthermore, in step S501, the trajectory drawn by the CPU 101 is not limited to the trajectory drawn based on the trajectory information. The CPU 101 may also draw the trajectory with a specific thickness to make it easier for the operator to select coordinates, or it may change the thickness of the trajectory depending on the situation. For example, the CPU 101 may draw a thinner trajectory the closer the distance to the operator is in the 3D space of the HMD 100, and a thicker trajectory the further away the operator is. The CPU 101 may also change the 3D coordinate information of the trajectory information depending on the situation. For example, the CPU 101 may change the 3D coordinate information of the trajectory information to absorb the error characteristics of the sensor, or it may change the 3D coordinate information of the trajectory information according to the image resolution.

[0030] The flowchart in Figure 5 can also be used as a flowchart for realizing the second display means 303 (second display step). In this case, in step S501, the CPU 101 draws a straight line connecting the start and end points of the three-dimensional coordinate information of the trajectory information held by the path information holding means 301 on the three-dimensional space of the HMD 100. Specifically, the CPU 101 displays the second path 210 shown in Figure 2(c) above. The rest is the same as in the flowchart for realizing the first display means 302 described above.

[0031] Figure 6 is a flowchart for implementing the first selection means 304 (first selection step). The flowchart in Figure 6 starts when the trajectory is drawn by the first display means 302. In step S601, the CPU 101 acquires operator operation information via the I / F 104. In step S602, the CPU 101 determines whether the operator has selected any coordinates on the trajectory drawn by the first display means 302. Specifically, the CPU 101 determines whether the operator has selected any coordinates on the first path 209 shown in Figure 2(c). This determination is made based on the operator operation information acquired in step S601. If the CPU 101 determines that the operator has not selected any coordinates on the trajectory drawn by the first display means 302, the process returns to step S601.

[0032] In response to this, if the CPU 101 determines that the operator has selected any coordinate on the trajectory drawn by the first display means 302, the flowchart in Figure 6 terminates. The flowchart in Figure 6 may also terminate if the CPU 101 determines that the operator has selected any coordinate on the straight line drawn by the second display means 303. In other words, the flowchart in Figure 6 may also terminate if the CPU 101 determines that the operator has selected any coordinate on the second path 210 shown in Figure 2(c). This determination is also made based on the operator's operation information obtained in step S601.

[0033] Furthermore, the flowchart in Figure 6 can also be used as a flowchart for realizing the second selection means 305 (second selection step). In this case, the flowchart in Figure 6 starts when a straight line is drawn by the second display means 303. Step S601 is as described above. In step S602, the CPU 101 determines whether the operator has selected any coordinates on the straight line drawn by the second display means 303. Specifically, the CPU 101 determines whether the operator has selected any coordinates on the second path 210 shown in Figure 2(c) above. This determination is made based on the operator's operation information obtained in step S601. If the CPU 101 determines that the operator has not selected any coordinates on the straight line drawn by the second display means 303, the process returns to step S601.

[0034] In response to this, if the CPU 101 determines that the operator has selected any coordinate on the straight line drawn by the second display means 303, the flowchart in Figure 6 terminates. The flowchart in Figure 6 may also terminate if the CPU 101 determines that the operator has selected any coordinate on the trajectory drawn by the first display means 302. Specifically, the flowchart in Figure 6 may also terminate if the CPU 101 determines that the operator has selected any coordinate on the first path 209 shown in Figure 2(c). This determination is also made based on the operator's operation information obtained in step S601.

[0035] Figure 7 is a flowchart for realizing the image control means 306. The flowchart in Figure 7 starts before the operator selects any coordinates, as described in the explanation of the software logic configuration in Figure 3 above. More specifically, the flowchart in Figure 7 starts at some point between the time the operator cancels editing of the 3D virtual object 201 and the time the operator selects any coordinates. In step S701, the CPU 101 determines whether the operator's hand movement speed is less than a predetermined value. The CPU 101 calculates the operator's hand movement speed by recognizing the operator's hand using sensors and image processing.

[0036] If the CPU 101 determines that the operator's hand movement speed is less than a specified value, the process proceeds to step S702. Conversely, if the CPU 101 determines that the operator's hand movement speed is not less than a specified value, the process proceeds to step S705. Note that while step S701 uses the operator's hand movement speed being less than a specified value as the condition for proceeding to step S702, the condition may also be that the operator's hand movement speed is equal to or less than a specified value. In step S702, the CPU 101 sets the polygon display detail level to the highest level in the three-dimensional space of the HMD 100. Specifically, the CPU 101 sets the polygon display detail level to the highest level for the virtual object 207 after cancellation, the virtual object 208 before cancellation, the first path 209, and the second path 210 shown in Figure 2(c) above.

[0037] In step S703, the CPU 101 places a light source in the three-dimensional space of the HMD 100. Specifically, the CPU 101 places the light source toward the virtual object 207 after cancellation, the virtual object 208 before cancellation, the first path 209, and the second path 210 shown in Figure 2(c). In step S704, the CPU 101 sets whether or not post-processing is required to perform post-processing. After that, the flowchart in Figure 7 ends. In step S705, the CPU 101 sets the polygon display detail level to the minimum in the three-dimensional space of the HMD 100. Specifically, the CPU 101 sets the polygon display detail level to the minimum for the virtual object 207 after cancellation, the virtual object 208 before cancellation, the first path 209, and the second path 210 shown in Figure 2(c). In step S706, the CPU 101 sets whether or not post-processing is required to not perform post-processing. After that, the flowchart in Figure 7 ends.

[0038] Note that the polygon display detail level set in step S702 is not limited to the highest level, but may be other than the highest level. Similarly, the polygon display detail level set in step S705 is not limited to the lowest level, but may be other than the lowest level. Also, the order in which steps S702, S703, and S704 are performed is not limited to the order shown in the flowchart of Figure 7, but may be in a different order. Similarly, the order in which steps S705 and S706 are performed is not limited to the order shown in the flowchart of Figure 7, but may be in a different order.

[0039] Figure 8 is a flowchart for implementing the decision means 307 (decision process). The flowchart in Figure 8 starts when the operator selects any coordinate, as described in the explanation of the software logic configuration in Figure 3 above. In step S801, the CPU 101 determines the state to which the object to be edited will transition based on the coordinate selected by the operator. Specifically, the CPU 101 displays the undo virtual object 207 shown in Figure 2(c) above with the size determined based on the coordinate selected by the operator. In step S802, the CPU 101 resets the image processing that was changed in the flowchart of Figure 7 above back to its initial settings. After that, the flowchart in Figure 8 ends.

[0040] As described above, in the first embodiment, when the operator performs an operation to undo editing of the 3D virtual object 201 in the HMD 100, the undo virtual object 207, which corresponds to the 3D virtual object 201 immediately before the operator performed the editing, is displayed. Furthermore, the HMD 100 displays a first path 209 showing the trajectory 202 of the hand during the hand interaction performed by the operator during editing, and a second path 210 showing the shortest distance from the operation start point 203 to the operation end point 204 of the trajectory 202. When the operator selects coordinates by moving their hand along the first path 209 or the second path 210, the size of the undo virtual object 207 is changed based on the selected coordinates. In this way, the HMD 100 can increase the likelihood of transitioning the 3D virtual object 201 after editing operations to the state intended by the operator.

[0041] In the above explanation, the second path 210 was defined as a straight line connecting the operation start point 203 and the operation end point 204 of the first path 209, but this is not the only definition. For example, the second path 210 may be a path represented by some function from the operation start point 203 to the operation end point 204 of the first path 209, or it may be a path represented by a combination of multiple different functions. Even in such cases, the HMD 100 can increase the likelihood of transitioning the 3D virtual object 201 after the editing operation to the state intended by the operator.

[0042] <Second Embodiment> The second embodiment will be described below with reference to Figures 9 to 11. Here, the differences from the first embodiment will be the main focus of the description. Figure 9 is a diagram showing an example of an image displayed by the HMD 100 in the second embodiment. Figures 9(a) and 9(b) are the same as Figures 2(a) and 2(b) above, so their explanation will be omitted. Figure 9(c) is a diagram showing the state immediately after the operator undoes the editing of the 3D virtual object 201. As shown in Figure 9(c), immediately after the operator undoes the editing of the 3D virtual object 201, in addition to the undoed virtual object 207, the pre-undoed virtual object 208, the first path 209, and the second path 210 described above, the first additional path 901 and the second additional path 902 are displayed.

[0043] The first additional route 901 represents a line segment that extends from the first route 209 in the direction of travel at the operation termination point 204 when proceeding to the operation termination point 204 according to the first route 209. The second additional route 902 represents a line segment that extends from the second route 210 in the direction of travel at the operation termination point 204 when proceeding to the operation termination point 204 according to the second route 210.

[0044] When the editing of the 3D virtual object 201 is undone, the operator can change the size of the undone virtual object 207 by moving their hand on the first additional path 901 or the second additional path 902, in addition to the first path 209 or the second path 210 described above. If the operator's desired size is among the sizes of the undone virtual object 207 changed by the operator's hand movements, the operator decides to undone the editing by manually selecting the coordinates on the first path 209 or the second path 210 that result in the desired size. Alternatively, the operator decides to undone the editing by manually selecting the coordinates on the first additional path 901 or the second additional path 902 that result in the desired size. In other words, the operator decides to undone the editing of the 3D virtual object 201 by manually selecting any of the coordinates on the first path 209, the second path 210, the first additional path 901, or the second additional path 902.

[0045] Figure 10 shows an example of the route information holding means 301 in the second embodiment. In addition to the trajectory information described in Figure 4 above, the route information holding means 301 holds endpoint vector information. The endpoint vector information is three-dimensional coordinate information indicating the position of the tip of the first additional route 901 that is on the opposite side of the operation termination point 204 in the first additional route 901.

[0046] Figure 11 is a flowchart for realizing the first display means 302 in the second embodiment. The flowchart in Figure 11 starts when the operator performs an operation to undo editing, as described in the explanation of the software logic configuration in Figure 3 above. Step S1101 is the same as step S501 in Figure 5 above, so the explanation is omitted. In step S1102, the CPU 101 draws an extended trajectory in the three-dimensional space of the HMD 100. An extended trajectory is a straight line connecting the endpoint of the three-dimensional coordinate information of the trajectory information held by the path information holding means 301 and one point of the three-dimensional coordinate information of the endpoint vector information. Specifically, the CPU 101 displays the first additional path 901 shown in Figure 9(c) above. After that, the flowchart in Figure 11 ends. In the second embodiment, the length of the first additional path 901 is not limited. For example, the length of the first additional path 901 may be a specific length, or it may be a length calculated based on the length of the first additional path 901 and the speed at which the operator's hand moves.

[0047] The flowchart in Figure 11 can also be used as a flowchart for realizing the second display means 303. In this case, in step S1101, the CPU 101 draws a straight line connecting the start and end points of the three-dimensional coordinate information of the trajectory information held by the path information holding means 301 in the three-dimensional space of the HMD 100. Specifically, the CPU 101 displays the second path 210 shown in Figure 9(c). In step S1102, the CPU 101 draws a line segment extending from the end point of the straight line drawn in step S1101 in the three-dimensional space of the HMD 100. Specifically, the CPU 101 displays the second additional path 902 shown in Figure 9(c). The rest is the same as in the flowchart for realizing the first display means 302 described above.

[0048] As described above, in the second embodiment, when the operator performs an operation to undo editing of the 3D virtual object 201 in the HMD 100, the undone virtual object 207, the first path 209, and the second path 210 are displayed, similar to the first embodiment. Furthermore, the HMD 100 displays the first additional path 901, which is an extension of the first path 209, and the second additional path 902, which is an extension of the second path 210. When the operator selects coordinates by moving their hand on the first path 209, the second path 210, the first additional path 901, or the second additional path 902, the size of the undone virtual object 207 is changed based on the selected coordinates. In this way, the HMD 100 can increase the likelihood of transitioning the 3D virtual object 201 after editing operations to the state intended by the operator.

[0049] <Other> Although preferred embodiments of the present invention have been described above, the present invention is not limited to the embodiments described above, and various modifications and changes are possible within the scope of its gist. The present invention can also be realized by supplying a program that implements one or more of the functions of the above embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. Furthermore, the present invention can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0050] Furthermore, in each embodiment, the HMD100 has a first display means 302 and a second display means 303, and a first selection means 304 and a second selection means 305, but it may have three or more display means and selection means.

[0051] Each embodiment of the disclosure includes the following configurations, methods, and programs. (Configuration 1) An image processing apparatus for which an operator performs operations related to editing a three-dimensional virtual object, comprising: a first display means for displaying a first path after the operation; a second display means for displaying a second path different from the first path after the operation; a first selection means for enabling the operator to select any coordinate on the first path; a second selection means for enabling the operator to select any coordinate on the second path; and a determination means for determining the state of the three-dimensional virtual object based on the coordinate selected by the operator using the first selection means or the second selection means. (Configuration 2) The image processing apparatus according to Configuration 1, characterized in that the operation is an operation to undo the editing of the three-dimensional virtual object. (Configuration 3) The image processing apparatus according to Configuration 1 or 2, characterized in that the editing of the three-dimensional virtual object is performed using the trajectory of the operator's hand in hand interaction. (Configuration 4) The image processing apparatus according to Configuration 3, characterized in that the first display means displays the trajectory as the first path. (Configuration 5) The image processing apparatus according to Configuration 3 or 4, characterized in that the second display means displays a straight line connecting the start and end points of the trajectory as the second path. (Configuration 6) The image processing apparatus according to Configuration 3 or 4, characterized in that the second display means displays the path from the starting point to the ending point of the trajectory represented by a function as the second path. (Configuration 7) The image processing apparatus according to Configuration 3 or 4, characterized in that the second display means displays the path from the starting point to the ending point of the trajectory, which is represented by a combination of multiple different functions, as the second path. (Configuration 8) The first display means displays the line segment extending from the endpoint of the first path as the first additional path, The image processing apparatus according to any one of configurations 1 to 7, characterized in that the first additional path extends in the direction of travel at the endpoint when traveling from the starting point of the first path to the endpoint according to the first path. (Configuration 9) The second display means displays the line segment extending from the endpoint of the second path as a second additional path. The image processing apparatus according to any one of configurations 1 to 8, characterized in that the second additional path extends in the direction of travel at the endpoint when traveling from the starting point of the second path to the endpoint according to the second path. (Configuration 10) An image processing apparatus according to any one of Configurations 1 to 9, characterized by comprising an image control means for controlling the image of the three-dimensional virtual object based on the movement speed of the operator's hand. (Configuration 11) The image processing apparatus according to Configuration 10, characterized in that the image control means changes the display detail of the three-dimensional virtual object. (Configuration 12) The image processing apparatus according to Configuration 10 or 11, characterized in that the image control means installs a light source that illuminates the three-dimensional virtual object. (Configuration 13) The apparatus according to any one of Configurations 10 to 12, characterized in that the image control means switches whether or not post-processing is required for the image of the three-dimensional virtual object. (Method 1) A control method for an image processing device in which an operator performs operations related to editing a three-dimensional virtual object, A first display step in which the first route is displayed after the above operation, A second display step, which displays a second route different from the first route after the operation described above, A first selection step that allows the operator to select any coordinate on the first path, A second selection step that allows the operator to select any coordinate on the second path, A control method for an image processing apparatus, comprising: a determination step of determining the state of the three-dimensional virtual object based on coordinates selected by the operator in the first selection step or the second selection step. (Program 1) A program for causing a computer to execute each of the means of the image processing apparatus described in any one of Configurations 1 to 13. [Explanation of symbols]

[0052] 100 HDM (Image Processing Unit) 201 3D virtual object 209 Route 1 210 Second Route 302 First display means 303 Second means of display 304 First Choice 305 Second alternative 307 Decision-making method

Claims

1. An image processing device in which an operator performs operations related to editing a three-dimensional virtual object, A first display means for displaying the first route after the aforementioned operation, A second display means that displays a second route different from the first route after the aforementioned operation, A first selection means that allows the operator to select any coordinate on the first path, A second selection means that allows the operator to select any coordinate on the second path, An image processing apparatus comprising: a determination means for determining the state of the three-dimensional virtual object based on coordinates selected by the operator using the first selection means or the second selection means.

2. The image processing apparatus according to claim 1, characterized in that the operation is an operation to undo the editing of the three-dimensional virtual object.

3. The image processing apparatus according to claim 1, characterized in that the editing of the three-dimensional virtual object is performed using the trajectory of the operator's hand in hand interaction.

4. The image processing apparatus according to claim 3, characterized in that the first display means displays the trajectory as the first path.

5. The image processing apparatus according to claim 3, characterized in that the second display means displays a straight line connecting the start and end points of the trajectory as the second path.

6. The image processing apparatus according to claim 3, characterized in that the second display means displays the path from the starting point to the ending point of the trajectory represented by a function as the second path.

7. The image processing apparatus according to claim 3, characterized in that the second display means displays the path from the starting point to the ending point of the trajectory, which is represented by a combination of multiple different functions, as the second path.

8. The first display means displays the line segment extending from the endpoint of the first path as the first additional path. The image processing apparatus according to claim 1, characterized in that the first additional path extends in the direction of travel at the endpoint when traveling from the starting point of the first path to the endpoint according to the first path.

9. The second display means displays the line segment extending from the endpoint of the second path as a second additional path. The image processing apparatus according to claim 1, characterized in that the second additional path extends in the direction of travel at the endpoint when traveling from the starting point of the second path to the endpoint according to the second path.

10. The image processing apparatus according to claim 1, further comprising image control means for controlling the image of the three-dimensional virtual object based on the speed of movement of the operator's hand.

11. The image processing apparatus according to claim 10, characterized in that the image control means changes the display detail of the three-dimensional virtual object.

12. The image processing apparatus according to claim 10, characterized in that the image control means installs a light source that illuminates the three-dimensional virtual object.

13. The image processing apparatus according to claim 10, characterized in that the image control means switches whether or not post-processing is required for the image of the three-dimensional virtual object.

14. A control method for an image processing device in which an operator performs operations related to editing a three-dimensional virtual object, A first display step in which the first route is displayed after the above operation, A second display step, which displays a second route different from the first route after the above operation, A first selection step that allows the operator to select any coordinate on the first path, A second selection step that allows the operator to select any coordinate on the second path, A control method for an image processing apparatus, comprising: a determination step of determining the state of the three-dimensional virtual object based on coordinates selected by the operator in the first selection step or the second selection step.

15. A program for causing a computer to execute each of the means of the image processing apparatus described in claim 1.

Images