Electronic device

Abstract

To solve the problem that it is impossible to appropriately control the display position or display effect of CG according to a situation or the purpose of a user when using the CG for learning in the use of a tool which is difficult to handle such as a musical instrument.SOLUTION: An electronic apparatus including an image capturing unit and a display unit includes an estimation unit configured to estimate a position and orientation of a tool used by a user based on an image captured by the image capturing unit, an acquisition unit configured to acquire an action of a hand of a person different from the user when the person uses the tool, and a display unit configured to display CG of the hand on the display unit based on the position and orientation acquired by the estimation unit and the action acquired by the acquisition unit. The display means displays CG of the hand at a position not overlapping the hand of the user on a tool.SELECTED DRAWING: Figure 17

Description

[Technical Field]

[0001] The present invention relates to an electronic device, and more particularly to an electronic device that displays CG images of a model hand. [Background technology]

[0002] Conventionally, a learning method using CG has been known as one of the learning methods for tools that are difficult to handle, such as musical instruments. For example, Patent Document 1 proposes showing a model CG to a user wearing an HMD (Head Mounted Display). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2011-215856 Summary of the Invention [Problem to be solved by the invention]

[0004] However, while the conventional technology disclosed in the above-mentioned patent document can show exemplary hand movements using CG, it has problems such as being unable to appropriately control the display position and display effects of the CG depending on the situation and the user's purpose. [Means for solving the problem]

[0005] In order to solve the above problem, the electronic device of the present invention is an electronic device having an imaging unit and a display unit, and includes: an estimation means for estimating the position and orientation of a tool used by a user based on an image captured by the imaging unit; an acquisition means for acquiring the hand movements of a person other than the user when the tool is used by the person; and a display means for displaying a CG of a hand on the display unit based on the position and orientation acquired by the estimation means and the movement acquired by the acquisition means, wherein the display means displays the CG of the hand at a position on the tool that does not overlap with the user's hand. [Effects of the Invention]

[0006] According to the present invention, it is possible to appropriately display a CG model of a hand. [Brief explanation of the drawings]

[0007] [Figure 1] 1 is a schematic diagram showing an example of a system configuration according to a first embodiment. [Figure 2] 1 is a block diagram showing an example of the configuration of an imaging and displaying device as an example of a device to which the configuration of the first embodiment can be applied. [Figure 3] 1 is a block diagram showing an example of the configuration of an imaging device as an example of a device to which the configuration of the first embodiment can be applied. [Figure 4] FIG. 1 is a diagram illustrating a functional configuration of a system according to a first embodiment. [Figure 5] 4 is a flowchart showing the flow of processing performed by the imaging and display device 101 according to the first embodiment. [Figure 6] 5 is a flowchart showing a position and orientation estimation process according to the first embodiment. [Figure 7] 6 is a flowchart showing a CG rendering process according to an operation mode in the first embodiment. [Figure 8] 10 is a flowchart showing a drawing process in operation mode 1 in the first embodiment. [Figure 9] 10 is a flowchart showing a drawing process in operation mode 2 in the first embodiment. [Figure 10] 10 is a flowchart showing a drawing process in operation mode 3 in the first embodiment. [Figure 11] 10 is a flowchart showing operation option processing in operation modes 1 and 2 in the first embodiment. [Figure 12] 4 is a flowchart showing the flow of processing performed by the image capture device 102 in the first embodiment. [Figure 13] 10 is an example of an operation mode setting screen in the first embodiment. [Figure 14]10 is a diagram illustrating an example of an operation mode change setting screen according to the first embodiment. [Figure 15] 11 is a table showing details of the control for drawing at a position close to the user's hand in the flowchart of FIG. 10 according to the first embodiment. [Figure 16] This is a display example when the process expressed in the first line of the detailed control diagram 15 for drawing at a position close to the user's hand in the first embodiment is performed. [Figure 17] This is a display example when the process expressed in the second line of the detailed control diagram 15 for drawing at a position close to the user's hand in the first embodiment is performed. [Figure 18] This is a display example when the process expressed in the third line of the detailed control diagram 15 for drawing at a position close to the user's hand in the first embodiment is performed. [Figure 19] This is a display example when the process expressed in the fourth line of the detailed control diagram 15 for drawing at a position close to the user's hand in the first embodiment is performed. [Figure 20] This is a display example when the process expressed in the fifth line of the detailed control diagram 15 for drawing at a position close to the user's hand in the first embodiment is performed. [Figure 21] This is a display example when the process expressed in the sixth line of the detailed control diagram 15 for drawing at a position close to the user's hand in the first embodiment is performed. [Figure 22] This is a display example when the process expressed in the seventh line of the detailed control diagram 15 for drawing at a position close to the user's hand in the first embodiment is performed. [Figure 23] This is a display example when the process expressed in lines 8 to 10 of detailed diagram 15 of the control for drawing at a position close to the user's hand in the first embodiment is performed. [Figure 24] 11 is a display example when control is exercised to draw at positions above and below an octave in the flowchart of FIG. 10 in the first embodiment. [Figure 25] 12 is a display example when control is performed to display the overlapping hands separately in the flowchart of FIG. 11 according to the first embodiment. [Figure 26]12 is a display example when control for normal display in the flowchart of FIG. 11 is performed in the first embodiment. [Figure 27] 12 is a display example when control is performed to give priority to displaying the hand that is being gazed at in the flowchart of FIG. 11 in the first embodiment. [Figure 28] FIG. 10 is a schematic diagram showing an example of a system configuration according to a second embodiment. [Figure 29] FIG. 10 is a diagram illustrating a functional configuration of a system according to a second embodiment. [Figure 30] 10 is a flowchart showing a drawing process in operation mode 1 in the second embodiment. [Figure 31] 10 is a flowchart showing a drawing process in an operation mode 2 according to the second embodiment. [Figure 32] 10 is a flowchart showing a drawing process in operation mode 3 in the second embodiment. [Figure 33] 10 is a diagram illustrating an example of an operation mode setting screen according to the second embodiment. [Figure 34] 10 is a display example when control is performed to draw at a position close to the user's hand in the second embodiment. [Figure 35] 10 is a display example when control is performed to draw at a position away by a fixed distance in the second embodiment. [Figure 36] 12 is a display example when control for normal display in the flowchart of FIG. 11 is performed in the second embodiment. [Figure 37] 12 is a display example when control is performed to display the overlapping hands separately in the flowchart of FIG. 11 according to the second embodiment. [Figure 38] 12 is a display example when control is performed to give priority to displaying the hand that is receiving the gaze in the flowchart of FIG. 11 according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] [First embodiment] A first preferred embodiment of the present invention will now be described with reference to the drawings.

[0009] 1 is a schematic diagram showing the system configuration of the first embodiment. The system is made up of an image capturing and displaying apparatus 101 and an image capturing apparatus 102.

[0010] The imaging and display device 101 includes a display unit and an imaging unit, and is capable of communicating with the imaging device 102. It can detect the user's line of sight by acquiring an image of the user's eyes observing the display unit, and acquires tool use actions such as fingering analysis results received from the imaging device 102, and uses this to perform display control suitable for the user and output it to the display unit. An example of the imaging and display device 101 is an HMD, but it is not limited to this, and electronic devices such as tablets with similar functions may also be used.

[0011] The imaging device 102 is capable of mutual communication with the imaging and display device 101, analyzes the fingering of the person performing in the video, and transmits the analysis results to the imaging and display device 101. The imaging device 102 may be equipped with a display unit such as a liquid crystal display and operation members such as a shutter button. An example of the imaging device 102 is a digital camera, but is not limited to this.

[0012] In the electronic device system of the first embodiment, a user uses a keyboard instrument as a tool, and a model CG is displayed on the keyboard, which is the tool, superimposed on a real image. CG, such as the fingering of the instrument analyzed by the imaging device 102, is matched with the position of the instrument in the user's real life, and rendering control is performed to impart rendering effects according to the user's settings. The CG is superimposed on the real image and output to the display unit of the imaging and display device 101, which the user can view.

[0013] FIG. 2 is a block diagram showing an example of the configuration of the image capturing and displaying device 101 according to the first embodiment.

[0014] The CPU 201 executes the programs stored in the nonvolatile memory 203 described above to realize each process of this embodiment, which will be described later.

[0015] The nonvolatile memory 203 is an electrically erasable and recordable memory, and may be, for example, a Flash-ROM. The nonvolatile memory 203 stores constants, programs, etc. for the operation of the CPU 201. The programs referred to here are computer programs for executing various flowcharts described later in this embodiment.

[0016] The main memory 202 is, for example, a RAM, and constants and variables for the operation of the CPU 201, programs read from the nonvolatile memory 203, and the like are loaded therein.

[0017] The display unit 204 uses a device such as an LCD or organic EL display that displays images and various information.

[0018] The line-of-sight detection unit 205 uses a sensor capable of line-of-sight detection, and acquires the detection result by detecting the line of sight of the user observing the display unit 204 .

[0019] The communication unit 206 can be connected to a wireless LAN (Local Area Network) or the Internet, and transmits and receives video signals, audio signals, and control signals. The communication unit 206 can transmit various data including images (including live view images) captured by the imaging unit 207, and can also receive images and various other information from external devices. The communication unit 206 acquires tool use actions such as fingering analysis results by transmitting and receiving them from the imaging device 102.

[0020] The imaging unit 207 is an imaging element configured with a CCD, CMOS element, or the like that converts an optical image into an electrical signal.

[0021] The image processing unit 208 performs predetermined pixel interpolation, resizing such as reduction, and color conversion. It also performs predetermined calculations using captured image data. The CPU 201 controls exposure and distance measurement based on the calculation results obtained by the image processing unit 208. This allows AF (autofocus) processing and AE (autoexposure) processing to be performed.

[0022] FIG. 3 is a block diagram showing an example of the configuration of the image capture device 102 according to the first embodiment.

[0023] The CPU 301 executes the programs stored in the nonvolatile memory 303 described above to realize each process of this embodiment, which will be described later.

[0024] The nonvolatile memory 303 is an electrically erasable and recordable memory, such as a Flash-ROM. The nonvolatile memory 303 stores constants, programs, etc. for the operation of the CPU 301. The programs referred to here are computer programs for executing various flowcharts described later in this embodiment.

[0025] The main memory 302 is, for example, a RAM, and constants and variables for the operation of the CPU 301, programs read from the nonvolatile memory 303, and the like are loaded therein.

[0026] The communication unit 304 can be connected to a wireless local area network (LAN) or the Internet, and transmits and receives video signals, audio signals, and control signals. The communication unit 304 can transmit various data including images (including live view images) captured by the imaging unit 305, and can also receive images and various other information from external devices.

[0027] The imaging unit 305 is an imaging element configured with a CCD, CMOS element, or the like that converts an optical image into an electrical signal.

[0028] The image processing unit 306 performs predetermined pixel interpolation, resizing such as reduction, and color conversion. It also performs predetermined calculations using captured image data. The CPU 301 controls exposure and distance measurement based on the calculation results obtained by the image processing unit 306. This allows AF (autofocus) processing and AE (autoexposure) processing to be performed.

[0029] FIG. 4 is a diagram showing the functional configuration of the system according to the first embodiment.

[0030] Each function of the image capturing and displaying device 101 is realized by the CPU 201. The image capturing and displaying device 101 is made up of a display control unit 401, a position and orientation estimation unit 402, a control unit 403, and CG data 404.

[0031] A display control unit 401 controls the display on the display unit 204 in accordance with various control processes of a control unit 403, which will be described later.

[0032] The position and orientation estimation unit 402 estimates the three-dimensional position and orientation of the image capturing and displaying device 101 and the keyboard instrument used by the user based on the images obtained from the image capturing unit 207 and image processing unit 208 .

[0033] The control unit 403 controls each process of the image capturing and displaying device 101, which will be described later.

[0034] The CG data 404 stores CG data for displaying CG by performing processing, described later, by the control unit 403 on the analysis results received from the image capture device 102.

[0035] Each function of the imaging device 102 is realized by the CPU 301 and is mainly composed of a fingering analysis unit 405 , a trained model 406 , and a control unit 407 .

[0036] The fingering analysis unit 405 estimates fingering and the like for playing a keyboard instrument from the video obtained from the imaging unit 305 and image processing unit 306 using a trained model 406 (described later).

[0037] The trained model 406 is a trained model used by the fingering analysis unit 405 when estimating fingering and the like for playing a keyboard instrument from video.

[0038] The control unit 407 controls each process of the imaging device 102, which will be described later.

[0039] The operation of the first embodiment will be described below with reference to Figures 5 to 12. In the image capture and display device 101, this processing is realized by loading a program recorded in the volatile memory 203 into the main memory 202 and executing it with the CPU 201. In the computer 102, this processing is realized by loading a program recorded in the volatile memory 303 into the main memory 302 and executing it with the CPU 301.

[0040] FIG. 5 is a flowchart showing the processing of the imaging and display device 101 in the first embodiment.

[0041] In S501, the CPU 201 instructs the display of an operation mode setting screen, accepts the user's settings of the operation mode and operation options, and stores them in the main memory 202. The operation mode setting screen is, for example, a screen such as that shown in Fig. 13. The operation mode and operation options may also be configured so that conditions for automatic switching by the system are set on a screen such as that shown in Fig. 14, and the system automatically switches when the conditions are met.

[0042] In S502, the CPU 201 performs position and orientation estimation processing, which will be described later with reference to FIG.

[0043] In S503, the CPU 201 performs CG rendering processing according to the operation mode, which will be described later with reference to FIG.

[0044] 6 is a flowchart showing the position and orientation estimation process of the image capture and display device 101 according to the first embodiment. Details of the position and orientation estimation process in S502 in FIG.

[0045] In S601, the CPU 201 estimates the positions and orientations of the imaging and display device 101 and the keyboard instrument used by the user based on the images obtained from the imaging unit 207 and image processing unit 208, and stores the estimates in the main memory 202. For example, since a keyboard instrument has the same repeated arrangement of keys, the imaging and display device 101 is moved to a position where the entire instrument is visible, and the relative positions of the imaging and display device 101 and the keyboard instrument are calculated.

[0046] In S602, the CPU 201 estimates the position and orientation of the user's hand if the user's hand is on the musical instrument, based on the image obtained from the imaging unit 207 and the image processing unit 208.

[0047] 7 is a flowchart relating to CG drawing according to the operation mode of the image capturing and displaying device 101 in the first embodiment. Details of the CG drawing process according to the operation mode in S503 of FIG. 5 will be described with reference to FIG.

[0048] In S701, the CPU 201 refers to the operation mode setting stored in the main memory 202 in S501 and determines which of S702 to S704 to proceed to. If the operation mode is operation mode 1, proceed to S702. If it is operation mode 2, proceed to S703. If it is operation mode 3, proceed to S703.

[0049] In S702, the CPU 201 performs drawing processing in operation mode 1 (display at the actual position on the musical instrument). The drawing processing in operation mode 1 will be described later with reference to FIG.

[0050] In S703, the CPU 201 performs drawing processing for operation mode 2 (display at a position outside the instrument). The drawing processing for operation mode 2 will be described later with reference to FIG.

[0051] In S704, the CPU 201 performs drawing processing for operation mode 3 (displaying at a position different from the actual position on the musical instrument). The drawing processing for operation mode 3 will be described later with reference to FIG.

[0052] 8 is a flowchart relating to the drawing process in operation mode 1 of the image capturing and displaying device 101 according to the first embodiment. The drawing process in operation mode 1 in S702 of FIG. 7 will be described in detail with reference to FIG.

[0053] In step S801, the CPU 201 stores the range of the user's musical instrument in the main memory 202 as the basic drawing position setting for CG.

[0054] In S802, the CPU 201 performs drawing including the operation options of operation modes 1 and 2. Drawing including the operation options of operation modes 1 and 2 will be described later with reference to FIG.

[0055] 9 is a flowchart showing the drawing process of the imaging and display device 101 in the first embodiment in operation mode 2. The drawing process of operation mode 2 in S703 in FIG.

[0056] In step S901, the CPU 201 stores the area outside the user's instrument in the main memory 202 as the basic drawing position setting for CG.

[0057] In S902, the CPU 201 performs drawing including the operation options of operation modes 1 and 2. Drawing including the operation options of operation modes 1 and 2 will be described later with reference to FIG.

[0058] 10 is a flowchart illustrating the drawing process of the imaging and display device 101 in the first embodiment in operation mode 3. The drawing process of operation mode 3 in S704 in FIG.

[0059] In S1001, the CPU 201 stores the range of the user's musical instrument in the main memory 202 as the basic drawing position setting of the CG.

[0060] In S1002, the CPU 201 refers to the operation option stored in the main memory 202 in S501 and determines which of S1003 and S1004 to proceed to. If the operation option is drawing close to the user's hand, the process proceeds to S1003. If it is an octave above or below, the process proceeds to S1004.

[0061] In S1003, the CPU 201 performs control to draw CG in a position close to the user's hand. An example of the relationship between the position of the user's hand on the instrument, the user's line of sight and the appropriate placement position of the CG, taking into consideration the range of the user's field of view, etc., is shown in Fig. 15. The control to draw CG in a position close to the user's hand will be described below with reference to Figs. 15 and 16 to 23.

[0062] An example of the display for the first row in Figure 15 is shown in Figure 16. Both of the user's hands are within the field of view, with the left and right hands overlapping. In this case, left hand CG 1603 is placed to the left of the user's left hand 1604, and right hand CG 1606 is placed to the right of the user's right hand 1605. Since the CGs of each hand are placed in positions different from the actual notes and it is difficult to understand from the hand CG alone, CGs of the keys are also created and drawn in addition to the hand CGs. The keyboard CGs are drawn from the lowest note being held down a semitone lower to the highest note being held down a semitone higher, and the creation of the hand and keyboard CGs is assumed to be similar in the explanations of Figures 10 and 11. Furthermore, when the hand and keyboard CGs are placed next to the user's hands, they are positioned so that the keys that are a semitone lower than the lowest note being held down overlap with the highest note of the keyboard CG. Alternatively, the keyboard that is a semitone higher than the highest note the user's hand is holding down can be positioned so that the lowest note of the keyboard CG overlaps with it. In the following explanations, the same drawing will be used when placing CG of the hand and keyboard next to the user's hand.

[0063] An example of the display of the second row in Figure 15 is shown in Figure 17. Both of the user's hands are within the field of view, and the left and right hands do not overlap or cross. In this case, left hand CG 1704 is placed to the right of the user's left hand 1703, and right hand CG 1705 is placed to the left of the user's right hand 1706. However, if the left and right CGs interfere with each other, left hand CG 1709 is placed to the left of the user's left hand 1710, and right hand CG 1712 is placed to the right of the user's right hand 1711.

[0064] An example of the display of the third row in Figure 15 is shown in Figure 18. Both of the user's hands are within the field of view, and the left and right hands do not overlap but cross each other. In this case, left hand CG 1805 is placed to the left of the user's left hand 1806, and right hand CG 1804 is placed to the right of the user's right hand 1803. However, if the left and right CGs interfere with each other, left hand CG 1812 is placed to the right of the user's left hand 1811, and right hand CG 1809 is placed to the left of the user's right hand 1810.

[0065] An example of the display of the fourth row in Figure 15 is shown in Figure 19. The user's left hand is outside the field of view, and the right hand is within the field of view, and the left and right hands do not intersect. In this case, left hand CG 1904 is placed to the right of the user's left hand 1903, and right hand CG 1905 is placed to the left of the user's right hand 1906. However, if the left and right CGs interfere with each other, priority is given to the position of right hand CG 1911, and left hand CG 1909 is placed to the left of the user's left hand 1910. Furthermore, if right hand CG 1918 interferes with the user's left hand 1916, left hand CG 1915 is placed to the left of the user's left hand 1916, and right hand CG 1918 is placed to the right of the user's right hand 1917.

[0066] An example of the display of the fifth row in FIG. 15 is shown in FIG. 20. The user's left hand is outside the field of view, while the right hand is within the field of view, with the left and right hands intersecting. In this case, left hand CG 2005 is placed to the left of the user's left hand 2006, and right hand CG 2004 is placed to the right of the user's right hand 2003. However, if the left and right CGs interfere with each other, priority is given to the position of right hand CG 2010, and left hand CG 2012 is placed to the right of the user's left hand 2011. Furthermore, if right hand CG 2015 interferes with the user's left hand 2017, left hand CG 2018 is placed to the right of the user's left hand 2017, and right hand CG 2015 is placed to the left of the user's right hand 2016.

[0067] An example of the display of the sixth row in FIG. 15 is shown in FIG. 21. The user's left hand is within the field of view, and the right hand is outside the field of view, and the left and right hands do not intersect. In this case, left hand CG 2104 is placed to the right of the user's left hand 2103, and right hand CG 2105 is placed to the left of the user's right hand 2106. However, if the left and right CGs interfere with each other, priority is given to the position of left hand CG 2110, and right hand CG 2112 is placed to the right of the user's right hand 2111. Furthermore, if left hand CG 2115 interferes with the user's right hand 2117, left hand CG 2115 is placed to the left of the user's left hand 2116, and right hand CG 2118 is placed to the right of the user's right hand 2117.

[0068] An example of the display of the seventh row in FIG. 15 is shown in FIG. 22. The user's left hand is within the field of view, and the right hand is outside the field of view, with the left and right hands intersecting. In this case, left hand CG 2205 is placed to the left of the user's left hand 2206, and right hand CG 2204 is placed to the right of the user's right hand 2203. However, if the left and right CGs interfere with each other, priority is given to the position of left hand CG 2211, and right hand CG 2209 is placed to the left of the user's right hand 2210. Furthermore, if left hand CG 2218 interferes with the user's right hand 2216, left hand CG 2218 is placed to the right of the user's left hand 2217, and right hand CG 2215 is placed to the left of the user's right hand 2216.

[0069] FIG. 23 shows possible display examples for lines 8 to 10 in FIG. 15. The first example describes a case where both of the user's hands are outside the field of view, and the left and right hands overlap. In this case, if the original pitch and display position of the next CG to be rendered are within the field of view, left hand CG 2303 and right hand CG 2304 are placed at positions within the field of view that match the original pitch. Otherwise, left hand CG 2310 and right hand CG 2309 are placed side by side at a keyboard position that is the center of the user's line of sight. The second example describes a case where both of the user's hands are outside the field of view, and the left and right hands do not overlap or intersect. In this case, if the original pitch and display position of the next CG to be rendered are within the field of view, left hand CG 2316 and right hand CG 2317 are placed at positions within the field of view that match the original pitch. Otherwise, left hand CG 2322 and right hand CG 2323 are placed side by side at a keyboard position that is the center of the user's line of sight. As a third example, consider a situation where both of the user's hands are outside the field of view, but the left and right hands are crossed without overlapping. In this case, if the original pitch and display position of the next CG to be rendered are within the field of view, the left hand CG 2328 and right hand CG 2329 are placed at a position within the field of view that matches the original pitch. If not, the left hand CG 2335 and right hand CG 2334 are placed side by side at the keyboard position that is the center of the user's line of sight. Note that the next CG to be rendered refers to the CG that is scheduled to be rendered within a certain period of time. This period could be, for example, one second or one measure of a piece of music, but is not limited to this. An example of a specific situation in Figure 23 is when there is a large jump to the next note on an instrument. When the user shifts their gaze near the next pitch, consciously aware of the jump to the next pitch, the CG is controlled to be displayed at the position of the next pitch.

[0070] In S1004, the CPU 201 performs control to draw CGs at positions one octave above and below. An example of a display in which CGs are drawn at positions one octave above and below can be seen in FIG. 24. A right-hand CG 2405 is placed one octave above the user's right hand 2404, and a left-hand CG 2402 is placed one octave below the user's left hand 2403. However, when the right hand one octave above or the left hand one octave below the user's left hand cannot fit on the instrument near the highest or lowest note on the keyboard, a right-hand CG 2409 and a left-hand CG 2408 are placed one octave below the right hand 2410. In this case, if placing a right-hand CG 2415 one octave below the right hand 2414 would interfere with the user's left hand 2413, the right-hand CG 2415 is placed closer to the user's right hand 2414, even if it does not fit on the keyboard. In this case, as in S1003, the keys are arranged so that the key that is a semitone lower than the lowest note held by the user's hand overlaps with the position of the highest note of the keyboard's CG. Alternatively, the keys are arranged so that the key that is a semitone higher than the highest note held by the user's hand overlaps with the position of the lowest note of the keyboard's CG. Although there are no keys that are a semitone lower than the lowest note or a semitone higher than the highest note on the instrument, they are arranged in the appropriate positions based on the width of the keyboard. An example of the right hand has been explained with reference to Figure 24, but the control near the lowest note for the left hand is also performed symmetrically and in the same way.

[0071] FIG. 11 is a flowchart showing the operation option processing of the operation modes 1 and 2 of the image capturing and displaying device 101 according to the first embodiment.

[0072] 11 is a flowchart relating to drawing including operation options.Details of drawing including operation options for operation modes 1 and 2 in S802 of FIG. 8 and S902 of FIG.

[0073] In S1101 , the CPU 201 acquires from the main memory 202 the image and fingering estimation information received from the imaging device 102 via the communication unit 206 .

[0074] In S1102, the CPU 201 refers to the fingering estimation information acquired in S1101 and determines whether or not both hands are overlapped. If both hands are overlapped, the process proceeds to S1104; if not, the process proceeds to S1103.

[0075] In S1103, the CPU 201 controls the normal display. An example of the normal display is shown in FIG. 26. The basic drawing position setting stored in the main memory 202 in S801 or S901 is referenced. If the basic drawing position setting is on the user's instrument, hand CGs 2602, 2603, 2605, and 2606 are drawn on the user's actual instrument. The position of the hand CG on the instrument is controlled by reference to the position and orientation estimation result stored in the main memory 202 in S601 and the hand position in the fingering estimation information acquired in S1101, so that the hand position in the fingering estimation information matches the pitch on the actual instrument. If the basic drawing position setting is outside the user's instrument, the CG of the keyboard and the CG of the hands are drawn at a fixed distance outside the user's actual instrument. This fixed distance may be, for example, about 30 cm, but this is merely an example and is not intended to be limiting. The positional relationship between the CG of the keyboard and the CG of the hands is the same as when the basic drawing position setting is on the user's instrument, as described above, and is controlled so that the hand position in the fingering estimation information matches the pitch on the CG of the keyboard. The drawing positions of the CG of the keyboard and the CG of the hands can be anywhere outside the actual instrument, and one example would be to draw them 30 cm behind the actual keyboard, but this is not limited to this. Note that being outside the instrument means that the object is not present at any point in the three-dimensional space, either vertically or horizontally, and in this state the object is considered to be outside the instrument.

[0076] In S1104, the CPU 201 refers to the setting of the operation option stored in the main memory 202 in S501 and determines whether or not to display the separated hands when both hands overlap. If the separated hands are to be displayed when both hands overlap, the process proceeds to S1105; otherwise, the process proceeds to S1106.

[0077] In S1105, the CPU 201 performs the normal display process of S1103 and also controls the separate display of the left and right hands in the CG. An example of the separate display of the left and right hands in the CG is shown in FIG. 25. The fingering estimation information acquired in S1101 is referenced to acquire the positions of the overlapping left and right hands, the pitches of the keys being pressed, and the hand shapes. CGs of the left and right hands are created by separating the overlapping hands, and are then placed on the user's actual keyboard. The hand CGs are created using the same process as in S1003, and the CGs are positioned so that the left-hand CG 2502 overlaps with the key that is a semitone lower than the lowest note pressed by either of the overlapping hands and the highest note of the left-hand CG 2502 key. The right-hand CG 2505 is positioned so that the key that is a semitone higher than the highest note pressed by either of the overlapping hands and the lowest note of the right-hand CG 2505 key overlaps.

[0078] In S1106, the CPU 201 determines whether to prioritize the gazed-up hand by referring to the setting of the operation option stored in the main memory 302 in S501. If the gazed-up hand is prioritized, the process proceeds to S1107; otherwise, the process proceeds to S1108.

[0079] In S1107, the CPU 201 performs the normal display process of S1103 and also performs control to prioritize the hand that is the subject of gaze. An example of a display that prioritizes the hand that is the subject of gaze is shown in FIG. 27. The gaze of the user observing the display unit 204, detected by the gaze detection unit 205, is acquired, and it is determined whether the CG processed for normal display in S1103 intersects with the user's gaze 2702. If the gaze intersects with the user's gaze, it is determined whether the CG on the left or right intersects with the gaze, and the hand 2703 where the gaze intersects is highlighted. The highlighting may be achieved by changing the color or transparency of the CG of the hand where the gaze intersects to make it stand out compared to the other hand, or by changing the color or transparency of the CG of the hand where the gaze does not intersect to make it less noticeable compared to the hand where the gaze intersects. These are examples of highlighting, and the present invention is not limited to these. Furthermore, because sound is an important element in playing a musical instrument, adding sound effect control linked to the highlighting of the hand where the gaze intersects as described above further emphasizes the hand where the gaze intersects. One example of a sound effect is to control the volume of the sound played by the hand whose line of sight intersects with the other hand, but this is not limiting. In the above-described control, there are several ways to respond when the user's line of sight intersects with the CG of the hand. For example, a method can be considered in which the above-described control is immediately applied to either the left or right CG. Another method can be considered in which the cumulative time of gaze within a certain period of time is measured to determine which CG to apply the control to. These are examples of processing application methods, and are not limiting.

[0080] FIG. 12 is a flowchart showing the processing of the image capturing device 102 in the first embodiment.

[0081] In S1201, the CPU 301 estimates the shape (fingering) and position of the instrument player's hands in the video using the video obtained from the imaging unit 305 and image processing unit 306 and the trained model 406, and stores the estimate in the main memory 302.

[0082] In S1202, the CPU 301 transmits the hand shape, position, and performance information stored in the main memory 302 to the imaging and display device 101 via the communication unit 304.

[0083] Fig. 13 is a display example of the operation mode setting screen in the flowchart of Fig. 5 according to the first embodiment. 1301, 1305, and 1309 are operation mode options in the first embodiment, and any one of a plurality of operation modes can be selected. For any one selected operation mode, operation options 1302 to 1304, 1306 to 1308, and 1310 to 1311 can be set. The user can arbitrarily set these setting items.

[0084] 1301 denotes operation mode 1, which is one of the operation mode options in the first embodiment, and either 1302 or 1303 can be set as an operation mode option.

[0085] 1302 is a display of the hands that are separated when the hands overlap, which is one of the options in operation mode 1 in the first embodiment.

[0086] 1303 is one of the options for operation mode 1 in the first embodiment, which displays the overlapping of both hands as they are when they overlap, and 1304 can be set as a detailed option item.

[0087] 1304 is a detailed item of one of the options for operation mode 1 in the first embodiment, which displays the overlapping hands as they are when they overlap, and the setting to prioritize the hand that is in line of sight can be enabled or disabled using a check box.

[0088] 1305 is an operation mode 2, which is one of the operation mode options in the first embodiment, and either 1306 or 1307 can be set as an operation mode option.

[0089] Reference numeral 1306 denotes a display of the hands that are separated when the hands overlap, which is one of the options in operation mode 2 in the first embodiment.

[0090] 1307 is one of the options for operation mode 2 in the first embodiment, which displays the overlapping of both hands as they are when they overlap, and 1308 can be set as a detailed option item.

[0091] 1308 is a detailed item of one of the options for operation mode 2 in the first embodiment, which displays the overlapping of both hands as they are when they overlap, and the setting to prioritize the hand that is in line of sight can be enabled or disabled using a check box.

[0092] Reference numeral 1309 denotes operation mode 3, which is one of the operation mode options in the first embodiment, and either 1310 or 1311 can be set as an operation mode option.

[0093] 1310 is a position close to the user's hand for one of the options of operation mode 3 in the first embodiment.

[0094] 1311 is the position one octave up or down of the option for operation mode 3 in the first embodiment.

[0095] FIG. 14 shows an example of a display of an operation mode setting change screen in a configuration example in which conditions are set when the operation mode and operation option are automatically switched by the system in the flowchart of FIG. 5 according to the first embodiment.

[0096] Reference numeral 1401 denotes the operation mode setting when the user's hands are detected on the instrument in the first embodiment, and can be selected from a set of options prepared in advance. The options include Mode 1 to Mode 3. When the state changes from when the user's hands are not on the instrument to when they are, this setting is applied and the instrument automatically switches to the set operation mode.

[0097] In the first embodiment, 1402 is an operation mode setting for when the user's hands are not detected on the instrument, and can be selected from pre-prepared options. The options include Mode 1 and Mode 2. This setting is applied when the state changes from one in which the user's hands are present on the instrument to one in which they are not present, and the operation mode automatically switches to the set mode.

[0098] Reference numeral 1403 denotes a setting for switching operation modes based on gaze in the first embodiment, and a checkbox can be used to enable or disable a setting for switching to operation mode 3 if the gaze is focused on the hands in operation mode 2. If the setting is enabled and the user's gaze is focused on the user's hands on the actual instrument while operating in operation mode 2, this setting is applied and operation mode 3 is automatically switched to. One possible criterion for determining that the gaze is focused on the hands is that the gaze is focused on the hands if the total time the gaze points to the instrument over the last 30 seconds is 15 seconds or more, but this is just one example and is not limiting.

[0099] Reference numeral 1404 denotes a setting for switching operation options by gaze in the first embodiment, and a checkbox can be used to enable or disable a setting for prioritizing the hand that is being gazed at when the gaze is focused on either the left or right CG in operation modes 1 and 2. If the setting is enabled and the gaze is focused on either the left or right CG in operation mode 1 or operation mode 2, this setting is applied, and the hand that is being gazed at is displayed with priority.

[0100] The standard for determining whether the gaze is focused on the left or right CG is that if the gaze points to either the left or right hand for a total of 15 seconds or more in the last 30 seconds, it is determined that the gaze is focused on the CG of the relevant hand. The above is an example and is not limited to this.

[0101] Fig. 15 is a table showing details of the control for drawing near the user's hands in the flowchart of Fig. 10 according to the first embodiment. There are conditions for the user's hands, such as the range of the field of view, left-right overlap, and hand intersection, and each row lists the placement positions of the left hand CG and right hand CG according to these conditions.

[0102] FIG. 16 is a display example of the first line of the detailed control diagram 15 for drawing near the user's hands in the flowchart of FIG. 10 according to the first embodiment. 1601 is the range of the user's field of view. 1602 is the real keyboard on which the user plays. 1603 and 1606 are the CG hands and the CG keyboard. 1604 and 1605 are the user's real hands.

[0103] FIG. 17 shows an example of the display of the second line of the detailed control diagram 15 for drawing near the user's hands in the flowchart of FIG. 10 according to the first embodiment. 1701 and 1707 are the range of the user's field of view. 1702 and 1708 are the real keyboard on which the user plays. 1704, 1705, 1709, and 1712 are the CG hands and the CG keyboard. 1703, 1706, 1710, and 1711 are the user's real hands.

[0104] FIG. 18 shows an example of the display of the third line of the detailed control diagram 15 for drawing near the user's hands in the flowchart of FIG. 10 according to the first embodiment. 1801 and 1807 are the range of the user's field of view. 1802 and 1808 are the real keyboard on which the user plays. 1804, 1805, 1809, and 1812 are the CG hands and the CG keyboard. 1803, 1806, 1810, and 1811 are the user's real hands.

[0105] FIG. 19 is a display example of the fourth line of the detailed control diagram 15 for drawing near the user's hands in the flowchart of FIG. 10 according to the first embodiment. 1901, 1907, and 1913 are the user's field of view. 1902, 1908, and 1914 are the real keyboard on which the user plays. 1904, 1905, 1909, 1911, 1915, and 1918 are the CG hands and the CG keyboard. 1903, 1906, 1910, 1912, 1916, and 1917 are the user's real hands.

[0106] FIG. 20 is a display example of the fifth line of the detailed control diagram 15 for drawing near the user's hands in the flowchart of FIG. 10 according to the first embodiment. 2001, 2007, and 2013 are the user's field of view. 2002, 2008, and 2014 are the real keys the user plays. 2004, 2005, 2010, 2012, 2015, and 2018 are the CG hands and the CG keys. 2003, 2006, 2009, 2011, 2016, and 2017 are the user's real hands.

[0107] FIG. 21 shows a display example of the sixth line of the detailed control diagram 15 for drawing near the user's hands in the flowchart of FIG. 10 according to the first embodiment. 2101, 2107, and 2113 are the user's field of view. 2102, 2108, and 2114 are the real keyboard on which the user plays. 2104, 2105, 2110, 2112, 2115, and 2118 are the CG hands and the CG keyboard. 2103, 2106, 2109, 2111, 2116, and 2117 are the user's real hands.

[0108] Figure 22 is a display example of the seventh line of the detailed control diagram 15 for drawing near the user's hands in the flowchart in Figure 10 according to the first embodiment. 2201, 2207, and 2213 are the user's field of view. 2202, 2208, and 2214 are the real keyboard on which the user plays. 2204, 2205, 2209, 2211, 2215, and 2218 are the CG hands and CG keyboard. 2203, 2206, 2210, 2212, 2216, and 2217 are the user's real hands.

[0109] 23 shows a display example of lines 8 to 10 of detailed control diagram 15 for drawing near the user's hands in flowchart FIG. 10 according to the first embodiment. References 2301, 2307, 2313, 2319, 2325, and 2331 indicate the user's field of view. References 2302, 2308, 2314, 2320, 2326, and 2332 indicate the actual keyboard on which the user plays. References 2303, 2304, 2309, 2310, 2316, 2317, 2322, 2323, 2328, 2329, 2334, and 2335 indicate the CG hands and CG keyboard. 2305, 2306, 2311, 2312, 2315, 2318, 2321, 2324, 2327, 2330, 2333, and 2336 are the actual hands of the user.

[0110] Figure 24 shows a display example when control is exercised to draw at positions one octave above and below in the flowchart of Figure 10 according to the first embodiment. Reference numerals 2401, 2406, and 2411 represent the actual keyboard played by the user. Reference numerals 2402, 2405, 2407, 2409, 2412, and 2415 represent CG hands and the CG keyboard. Reference numerals 2403, 2404, 2408, 2410, 2413, and 2414 represent the actual hands of the user.

[0111] Figure 25 shows an example of a display when control is exercised to separate and display the overlapping hands in the flowchart of Figure 11 according to the first embodiment. 2501 is a real keyboard on which the user plays. 2502 and 2505 are CG hands and a CG keyboard with the overlapping hands separated. 2503 and 2504 are CG hands in their original positions with the overlapping hands not separated.

[0112] 26 shows an example of a display when normal display control is performed in the flowchart of FIG. 11 according to the first embodiment. 2601 is a real keyboard on which the user plays. 2602 to 2606 are CG hands.

[0113] Figure 27 shows an example of a display when control is exercised to prioritize display of the hand that the user's gaze is directed at in the flowchart of Figure 11 according to the first embodiment. 2701 is the actual keyboard that the user plays. 2702 is the user's gaze. 2703 is the CG hand (left hand) that the user's gaze is directed at. 2704 is the CG hand (right hand) that the user is not looking at.

[0114] [Second embodiment] Hereinafter, the second embodiment of the present invention will be described with reference to the drawings, focusing on the differences from the first embodiment. In the drawings of the second embodiment, the same parts as those in the first embodiment are given the same reference numerals as those in the first embodiment, and detailed descriptions thereof will be omitted.

[0115] 28 is a schematic diagram showing the system configuration of the second embodiment. The system is made up of an image capturing and displaying device 101.

[0116] FIG. 29 is a diagram illustrating the functional configuration of a system according to the second embodiment.

[0117] Each function of the image capturing and displaying device 101 is realized by the CPU 201. The image capturing and displaying device 101 is made up of a display control unit 401, a position and orientation estimation unit 402, a control unit 403, CG data 404, and model data 2901.

[0118] The position and orientation estimation unit 402 estimates the three-dimensional position and orientation of the image capturing and displaying device 101 and the cooking utensils used by the user based on the images obtained from the image capturing unit 207 and image processing unit 208.

[0119] The CG data 404 stores CG data when the control unit 403 performs processing (described later) on the model data 2901 to display CG.

[0120] The model data 2901 stores CG data for displaying CG by performing processing, described below, by the control unit 403 on the analysis results received from the image capture device 102 .

[0121] The operation of the second embodiment will be described below with reference to the drawings. This processing is realized by loading a program recorded in the nonvolatile memory 203 of the imaging and displaying device 101 into the main memory 202 and executing it by the CPU 201.

[0122] The differences from the first embodiment in FIG. 5 will be described.

[0123] In S501, the CPU 201 instructs the display of an operation mode setting screen, accepts the user's settings of the operation mode and operation options, and stores them in the main memory 202. The operation mode setting screen is, for example, a screen such as that shown in FIG.

[0124] The differences from the first embodiment in FIG. 6 will be described.

[0125] In S601, the CPU 201 estimates the positions and orientations of the imaging and display device 101 and the cooking utensils 2801 (cutting board placed in front of the user) used by the user based on the images obtained from the imaging unit 207 and image processing unit 208, and stores the estimates in the main memory 202. For example, the relative positions of the imaging and display device 101 and the cooking utensils are calculated by moving the imaging and display device 101 to a position where the entire cooking utensils are visible.

[0126] In S602, the CPU 201 estimates the position and orientation of the user's hand if the user's hand is on a cooking utensil, based on the image obtained from the imaging unit 207 and the image processing unit 208.

[0127] The differences from the first embodiment in FIG. 7 will be described.

[0128] In S702, the CPU 201 performs drawing processing in operation mode 1 (displaying at the actual position on the cooking utensil). The drawing processing in operation mode 1 will be described later with reference to FIG.

[0129] In S703, the CPU 201 performs drawing processing for operation mode 2 (displaying at a position outside the cooking utensils). The drawing processing for operation mode 2 will be described later with reference to FIG.

[0130] In S704, the CPU 201 performs drawing processing for operation mode 3 (displaying at a position different from the actual position on the cooking utensil). The drawing processing for operation mode 3 will be described later with reference to FIG.

[0131] The differences from the first embodiment in FIG. 11 will be described.

[0132] In S1101, the CPU 201 acquires the model data 2901 that has been read into the main memory 202.

[0133] In S1102, the CPU 201 determines whether or not both hands are overlapping with reference to the model data 2901 acquired in S1101. If both hands are overlapping, the process proceeds to S1104, and if not, the process proceeds to S1103.

[0134] In S1103, the CPU 201 references the basic drawing position setting stored in the main memory 202 in S3101 or S3201. If the basic drawing position setting is on the user's cooking utensils, it draws a CG image of the hand on the user's actual cooking utensils. The position of the CG image of the hand on the cooking utensils is determined by referencing the position and orientation estimation results stored in the main memory 202 in S601 and the hand position in the model data acquired in S1101, and controlling the position of the hand in the model data to match its position on the actual cooking utensils. If the basic drawing position setting is outside the user's cooking utensils, the CG image of the cooking utensils and the CG image of the hand are drawn at a fixed distance outside the user's actual cooking utensils. This fixed distance may be, for example, approximately 30 cm, but is not limited to this example. The drawing positions of the CG image of the cooking utensils and the CG image of the hand may be anywhere outside the actual cooking utensils. For example, the drawing may be 30 cm behind the actual cooking utensils, but is not limited to this example. Note that being outside the cooking utensil means that the object does not exist at any point in the vertical direction in three-dimensional space, and in this state the object is considered to be outside the cooking utensil.

[0135] In S1105, the CPU 201 performs the normal display process of S1103, and also performs processing to separate and display the left and right CG hands. The model data acquired in S1101 is referenced to acquire the position and shape of each of the overlapping hands, and CG images of the left and right hands are created by separating the overlapping hands, and are placed on the user's actual cooking utensils. The separated left and right CG hands are both placed 5 cm from the boundary edge of the overlapping CG hands.

[0136] 30 is a flowchart showing the drawing process in the operation mode 1 of the image capturing and displaying device 101 according to the second embodiment. The drawing process in the operation mode 1 in S702 of FIG.

[0137] In S3001, the CPU 201 stores the range of the user's cooking tools in the main memory 202 as the basic drawing position setting for CG.

[0138] 31 is a flowchart showing the drawing process of the imaging and display device 101 in the second embodiment in operation mode 2. The drawing process of operation mode 2 in S703 in FIG.

[0139] In step S3101, the CPU 201 stores the area outside the user's cooking utensils in the main memory 202 as the basic drawing position setting for CG.

[0140] 32 is a flowchart of the drawing process of the imaging and display device 101 in the second embodiment in operation mode 3. The drawing process of operation mode 3 in S704 in FIG.

[0141] In S3201, the CPU 201 stores the range of the user's cooking tools in the main memory 202 as the basic drawing position setting for CG.

[0142] In S3202, the CPU 201 references the operation option stored in the main memory 202 in S501 and determines whether to proceed to S3203 or S3204. If the operation option is to draw the hand close to the user's hand, the process proceeds to S3203. If it is to draw the hand at a certain distance away, the process proceeds to S3204.

[0143] In S3203, the CPU 201 performs control to render CG in a position close to the user's hand. An example of the relationship between the position of the user's hand on the cooking utensil, the user's line of sight, the user's field of view, and other factors can be seen in Figure 34. When the user's hand is on the cooking utensil and within the field of view, CG hands 3405 and 3406 are positioned to the right of the user's hands 3403 and 3404 in the field of view. If the CG position becomes inappropriate due to a change in the user's field of view, for example, CG hands 3408 and 3409 are positioned to the left of the user's hands 3410 and 3411. Examples of inappropriate CG positions include when the user's field of view changes and the CG falls out of the field of view, or when the area of ​​the CG that was not within the field of view increases. In such cases, if the CG would fit within a larger field of view if positioned to the left of the user's hand, the CG position is determined to be inappropriate, and control can be exercised to position the CG to the left of the user's hand. The distance from the user's hand to the right and left of the device is the same, for example, about 5 cm, but this is just an example and is not limiting.

[0144] In S3204, the CPU 201 performs control to draw a CG at a fixed distance. An example of a display in which a CG is drawn at a fixed distance is shown in FIG. 35. The CG is placed at a fixed distance from the user's hands. The CG is basically placed to the right of the user's hands 3502 and 3503. If the movement of the user's hands 3510 and 3511 determines that the left side is more appropriate than the right side, CG hands 3508 and 3509 are placed at a fixed distance to the left of the user's hands 3510 and 3511. An example of a case in which the left side is determined to be more appropriate is when placing the CG to the right would result in more than half of the CG being outside the cooking utensils, and placing the CG to the left would result in a larger area fitting within the field of view. In such a case, it may be considered appropriate to place the CG on the left, but this is merely an example and is not limiting.

[0145] FIG. 33 shows an example of the display of the operation mode setting screen in the flowchart of FIG. 5 according to the second embodiment.

[0146] 1301, 1305, and 1309 are operation mode options in the first embodiment, and any one of a plurality of operation modes can be selected. Operation options 1302 to 1304, 1306 to 1308, 1310 to 1310, and 3301 can be set for any one selected operation mode. The user can arbitrarily set these setting items.

[0147] 3301 is a fixed distance away position of one of the options of operation mode 3 in the second embodiment.

[0148] Reference numeral 3302 denotes an input box for the value of the fixed distance to be set when one of the options for operation mode 3 in the second embodiment, a position at a fixed distance, is selected.

[0149] Figure 34 is a display example when control is exercised to draw the hand close to the user's hand in the flowchart of Figure 32 according to the second embodiment. 3401 and 3407 are the range of the user's field of view. 3402 and 3412 are actual cooking utensils used by the user. 3405, 3406, 3408, and 3409 are CG hands. 3403, 3404, 3410, and 3411 are the user's actual hands.

[0150] Figure 35 is a display example when control is exercised to draw at a position a certain distance away in the flowchart of Figure 32 according to the second embodiment. 3506 and 3507 are the range of the user's field of view. 3501 and 3512 are real cooking utensils used by the user. 3504, 3505, 3508, and 3509 are CG hands. 3502, 3503, 3510, and 3511 are the user's real hands.

[0151] 36 shows an example of a display when normal display control is performed in the flowchart of FIG. 11 according to the second embodiment. Reference numeral 3601 denotes real cooking utensils used by the user. Reference numerals 3602 and 3603 denote CG hands.

[0152] Figure 37 shows an example of a display when control is exercised to separate and display overlapping hands in the flowchart of Figure 11 according to the second embodiment. Reference numeral 3701 denotes real cooking utensils used by the user. Reference numerals 3702 and 3705 denote CG hands with overlapping hands separated. Reference numerals 3703 and 3704 denote CG hands in their original positions with overlapping hands not separated.

[0153] Figure 38 is a display example when control is exercised to give priority to displaying the hand that the user is looking at in the flowchart in Figure 11 according to the second embodiment. 3801 is the actual cooking utensil used by the user. 3802 is the user's line of sight. 3803 is the CG hand (left hand) that the user is looking at. 3804 is the CG hand (right hand) that the user is not looking at.

[0154] [Other embodiments] The present invention can also be realized by executing the following process. That is, software (programs) that realize the functions of the above-described embodiments are supplied to a system or device via a network or various storage media, and the computer (or CPU, MPU, etc.) of the system or device reads and executes the program code. In this case, the program and the storage media storing the program constitute the present invention. [Explanation of symbols]

[0155] 101 Imaging and display devices (electronic devices) 204 Display section 205 Detection unit (detection means) 206 Communication unit (acquisition means) 207 Imaging unit 401 Display control unit (display control means) 402 Position and orientation estimation unit (estimation means)

Claims

1. An electronic device having an imaging unit and a display unit, an estimation means for estimating a position and orientation of a tool used by a user based on an image captured by the imaging unit; an acquisition means for acquiring a hand motion of a person different from the user when the person uses a tool; a display unit that displays a CG image of a hand on the display unit based on the position and orientation acquired by the estimation unit and the motion acquired by the acquisition unit, The display means displays the CG of the hand on the tool at a position that does not overlap with the user's hand. An electronic device characterized by:

2. a detection means for detecting the line of sight of the user; The display means changes the position where the CG image of the hand is displayed based on the detection result of the detection means.

2. The electronic device according to claim 1, wherein the electronic device is a semiconductor device.

3. The display means It operates in one of several modes of operation, In a first mode of operation, displaying a CG image of the hand in its actual position on the tool; In the second operation mode, the CG of the hand is displayed at a position outside the tool; In the third operation mode, a CG image of the user's hand is displayed on the tool at a position that does not overlap the user's hand.

2. The electronic device according to claim 1, wherein the electronic device is a semiconductor device.

4. In the first operation mode, the display means displays a CG image of a hand so that the shape of the hand and the position of the hand on the tool that match the operation acquired by the acquisition means also match the user's tool on the actual image.

4. The electronic device according to claim 3.

5. In the second operation mode, the display means displays a CG image of the hand and the tool at a position on the tool that is not on the user's tool in a real image, in accordance with the hand shape and the position of the hand on the tool that match the operation acquired by the acquisition means.

4. The electronic device according to claim 3.

6. In the third operation mode, the display means changes the positions of the left and right hands to positions where they do not overlap when the left and right hands overlap in the operation acquired by the acquisition means, and displays the CG of the hands and the tool.

4. The electronic device according to claim 3.

7. The display means applies different drawing effects to the CG drawing effect of the hand that is the target of the line of sight obtained by the detection means and the CG drawing effect of the hand that is not the target of the line of sight obtained by the detection means when the left and right hands overlap in the action obtained by the acquisition means.

3. The electronic device according to claim 2.

8. The display means takes into consideration the position of the line of sight and the positions and shapes of both hands obtained by the detection means, and displays the hand shape and hand position on the tool in an appropriate position that does not match the user's movement on the tool, in addition to the hand shape and hand position on the tool that match the movement obtained by the acquisition means.

3. The electronic device according to claim 2.

9. A user can set any one of the plurality of operation modes in advance, The display means operates in the set operation mode.

4. The electronic device according to claim 3.

Images