Information processing apparatus, information processing system, information processing method, and information processing program

Abstract

To provide an information processing apparatus, an information processing system, an information processing method, and an information processing program which can improve operability of a user.SOLUTION: An information processing apparatus according to an embodiment of the present invention has a position detection unit, a movement detection unit, a processing execution unit, and an operation area adjustment unit. The position detection unit detects a position of an operator. The movement detection unit detects a predetermined movement of the operator in an operation area indicating an area defined by linking the position of the operator detected by the position detection unit with an operation target area set on an electronic device. The processing execution unit carries out processing associated with the operation target area when the predetermined movement is detected by the movement detection unit. The operation area adjustment unit enlarges the operation area when the position of the operator detected by the position detection unit is located outside of an area in front of the electronic device.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present disclosure relates to an information processing device, an information processing system, an information processing method, and an information processing program. [Background technology]

[0002] Conventionally, in various electronic devices, a technique is known that executes a process associated with an object to be operated by a user's hand movement when the user moves his / her hand without directly touching the electronic device (for example, see Patent Documents 1-2). For example, when the electronic device is a display, a technique is known that changes the display content of the display as a process associated with an object to be operated by the hand movement (for example, a button arranged on the screen) when the user moves his / her hand without directly touching the display. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6463963 [Patent Document 2] Patent No. 5341126 Summary of the Invention [Problem to be solved by the invention]

[0004] In the above-mentioned technology, for example, when the user's position is outside the front area of ​​the electronic device, the user has a harder time executing a process linked to an operation target such as a button, i.e., it is more difficult to operate the device, than when the user's position is in the front area.

[0005] The present disclosure provides an information processing device, an information processing system, an information processing method, and an information processing program capable of improving user operability. [Means for solving the problem]

[0006] An information processing device according to an aspect of the present disclosure includes a position detection unit, a motion detection unit, a process execution unit, and an operation area adjustment unit. The position detection unit detects the position of an operator. The motion detection unit detects a predetermined motion of the operator in an operation area indicating an area defined by connecting the position of the operator detected by the position detection unit and an operation target area set on the electronic device. When the predetermined motion is detected by the motion detection unit, the process execution unit performs a process associated with the operation target area. When the position of the operator detected by the position detection unit is outside the area in front of the electronic device, the operation area adjustment unit expands the operation area. Effect of the Invention

[0007] According to the information processing device of the present disclosure, it is possible to improve the operability of the user. Note that the effects described herein are not necessarily limited to those described herein, and may be any of the effects described in this specification. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a schematic diagram illustrating an example of an information processing device. [Diagram 2] FIG. 2 is a schematic diagram for explaining an operation process of an information processing device according to a first comparative example. [Diagram 3] FIG. 3 is a schematic diagram for explaining the operation process of the information processing device according to the first embodiment. [Figure 4] FIG. 4 is a diagram showing a schematic configuration of an information system including a control device which is an information processing device according to the first embodiment. [Diagram 5] FIG. 5 is a schematic diagram for explaining an example of an enlarged operation region according to the first embodiment. [Figure 6] FIG. 6 is a flowchart showing a processing procedure of the control device according to the first embodiment. [Figure 7] FIG. 7 is a schematic diagram for explaining an example of an enlarged operation target region according to the first modified example. [Figure 8] FIG. 8 is a schematic diagram for explaining an example of an expanding operation region according to the second modified example. [Figure 9] FIG. 9 is a schematic diagram for explaining an enlarged operation target region according to the third modified example. [Figure 10] FIG. 10 is a schematic diagram for explaining an enlarged operation target region according to the third modified example. [Figure 11] FIG. 11 is a schematic diagram for explaining an example of an expanding operation region according to the fourth modified example. [Figure 12] FIG. 12 is a schematic diagram for explaining an example of an expanding operation region according to the fifth modified example. [Figure 13] FIG. 13 is a schematic diagram for explaining an example of an expanding operation region according to the sixth modified example. [Figure 14] FIG. 14 is a schematic diagram for explaining an example of an expanding operation region according to the sixth modified example. [Figure 15] FIG. 15 is a schematic diagram for explaining an operation process of an information processing device according to the seventh modification. [Figure 16] FIG. 16 is a schematic diagram for explaining an operation process of an information processing device according to a second comparative example. [Figure 17] FIG. 17 is a schematic diagram for explaining the operation process of the information processing device according to the second embodiment. [Figure 18] FIG. 18 is a diagram showing a schematic configuration of an information system including a control device which is an information processing device according to the second embodiment. [Figure 19] FIG. 19 is a schematic diagram for explaining an example of an enlarged operation region according to the second embodiment. [Figure 20] FIG. 20 is a schematic diagram for explaining an example of an enlarged operation target region according to the eighth modified example. [Figure 21] FIG. 21 is a schematic diagram for explaining an example of an expanding operation region according to the ninth modified example. [Figure 22] FIG. 22 is a schematic diagram for explaining an example of an enlarged operation target region according to the third embodiment. [Figure 23] FIG. 23 is a schematic diagram for explaining an example of an enlarged operation target region according to the third embodiment. [Figure 24] FIG. 24 is a schematic diagram for explaining an example of an enlarged operation target region according to the third embodiment. [Diagram 25] FIG. 25 is a schematic diagram for explaining an example of an enlarged operation target region according to the tenth modified example. [Figure 26] FIG. 26 is a schematic diagram for explaining an example of an enlarged operation target region according to the tenth modified example. [Figure 27] FIG. 27 is a schematic diagram for explaining an example of an enlarged operation target region according to the eleventh modified example. [Figure 28] FIG. 28 is a schematic diagram for explaining an example of an enlarged operation target region according to the eleventh modified example. [Figure 29] FIG. 29 is a schematic diagram for explaining an example of an enlarged operation target region according to the eleventh modified example. [Diagram 30] FIG. 30 is a schematic diagram for explaining an example of an enlarged operation target region according to the eleventh modified example. [Diagram 31] FIG. 31 is a schematic diagram for explaining an operation process of an information processing device according to a third comparative example. [Diagram 32] FIG. 32 is a schematic diagram for explaining an example of an enlarged operation target region according to the fourth embodiment. [Diagram 33] FIG. 33 is a schematic diagram showing an example of the contents of the operation process of the enlarged operation target region according to the fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of an information processing device according to the present disclosure will be described with reference to the drawings.

[0010] (First embodiment) Before describing the information processing device according to the first embodiment, a device that operates based on the movement of a user's hand will be described.

[0011] FIG. 1 shows an example of an information processing device that operates based on the movement of a user's hand. The information processing device has a sensor 500 and an electronic device 600. The sensor 500 is, for example, a camera. The sensor 500 captures an image of a user UR who is an operator. The user UR has an arm. Here, the arm includes not only the upper arm and the forearm but also the hand. The hand also includes fingertips 72. The electronic device 600 is, for example, a display device such as a display. The information processing device controls data displayed by the electronic device 600 based on the position of the fingertips 72 of the user UR.

[0012] In addition, the information processing device is provided with a virtual operation area AR100 for receiving an operation on a device (e.g., electronic device 600) and a detection effective area AR200 that is an area in which the movement of a fingertip 72 of a user UR can be detected. The detection effective area AR200 is set within the sensing range of the sensor 500, for example, in front of the display surface of the electronic device 600. When the sensor 500 is a camera, the sensing range may be referred to as an imaging range. The virtual operation area AR100 is included in the detection effective area AR200, and is set as a plane parallel to the display surface of the electronic device 600, for example, as shown in FIG. 1.

[0013] In the configuration of Figure 1, when the fingertip 72 enters a range closer to the electronic device 600 than the virtual operation area AR100, the information processing device determines that an operation instruction has been given to the electronic device 600, executes processing based on the operation instruction, and displays the execution result on the electronic device 600.

[0014] 1 and the drawings relating to electronic device 600 described below, the mutually orthogonal X-axis, Y-axis, and Z-axis respectively refer to the left-right direction, up-down direction, and front-rear direction of electronic device 600. In the following description, when the X-direction, Y-direction, or Z-direction is simply mentioned, it refers to the respective axial direction and includes the two opposite directions.

[0015] In addition, when the positive direction of the X axis is specified, it is one direction from the left hand direction to the right hand direction of the user UR, when the positive direction of the Y axis is specified, it is one direction from the bottom to the top of the user UR, when the positive direction of the Z axis is specified, it is one direction from the front to the back of the user UR, when the negative direction of the X axis is specified, it is one direction from the right hand direction to the left hand direction of the user UR, when the negative direction of the Y axis is specified, it is one direction from the top to the bottom of the user UR, when the negative direction of the Z axis is specified, it is one direction from the back to the front of the user UR.

[0016] Fig. 2 is a schematic diagram for explaining the operation process of the information processing device according to the first comparative example. Fig. 2 shows an electronic device 600, a user UR who operates the electronic device 600, and a first button 41 and a second button 43 output by the electronic device 600. The first button 41 and the second button 43 are the same size and positioned approximately parallel to each other. It is also assumed that the positions of the first button 41 and the second button 43 are symmetrical with respect to the center of the electronic device 600.

[0017] Here, a description will be given of the operation process of the information processing device when the user UR operates the electronic device 600 in midair. For example, when the user UR is positioned in front of and in the center of the electronic device 600, the positional relationship between the user UR and the first button 41 and the second button 43 is the same.

[0018] In addition, regarding the operation angle between the user UR and the first button 41, for example, a first operation angle 413 can be calculated from a first line 411 connecting the user UR and the right end of the first button 41 and a second line 412 connecting the user UR and the left end of the first button 41, with the user UR as a base point. Regarding the operation angle between the user UR and the second button 43, for example, a second operation angle 433 can be calculated from a third line 431 connecting the user UR and the right end of the second button 43 and a fourth line 432 connecting the user UR and the left end of the second button 43, with the user UR as a base point.

[0019] When the user UR is positioned in front of and in the center of the electronic device 600, the positional relationship between the user UR and the first button 41 and the second button 43 is equal, so the first operation angle 413 and the second operation angle 433 are also equal. Therefore, when the user UR operates the first button 41 and the second button 43 output by the electronic device 600 in the air, the first operation angle 413 and the second operation angle 433 are equal, so that the user UR can obtain the same operation feeling.

[0020] Fig. 3 is a schematic diagram for explaining the operation process of the information processing device according to the first embodiment. Like Fig. 2, Fig. 3 shows an electronic device 600, a user UR who operates the electronic device 600, and a first button 41 and a second button 43 output by the electronic device 600.

[0021] For example, when the user UR is located outside the front area of ​​the electronic device 600, the positional relationship between the user UR and the first button 41 and the second button 43 is different. In the positional relationship between the user UR and the electronic device 600 shown in FIG. 3, the first button 41 is closer than the second button 43 when the user UR is taken as the base point. Therefore, the lengths of the lines connecting the user UR and the buttons are as follows: fourth line 432>third line 431>second line 412>first line 411>0. In other words, the first operation angle 413 is larger than the second operation angle 433, and the operation angles are different.

[0022] As a result, when the user UR operates the first button 41 and the second button 43 output by the electronic device 600 in mid-air, the user UR may feel that it is difficult to operate them because the first operation angle 413 and the second operation angle 433 are different. Therefore, this embodiment provides an information processing device that can improve the operability of the user UR when the user UR is located outside the front area of ​​the electronic device 600.

[0023] 4 is a diagram showing a schematic configuration of an information system including a control device 1 which is an information processing device according to the first embodiment. The information processing device according to the first embodiment includes a sensor 500, an electronic device 600, and the control device 1.

[0024] The sensor 500 is, for example, a camera. As an example, the sensor 500 is a visible light camera. The sensor 500 outputs an image of the user UR to the control device 1. The sensor 500 is an example of an imaging unit. The sensor 500 continuously executes imaging processing and outputs the image to the control device 1. The electronic device 600 is a display unit that displays various data. The electronic device 600 is, for example, a display device such as a liquid crystal display. It is assumed that there is one electronic device 600.

[0025] The control device 1 executes processing of the data displayed on the electronic device 600 in response to the movement of the fingertip 72 of the user UR detected via the sensor 500.

[0026] The control device 1 includes a control unit 10 and a memory unit 30. The control unit 10 is configured, for example, as a CPU (Central Processing Unit) and controls the overall operation of each unit of the control device 1. The control device 1 according to this embodiment includes a ROM (Read Only Memory) and a RAM (Random Access Memory), which are not shown. The ROM stores various programs. The RAM is a working area when the CPU executes the programs. The control device 1 includes, for example, a processor and a memory, and the processor executes the programs stored in the memory to realize the functions of the control unit 10 and each functional block of the control unit 10. The CPU is an example of a processor. The memory unit 30 is an example of a memory.

[0027] The CPU uses the RAM as a working area and executes the program stored in the ROM, thereby realizing the image acquisition unit 11, the first judgment unit 12, the position detection unit 13, the operation area generation unit 14, the second judgment unit 15, the operation area adjustment unit 16, the motion detection unit 17, the third judgment unit 18, and the process execution unit 19, as shown in Fig. 2. This may be said as the control device 1 including the image acquisition unit 11, the first judgment unit 12, the position detection unit 13, the operation area generation unit 14, the second judgment unit 15, the operation area adjustment unit 16, the motion detection unit 17, the third judgment unit 18, and the process execution unit 19. The image acquisition unit 11, the first judgment unit 12, the position detection unit 13, the operation area generation unit 14, the second judgment unit 15, the operation area adjustment unit 16, the motion detection unit 17, the third judgment unit 18, and the process execution unit 19 may be realized by different hardware.

[0028] The storage unit 30 stores various information. The storage unit 30 is realized by hardware for storing information (in other words, data), such as a memory or storage. Specifically, the storage unit 30 stores coordinate information 31, detection effective area information 32, operation area information 33, extended operation area information 34, and device operation amount information 35.

[0029] The coordinate information 31 includes 3D coordinates of the installation position of the sensor 500, information on the mounting angle of the sensor 500, and 3D coordinates of the position of the electronic device 600. The detection effective area information 32 is coordinate information related to the setting of the detection effective area. The detection effective area corresponds to the above-mentioned detection effective area AR200, and is an area in which the position and movement of the fingertip 72 of the user UR can be detected.

[0030] The operation area information 33 is coordinate information related to the setting of the operation area. The operation area corresponds to the above-mentioned virtual operation area AR100, and is an area for receiving an operation by the fingertip 72 of the user UR. The expanded operation area information 34 is coordinate information related to the setting for expanding the above-mentioned operation area. The operation area and the setting for expanding the operation area will be described later.

[0031] The device operation amount information 35 is information indicating an operation of the electronic device 600 corresponding to the position of the fingertip 72 of the user UR. The operation of the electronic device 600 is, for example, a pressing operation of a button output by the electronic device 600. The operation of the electronic device 600 is also called a predetermined operation.

[0032] Here, the coordinate systems in the coordinate information 31, the detection effective area information 32, the operation area information 33, the extended operation area information 34, and the device operation amount information 35 are absolute coordinates. Note that the coordinate system is not limited to absolute coordinates. For example, the coordinate system may be a relative coordinate system based on the electronic device 600. In other words, the coordinate system may be a relative coordinate system based on the positional relationship between the user UR and the electronic device 600.

[0033] The image acquisition unit 11 acquires, from the sensor 500, a captured image of a user UR who is an operator.

[0034] The first determination unit 12 determines whether an operator has been detected. Specifically, the first determination unit 12 determines whether a user UR who will be the operator has been detected from the captured image acquired by the image acquisition unit 11. For example, the first determination unit 12 determines that an operator has been detected when a user UR who will be the operator is present in the captured image acquired by the image acquisition unit 11. Also, for example, the first determination unit 12 determines that an operator has not been detected when a user UR who will be the operator is not present in the captured image acquired by the image acquisition unit 11.

[0035] The position detection unit 13 detects the position of the user UR based on the captured image in which the user UR is captured. Specifically, when the first determination unit 12 determines that the user UR who is the operator has been detected from the captured image acquired by the image acquisition unit 11, the position detection unit 13 detects the position of the user UR based on the captured image in which the user UR is captured.

[0036] The operation area generating unit 14 generates an operation area indicating an area defined by connecting the position of the user UR detected by the position detecting unit 13 and an operation target area set on the electronic device based on a captured image of the user UR. Here, the operation area generated by the operation area generating unit 14 will be described with reference to FIG. 3.

[0037] Here, in Fig. 3, the first button 41 and the second button 43 are each an operation target area set on the electronic device. For example, when the electronic device 600 is a display, there are at least two operation target areas on the display: a nearby button area indicating an area of ​​a nearby button arranged on the side closer to the operator, and a distant button area indicating an area of ​​a distant button arranged on the side farther from the operator. In Fig. 3, the nearby button is the first button 41, and the distant button is the second button 43.

[0038] 3, the first operation angle 413 and the second operation angle 433 are operation areas that indicate areas that are determined by connecting the position of the user UR and the operation target area set on the electronic device. One operation area is, for example, an area set from a first angle that indicates an angle between a first straight line that connects one end of the operation area corresponding to the nearby button area in the direction of a center line C1 that connects the center of the nearby button area and the center of the far button area, and the position of the operator, and a second straight line that connects the other end of the operation area corresponding to the nearby button area in the direction of the center line C1, and the position of the operator.

[0039] In Figure 3, for example, if one end of the first button 41 in the direction of the center line C1 is the right end and the other end of the first button 41 in the direction of the center line C1 is the left end, a first angle indicating the angle between a first line 411 connecting the positions of the users UR and a second line 412 connecting the positions of the users UR corresponds to a first operation angle 413.

[0040] In addition, the other operation area is an area set from a second angle indicating the angle between, for example, one end of the operation area corresponding to the distant button area in the direction of the center line C1 and a third straight line connecting the position of the operator, and the other end of the operation area corresponding to the distant button area in the direction of the center line C1 and a fourth straight line connecting the position of the operator.

[0041] In Figure 3, for example, if one end of the second button 43 in the direction of the center line C1 is the right end and the other end of the second button 43 in the direction of the center line C2 is the left end, a second angle indicating the angle between a third line 431 connecting the positions of the users UR and a fourth line 432 connecting the positions of the users UR corresponds to a second operation angle 433.

[0042] 4, the second determination unit 15 determines whether the position of the user UR is outside the front area of ​​the electronic device 600. Specifically, the second determination unit 15 determines, from the position of the user UR detected by the position detection unit 13, whether the position of the user UR is outside the front area of ​​the electronic device 600.

[0043] Here, the area on the front side of the electronic device 600 may be referred to as the front area. A straight line connecting the center of the width of the front area of ​​the electronic device 600 when viewed from the negative direction of the Y axis to the center of the width of the electronic device 600 is perpendicular to the width direction of the front area. The width of the front area is equal to the width of the electronic device 600 in the X-axis direction. The width of the front area can be set individually. The width of the front area may be smaller than the width of the electronic device 600.

[0044] For example, when the electronic device 600 is long in the horizontal direction, if the user UR is located near an end of the electronic device 600, it may be difficult for the user UR to operate the opposite end of the electronic device 600. In such a case, the same effect can be obtained by setting the width of the front region to be smaller than the width of the electronic device 600.

[0045] The width of the front region is set to be equal to or larger than the shoulder width of the user UR. Since the average shoulder width of a person is approximately 0.22 times the height of a person, for example, the position detection unit 13 may detect the height of the user UR from a captured image in which the user UR is captured, and may set the value obtained by multiplying the detected height value by 0.22 as the width of the front region.

[0046] In the positional relationship between the user UR and the electronic device 600 shown in Fig. 2, the second determination unit 15 determines that the position of the user UR is inside the front area of ​​the electronic device 600. In the positional relationship between the user UR and the electronic device 600 shown in Fig. 3, the second determination unit 15 determines that the position of the user UR is outside the front area of ​​the electronic device 600.

[0047] Returning to Fig. 4, the operation area adjustment unit 16 enlarges the operation area when the position of the operator detected by the position detection unit 13 is outside the front area of ​​the electronic device 600. Specifically, when the second determination unit 15 determines that the position of the operator detected by the position detection unit 13 is outside the front area of ​​the electronic device 600, the operation area adjustment unit 16 enlarges the operation area generated by the operation area generation unit 14. Here, the operation area enlarged by the operation area adjustment unit 16 will be described with reference to Fig. 5.

[0048] Fig. 5 is a schematic diagram for explaining an example of an expanding operation area according to the first embodiment. Like Fig. 3, Fig. 5 shows an electronic device 600, a user UR who operates the electronic device 600, and a first button 41 and a second button 43 output by the electronic device 600.

[0049] When the second determination unit 15 determines that the position of the operator detected by the position detection unit 13 is outside the front area of ​​the electronic device 600, the operation area adjustment unit 16 sets operation areas corresponding to each of the first button area and the second button area so that the first operation angle 413 is equal to the second operation angle 433.

[0050] 5, first, when adjusting the second operation angle 433 to be equal to the first operation angle 413, the operation area adjustment unit 16 expands the operation target area of ​​the second button 43 so that the right end and left end of the second button 43 spread from the center of the second button 43, and sets the operation target area 44 of the second button 43. Then, the operation area adjustment unit 16 sets the size of the angle so that the second operation angle 443 indicating the angle between a third straight line 441 connecting the right end of the operation target area 44 and the position of the operator in the direction of the center line C1 and a fourth straight line 442 connecting the left end of the operation target area 44 and the position of the operator in the direction of the center line C1 is equal to the first operation angle 413 in the operation area corresponding to the expanded operation target area 44.

[0051] Thereby, operation area adjustment unit 16 can make first operation angle 413 equal to second operation angle 443. Then, operation area adjustment unit 16 sets an operation area corresponding to second operation angle 443.

[0052] 4, the motion detection unit 17 detects a predetermined motion of the operator in the operation area based on the captured image of the user UR. Specifically, the motion detection unit 17 detects a predetermined motion of the operator in the operation area generated by the operation area generation unit 14 based on the captured image of the user UR.

[0053] Further, the action detection unit 17 detects a predetermined action of the operator in the operation area enlarged by the operation area adjustment unit 16 based on the captured image in which the user UR is captured. Here, the predetermined action refers to, for example, a state in which the fingertip 72 of the user UR is present in the operation area and touches the operation area. Note that the predetermined action is not limited to this.

[0054] The third determination unit 18 determines whether a predetermined motion has been detected. Specifically, the third determination unit 18 determines whether the motion detection unit 17 has detected a predetermined motion of the operator in the operation area generated by the operation area generation unit 14. The third determination unit 18 also determines whether the motion detection unit 17 has detected a predetermined motion of the operator in the operation area enlarged by the operation area adjustment unit 16.

[0055] The processing execution unit 19 performs processing corresponding to the predetermined motion on the operation target area when a predetermined motion is detected by the motion detection unit 17. Specifically, when the third determination unit 18 determines that a predetermined motion has been detected by the motion detection unit 17, the processing execution unit 19 performs processing corresponding to the predetermined motion on the operation target area.

[0056] Next, the processing procedure of the control device 1 according to the first embodiment will be described with reference to Fig. 6. Fig. 6 is a flowchart showing the processing procedure of the control device 1 according to the first embodiment.

[0057] First, the image acquisition unit 11 acquires a captured image of the user UR from the sensor 500 (step S61). Next, the first determination unit 12 determines whether the user UR who will be the operator has been detected from the captured image acquired by the image acquisition unit 11 (step S62). Here, if the first determination unit 12 determines that the operator has not been detected (step S62: No), the process proceeds to step S61. On the other hand, if the first determination unit 12 determines that the operator has been detected (step S62: Yes), the process proceeds to step S63.

[0058] In step S63, the position detection unit 13 detects the position of the user UR who will be the operator based on the captured image (step S63). Next, the operation area generation unit 14 generates an operation area indicating an area defined by connecting the position of the user UR detected by the position detection unit 13 and an operation target area set on the electronic device based on the captured image in which the user UR is captured (step S64).

[0059] The second determination unit 15 determines whether the position of the user UR is outside the front area of ​​the electronic device 600 from the position of the user UR detected by the position detection unit 13 (step S65). If the second determination unit 15 determines that the position of the user UR is inside the front area of ​​the electronic device 600 (step S65: No), the process proceeds to step S67. On the other hand, if the second determination unit 15 determines that the position of the user UR is outside the front area of ​​the electronic device 600 (step S65: Yes), the process proceeds to step S66.

[0060] In step S66, the operation area adjustment unit 16 enlarges the operation area generated by the operation area generation unit 14 (step S66). In step S67, the motion detection unit 17 detects a predetermined motion of the operator in the operation area generated by the operation area generation unit 14 based on the captured image in which the user UR is captured (step S67).

[0061] Next, the third determination unit 18 determines whether the motion detection unit 17 has detected a predetermined motion of the operator in the operation area generated by the operation area generation unit 14 (step S68). If the third determination unit 18 determines that the predetermined motion has not been detected (step S68: No), the process proceeds to step S67. On the other hand, if the third determination unit 18 determines that the predetermined motion has been detected (step S68: Yes), the process proceeds to step S71.

[0062] In step S69, the motion detection unit 17 detects a predetermined motion of the operator in the operation area enlarged by the operation area adjustment unit 16 based on the captured image in which the user UR is captured (step S69). Next, the third determination unit 18 determines whether the motion detection unit 17 has detected a predetermined motion of the operator in the operation area enlarged by the operation area adjustment unit 16 (step S70). Here, if the third determination unit 18 determines that the predetermined motion has not been detected (step S70: No), the process proceeds to step S69. On the other hand, if the third determination unit 18 determines that the predetermined motion has been detected (step S70: Yes), the process proceeds to step S71.

[0063] In step S71, the process execution unit 19 performs a process corresponding to a predetermined operation on the operation target area (step S71). When the process of step S71 ends, the process proceeds to step S61, and this process is continued while the control device 1 is being executed.

[0064] As described above, the control device 1 according to an embodiment of the present disclosure detects the position of the operator, and detects a predetermined action of the operator in an operation area that indicates an area defined by connecting the detected position of the operator and an operation target area set on the electronic device. When the control device 1 detects the predetermined action, it performs a process associated with the operation target area, and when the position of the operator is outside the area in front of the electronic device, it expands the operation area.

[0065] As a result, when the operator is located outside the area in front of the electronic device 600, the control device 1 expands the operation area, so that the operator can operate a distant button displayed on the electronic device 600 in the same way as a nearby button. Therefore, the control device 1 can improve the operability for the user.

[0066] The above-described embodiment can be modified as appropriate by changing a part of the configuration or function of each of the above-described devices. Therefore, some modifications of the above-described embodiment will be described below as other embodiments. The following mainly describes the differences from the above-described embodiment, and a detailed description of the commonalities with the contents already described will be omitted.

[0067] (First Modification) For example, the control device 1 may expand the operation target area so that the operation target area corresponding to the button of the electronic device 600 overlaps with an area outside the electronic device 600, so long as it is within the detection effective area AR200. Here, the expanded operation target area according to the first modified example will be described with reference to Fig. 7. Fig. 7 is a schematic diagram for explaining an example of the expanded operation target area according to the first modified example.

[0068] 7 shows an electronic device 600, and a first button 41 and a second button 43 output by the electronic device 600. When the control device 1 sets the detection effective area AR200 to a range larger than the electronic device 600, the operation area adjustment section 16 of the control section 10 expands the operation target area 44 of the second button 43 within the range inside the detection effective area AR200.

[0069] (Second Modification) For example, the operability of the buttons output by the electronic device 600 differs when the user UR is located in the area in front of the electronic device 600 compared to when the user UR is located outside the area in front of the electronic device 600. Specifically, the size of the operation area corresponding to the button when the user UR is located in the area in front of the electronic device 600 differs from the size of the operation area corresponding to the button when the user UR is located outside the area in front of the electronic device 600, and the operation area corresponding to the button when the user UR is located outside the area in front of the electronic device 600 is smaller than the operation area corresponding to the button when the user UR is located in the area in front of the electronic device 600.

[0070] Therefore, the control device 1 may enlarge not only the operation target area of ​​the distant button but also the operation target area of ​​the nearby button in the X-axis direction to the left and right, thereby enlarging the operation areas of the distant button and the nearby button. Here, the enlarged operation area according to the second modified example will be described with reference to Fig. 8. Fig. 8 is a schematic diagram for explaining an example of the enlarged operation area according to the second modified example.

[0071] 8 shows an electronic device 600, a user UR who operates the electronic device 600, and a first button 41 and a second button 43 output by the electronic device 600. The first button 41 is a nearby button, and the second button 43 is a distant button. When the second determination unit 15 of the control device 1 determines that the position of the user UR is outside the front area of ​​the electronic device 600, the operation area adjustment unit 16 expands the operation area generated by the operation area generation unit 14.

[0072] 8, the operation area adjustment unit 16 expands the operation target area of ​​the first button 41 so that the right and left ends of the first button 41 spread out from the center of the first button 41, thereby setting an operation target area 42 for the first button 41. In addition, the operation area adjustment unit 16 expands the operation target area of ​​the second button 43 so that the right and left ends of the second button 43 spread out from the center of the second button 43, thereby setting an operation target area 44 for the second button 43.

[0073] Then, the operation area adjustment unit 16 sets a first operation angle 423 indicating an angle between a first straight line 421 connecting the right end of the operation target area 42 and the position of the operator in the direction of the center line C1, and a second straight line 422 connecting the left end of the operation target area 42 and the position of the operator in the direction of the center line C1, in the operation area corresponding to the enlarged operation target area 42. Also, the operation area adjustment unit 16 sets a second operation angle 443 indicating an angle between a third straight line 441 connecting the right end of the operation target area 44 and the position of the operator in the direction of the center line C1, and a fourth straight line 442 connecting the left end of the operation target area 44 and the position of the operator in the direction of the center line C1, in the operation area corresponding to the enlarged operation target area 44, to be equal to the angle of the first operation angle 423.

[0074] As a result, operation area adjustment unit 16 can increase first operation angle 413 to first operation angle 423, and equalize first operation angle 423 and second operation angle 443. Then, operation area adjustment unit 16 sets operation areas corresponding to first operation angle 423 and second operation angle 443, respectively.

[0075] (Third Modification) For example, when the control device 1 enlarges the operation area, the operation target areas may overlap as shown in Fig. 9. Fig. 9 is a schematic diagram for explaining the enlarged operation target area according to the third modified example. Fig. 9 shows an electronic device 600, a first button 41, a second button 43, and a third button 45 output by the electronic device 600, an enlarged operation target area 42 corresponding to the first button 41, an enlarged operation target area 44 corresponding to the second button 43, and an enlarged operation target area 46 corresponding to the third button 45.

[0076] For example, when the operation area adjustment unit 16 adjusts the operation areas for each of the operation target area 42, the operation target area 44, and the operation target area 46, the operation area adjustment unit 16 may erroneously detect the action of the user UR, and it may be difficult to adjust a plurality of operation areas. Therefore, the control device 1 according to the third modification may expand the operation target areas, and when the expanded operation target areas overlap, expand the operation target areas so that they do not overlap, according to the configuration of the buttons output by the electronic device 600.

[0077] Fig. 10 is a schematic diagram for explaining an enlarged operation target area according to the third modified example. Fig. 10 shows an electronic device 600, a user UR who operates the electronic device 600, a first button 41, a second button 43, a third button 45 output by the electronic device 600, an enlarged operation target area 42 corresponding to the first button 41, an enlarged operation target area 44 corresponding to the second button 43, and an enlarged operation target area 46 corresponding to the third button 45. In addition, when the position of the user UR is taken as a base point, the positions of the respective buttons are located closer to the user UR in the order of the first button 41, the third button 45, and the second button 43.

[0078] When the user UR is located outside the area in front of the electronic device 600, the operation area adjustment unit 16 enlarges the operation target areas of the first button 41, the second button 43, and the third button 45 so that the operation target areas 42, 44, and 46 do not overlap. In this case, the operation angles of the buttons are not the same, but the operation target areas of the first button 41, the second button 43, and the third button 45 are wider than before the operation target areas are enlarged, thereby improving the operability of the user UR.

[0079] (Fourth Modification) In the above embodiment, the case where the buttons output by the electronic device 600 are the same size has been described, but the present invention is not limited to this. In the fourth modification, a case where the buttons output by the electronic device 600 are different sizes will be described with reference to FIG.

[0080] FIG. 11 is a schematic diagram for explaining an example of an enlarged operation area according to the fourth modified example. FIG. 11 shows an electronic device 600, a user UR who operates the electronic device 600, and a first button 41 and a second button 43 output by the electronic device 600. The first button 41 is a nearby button, and the second button 43 is a distant button. The first button 41 is smaller than the second button 43 and is positioned approximately parallel to the second button 43. When the second determination unit 15 of the control device 1 determines that the position of the user UR is outside the front area of ​​the electronic device 600, the operation area adjustment unit 16 enlarges the operation area generated by the operation area generation unit 14.

[0081] 11, the operation area adjustment unit 16 expands the operation target area of ​​the first button 41 so that the right and left ends of the first button 41 spread out from the center of the first button 41, and sets an operation target area 42 for the first button 41. The operation area adjustment unit 16 also expands the operation target area of ​​the second button 43 so that the right and left ends of the second button 43 spread out from the center of the second button 43, and sets an operation target area 44 for the second button 43. At this time, the operation area adjustment unit 16 sets the operation target area 42 and the operation target area 44 so that they do not overlap.

[0082] Then, the operation area adjustment unit 16 sets a first operation angle 423 indicating an angle between a first straight line 421 connecting the right end of the operation target area 42 and the position of the operator in the direction of the center line C1, and a second straight line 422 connecting the left end of the operation target area 42 and the position of the operator in the direction of the center line C1, in the operation area corresponding to the enlarged operation target area 42. Also, the operation area adjustment unit 16 sets a second operation angle 443 indicating an angle between a third straight line 441 connecting the right end of the operation target area 44 and the position of the operator in the direction of the center line C1, and a fourth straight line 442 connecting the left end of the operation target area 44 and the position of the operator in the direction of the center line C1, in the operation area corresponding to the enlarged operation target area 44, to be equal to the angle of the first operation angle 423.

[0083] As a result, operation area adjustment unit 16 can increase first operation angle 413 to first operation angle 423, and equalize first operation angle 423 and second operation angle 443. Then, operation area adjustment unit 16 sets operation areas corresponding to first operation angle 423 and second operation angle 443, respectively.

[0084] (Fifth Modification) In the above-described embodiment, the case where the buttons output by the electronic device 600 are positioned substantially parallel is described, but the present invention is not limited thereto. In the fifth modified example, a case where the buttons output by the electronic device 600 are positioned substantially vertically is described with reference to FIG.

[0085] Fig. 12 is a schematic diagram for explaining an example of an enlarged operation area according to the fifth modified example. Fig. 12 shows an electronic device 600, a user UR who operates the electronic device 600, and a first button 41 and a second button 43 output by the electronic device 600. The first button 41 and the second button 43 are the same size and positioned substantially vertically. The first button 41 is a nearby button, and the second button 43 is a distant button. When the second determination unit 15 of the control device 1 determines that the position of the user UR is outside the area in front of the electronic device 600, the operation area adjustment unit 16 enlarges the operation area generated by the operation area generation unit 14.

[0086] In the case of Figure 12, the operation area generation unit 14 generates an operation area corresponding to the nearby button area of ​​the first button 41 from a third operation angle 416 indicating the angle between a fifth line 414 connecting the upper end of the first button 41 and the position of the operator in the direction of a center line C2 connecting the center of the nearby button area of ​​the first button 41 and the center of the distant button area of ​​the second button 43, and a sixth line 415 connecting the lower end of the first button 41 and the position of the operator in the direction of the center line C2.

[0087] When adjusting the fourth operation angle 446 to be equal to the third operation angle 416, the operation area adjustment unit 16 expands the operation target area of ​​the second button 43 so that the upper end and the lower end of the second button 43 spread from the center of the second button 43, and sets the operation target area 44 of the second button 43. Then, the operation area adjustment unit 16 sets the size of the angle such that the fourth operation angle 446 indicating the angle formed by a seventh straight line 444 connecting the upper end of the operation target area 44 in the direction of the center line C2 and the position of the operator and an eighth straight line 445 connecting the lower end of the operation target area 44 in the direction of the center line C2 and the position of the operator in the operation area corresponding to the expanded operation target area 44 is equal to the third operation angle 416.

[0088] In this way, operation area adjustment unit 16 can make fourth operation angle 446 equal to third operation angle 416. Then, operation area adjustment unit 16 sets an operation area corresponding to fourth operation angle 446.

[0089] (Sixth Modification) In the sixth modified example, a case where the buttons output by the electronic device 600 are located substantially diagonally will be described with reference to Fig. 13. Fig. 13 and Fig. 14 are schematic diagrams for explaining an example of an expanding operation region according to the sixth modified example.

[0090] 13 and 14 show an electronic device 600, a user UR who operates the electronic device 600, and a first button 41 and a second button 43 output by the electronic device 600. The first button 41 and the second button 43 are the same size and are located approximately diagonally. In FIG. 13, the second button 43 is located at the upper left of the first button 41. The first button 41 is a nearby button, and the second button 43 is a distant button. When the second determination unit 15 of the control device 1 determines that the position of the user UR is outside the front area of ​​the electronic device 600, the operation area adjustment unit 16 expands the operation area generated by the operation area generation unit 14.

[0091] In the case of Fig. 13, the operation area generating unit 14 sets a first operation angle 413 indicating an angle between a first straight line 411 connecting the right end of the first button 41 and the position of the operator in the direction of a center line C3 connecting the center of the near button area of ​​the first button 41 and the center of the far button area of ​​the second button 43, and a second straight line 412 connecting the left end of the first button 41 and the position of the operator in the direction of the center line C3, in the operation area corresponding to the near button area of ​​the first button 41. In addition, the operation area generating unit 14 sets a third operation angle 416 indicating an angle between a fifth straight line 414 connecting the upper end of the first button 41 and the position of the operator in the direction of the center line C3, and a sixth straight line 415 connecting the lower end of the first button 41 and the position of the operator in the direction of the center line C3, in the operation area corresponding to the near button area of ​​the first button 41.

[0092] 14, when the operation area adjustment unit 16 adjusts the second operation angle 443 of the second button 43 to be equal to the first operation angle 413, the operation area adjustment unit 16 enlarges the operation target area of ​​the second button 43 so that the right end and left end of the second button 43 spread from the center of the second button 43, and sets the operation target area 44 of the second button 43. Then, the operation area adjustment unit 16 sets the size of the angle such that the second operation angle 443 indicating the angle formed by a third line 441 connecting the right end of the operation target area 44 and the position of the operator in the direction of the center line C3 and a fourth line 442 connecting the left end of the operation target area 44 and the position of the operator in the direction of the center line C3, in the operation area corresponding to the enlarged operation target area 44, is equal to the first operation angle 413.

[0093] Furthermore, when adjusting the fourth operation angle 446 of the second button 43 to be equal to the third operation angle 416, the operation area adjustment unit 16 further expands the operation target area of ​​the second button 43 so that the upper end and the lower end of the second button 43 spread from the center of the second button 43, and sets the operation target area 44 of the second button 43. Then, the operation area adjustment unit 16 sets the size of the angle such that a fourth operation angle 446 indicating an angle formed by a seventh line 444 connecting the upper end of the operation target area 44 in the direction of the center line C3 and the position of the operator and an eighth line 445 connecting the lower end of the operation target area 44 in the direction of the center line C3 and the position of the operator, in the operation area corresponding to the further expanded operation target area 44, is equal to the third operation angle 416.

[0094] Thereby, the operation area adjustment unit 16 can make the second operation angle 443 equal to the first operation angle 413. Also, the operation area adjustment unit 16 can make the fourth operation angle 446 equal to the third operation angle 416. Then, the operation area adjustment unit 16 sets an operation area corresponding to the second operation angle 443 and the fourth operation angle 446.

[0095] (Seventh Modification) In the above-described embodiment, the embodiment in which the electronic device 600 is one is described, but the present invention is not limited to this. In the seventh modification, the embodiment in which the electronic device is two is described with reference to Fig. 15. Fig. 15 is a schematic diagram for explaining the operation process of the information processing device according to the seventh modification.

[0096] 15 shows a first electronic device 601, a second electronic device 602, and a user UR who operates the first electronic device 601 and the second electronic device 602. The first electronic device 601 is a light, and the second electronic device 602 is a display.

[0097] The control device 1 of the seventh variant has a first electronic device 601 and a second electronic device 602 as electronic devices, and the operation area adjustment unit 16 expands the operation area of ​​at least one of the first electronic device 601 and the second electronic device 602 when the position of the operator detected by the position detection unit 13 is outside the front area corresponding to the front of the first electronic device 601 and the second electronic device 602.

[0098] For example, as shown in Fig. 15, the operation area generating unit 14 generates an operation area indicating an area defined by connecting the position of the user UR detected by the position detecting unit 13 and operation target areas set on a plurality of electronic devices based on a captured image in which the user UR is captured. In Fig. 14, the operation target areas are a first electronic device area 47 indicating an area of ​​the first electronic device 601 located closer to the user UR, and a second electronic device area 48 indicating an area of ​​the first electronic device 601 located farther from the user UR.

[0099] In addition, when the position of the user UR is outside the front area corresponding to the front of the first electronic device 601 and the second electronic device 602, the operation area adjustment unit 16 sets the operation areas corresponding to each of the first electronic device area 47 and the second electronic device area 48 so that the first operation angle 473 is equal to the second operation angle 483.

[0100] The first operation angle 473 is an angle between a first straight line 471 connecting the right end of the operation area corresponding to the first electronic device area 47 in the direction of the center line C4 connecting the center of the first electronic device area 47 and the center of the second electronic device area 48 and the position of the operator, and a second straight line 472 connecting the left end of the operation area corresponding to the first electronic device area 47 in the direction of the center line C4 and the position of the operator. The second operation angle 483 is an angle between a third straight line 481 connecting the right end of the operation area corresponding to the second electronic device area 48 in the direction of the center line C4 and the position of the operator, and a fourth straight line 482 connecting the left end of the operation area corresponding to the second electronic device area 48 in the direction of the center line C4 and the position of the operator.

[0101] Second embodiment In the above-described embodiment, the predetermined action refers to a state in which the fingertip 72 of the user UR is present in the operation area and touches the operation area, but the present invention is not limited to this. In the second embodiment, the predetermined action refers to a scroll operation.

[0102] Fig. 16 is a schematic diagram for explaining the operation process of an information processing device according to a second comparative example. Fig. 16 shows an electronic device 600 and a user UR who operates the electronic device 600. It is assumed that the user UR is located in front of the electronic device 600 and that the distance between the user UR and the electronic device 600 is the shortest. It is also assumed that the electronic device 600 is located perpendicular to the Y-axis direction.

[0103] Here, an operation process of the information processing device will be described when the user UR scrolls the electronic device 600. The operation angle between the user UR and the electronic device 600 is generated as a first operation angle 53 from a first straight line 51 connecting the user UR and the right end of the electronic device 600 and a second straight line 52 connecting the user UR and the left end of the electronic device 600, with the user UR as the base point. In addition, an operation target movement distance 54 of an operation target area where the user UR scrolls the electronic device 600 corresponds to an operation movement distance 55 in the operation area where the fingertip of the user UR scrolls.

[0104] Furthermore, when the user UR scrolls the electronic device 600, the user UR and the electronic device 600 are positioned parallel to each other in the X-axis direction and are equal in distance in the Z-axis direction, so that the operation target movement distance 54 is proportional to the movement speed of the scroll operation of the operation movement distance 55. Therefore, in the positional relationship between the user UR and the electronic device 600 in FIG. 16, the user UR can operate the electronic device 600 without feeling uncomfortable.

[0105] Fig. 17 is a schematic diagram for explaining the operation process of the information processing device according to the second embodiment. Fig. 17 shows an electronic device 600 and a user UR who operates the electronic device 600, similarly to Fig. 16. The operation angle between the user UR and the electronic device 600 is generated as a first operation angle 53 from a first straight line 51 that connects the user UR and the right end of the electronic device 600 and a second straight line 52 that connects the user UR and the left end of the electronic device 600, with the user UR as the base point. In addition, an operation target movement distance 54 of an operation target area where the user UR scrolls the electronic device 600 corresponds to an operation movement distance 55 in the operation area where the fingertip of the user UR scrolls.

[0106] Here, when the user UR is located outside the front area of ​​the electronic device 600, for example, when the user UR is located to the right of the front area of ​​the electronic device 600 as shown in Fig. 17, the positional relationship between the user UR and the electronic device 600 is not parallel. Also, when the user UR is located to the right of the front area of ​​the electronic device 600, the first straight line 51 is shorter than the second straight line 52.

[0107] When the user UR scrolls from the right end to the left end of the electronic device 600, the user may feel that the movement speed is faster as the user scrolls to the left end. This is because the operation object movement distance 56 when the user UR and the electronic device 600 are parallel to each other in the X-axis direction and the distance in the Z-axis direction is equal is different from the operation object movement distance 54 corresponding to the operation movement distance 55. For example, in FIG. 17, when the operation object movement distance 56 is applied to a position parallel to the operation movement distance 55, the operation object movement distance 56 is shorter than the operation object movement distance 54 when the user UR and the electronic device 600 are parallel to each other.

[0108] Therefore, when the user UR is located to the right of the front area of ​​the electronic device 600, the operation object movement distance 54 is not proportional to the movement speed of the scroll operation of the operation movement distance 55. Therefore, when the user UR scrolls from the right end to the left end of the electronic device 600, the user UR may feel that the movement speed is faster as the scroll operation moves toward the left end, and the user UR may feel that the scrolling operation is difficult due to the change in positional relationship with the electronic device 600.

[0109] In view of this, in the second embodiment, an information processing device capable of improving operability for the user UR when the user UR is positioned outside the front area of ​​the electronic device 600 during a scroll operation by the user UR is provided.

[0110] Specifically, the information processing device according to the second embodiment determines the size of the enlarged operation area based on the positional relationship between the electronic device 600 and the operator. For example, when the electronic device 600 is a display, the information processing device according to the second embodiment determines the size of the enlarged operation area based on the size of the operation area when the display is placed in front of the operator with the shortest distance between the display and the operator being equal.

[0111] Fig. 18 is a diagram showing a schematic configuration of an information system including a control device 2 which is an information processing device according to the second embodiment. The information processing device according to the second embodiment includes a sensor 500, an electronic device 600, and a control device 2. Note that in Fig. 18, detailed description of points common to the control device 1 according to the first embodiment shown in Fig. 4 will be omitted.

[0112] The control device 2 includes a control unit 10 and a storage unit 30. The control unit 10 is configured as, for example, a CPU, and controls the overall operation of each unit of the control device 2. The control device 2 according to this embodiment includes a ROM and a RAM (not shown). The ROM stores various programs. The RAM is a working area when the CPU executes the programs.

[0113] The CPU uses the RAM as a working area and executes the programs stored in the ROM, thereby realizing an image acquisition unit 11, a first judgment unit 12, a position detection unit 13, an operation area generation unit 14, a second judgment unit 15, an operation area adjustment unit 16, a motion detection unit 17, a third judgment unit 18, and a processing execution unit 19, as shown in FIG. 18.

[0114] The storage unit 30 stores coordinate information 31, detection effective area information 32, operation area information 33, expanded operation area information 34, device operation amount information 35, and movement amount information 36. The movement amount information 36 is information indicating the amount of movement of the fingertip 72 of the user UR. The device operation amount information 35 is information indicating an operation of the electronic device 600 corresponding to the amount of movement of the fingertip 72 of the user UR. The operation of the electronic device 600 includes, for example, a scroll operation on the electronic device 600.

[0115] The operation area generating unit 14 generates an operation area indicating an area determined by connecting the position of the user UR detected by the position detecting unit 13 and an operation target area set on the electronic device based on a captured image of the user UR. For example, the operation area generating unit 14 determines the size of the enlarged operation area based on the size of the operation area when the electronic device 600 is a display and the display is placed in front of the operator with the shortest distance between the display and the operator being equal. Here, the operation area generated by the operation area generating unit 14 will be described with reference to FIG. 17.

[0116] Here, the operation target area is, for example, an area on the display where scrolling is performed from one end of the display to the other end when the electronic device 600 is a display. In FIG. 17, one end of the display is the right end of the electronic device 600, and the other end of the display is the left end of the electronic device 600.

[0117] 17, the first operation angle 53 is an operation area indicating an area determined by connecting the position of the user UR and an operation target area set on the electronic device. The operation area is an area set from the first operation angle 53 indicating an angle between a first straight line 51 connecting the right end of the electronic device 600 and the position of the operator and a second straight line 52 connecting the left end of the electronic device 600 and the position of the operator, for example.

[0118] 18, when the second determination unit 15 determines that the position of the operator detected by the position detection unit 13 is outside the front area of ​​the electronic device 600, the operation area adjustment unit 16 enlarges the operation area generated by the operation area generation unit 14. Here, the operation area enlarged by the operation area adjustment unit 16 will be described with reference to FIG.

[0119] Fig. 19 is a schematic diagram for explaining an example of an expanding operation area according to the second embodiment. Fig. 19 shows an electronic device 600 and a user UR who operates the electronic device 600, similar to Fig. 17. Here, the distance of the operation target movement distance 54 when the user UR and the electronic device 600 are parallel is d [mm], the distance from the user UR to the fingertip is a [mm], the distance of the first straight line 51 is b [mm], and the angle of the first operation angle 53 is θ [degree].

[0120] 19, the moving distance P [mm] of the fingertip of the user UR can be calculated as P = a × sin(θ / 2). In addition, when the electronic device 600 is located parallel to the user UR at the distance of the first straight line 51, if the operation target moving distance 56 of the electronic device 600 is Q [mm], it can be calculated as Q = 2b × sin(θ / 2).

[0121] In this case, when the fingertip of the user UR scrolls a distance P [mm], scrolling up to Q [mm] on the electronic device 600 will be the same as the distance moved on the electronic device 600 when the user UR is located in the area in front of the electronic device 600, and therefore the scrolling speed will be the same as when the user UR is located in the area in front of the electronic device 600.

[0122] On the other hand, since the user UR is located to the right of the front area of ​​the electronic device 600, the operation target movement distance 54 when the user UR and the electronic device 600 are parallel leaves an operation target area of ​​dQ [mm]. Therefore, the operation area adjustment unit 16 enlarges the operation target movement distance 56 by dQ [mm] as the enlarged operation target movement distance 561 and determines the size of the enlarged operation area in order to scroll at the same scroll speed as when the user UR is located in the front area of ​​the electronic device 600 with the shortest distance between the electronic device 600 and the user UR being equal, for the remaining distance dQ [mm] of the operation target area. As a result, the enlarged operation target movement distance 561 becomes 2d-Q [mm].

[0123] Furthermore, the operation area adjustment unit 16 enlarges the operation movement distance 551 corresponding to the enlargement operation target movement distance 561 by a×(d / b−sin(θ / 2)) [mm] and determines the size of the enlarged operation area. As a result, the operation movement distance 551 corresponding to the enlargement operation target movement distance 561 becomes a×(1+d / b−sin(θ / 2)) [mm].

[0124] Furthermore, the operation area adjustment unit 16 enlarges the enlarged first operation angle 531 corresponding to the enlargement operation target movement distance 561 by (θ-2a*sin(d / b)-a*θ) [degrees] and determines the size of the enlarged operation area. As a result, the first operation angle 531 corresponding to the enlargement operation target movement distance 561 becomes (2θ-2a*sin(d / b)-a*θ) [degrees].

[0125] As described above, the control device 2 according to the second modification of the present disclosure determines the size of the enlarged operation area based on the positional relationship between the electronic device 600 and the operator. The control device 2 determines the size of the enlarged operation area based on the size of the operation area when the electronic device 600 is a display and the display is placed in front of the operator with the shortest distance between the display and the operator being equal.

[0126] As a result, when an operator performs a scroll operation, for example, if the operator is located outside the area in front of the electronic device 600, the control device 2 expands the operation area, allowing the operator to perform the same scroll operation as when the operator is located in the area in front of the electronic device 600. Therefore, the control device 2 can improve operability for the user.

[0127] (Eighth Modification) For example, the control device 2 may expand the operation target area of ​​the electronic device 600 so that the operation target area overlaps with an area outside the electronic device 600 as long as the operation target area is within the detection effective area AR200. Here, the expanded operation target area according to the eighth modified example will be described with reference to Fig. 20. Fig. 20 is a schematic diagram for explaining an example of the expanded operation target area according to the eighth modified example.

[0128] Fig. 20 corresponds to the contents shown in Fig. 19. Also, the length of electronic device 600 is L1 [mm], the length of detection effective area AR200 of electronic device 600 is L2 [mm], and L2>L1>0. When control device 2 sets detection effective area AR200 to a range larger than electronic device 600, operation area adjustment section 16 of control section 10 expands operation target movement distance 56 within the range inside detection effective area AR200, and determines the size of the expanded operation area corresponding to expanded operation target movement distance 561.

[0129] (Ninth Modification) In the above-described embodiment, the electronic device 600 is described as being positioned perpendicular to the Y-axis direction, but is not limited thereto. In the ninth modification, a form in which the electronic device 600 is inclined in the positive direction of the Z-axis with respect to the user UR will be described with reference to FIG.

[0130] Fig. 21 is a schematic diagram for explaining an example of an expanding operation region according to the ninth modification. Fig. 21 shows an electronic device 600 and a user UR who operates the electronic device. The electronic device 600 is positioned perpendicular to the Y-axis direction. The electronic device 603 is inclined in the positive direction of the Z-axis with respect to the user UR.

[0131] The operation area generating unit 14 generates an operation target moving distance 64 of an operation target area where the user UR scrolls the electronic device 600 located vertically from a first operation angle 63 indicating an angle between a first straight line 61 connecting the lower end of the electronic device 600 and the position of the operator and a second straight line 62 connecting the upper end of the electronic device 600 and the position of the operator. The operation target moving distance 64 in the case of FIG. 21 is the scroll distance for scrolling in the Y-axis direction.

[0132] Then, the operation area adjustment unit 16 determines the size of the enlarged operation area by setting the operation target movement distance 66 corresponding to the electronic device 603 as the enlarged operation target movement distance 661 in order to perform a scroll operation on the tilted electronic device 603 at the same scroll speed as when the user UR is positioned vertically to the electronic device 600. Specifically, the operation area adjustment unit 16 determines the operation target movement distance 64 and the enlarged operation target movement distance 661 so that they are equal to each other.

[0133] Also, the operation area adjustment unit 16 determines the size of the enlarged operation area for the enlarged operation movement distance 651 corresponding to the enlargement operation target movement distance 661. Furthermore, the operation area adjustment unit 16 determines the size of the enlarged operation area for the enlarged first operation angle 631 corresponding to the enlargement operation target movement distance 661.

[0134] (Tenth Modification) The sensor 500 is not limited to a visible light camera, but may be, for example, a charge coupled device (CCD) camera, a complementary metal oxide semiconductor (CMOS) camera, or an RGB camera. The sensor 500 is not limited to a camera device, but may be a time of flight (TOF) sensor or an infrared sensor.

[0135] (Eleventh Modification) The position detection unit 13 detects the fingertips of the user UR based on an image of the user who is the operator, but is not limited to this. For example, the position detection unit 13 may detect the hand, palm, or arm of the user UR.

[0136] Third embodiment In the third embodiment, a specific action is performed when the fingertip 72 of the user UR is present in the enlarged operation target area and the enlarged operation target area is selected. Specifically, when the action detection unit 17 detects an action of selecting one of the multiple operation areas, the operation area adjustment unit 16 of the control unit 10 determines the size of the enlarged operation target area according to the specific action.

[0137] Fig. 22 is a schematic diagram for explaining an example of an enlarged operation target area according to the third embodiment. Fig. 22 shows an electronic device 600 and a button 81, a button 82, a button 83, a button 84, and a button 85 output by the electronic device 600. Fig. 22 also shows an enlarged operation target area 831 corresponding to the button 83, an enlarged operation target area 841 corresponding to the button 84, and an enlarged operation target area 851 corresponding to the button 85. Here, in Fig. 22, the operation target area 831, the operation target area 842, and the operation target area 851 are equal in size.

[0138] 22, the motion detection unit 17 of the control unit 10 detects the motion of the user UR selecting the button 84 in the operation target area enlarged by the operation area adjustment unit 16 based on the captured image in which the user UR is captured. For example, the motion detection unit 17 detects the motion of the user UR selecting the button 84 in the operation target area enlarged by the operation area adjustment unit 16 when the fingertip 72 of the user UR enters a range closer to the electronic device 600 than the virtual operation area AR100. Here, the motion of the user UR selecting the button 84 includes the motion of the fingertip 72 of the user UR moving toward the button 84 and preceding the selection of the button 84.

[0139] Then, as shown in FIG. 23, the operation area adjustment unit 16 enlarges the size of the operation target area 841 corresponding to the button 84 before being selected, and reduces the sizes of the operation target area 831 and the operation target area 851 adjacent to the operation target area 841.

[0140] Fig. 23 is a schematic diagram for explaining an example of an enlarged operation target area according to the third embodiment. Fig. 23 is a schematic diagram of a state before the user UR shown in Fig. 22 selects the button 84. Fig. 23 shows the electronic device 600 and the buttons 81, 82, 83, 84, and 85 output by the electronic device 600, similar to Fig. 22. Fig. 23 also shows a reduced operation target area 832 corresponding to the button 83, an enlarged operation target area 842 corresponding to the button 84, and a reduced operation target area 852 corresponding to the button 85.

[0141] 23, operation target area 842 is the largest in size, followed by operation target area 832 and operation target area 852 adjacent to operation target area 842, which are equal in size and smaller than operation target area 842. That is, when the sizes of the enlarged operation target areas corresponding to a plurality of operation areas are equal, operation area adjustment unit 16 enlarges the size of the enlarged operation target area corresponding to the operation area to be selected, and reduces the sizes of the enlarged operation target areas corresponding to the other operation areas adjacent to the operation area to be selected.

[0142] Next, a state before the user UR selects button 85 by performing a sliding action in the positive direction of the X-axis from a state before selecting button 84 shown in Fig. 23 will be described. For example, in Fig. 23, the action detection unit 17 detects an action of the user UR selecting button 85 in the operation target area enlarged by the operation area adjustment unit 16 based on a captured image in which the user UR is captured. Then, as shown in Fig. 24, the operation area adjustment unit 16 enlarges the size of the operation target area 852 corresponding to the button 85 before selection, and reduces the size of the operation target area 842 adjacent to the operation target area 852.

[0143] Fig. 24 is a schematic diagram for explaining an example of an enlarged operation target area according to the third embodiment. Fig. 24 is a schematic diagram of a state in which the user UR shown in Fig. 23 has selected the button 85. Fig. 24 shows the electronic device 600 and the buttons 81, 82, 83, 84, and 85 output by the electronic device 600, similarly to Figs. 22 and 23. Fig. 24 also shows an enlarged operation target area 833 corresponding to the button 83, a reduced operation target area 843 corresponding to the button 84, and an enlarged operation target area 853 corresponding to the button 85.

[0144] 24, operation target area 853 is the largest in size, operation target area 833 is next smaller in size than operation target area 853, and operation target area 842 adjacent to operation target area 853 is smaller in size than operation target area 853. That is, operation area adjustment unit 16 enlarges the size of the operation target area corresponding to the operation target area before selection, reduces the sizes of other operation target areas adjacent to the operation target area before selection, and maintains the sizes of other operation target areas not adjacent to the operation target area before selection.

[0145] As described above, when an action of selecting one operation area from among multiple operation areas is detected, the control device according to the third embodiment of the present disclosure determines the size of the enlarged operation target area in accordance with a predetermined action.

[0146] In this way, for example, when an operator selects an operation area, the control device can determine the size of the operation target area corresponding to the operation area in accordance with the selecting action, thereby improving the operability of the control device for the user.

[0147] (Tenth Modification) In the above-described third embodiment, a form in which the enlarged operation target areas corresponding to the buttons output by the electronic device 600 are uniform is described, but the present invention is not limited to this. In the tenth modification, a form in which the enlarged operation target areas corresponding to the buttons output by the electronic device 600 are different is described.

[0148] Fig. 25 is a schematic diagram for explaining an example of an enlarged operation target area according to Modification 10. Like Fig. 22, Fig. 25 shows electronic device 600 and buttons 81, 82, 83, 84, and 85 output by electronic device 600. Fig. 25 also shows enlarged operation target area 834 corresponding to button 83, enlarged operation target area 844 corresponding to button 84, and enlarged operation target area 854 corresponding to button 85.

[0149] 25, the operation target area 834, the operation target area 844, and the operation target area 854 have different sizes, and the sizes increase in the order of the operation target area 834, the operation target area 844, and the operation target area 854. For example, when the user UR is located in front of the button 83, the operation area adjustment unit 16 increases the size of the operation target area corresponding to the button output by the electronic device 600 as it moves away from the center of the user UR.

[0150] 25, the action detection unit 17 detects, based on a captured image of the user UR, an action of the user UR selecting a button 84 in the operation target area enlarged by the operation area adjustment unit 16. Then, as shown in Fig. 26, the operation area adjustment unit 16 enlarges the size of the operation target area 844 corresponding to the button 84 before selection, and reduces the sizes of the operation target area 834 and the operation target area 854 adjacent to the operation target area 844.

[0151] Fig. 26 is a schematic diagram for explaining an example of an enlarged operation target area according to the tenth modified example. Fig. 26 is a schematic diagram of a state before the user UR shown in Fig. 25 selects the button 84. Fig. 26 shows the electronic device 600 and the buttons 81, 82, 83, 84, and 85 output by the electronic device 600, similar to Fig. 25. Fig. 26 also shows a reduced operation target area 835 corresponding to the button 83, an enlarged operation target area 845 corresponding to the button 84, and a reduced operation target area 855 corresponding to the button 85.

[0152] 26, the operation target area 845 corresponding to the button 84 before the user UR selects it is the largest, followed by the operation target area 855 adjacent to the operation target area 845 and far from the center of the user UR, and then the operation target area 835 adjacent to the operation target area 845 and located in front of the user UR, which is smaller than the operation target area 845. In other words, when the sizes of the operation target areas after enlargement corresponding to a plurality of operation areas are different, the operation target area adjustment unit 16 enlarges the size of the operation target area corresponding to the selected operation area and reduces the sizes of the other operation target areas adjacent to the selected operation target area.

[0153] (Eleventh Modification) In the eleventh modification, a form will be described in which the enlarged operation target areas corresponding to the buttons output by the electronic device 600 are selected in a case where the enlarged operation target areas overlap each other. Specifically, when the operation detection section 17 detects an operation of selecting one of the multiple operation areas, the operation area adjustment section 16 moves the enlarged operation target area.

[0154] Fig. 27 is a schematic diagram for explaining an example of an enlarged operation target area according to Modification 11. Like Fig. 22, Fig. 27 shows electronic device 600 and buttons 81, 82, 83, 84, and 85 output by electronic device 600. Fig. 27 also shows enlarged operation target area 836 corresponding to button 83, enlarged operation target area 846 corresponding to button 84, and enlarged operation target area 856 corresponding to button 85.

[0155] 27, the operation target area 836, the operation target area 846, and the operation target area 856 are all equal in size, and the right side of the operation target area 836 overlaps with the left side of the operation target area 846, and the right side of the operation target area 846 overlaps with the left side of the operation target area 856. In addition, the operation target area 836, the operation target area 846, and the operation target area 856 are all equal in size.

[0156] For example, in Fig. 27, the action detection unit 17 detects an action performed by the user UR before selecting a button in the operation target area enlarged by the operation area adjustment unit 16 based on a captured image of the user UR. Then, as shown in Fig. 28, the operation area adjustment unit 16 moves the operation target area in response to the action of the fingertip 72 of the user UR.

[0157] Here, a process performed by the operation area adjustment unit 16 in a state in which the fingertip 72 of the user UR is moving to the right of the electronic device 600 in the positive direction of the X axis will be described with reference to FIG.

[0158] Fig. 28 is a schematic diagram for explaining an example of an enlarged operation target area according to Modification Example 11. Like Fig. 27, Fig. 28 shows electronic device 600 and buttons 81, 82, 83, 84, and 85 output by electronic device 600. Fig. 28 also shows operation target area 837 corresponding to button 83, operation target area 847 corresponding to button 84, and operation target area 857 corresponding to button 85.

[0159] The positions of the operation target area 837, the operation target area 847, and the operation target area 857 in Fig. 28 in the X-axis direction have moved to the right compared to the positions of the operation target area 836, the operation target area 846, and the operation target area 856 in Fig. 27 in the X-axis direction. This is a state resulting from the operation area adjustment unit 16 moving the positions of the operation target area 837, the operation target area 847, and the operation target area 857 in the X-axis direction in the positive direction of the X-axis in response to the action of the fingertip 72 of the user UR moving in the positive direction of the X-axis as shown by the arrow M1. That is, when the action detection unit 17 detects an action toward a first direction indicating the left and right of the front of the multiple operation areas, the operation area adjustment unit 16 moves the enlarged operation target area so as to move in the first direction.

[0160] Next, a process performed by the operation area adjustment unit 16 in a state in which the fingertip 72 of the user UR is moving to the left side of the electronic device 600 in the negative direction of the X axis will be described with reference to FIG.

[0161] Fig. 29 is a schematic diagram for explaining an example of an enlarged operation target area according to Modification Example 11. Like Fig. 27, Fig. 29 shows electronic device 600 and buttons 81, 82, 83, 84, and 85 output by electronic device 600. Fig. 29 also shows operation target area 838 corresponding to button 83, operation target area 848 corresponding to button 84, and operation target area 858 corresponding to button 85.

[0162] The positions of the operation target area 838, the operation target area 848, and the operation target area 858 in Fig. 29 in the X-axis direction have moved to the left side compared to the positions of the operation target area 836, the operation target area 846, and the operation target area 856 in the X-axis direction in Fig. 27. This is a state resulting from the operation target area adjustment unit 16 moving the positions of the operation target area 838, the operation target area 848, and the operation target area 858 in the negative direction of the X-axis in response to the action of the fingertip 72 of the user UR moving in the negative direction of the X-axis as shown by the arrow M2. That is, when the action detection unit 17 detects an action toward a first direction indicating the left and right of the front of the multiple operation areas, the operation target area adjustment unit 16 moves the enlarged operation target area so as to move in the first direction.

[0163] Next, the operation area adjustment unit 16 will be described with respect to an operation in which the fingertip 72 of the user UR selects a button after the enlarged operation target area is moved in accordance with the operation of the fingertip 72 of the user UR.

[0164] 28, the action detection unit 17 detects, based on a captured image of the user UR, an action of the user UR selecting a button 84 in the operation target area enlarged by the operation area adjustment unit 16. For example, when the fingertip 72 of the user UR enters a range closer to the electronic device 600 than the virtual operation area AR100 and the fingertip 72 of the user UR remains stationary for a certain period of time in the detection effective area AR200, the action detection unit 17 detects an action of the user UR selecting a button 84 in the operation target area enlarged by the operation area adjustment unit 16.

[0165] Then, the operation area adjustment section 16 enlarges the size of the operation target area 847 corresponding to the selected button 84 , and reduces the sizes of the operation target area 837 and the operation target area 857 adjacent to the operation target area 847 .

[0166] Fig. 30 is a schematic diagram for explaining an example of an enlarged operation target area according to the 11th modified example. Fig. 30 is a schematic diagram showing a state in which the user UR shown in Fig. 28 has selected the button 84. Fig. 30 shows the electronic device 600 and the buttons 81, 82, 83, 84, and 85 output by the electronic device 600, similar to Fig. 28. Fig. 30 also shows a reduced operation target area 860 corresponding to the button 83, an enlarged operation target area 861 corresponding to the button 84, and a reduced operation target area 862 corresponding to the button 85.

[0167] 30, the operation target area 861 corresponding to the button 84 selected by the user UR is the largest, followed by the operation target area 862 adjacent to the operation target area 861 and far from the center of the user UR, and then the operation target area 860 adjacent to the operation target area 861 and located in front of the user UR, all of which are smaller than the operation target area 861. That is, when the operation detection unit 17 detects an action of selecting one operation area out of multiple operation areas, the operation target area adjustment unit 16 enlarges the size of the operation target area corresponding to the selected operation area and reduces the sizes of the other operation target areas adjacent to the selected operation target area.

[0168] Note that Figure 30 describes the state in which user UR shown in Figure 28 selects button 84, but it can also be applied to the states in which user UR selects button 84 in the schematic diagrams shown in Figures 23, 24, 26, and 29.

[0169] For example, when the fingertip 72 of the user UR moves along the Z axis toward the electronic device 600 in front of the overlapping area of ​​the multiple operation target areas in FIG. 27, the motion detection unit 17 detects the motion of the user UR selecting a button. In this case, the motion detection unit 17 determines that the button to which the fingertip of the user UR is closer is selected. Furthermore, for example, when the fingertip 72 of the user UR performs a selection motion in an area closer to the button 84 than the button 83 in front of the overlapping area of ​​the operation target area 836 and the operation target area 846, the motion detection unit 17 detects the motion of the user UR selecting the button 84.

[0170] (Fourth embodiment) In the fourth embodiment, a selection operation in which the predetermined motion corresponds to a swinging motion of the fingertip 72 of the user UR will be described.

[0171] Fig. 31 is a schematic diagram for explaining operation processing of an information processing device according to a third comparative example. Fig. 31 shows an electronic device 600, the eyes of a user UR who operates the electronic device 600, a fingertip 72 of the user UR who operates the electronic device 600, and multiple buttons output by the electronic device 600. The multiple buttons are, for example, button 91, button 92, button 93, button 94, and button 95. Also, arrows 931, 941, and 951 indicate the expanding operation areas of button 93, button 94, and button 95, respectively.

[0172] For example, the schematic diagram shown in FIG. 31 is a schematic diagram of a case where a user UR performs a sliding motion using fingertip 72 of the user UR in the X-axis direction as indicated by arrow M5 from button 91 located in front of the user UR to button 93, and the fingertip 72 of the user UR selects button 93.

[0173] 31 is a schematic diagram showing a case where the user UR performs a sliding motion in the Z-axis direction with the fingertip 72 of the user UR as indicated by the arrow M6 while the button 93 is selected. The sliding motion of the fingertip of the user UR in the Z-axis direction refers to, for example, a swinging motion such as a motion of pulling the fingertip 72 of the user UR toward the user UR. In other words, the swinging motion refers to a motion toward a third direction away from the front of the operation target area corresponding to the selected operation area. At this time, the position of the eyes of the user UR has not changed.

[0174] Here, when the user UR performs a sliding motion with the fingertip 72 of the user UR in the Z-axis direction as indicated by the arrow M6 while the button 93 is selected, the direction between the eyes of the user UR and the fingertip 72 of the user UR may move from the arrow 722 to the arrow 723 corresponding to the button 95, resulting in a state in which the button 95 is selected. This is because the direction connecting the eyes of the user UR and the fingertip 72 changes due to the user UR performing a swinging motion, and the operation angle between the user UR and the button 93 is shifted, and the operation angle between the user UR and the button 95 is shifted, resulting in a state in which the button 95 is selected.

[0175] As a result, the user UR may feel that it is difficult to operate the button 95 because the button 95 is selected even though the button 93 is selected. In view of this, the fourth embodiment describes an information processing device capable of improving a selection operation corresponding to a swinging motion of the fingertip 72 of the user UR.

[0176] Fig. 32 is a schematic diagram for explaining an example of an enlarged operation target area according to the fourth embodiment. Fig. 32 shows an electronic device 600, a fingertip 72 of a user UR who operates the electronic device 600, and buttons 81, 82, 83, 84, and 85 output by the electronic device 600. Fig. 32 is also a schematic diagram showing a state in which the user UR transitions from the state of the fingertip 72 of the user UR shown in Fig. 31 to a swinging motion.

[0177] For example, in FIG. 32, when the fingertip 72 of the user UR selects the button 84 and moves the fingertip 72 of the user UR in the positive direction of the Z axis as indicated by the arrow M7, the operation area adjustment unit 16 further expands the operation target area 861 corresponding to the button 84 to become an operation target area 863 larger than the operation target area 861.

[0178] That is, when the operation area adjustment unit 16 detects a motion in a third direction away from the front of the operation target area corresponding to the selected operation area by the motion detection unit 17, the operation area adjustment unit 16 further expands the size of the operation target area corresponding to the selected operation area. Then, after a certain period of time has elapsed, the operation area adjustment unit 16 returns the size of the operation target area to the size before expansion. Here, the certain period of time is one second. The certain period of time is not limited to one second and can be set arbitrarily. This allows the operation area adjustment unit 16 to set an operation area according to the swinging motion of the user UR.

[0179] Fig. 33 is a schematic diagram showing an example of the contents of the operation process of the enlarged operation target area according to the fourth embodiment. In Fig. 33, the process of detecting a swinging motion of the user UR by the motion detection unit 17 of the control unit 10 will be described.

[0180] 33 shows an electronic device 600, the eyes of a user UR who operates the electronic device 600, a fingertip 72 of the user UR who operates the electronic device 600, and multiple buttons output by the electronic device 600. The multiple buttons are, for example, buttons 91, 92, 93, 94, and 95. The direction of the eyes of the user UR and the fingertip 72 of the user UR is an arrow 722 pointing to the button 93. FIG 33 shows a state in which the fingertip 72 of the user UR selects the button 93 and then swings the fingertip 72 of the user UR along the arrows M61 and M62 in that order.

[0181] When the motion detection unit 17 determines that at least one of the movement speed, movement direction, and movement distance of the movement of the fingertip 72 of the user UR is likely to be a swing-back motion, it determines that the motion is a swing-back motion.

[0182] Specifically, as shown in FIG. 33, when the moving speed in the Z direction of the fingertip 72 of the user UR is equal to or greater than a predetermined speed threshold, the motion detection unit 17 determines that there is a possibility of a swing-back. For example, the motion detection unit 17 may detect the moving speed in the Z direction at a time t x and time t x+1 If the moving speed corresponding to the moving distance of the fingertip 72 of the user UR in the Z direction is 0.1 [m / s] or more, it is determined that there is a possibility of a swing-up. Note that, although the predetermined speed threshold has been described as 0.1 [m / s], it is not limited to this.

[0183] Specifically, the motion detection unit 17 determines that there is a possibility of a swing-back when the fingertip of the user UR is present in the approximately triangular ranges R61 and R62 shown in FIG. 33 and moves in the positive direction of the Z direction. For example, the motion detection unit 17 detects at time t x Based on the position of , time t x+1 The direction of the movement is calculated, and if the calculated movement direction is less than a predetermined angle, it is determined that there is a possibility of a swing-back. x Whereas, t x+y Here, y is an arbitrary value.

[0184] More specifically, the motion detection unit 17 determines that there is a possibility of a swing-up when the movement distance in the Z direction of the fingertip 72 of the user UR is equal to or greater than a predetermined distance threshold, as indicated by arrows M61 and M62 in Fig. 33. For example, the motion detection unit 17 identifies the movement distance in the Z direction of the fingertip 72 of the user UR from a captured image of the user UR, and determines that there is a possibility of a swing-up when there are five or more consecutive frames in which the fingertip 72 of the user UR moves 10 cm or more in the Z direction in the captured image.

[0185] The predetermined distance threshold has been described as 10 cm, but is not limited to this. Also, the number of frames for determining that a swing-over is possible is described as 5 or more, but is not limited to this. Also, for example, the motion detection unit 17 may calculate the distance at time tx+1 based on the position at time tx, and determine that a swing-over is possible if the calculated movement distance is equal to or greater than a predetermined distance.

[0186] As described above, when a control device according to the fourth embodiment of the present disclosure detects a movement in a third direction away from the front of the operation target area corresponding to the selected operation area, the control device further expands the size of the operation target area corresponding to the selected operation area, and after a certain period of time, returns the size of the operation target area to the size before it was expanded.

[0187] In this way, when the operator performs a swing motion after selecting an operation area, the control device can prevent the operator from selecting another operation area against his / her will. Therefore, the control device can improve the operability for the user.

[0188] (Twelfth Modification) The above-described third embodiment, fourth embodiment, tenth modification, and eleventh modification have been described with respect to the case where the user UR is located outside the front area of ​​the electronic device 600, but are not limited thereto. The above-described third embodiment, fourth embodiment, tenth modification, and eleventh modification may also be applied to the case where the user UR is located in front of and at the center of the electronic device 600, or the case where the user UR is located inside the front area of ​​the electronic device 600.

[0189] Although the embodiments of the present disclosure have been described above, the above-mentioned embodiments are presented as examples and are not intended to limit the scope of the invention. These new embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the gist of the invention. These new embodiments and their modifications are included in the scope and gist of the invention, and are included in the scope of the invention and its equivalents described in the claims. Furthermore, components across different embodiments and modifications may be appropriately combined.

[0190] In addition, the notation "... part" in the above-mentioned embodiments may be replaced with other notations such as "... circuitry", "... assembly", "... device", "... unit", or "... module".

[0191] In each of the above embodiments, the present disclosure has been described as an example configured using hardware, but the present disclosure can also be realized by software in cooperation with hardware.

[0192] Moreover, each functional block used in the description of each of the above embodiments is typically realized as an LSI, which is an integrated circuit. The integrated circuit controls each functional block used in the description of the above embodiments and may have input terminals and output terminals. These may be individually integrated into one chip, or may be integrated into one chip that includes some or all of them. Here, the term LSI is used, but depending on the degree of integration, it may also be called an IC, a system LSI, a super LSI, or an ultra LSI.

[0193] The method of integration is not limited to LSI, but may be realized using a dedicated circuit or a general-purpose processor and memory. After LSI manufacture, a programmable FPGA (Field Programmable Gate Array) or a reconfigurable processor that allows reconfiguration of the connections or settings of circuit cells inside the LSI may also be used.

[0194] Furthermore, if a new integrated circuit technology that can replace LSI appears due to the progress of semiconductor technology or a derivative technology, it is possible to integrate the functional blocks using that technology. The application of biotechnology is also a possibility.

[0195] Furthermore, the effects of the embodiments described in this specification are merely examples and are not limiting, and other effects may also be provided.

[0196] (Additional Note) The above description of the embodiments discloses the following techniques. (Technology 1) A position detection unit that detects the position of an operator; a motion detection unit that detects a predetermined motion of the operator within an operation area that indicates an area defined by connecting the position of the operator detected by the position detection unit and an operation target area set on an electronic device; a processing execution unit that performs processing associated with the operation target area when the predetermined motion is detected by the motion detection unit; an operation area adjustment unit that expands the operation area when the position of the operator detected by the position detection unit is outside a front area of ​​the electronic device; An information processing device comprising: (Technology 2) the electronic device is a display, On the display, there are at least two operation target areas, a nearby button area indicating an area of ​​nearby buttons arranged on a side closer to the operator, and a distant button area indicating an area of ​​distant buttons arranged on a side farther from the operator, The operation area adjustment unit is a first angle indicating an angle between a first line connecting one end of the operation area corresponding to the nearby button area in a direction of a center line connecting the center of the nearby button area and the center of the distant button area, and a position of the operator, and a second line connecting the other end of the operation area corresponding to the nearby button area in the direction of the center line, and a position of the operator, the operation areas corresponding to the near button area and the far button area are set so as to be equal to a second angle indicating an angle formed by a third line connecting one end of the operation area corresponding to the far button area in the direction of the center line and a position of the operator, and a fourth line connecting the other end of the operation area corresponding to the far button area in the direction of the center line and a position of the operator; The information processing device according to the above technique 1. (Technology 3) the first straight line is a straight line connecting a right end of the adjacent button area and a position of the operator, the second straight line is a straight line connecting a left end of the adjacent button area and a position of the operator, the third straight line is a straight line connecting a right end of the far button area and a position of the operator, The fourth straight line is a straight line connecting a left end of the far button area and a position of the operator. The information processing device according to the above-mentioned technique 2. (Technology 4) the first straight line is a straight line connecting an upper end of the adjacent button area and a position of the operator, the second straight line is a straight line connecting a lower end of the adjacent button area and a position of the operator, the third straight line is a straight line connecting an upper end of the far button area and a position of the operator, The fourth straight line is a straight line connecting a lower end of the far button area and a position of the operator. The information processing device according to the above-mentioned technique 2. (Technology 5) the electronic device is a display, on the display, there are at least two operation target areas, a first button area indicating an area where a first button is arranged, and a second button area indicating an area where a second button is arranged, a length of the first button region is smaller than a length of the second button region in a direction of a center line connecting a center of the first button region and a center of the second button region; The operation area adjustment unit is a first angle indicating an angle between a first straight line connecting one end of the operation area corresponding to the first button area in the direction of the center line and the position of the operator and a second straight line connecting the other end of the operation area corresponding to the first button area in the direction of the center line and the position of the operator, setting the operation areas corresponding to each of the first button area and the second button area so that the angle is equal to a second angle indicating an angle formed by a third line connecting one end of the operation area corresponding to the second button area in the direction of the center line and a position of the operator, and a fourth line connecting the other end of the operation area corresponding to the second button area in the direction of the center line and a position of the operator; The information processing device according to the above technique 1. (Technology 6) The electronic devices include a first electronic device and a second electronic device, the operation area adjustment unit expands the operation area of ​​at least one of the first electronic device and the second electronic device when the position of the operator detected by the position detection unit is outside a front area corresponding to a front surface of the first electronic device and the second electronic device. The information processing device according to the above technique 1. (Technology 7) the operation area adjustment unit determines a size of the enlarged operation area based on a positional relationship between the electronic device and the operator. The information processing device according to the above technique 1. (Technology 8) the electronic device is a display, the operation area adjustment unit determines a size of the enlarged operation area based on a size of the operation area when the display is disposed in front of the operator with the shortest distance between the display and the operator being equal; The information processing device according to the above technique 7. (Technology 9) the electronic device is a display, When there are multiple operation areas on the display, when the action detection unit detects an action of selecting one of the plurality of operation areas, the operation area adjustment unit determines a size of the enlarged operation area in response to the predetermined action. The information processing device according to the above technique 1. (Technology 10) When the sizes of the enlarged operation target areas corresponding to the plurality of operation areas are uniform, the operation area adjustment unit enlarging a size of the enlarged operation target area corresponding to the selected operation area; reducing a size of the enlarged operation target area corresponding to the other operation area adjacent to the selected operation area; The information processing device according to the above technique 9. (Technology 11) When the sizes of the enlarged operation target areas corresponding to the plurality of operation areas are different, the operation area adjustment unit enlarging a size of the enlarged operation target area corresponding to the selected operation area; reducing a size of the enlarged operation target area corresponding to the other operation area adjacent to the selected operation area; The information processing device according to the above technique 9. (Technology 12) the electronic device is a display, When there are multiple operation areas on the display, when the operation detection unit detects an operation of selecting one of the plurality of operation areas, the operation area adjustment unit moves the enlarged operation target area in response to the predetermined operation. The information processing device according to the above technique 1. (Technology 13) when the motion detection unit detects a motion toward a first direction indicating the left or right of a front of the plurality of operation areas, the operation area adjustment unit moves the enlarged operation target area so as to move in the first direction. The information processing device according to the above technique 12. (Technology 14) when the action detection unit detects an action of selecting one of the plurality of operation areas, the operation area adjustment unit enlarges a size of an operation target area corresponding to the selected operation area and reduces sizes of other operation target areas adjacent to the selected operation target area; The information processing device according to the above-mentioned technique 9 or 12. (Technology 15) when the motion detection unit detects a motion in a third direction away from a front of the operation target area corresponding to the selected operation area, the operation area adjustment unit further enlarges a size of the operation target area corresponding to the selected operation area, and after a certain time has elapsed, returns the size of the operation target area to the size before enlargement. The information processing device according to the above technique 14. (Technology 16) An information processing system including an imaging unit and an information processing device, The imaging unit images a user who is an operator, The information processing device includes: a position detection unit that detects a position of an operator based on an image captured by the imaging unit; a motion detection unit that detects a predetermined motion of the operator within an operation area that indicates an area defined by connecting the position of the operator detected by the position detection unit and an operation target area set on an electronic device; a processing execution unit that performs processing associated with the operation target area when the predetermined motion is detected by the motion detection unit; and an operation area adjustment unit that expands the operation area when the position of the operator detected by the position detection unit is outside a front area of ​​the electronic device. Information processing system. (Technology 17) An information processing method executed by an information processing device, a position detection step of detecting a position of an operator; a motion detection step of detecting a predetermined motion of the operator in an operation area indicating an area defined by connecting the position of the operator detected in the position detection step and an operation target area set on an electronic device; a process execution step of performing a process associated with the operation target area when the predetermined action is detected by the action detection step; an operation area adjustment step of expanding the operation area when the position of the operator detected by the position detection step is outside a front area of ​​the electronic device; An information processing method comprising: (Technology 18) On the computer, a position detection step of detecting a position of an operator; a motion detection step of detecting a predetermined motion of the operator in an operation area indicating an area defined by connecting the position of the operator detected in the position detection step and an operation target area set on an electronic device; a process execution step of performing a process associated with the operation target area when the predetermined action is detected by the action detection step; an operation area adjustment step of expanding the operation area when the position of the operator detected by the position detection step is outside a front area of ​​the electronic device; An information processing program that executes the above. [Explanation of symbols]

[0197] 1 Control device 10 Control section 11 Image acquisition section 12 1st Judgment Section 13 Position detection unit 14 Operation area generation section 15 Second Judgment Section 16 Operation area adjustment section 17 Motion detection unit 18 Third Judgment Section 19 Processing Execution Unit 30 Storage section 500 Sensors 600 Electronic equipment

Claims

1. A position detection unit that detects the position of an operator; a motion detection unit that detects a predetermined motion of the operator within an operation area that indicates an area defined by connecting the position of the operator detected by the position detection unit and an operation target area set on an electronic device; a processing execution unit that performs processing associated with the operation target area when the predetermined motion is detected by the motion detection unit; an operation area adjustment unit that expands the operation area when the position of the operator detected by the position detection unit is outside a front area of ​​the electronic device; An information processing device comprising:

2. the electronic device is a display, On the display, there are at least two operation target areas, a nearby button area indicating an area of ​​nearby buttons arranged on a side closer to the operator, and a distant button area indicating an area of ​​distant buttons arranged on a side farther from the operator, The operation area adjustment unit is a first angle indicating an angle between a first line connecting one end of the operation area corresponding to the nearby button area in a direction of a center line connecting the center of the nearby button area and the center of the distant button area, and a position of the operator, and a second line connecting the other end of the operation area corresponding to the nearby button area in the direction of the center line, and a position of the operator, the operation areas corresponding to the near button area and the far button area are set so as to be equal to a second angle indicating an angle formed by a third line connecting one end of the operation area corresponding to the far button area in the direction of the center line and a position of the operator, and a fourth line connecting the other end of the operation area corresponding to the far button area in the direction of the center line and a position of the operator; The information processing device according to claim 1 .

3. the first straight line is a straight line connecting a right end of the adjacent button area and a position of the operator, the second straight line is a straight line connecting a left end of the adjacent button area and a position of the operator, the third straight line is a straight line connecting a right end of the far button area and a position of the operator, the fourth straight line is a straight line connecting a left end of the far button area and a position of the operator; The information processing device according to claim 2 .

4. the first straight line is a straight line connecting an upper end of the adjacent button area and a position of the operator, the second straight line is a straight line connecting a lower end of the adjacent button area and a position of the operator, the third straight line is a straight line connecting an upper end of the far button area and a position of the operator, the fourth straight line is a straight line connecting a lower end of the far button area and a position of the operator; The information processing device according to claim 2 .

5. the electronic device is a display, On the display, there are at least two operation target areas, a first button area indicating an area where a first button is arranged, and a second button area indicating an area where a second button is arranged, a length of the first button region is smaller than a length of the second button region in a direction of a center line connecting a center of the first button region and a center of the second button region; The operation area adjustment unit is a first angle indicating an angle between a first straight line connecting one end of the operation area corresponding to the first button area in the direction of the center line and a position of the operator and a second straight line connecting the other end of the operation area corresponding to the first button area in the direction of the center line and a position of the operator, the operation areas corresponding to the first button area and the second button area are set so as to be equal to a second angle indicating an angle formed by a third line connecting one end of the operation area corresponding to the second button area in the direction of the center line and a position of the operator, and a fourth line connecting the other end of the operation area corresponding to the second button area in the direction of the center line and a position of the operator; The information processing device according to claim 1 .

6. The electronic devices include a first electronic device and a second electronic device, the operation area adjustment unit expands the operation area of ​​at least one of the first electronic device and the second electronic device when the position of the operator detected by the position detection unit is outside a front area corresponding to a front surface of the first electronic device and the second electronic device. The information processing device according to claim 1 .

7. the operation area adjustment unit determines a size of the enlarged operation area based on a positional relationship between the electronic device and the operator. The information processing device according to claim 1 .

8. the electronic device is a display, the operation area adjustment unit determines a size of the enlarged operation area based on a size of the operation area when the display is disposed in front of the operator with the shortest distance between the display and the operator being equal; The information processing device according to claim 7.

9. the electronic device is a display, When there are multiple operation areas on the display, when the operation detection unit detects an operation of selecting one of the plurality of operation areas, the operation area adjustment unit determines a size of the enlarged operation target area in response to the predetermined operation. The information processing device according to claim 1 .

10. When the sizes of the enlarged operation target areas corresponding to the plurality of operation areas are uniform, the operation area adjustment unit enlarging a size of the enlarged operation target area corresponding to the selected operation area; reducing a size of the enlarged operation target area corresponding to the other operation area adjacent to the selected operation area; The information processing device according to claim 9.

11. When the sizes of the enlarged operation target areas corresponding to the plurality of operation areas are different, the operation area adjustment unit enlarging a size of the enlarged operation target area corresponding to the selected operation area; reducing a size of the enlarged operation target area corresponding to the other operation area adjacent to the selected operation area; The information processing device according to claim 9.

12. the electronic device is a display, When there are multiple operation areas on the display, the operation area adjustment unit, when a motion of selecting one of the plurality of operation areas is detected by the motion detection unit, moves the enlarged operation target area in response to the predetermined motion. The information processing device according to claim 1 .

13. when the motion detection unit detects a motion toward a first direction indicating the left or right of a front of the plurality of operation areas, the operation area adjustment unit moves the enlarged operation target area so as to move in the first direction. The information processing device according to claim 12.

14. when the action detection unit detects an action of selecting one of the plurality of operation areas, the operation area adjustment unit enlarges a size of an operation target area corresponding to the selected operation area and reduces sizes of other operation target areas adjacent to the selected operation target area.

13. The information processing device according to claim 9 or 12.

15. when the motion detection unit detects a motion in a third direction away from a front of the operation target area corresponding to the selected operation area, the operation area adjustment unit further enlarges a size of the operation target area corresponding to the selected operation area, and after a certain time has elapsed, returns the size of the operation target area to the size before enlargement. The information processing device according to claim 14.

16. An information processing system including an imaging unit and an information processing device, The imaging unit images a user who is an operator, The information processing device includes: a position detection unit that detects a position of an operator based on an image captured by the imaging unit; a motion detection unit that detects a predetermined motion of the operator within an operation area that indicates an area defined by connecting the position of the operator detected by the position detection unit and an operation target area set on an electronic device; a processing execution unit that performs processing associated with the operation target area when the predetermined motion is detected by the motion detection unit; and an operation area adjustment unit that expands the operation area when the position of the operator detected by the position detection unit is outside a front area of ​​the electronic device. Information processing system.

17. An information processing method executed by an information processing device, a position detection step of detecting a position of an operator; a motion detection step of detecting a predetermined motion of the operator in an operation area indicating an area defined by connecting the position of the operator detected in the position detection step and an operation target area set on an electronic device; a process execution step of performing a process associated with the operation target area when the predetermined action is detected by the action detection step; an operation area adjustment step of expanding the operation area when the position of the operator detected by the position detection step is outside a front area of ​​the electronic device; An information processing method comprising:

18. On the computer, a position detection step of detecting a position of an operator; a motion detection step of detecting a predetermined motion of the operator in an operation area indicating an area defined by connecting the position of the operator detected in the position detection step and an operation target area set on an electronic device; a process execution step of performing a process associated with the operation target area when the predetermined action is detected by the action detection step; an operation area adjustment step of expanding the operation area when the position of the operator detected by the position detection step is outside a front area of ​​the electronic device; An information processing program that executes the above.

Images