Information processing device, information processing system, information processing method, information processing program product, and computer-readable storage medium

Abstract

The disclosure relates to an information processing apparatus, an information processing system, an information processing method, an information processing program product, and a computer-readable storage medium. An information processing device according to one embodiment of the present disclosure is provided with a position detection unit, an operation detection unit, a process execution unit, and an operation region adjustment unit. The position detection unit detects the position of an operator. The motion detection unit detects a predetermined motion of the operator in an operation region that is determined by linking the position of the operator detected by the position detection unit and an operation target region set in the at least one electronic device. When a predetermined operation is detected by the operation detection unit, the process execution unit performs a process associated with the operation target region. When the position of the operator detected by the position detection unit is outside the front surface area of the at least one electronic device, the operation area adjustment unit expands the operation area.

Description

Technical Field

[0001] The present disclosure relates to an information processing device, an information processing system, an information processing method, and a computer-readable storage medium. Background Art

[0002] Conventionally, the following technology is known: in various electronic devices, when a user moves a hand without directly touching the electronic device, a process associated with an operation object of the hand movement is executed (for example, refer to Patent Documents 1-2). For example, the following technology is known: when the electronic device is a display, when a user moves a hand without directly touching the display, the display content of the display is changed as a process associated with an operation object of the hand movement (for example, a button arranged on a screen, etc.).

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent No. 6463963

[0006] Patent Document 2: Japanese Patent No. 5341126 Summary of the invention

[0007] Problem that the invention aims to solve

[0008] In the technology as described above, there is, for example, the following problem: when the user's position is outside the front area of ​​the electronic device, it is difficult for the user to perform processing associated with operation objects such as buttons, that is, it is difficult to operate, compared to the case where the user's position is in the front area.

[0009] The present disclosure provides an information processing device, an information processing system, an information processing method, and an information processing program capable of improving user operability.

[0010] Solutions for solving problems

[0011] An information processing device according to one embodiment of the present disclosure includes a position detection unit, a motion detection unit, a processing execution unit, and an operation area adjustment unit. The position detection unit detects the position of an operator. The motion detection unit detects a prescribed motion of the operator in an operation area, where the operation area represents an area determined by linking the position of the operator detected by the position detection unit with an operation object area set on at least one electronic device. When the prescribed motion is detected by the motion detection unit, the processing execution unit performs processing associated with the operation object area. When the position of the operator detected by the position detection unit is outside the area on the front of at least one electronic device, the operation area adjustment unit expands the operation area.

[0012] Effects of the Invention

[0013] According to the information processing device involved in the present disclosure, the user's operability can be improved. In addition, the effects described here are not necessarily limited, and may be any effect described in this specification. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a schematic diagram showing an example of an information processing device.

[0015] Figure 2 It is a schematic diagram for explaining the operation process of the information processing device involved in the first comparative example.

[0016] Figure 3 It is a schematic diagram for explaining the operation process of the information processing device involved in the first embodiment.

[0017] Figure 4 1 is a diagram showing a schematic configuration of an information system including a control device as an information processing device according to the first embodiment.

[0018] Figure 5 This is a schematic diagram for explaining an example of the operation area to be expanded according to the first embodiment.

[0019] Figure 6 This is a flowchart showing a processing procedure of the control device according to the first embodiment.

[0020] Figure 7 It is a schematic diagram for explaining an example of the operation target area to be expanded according to the first modification example.

[0021] Figure 8 This is a schematic diagram for explaining an example of an operation area to be expanded according to the second modification.

[0022] Fig. 9 It is a schematic diagram for explaining the operation target area to be expanded according to the third modification example.

[0023] Fig.10 It is a schematic diagram for explaining the operation target area to be expanded according to the third modification example.

[0024] Fig.11 This is a schematic diagram for explaining an example of an operation area to be expanded according to the fourth modification.

[0025] Fig.12 This is a schematic diagram for explaining an example of an operation area to be expanded according to the fifth modification.

[0026] Fig.13It is a schematic diagram for explaining an example of an operation area to be expanded according to the sixth modification.

[0027] Fig.14 It is a schematic diagram for explaining an example of an operation area to be expanded according to the sixth modification.

[0028] Fig.15 It is a schematic diagram for explaining the operation process of the information processing device involved in the seventh modification example.

[0029] Fig.16 It is a schematic diagram for explaining the operation process of the information processing device involved in the second comparative example.

[0030] Fig.17 It is a schematic diagram for explaining the operation process of the information processing device involved in the second embodiment.

[0031] Fig.18 1 is a diagram showing a schematic configuration of an information system including a control device as an information processing device according to a second embodiment.

[0032] Fig.19 This is a schematic diagram for explaining an example of an operation area to be expanded according to the second embodiment.

[0033] Fig. 20 It is a schematic diagram for explaining an example of an operation target region to be expanded according to the eighth modification example.

[0034] Fig.21 It is a schematic diagram for explaining an example of an operation area to be expanded according to the ninth modification.

[0035] Fig. 22 It is a schematic diagram for explaining an example of the expanded operation target region according to the third embodiment.

[0036] Fig.23 It is a schematic diagram for explaining an example of the expanded operation target region according to the third embodiment.

[0037] Fig.24 It is a schematic diagram for explaining an example of the expanded operation target region according to the third embodiment.

[0038] Fig.25 It is a schematic diagram for explaining an example of an expanded operation target area according to the tenth modification example.

[0039] Fig.26 It is a schematic diagram for explaining an example of an expanded operation target area according to the tenth modification example.

[0040] Fig. 27It is a schematic diagram for explaining an example of the expanded operation target area involved in the eleventh modification.

[0041] Fig.28 It is a schematic diagram for explaining an example of the expanded operation target area involved in the eleventh modification.

[0042] Fig.29 It is a schematic diagram for explaining an example of the expanded operation target area involved in the eleventh modification.

[0043] Fig.30 It is a schematic diagram for explaining an example of the expanded operation target area involved in the eleventh modification.

[0044] Fig.31 It is a schematic diagram for explaining the operation process of the information processing device involved in the third comparative example.

[0045] Fig.32 It is a schematic diagram for explaining an example of the expanded operation target region according to the fourth embodiment.

[0046] Fig.33 It is a schematic diagram showing an example of the content of the operation process of the expanded operation target area according to the fourth embodiment. DETAILED DESCRIPTION

[0047] Next, refer to the attached Figure 1 Embodiments of the information processing device according to the present disclosure will be described.

[0048] (First Embodiment)

[0049] Before describing the information processing device according to the first embodiment, a device that operates based on the movement of a user's hand will be described.

[0050] exist Figure 1 An example of an information processing device that operates based on the movement of a user's hand is shown in FIG. The information processing device has a sensor 500 and an electronic device 600. The sensor 500 is, for example, a camera. The sensor 500 captures a user UR who is an operator. The user UR has an arm. Here, the arm includes not only an upper arm and a forearm, but also a hand. The hand also includes a fingertip 72. The electronic device 600 is, for example, a display device such as a monitor. The information processing device controls the data displayed by the electronic device 600 based on the position of the fingertip 72 of the user UR.

[0051] In addition, a virtual operation area AR100 and a detection effective area AR200 are set in the information processing device, wherein the virtual operation area AR100 is used to accept operations on the device (for example, the electronic device 600), and the detection effective area AR200 is an area that can detect the movement of the fingertip 72 of the user UR. The detection effective area AR200 is set within the sensing range of the sensor 500, for example, in front of the display surface of the electronic device 600. In the case where the sensor 500 is a camera, the sensing range is sometimes referred to as the camera range. The virtual operation area AR100 is included in the detection effective area AR200, for example, Figure 1 As shown, it is set as a plane parallel to the display surface of the electronic device 600.

[0052] exist Figure 1 In the structure, when the fingertip 72 enters a range closer to the electronic device 600 than the virtual operation area AR100, the information processing device determines that an operation instruction for the electronic device 600 has been made, thereby executing processing based on the operation instruction and displaying the execution result to the electronic device 600.

[0053] In addition, Figure 1 In the drawings related to the electronic device 600 described below, the mutually orthogonal X-axis, Y-axis, and Z-axis respectively refer to the left-right direction, the up-down direction, and the front-back direction of the electronic device 600. In the following description, when only the X-direction, the Y-direction, or the Z-direction is described, it refers to each axial direction, and is assumed to include two directions in opposite directions.

[0054] In addition, when the positive direction of the X-axis is determined, it is a direction from the left hand direction to the right hand direction of the user UR, when the positive direction of the Y-axis is determined, it is a direction from the lower side to the upper side of the user UR, and when the positive direction of the Z-axis is determined, it is a direction from the front to the back of the user UR. When the negative direction of the X-axis is determined, it is a direction from the right hand direction to the left hand direction of the user UR, when the negative direction of the Y-axis is determined, it is a direction from the upper side to the lower side of the user UR, and when the negative direction of the Z-axis is determined, it is a direction from the back to the front of the user UR.

[0055] Figure 2 FIG. 2 is a schematic diagram for explaining the operation process of the information processing device involved in the first comparative example. Figure 2 , an electronic device 600, a user UR operating the electronic device 600, a first button 41 and a second button 43 for outputting the electronic device 600 are shown. The first button 41 and the second button 43 are of the same size and are located at approximately parallel positions. In addition, it is assumed that the positions of the first button 41 and the second button 43 are located at positions that are symmetrical with respect to the center of the electronic device 600.

[0056] Here, the operation processing of the information processing device when the user UR operates the electronic device 600 in mid-air is described. For example, when the user UR is located in the front and center of the electronic device 600, the positional relationship between the user UR and the first button 41 is equal to the positional relationship between the user UR and the second button 43.

[0057] In addition, regarding the operation angle between the user UR and the first button 41, for example, the first operation angle 413 can be calculated based on the first straight line 411 connecting the user UR to the right end of the first button 41 and the second straight line 412 connecting the user UR to the left end of the first button 41 with the user UR as the base point. Regarding the operation angle between the user UR and the second button 43, for example, the second operation angle 433 can be calculated based on the third straight line 431 connecting the user UR to the right end of the second button 43 and the fourth straight line 432 connecting the user UR to the left end of the second button 43 with the user UR as the base point.

[0058] Regarding the first operation angle 413 and the second operation angle 433, when the user UR is located in the front and center of the electronic device 600, the positional relationship between the user UR and the first button 41 is equal to the positional relationship between the user UR and the second button 43, so the first operation angle 413 is also equal to the second operation angle 433. Therefore, when the user UR operates the first button 41 and the second button 43 for output of the electronic device 600 in mid-air, since the first operation angle 413 is equal to the second operation angle 433, the user UR can obtain an equal operating feeling.

[0059] Figure 3 FIG. 2 is a schematic diagram for explaining the operation process of the information processing device involved in the first embodiment. Figure 3 In, with Figure 2 Likewise, the electronic device 600 , the user UR operating the electronic device 600 , and the first button 41 and the second button 43 for output of the electronic device 600 are shown.

[0060] For example, when the user UR is located outside the area of ​​the front surface of the electronic device 600, the positional relationship between the user UR and the first button 41 is different from the positional relationship between the user UR and the second button 43. Figure 3 In the positional relationship between the user UR and the electronic device 600 shown, if the user UR is set as the base point, the first button 41 is located closer than the second button 43. Therefore, the lengths of the straight lines connecting the user UR and the buttons are fourth straight line 432> third straight line 431> second straight line 412> first straight line 411> 0. In other words, the first operation angle 413 is larger than the second operation angle 433, and the operation angles are different.

[0061] Thus, when the user UR operates the first button 41 and the second button 43 for outputting the electronic device 600 in mid-air, the user UR sometimes feels that the operation is difficult because the first operation angle 413 is different from the second operation angle 433. Therefore, the present embodiment provides an information processing device that can improve the operability of the user UR when the user UR is located outside the area in front of the electronic device 600.

[0062] Figure 4 1 is a diagram showing a schematic configuration of an information system including a control device 1 as an information processing device according to the first embodiment. The information processing device according to the first embodiment includes a sensor 500 , an electronic device 600 , and the control device 1 .

[0063] The sensor 500 is, for example, a camera. As an example, the sensor 500 is a visible light camera. The sensor 500 outputs an image obtained by photographing the user UR to the control device 1. The sensor 500 is an example of a camera unit. The sensor 500 continuously performs camera processing and outputs an image to the control device 1. The electronic device 600 is a display unit for displaying various data. The electronic device 600 is, for example, a display device such as a liquid crystal display. In addition, it is assumed that the electronic device 600 is one unit.

[0064] The control device 1 executes processing of data displayed on the electronic device 600 according to the movement of the fingertip 72 of the user UR detected via the sensor 500 .

[0065] The control device 1 includes a control unit 10 and a storage unit 30. The control unit 10 is configured as, for example, a CPU (Central Processing Unit) to uniformly control the operations of the various parts of the control device 1. The control device 1 involved in the present embodiment has a ROM (Read Only Memory) and a RAM (Random Access Memory) which are not shown in the figure. The ROM stores various programs. The RAM is an operating area when the CPU executes the program. The control device 1 includes, for example, a processor and a memory, and the functions of the control unit 10 and the various functional blocks of the control unit 10 are realized by the processor executing the program stored in the memory. The CPU is an example of a processor. The storage unit 30 is an example of a memory.

[0066] The CPU uses the RAM as a work area to execute the program stored in the ROM, thereby Figure 4The image acquisition unit 11, the first determination unit 12, the position detection unit 13, the operation region generation unit 14, the second determination unit 15, the operation region adjustment unit 16, the motion detection unit 17, the third determination unit 18, and the process execution unit 19 are implemented as shown. The content may be changed to that the control device 1 includes the image acquisition unit 11, the first determination unit 12, the position detection unit 13, the operation region generation unit 14, the second determination unit 15, the operation region adjustment unit 16, the motion detection unit 17, the third determination unit 18, and the process execution unit 19. The image acquisition unit 11, the first determination unit 12, the position detection unit 13, the operation region generation unit 14, the second determination unit 15, the operation region adjustment unit 16, the motion detection unit 17, the third determination unit 18, and the process execution unit 19 may also be implemented by different hardware.

[0067] The storage unit 30 stores various information. The storage unit 30 is implemented by hardware such as a memory or storage for storing information (in other words, data). Specifically, the storage unit 30 stores coordinate information 31, detection effective area information 32, operation area information 33, extended operation area information 34, and device operation amount information 35.

[0068] The coordinate information 31 includes the 3D coordinates of the installation position of the sensor 500, the information of the installation angle of the sensor 500, the 3D coordinates of the position of the electronic device 600, etc. The detection effective area information 32 is coordinate information related to the setting of the detection effective area. The detection effective area corresponds to the above-mentioned detection effective area AR200, and is an area in which the position and movement of the fingertip 72 of the user UR can be detected.

[0069] The operation area information 33 is coordinate information related to the setting of the operation area. The operation area corresponds to the virtual operation area AR100 described above, and is an area for receiving operations performed by the fingertip 72 of the user UR. The extended operation area information 34 is coordinate information related to the setting for expanding the above-mentioned operation area. The contents related to the operation area and the setting for expanding the operation area will be described later.

[0070] The device operation amount information 35 is information indicating the operation of the electronic device 600 corresponding to the position of the user UR's fingertip 72. The operation of the electronic device 600 is, for example, a pressing operation of a button output by the electronic device 600. The operation of the electronic device 600 is also called a predetermined operation.

[0071] Here, the coordinate system of the coordinate information 31, the detection effective area information 32, the operation area information 33, the extended operation area information 34, and the device operation amount information 35 is an absolute coordinate. In addition, the coordinate system is not limited to absolute coordinates. For example, the coordinate system may also be a relative coordinate based on the electronic device 600. In other words, the coordinate system may also be a relative coordinate based on the positional relationship between the user UR and the electronic device 600.

[0072] The image acquisition unit 11 acquires a captured image of the user UR serving as an operator from the sensor 500 .

[0073] The first determination unit 12 determines whether an operator is detected. Specifically, the first determination unit 12 determines whether a user UR serving as an operator is detected based on the camera image acquired by the image acquisition unit 11. For example, when the user UR serving as an operator is present in the camera image acquired by the image acquisition unit 11, the first determination unit 12 determines that the operator is detected. In addition, for example, when the user UR serving as an operator is not present in the camera image acquired by the image acquisition unit 11, the first determination unit 12 determines that the operator is not detected.

[0074] The position detection unit 13 detects the position of the user UR based on the camera image that captured the user UR. Specifically, when the first determination unit 12 determines that the user UR who is the operator is detected based on the camera image acquired by the image acquisition unit 11, the position detection unit 13 detects the position of the user UR based on the camera image that captured the user UR.

[0075] The operation area generating unit 14 generates an operation area based on the camera image of the user UR. The operation area indicates an area determined by linking the position of the user UR detected by the position detecting unit 13 with the operation target area set on the electronic device. Figure 3 The operation area generated by the operation area generating unit 14 will be described.

[0076] Here, in Figure 3 In the embodiment, the first button 41 and the second button 43 are respectively the operation target areas set on the electronic device. For example, when the electronic device 600 is a display, on the display, as at least two operation target areas, the operation target areas are the near button area and the far button area, the near button area represents the area of ​​the near button configured on the side close to the operator, and the far button area represents the area of ​​the far button configured on the side far from the operator. Figure 3 In the example, the near button is the first button 41 and the far button is the second button 43 .

[0077] In addition, Figure 3, the first operation angle 413 and the second operation angle 433 are operation areas respectively indicating an area determined by connecting the position of the user UR with the operation object area set on the electronic device. An operation area is, for example, an area set according to a first angle indicating an angle formed by a first straight line and a second straight line, wherein the first straight line is a straight line connecting one end in the direction of a center line C1 connecting the center of the near button area and the center of the far button area in the operation area corresponding to the near button area with the position of the operator, and the second straight line is a straight line connecting the other end in the direction of the center line C1 in the operation area corresponding to the near button area with the position of the operator.

[0078] exist Figure 3 For example, if one end of the first button 41 in the direction of the center line C1 is set as the right end and the other end of the first button 41 in the direction of the center line C1 is set as the left end, the first angle formed by the first straight line 411 connecting the position of the user UR and the second straight line 412 connecting the position of the user UR corresponds to the first operation angle 413.

[0079] In addition, another operation area, for example, is an area set based on a second angle representing an angle formed by a third straight line and a fourth straight line, wherein the third straight line is a straight line connecting one end in the direction of the center line C1 in the operation area corresponding to the distant button area and the position of the operator, and the fourth straight line is a straight line connecting the other end in the direction of the center line C1 in the operation area corresponding to the distant button area and the position of the operator.

[0080] exist Figure 3 For example, if one end of the second button 43 in the direction of the center line C1 is set as the right end and the other end of the second button 43 in the direction of the center line C1 is set as the left end, the second angle representing the angle formed by the third straight line 431 connecting the position of the user UR and the fourth straight line 432 connecting the position of the user UR corresponds to the second operation angle 433.

[0081] Return to Figure 4 The second determination unit 15 determines whether the position of the user UR is outside the front area of ​​the electronic device 600. Specifically, the second determination unit 15 determines whether the position of the user UR is outside the front area of ​​the electronic device 600 based on the position of the user UR detected by the position detection unit 13.

[0082] Here, the area on the front side of the electronic device 600 is sometimes referred to as the front area. When viewed from the negative direction of the Y axis, a straight line connecting the center of the lateral width of the front area of ​​the electronic device 600 and the center of the lateral width of the electronic device 600 is perpendicular to the lateral width direction of the front area. In addition, the lateral width of the front area is equal to the lateral width of the electronic device 600 in the X-axis direction. The lateral width of the front area can be set separately. The lateral width of the front area can also be smaller than the lateral width of the electronic device 600.

[0083] For example, when the electronic device 600 is horizontally long, if the user UR is located near an end of the electronic device 600, it may be difficult to operate near the end on the opposite side of the electronic device 600. In such a case, the same effect can be obtained by setting the horizontal width of the front area to be smaller than the horizontal width of the electronic device 600.

[0084] In addition, the lateral width of the front area is set to be greater than the shoulder width of the user UR, for example. Since the average shoulder width of a person is approximately 0.22 times the height of the person, the position detection unit 13 may detect the height of the user UR based on a camera image of the user UR, and set the value obtained by multiplying the detected height by 0.22 times as the lateral width of the front area.

[0085] In addition, Figure 2 In the positional relationship between the user UR and the electronic device 600 shown in FIG. 1 , the second determination unit 15 determines that the position of the user UR is inside the front area of ​​the electronic device 600. Figure 3 In the illustrated positional relationship between the user UR and the electronic device 600 , the second determination unit 15 determines that the position of the user UR is outside the front area of ​​the electronic device 600 .

[0086] Return to Figure 4 When the position of the operator detected by the position detection unit 13 is outside the front area of ​​the electronic device 600, the operation area adjustment unit 16 expands the operation area. Specifically, when the second determination unit 15 determines that the position of the operator detected by the position detection unit 13 is outside the front area of ​​the electronic device 600, the operation area adjustment unit 16 expands the operation area generated by the operation area generation unit 14. Figure 5 The operation area to be expanded by the operation area adjustment unit 16 will be described.

[0087] Figure 5 This is a schematic diagram for explaining an example of the operation area to be expanded according to the first embodiment. Figure 5 and Figure 3Likewise, the electronic device 600 , the user UR operating the electronic device 600 , and the first button 41 and the second button 43 for output of the electronic device 600 are shown.

[0088] When the second determination unit 15 determines that the position of the operator detected by the position detection unit 13 is outside the front area of ​​the electronic device 600, the operation area adjustment unit 16 sets the operation areas corresponding to the first button area and the second button area, respectively, so that the first operation angle 413 is equal to the second operation angle 433.

[0089] For example, Figure 5 As shown, first, when the second operation angle 433 is to be adjusted to be equal to the first operation angle 413, the operation area adjustment unit 16 expands the operation target area of ​​the second button 43 in such a manner that the right end and the left end of the second button 43 are widened from the center of the second button 43, thereby setting the operation target area 44 of the second button 43. Then, the operation area adjustment unit 16 sets the size of the angle in such a manner that the second operation angle 443 representing the angle formed by the third straight line 441 and the fourth straight line 442 in the operation area corresponding to the expanded operation target area 44 is equal to the first operation angle 413, wherein the third straight line 441 is a straight line connecting the right end of the operation target area 44 in the direction of the center line C1 and the position of the operator, and the fourth straight line 442 is a straight line connecting the left end of the operation target area 44 in the direction of the center line C1 and the position of the operator.

[0090] Thus, the operation region adjustment unit 16 can make the first operation angle 413 equal to the second operation angle 443. Then, the operation region adjustment unit 16 sets the operation region corresponding to the second operation angle 443.

[0091] Return to Figure 4 The action detection unit 17 detects a predetermined action of the operator in the operation area based on the camera image of the user UR. Specifically, the action detection unit 17 detects a predetermined action of the operator in the operation area generated by the operation area generation unit 14 based on the camera image of the user UR.

[0092] In addition, the action detection unit 17 detects a predetermined action of the operator in the operation area expanded by the operation area adjustment unit 16 based on the camera image of the user UR. Here, the predetermined action refers to, for example, a state in which the fingertip 72 of the user UR is present in the operation area and contacts the operation area. In addition, the predetermined action is not limited to this.

[0093] The third determination unit 18 determines whether a predetermined action is detected. Specifically, the third determination unit 18 determines whether the action detection unit 17 detects the predetermined action of the operator in the operation area generated by the operation area generation unit 14. In addition, the third determination unit 18 determines whether the action detection unit 17 detects the predetermined action of the operator in the operation area expanded by the operation area adjustment unit 16.

[0094] When the motion detection unit 17 detects the predetermined motion, the processing execution unit 19 performs processing corresponding to the predetermined motion on the operation target area. Specifically, when the third determination unit 18 determines that the motion detection unit 17 detects the predetermined motion, the processing execution unit 19 performs processing corresponding to the predetermined motion on the operation target area.

[0095] Next, use Figure 6 The processing procedure of the control device 1 according to the first embodiment will be described. Figure 6 It is a flowchart showing the processing procedure of the control device 1 according to the first embodiment.

[0096] First, the image acquisition unit 11 acquires a camera image obtained by photographing the user UR from the sensor 500 (step S61). Next, the first determination unit 12 determines whether the user UR serving as the operator is detected based on the camera image acquired by the image acquisition unit 11 (step S62). Here, when the first determination unit 12 determines that the operator is not detected (step S62: "No"), the process proceeds to step S61. On the other hand, when the first determination unit 12 determines that the operator is detected (step S62: "Yes"), the process proceeds to step S63.

[0097] In step S63, the position detection unit 13 detects the position of the user UR who is the operator based on the camera image (step S63). Next, the operation area generation unit 14 generates an operation area based on the camera image that captured the user UR, and the operation area represents an area determined by linking the position of the user UR detected by the position detection unit 13 with the operation target area set on the electronic device (step S64).

[0098] The second determination unit 15 determines whether the position of the user UR is outside the area of ​​the front face of the electronic device 600 based on the position of the user UR detected by the position detection unit 13 (step S65). Here, when the second determination unit 15 determines that the position of the user UR is within the area of ​​the front face of the electronic device 600 (step S65: No), the process proceeds to step S67. On the other hand, when the second determination unit 15 determines that the position of the user UR is outside the area of ​​the front face of the electronic device 600 (step S65: Yes), the process proceeds to step S66.

[0099] In step S66, the operation area adjustment unit 16 expands the operation area generated by the operation area generation unit 14 (step S66). In step S67, the action detection unit 17 detects a predetermined action of the operator in the operation area generated by the operation area generation unit 14 based on the camera image of the user UR (step S67).

[0100] Next, the third determination unit 18 determines whether the action detection unit 17 has detected a predetermined action of the operator in the operation area generated by the operation area generation unit 14 (step S68). Here, when the third determination unit 18 determines that the predetermined action has not been detected (step S68: No), the process proceeds to step S67. On the other hand, when the third determination unit 18 determines that the predetermined action has been detected (step S68: Yes), the process proceeds to step S71.

[0101] In step S69, the action detection unit 17 detects the operator's prescribed action in the operation area expanded by the operation area adjustment unit 16 based on the camera image that captured the user UR (step S69). Next, the third determination unit 18 determines whether the action detection unit 17 has detected the operator's prescribed action in the operation area expanded by the operation area adjustment unit 16 (step S70). Here, when the third determination unit 18 determines that the prescribed action is not detected (step S70: "No"), the process proceeds to step S69. On the other hand, when the third determination unit 18 determines that the prescribed action is detected (step S70: "Yes"), the process proceeds to step S71.

[0102] In step S71, the process execution unit 19 performs a process corresponding to a predetermined action on the operation target area (step S71). When the process of step S71 is completed, the process proceeds to step S61, and the process is continued while the control device 1 executes this process.

[0103] As described above, the control device 1 involved in one embodiment of the present disclosure detects the position of the operator and detects the prescribed action of the operator in the operation area, which indicates an area determined by linking the detected position of the operator with the operation target area set on the electronic device. Furthermore, the control device 1 performs processing associated with the operation target area when the prescribed action is detected, and expands the operation area when the position of the operator is outside the area of ​​the front of the electronic device.

[0104] Thus, the control device 1 expands the operation area when the operator is located outside the front area of ​​the electronic device 600, so that the operator can operate the far button and the near button displayed on the electronic device 600 in the same manner. Therefore, the control device 1 can improve the operability of the user.

[0105] In addition, the above-mentioned embodiments can also be appropriately deformed and implemented by changing a part of the structure or function of each of the above-mentioned devices. Therefore, several modified examples involved in the above-mentioned embodiments are set as other embodiments for description. In addition, the following mainly describes the points different from the above-mentioned embodiments, and the detailed description is omitted for the points common to the contents already described.

[0106] (First Modification)

[0107] For example, the operation target area corresponding to the button of the electronic device 600 only needs to be within the detection effective area AR200, and the control device 1 may expand the operation target area so as to overlap with the area outside the electronic device 600. Figure 7 The operation target area to be expanded according to the first modification example will be described. Figure 7 It is a schematic diagram for explaining an example of the operation target area to be expanded according to the first modification example.

[0108] exist Figure 7 6 shows an electronic device 600, a first button 41 for output of the electronic device 600, and a second button 43. When the control device 1 sets the detection effective area AR200 to a larger range than the electronic device 600, the operation area adjustment unit 16 of the control unit 10 expands the operation target area 44 of the second button 43 within the range inside the detection effective area AR200.

[0109] (Second Modification)

[0110] For example, if the user UR compares the case of the area located in the front of the electronic device 600 with the case of the area outside the front of the electronic device 600, the operability of the button output by the electronic device 600 is different. Specifically, the size of the operation area corresponding to the button in the case of the area located in the front of the electronic device 600 is different from the size of the operation area corresponding to the button in the case of the area outside the front of the electronic device 600, and the operation area corresponding to the button in the case of the area outside the front of the electronic device 600 is smaller than the operation area corresponding to the button in the case of the area located in the front of the electronic device 600.

[0111] Therefore, the control device 1 is not limited to expanding the operation target area of ​​the far button, but can also expand the operation target area of ​​the near button in the X-axis direction to the left and right to expand the operation area of ​​the far button and the near button. Figure 8 The operation area to be expanded according to the second modification example will be described. Figure 8 This is a schematic diagram for explaining an example of an operation area to be expanded according to the second modification.

[0112] exist Figure 8 6 shows the electronic device 600, the user UR who operates the electronic device 600, and the first button 41 and the second button 43 for output of the electronic device 600. In addition, the first button 41 is set as a near button, and the second button 43 is set as a far button. When the second determination unit 15 of the control device 1 determines that the position of the user UR is outside the front area of ​​the electronic device 600, the operation area adjustment unit 16 expands the operation area generated by the operation area generation unit 14.

[0113] For example, Figure 8 As shown, the operation area adjustment unit 16 expands the operation target area of ​​the first button 41 in such a manner that the right end and the left end of the first button 41 are widened from the center of the first button 41, thereby setting the operation target area 42 of the first button 41. In addition, the operation area adjustment unit 16 expands the operation target area of ​​the second button 43 in such a manner that the right end and the left end of the second button 43 are expanded from the center of the second button 43, thereby setting the operation target area 44 of the second button 43.

[0114] Then, the operation area adjustment unit 16 sets a first operation angle 423 indicating an angle formed by a first straight line 421 and a second straight line 422 in the operation area corresponding to the expanded operation target area 42, wherein the first straight line 421 is a straight line connecting the right end of the operation target area 42 in the direction of the center line C1 and the position of the operator, and the second straight line 422 is a straight line connecting the left end of the operation target area 42 in the direction of the center line C1 and the position of the operator. In addition, the operation area adjustment unit 16 sets a second operation angle 443 indicating an angle formed by a third straight line 441 and a fourth straight line 442 in the operation area corresponding to the expanded operation target area 44 to be equal in size to the first operation angle 423.

[0115] Thus, operation region adjustment unit 16 can increase first operation angle 413 toward first operation angle 423 and make first operation angle 423 equal to second operation angle 443. Then, operation region adjustment unit 16 sets operation regions corresponding to first operation angle 423 and second operation angle 443.

[0116] (Third Modification)

[0117] For example, when the control device 1 expands the operating area, Fig. 9 As shown in the figure, there is a case where the operation target areas overlap. Fig. 9FIG. 2 is a schematic diagram for explaining an operation target area to be expanded according to the third modification example. Fig. 9 Shown is an electronic device 600, a first button 41, a second button 43, a third button 45 for output of the electronic device 600, an enlarged operation object area 42 corresponding to the first button 41, an enlarged operation object area 44 corresponding to the second button 43, and an enlarged operation object area 46 corresponding to the third button 45.

[0118] For example, when the operation area adjustment unit 16 adjusts the operation areas for the operation object area 42, the operation object area 44, and the operation object area 46, there is a case where the action of the user UR is misdetected, and it is difficult to adjust the multiple operation areas. Therefore, the control device 1 involved in the third modification example can also expand the operation object area in a non-overlapping manner according to the structure of the button output by the electronic device 600 when the operation object area is expanded and the expanded operation object area overlaps.

[0119] Fig.10 It is a schematic diagram for explaining the operation target area to be expanded according to the third modification example. Fig.10 The electronic device 600, the user UR who operates the electronic device 600, the first button 41, the second button 43, the third button 45 for output of the electronic device 600, the enlarged operation object area 42 corresponding to the first button 41, the enlarged operation object area 44 corresponding to the second button 43, and the enlarged operation object area 46 corresponding to the third button 45 are shown. In addition, if the position of the user UR is taken as the base point, the positions of the buttons are located close to the user UR in the order of the first button 41, the third button 45, and the second button 43.

[0120] When the user UR is outside the area on the front of the electronic device 600, the operation area adjustment unit 16 expands the operation target areas of the first button 41, the second button 43, and the third button 45 so that the operation target areas 42, the operation target areas 44, and the operation target areas 46 do not overlap. In this case, although the operation angles of the buttons are different, the operation target areas of the first button 41, the second button 43, and the third button 45 are widened compared to before the expansion of each operation target area, so the operability of the user UR can be improved.

[0121] (Fourth Modification)

[0122] In the above-mentioned embodiment, the size of the buttons output by the electronic device 600 is equal, but the present invention is not limited to this. Fig.11 The following describes a case where the sizes of buttons output by electronic device 600 are different.

[0123] Fig.11 1 is a schematic diagram for explaining an example of an operation area to be expanded according to the fourth modification. Fig.11 , an electronic device 600, a user UR operating the electronic device 600, a first button 41 and a second button 43 for outputting the electronic device 600 are shown. In addition, the first button 41 is set as a near button, and the second button 43 is set as a far button. In addition, the first button 41 is smaller than the second button 43 and is located at a substantially parallel position. When the second determination unit 15 of the control device 1 determines that the position of the user UR is outside the area of ​​the front of the electronic device 600, the operation area adjustment unit 16 expands the operation area generated by the operation area generation unit 14.

[0124] For example, Fig.11 As shown, the operation area adjustment unit 16 expands the operation target area of ​​the first button 41 in such a manner that the right end and the left end of the first button 41 are widened from the center of the first button 41, thereby setting the operation target area 42 of the first button 41. In addition, the operation area adjustment unit 16 expands the operation target area of ​​the second button 43 in such a manner that the right end and the left end of the second button 43 are widened from the center of the second button 43, thereby setting the operation target area 44 of the second button 43. At this time, the operation area adjustment unit 16 sets the operation target area 42 and the operation target area 44 so as not to overlap.

[0125] Then, the operation area adjustment unit 16 sets a first operation angle 423 indicating an angle formed by a first straight line 421 and a second straight line 422 in the operation area corresponding to the expanded operation target area 42, wherein the first straight line 421 is a straight line connecting the right end of the operation target area 42 in the direction of the center line C1 and the position of the operator, and the second straight line 422 is a straight line connecting the left end of the operation target area 42 in the direction of the center line C1 and the position of the operator. In addition, the operation area adjustment unit 16 sets a second operation angle 443 indicating an angle formed by a third straight line 441 and a fourth straight line 442 in the operation area corresponding to the expanded operation target area 44 to be equal to the first operation angle 423.

[0126] Thus, operation region adjustment unit 16 can increase first operation angle 413 toward first operation angle 423 and make first operation angle 423 equal to second operation angle 443. Then, operation region adjustment unit 16 sets operation regions corresponding to first operation angle 423 and second operation angle 443.

[0127] (Fifth Modification)

[0128] In the above-mentioned embodiment, the case where the buttons for output of the electronic device 600 are located in substantially parallel positions is described, but the present invention is not limited thereto. Fig.12 The case where the button for output of the electronic device 600 is located in a substantially vertical position is described.

[0129] Fig.12 This is a schematic diagram for explaining an example of an operation area to be expanded according to the fifth modification. Fig.12 The electronic device 600, the user UR who operates the electronic device 600, and the first button 41 and the second button 43 for output of the electronic device 600 are shown. The first button 41 and the second button 43 are of the same size and are located at a substantially vertical position. In addition, the first button 41 is set as a near button, and the second button 43 is set as a far button. When the second determination unit 15 of the control device 1 determines that the position of the user UR is outside the area of ​​the front of the electronic device 600, the operation area adjustment unit 16 expands the operation area generated by the operation area generation unit 14.

[0130] exist Fig.12 In this case, the operation area generating unit 14 generates an operation area corresponding to the near button area of ​​the first button 41 based on the third operation angle 416 representing the angle formed by the fifth straight line 414 and the sixth straight line 415, wherein the fifth straight line 414 is a straight line connecting the upper end of the first button 41 and the position of the operator in the direction of the center line C2 connecting the center of the near button area of ​​the first button 41 and the center of the far button area of ​​the second button 43, and the sixth straight line 415 is a straight line connecting the lower end of the first button 41 in the direction of the center line C2 and the position of the operator.

[0131] When the fourth operation angle 446 is adjusted to be equal to the third operation angle 416, the operation area adjustment unit 16 expands the operation target area of ​​the second button 43 so that the upper end and the lower end of the second button 43 are widened from the center of the second button 43, thereby setting the operation target area 44 of the second button 43. Then, the operation area adjustment unit 16 sets the size of the angle so that the fourth operation angle 446 indicating the angle formed by the seventh straight line 444 and the eighth straight line 445 in the operation area corresponding to the expanded operation target area 44, which connects the upper end of the operation target area 44 in the direction of the center line C2 with the position of the operator, and the eighth straight line 445, which connects the lower end of the operation target area 44 in the direction of the center line C2 with the position of the operator, are equal to the third operation angle 416.

[0132] Thus, the operation region adjustment unit 16 can make the fourth operation angle 446 equal to the third operation angle 416. Then, the operation region adjustment unit 16 sets the operation region corresponding to the fourth operation angle 446.

[0133] (Sixth Modification)

[0134] In the sixth variation, using Fig.13 The following describes a case where the buttons for output of the electronic device 600 are located at approximately diagonal positions. Fig.13 and Fig.14 It is a schematic diagram for explaining an example of an operation area to be expanded according to the sixth modification.

[0135] Fig.13 and Fig.14 The electronic device 600, the user UR operating the electronic device 600, the first button 41 and the second button 43 for outputting the electronic device 600 are shown. The first button 41 and the second button 43 are of the same size and are located at approximately diagonal positions. Fig.13 , the second button 43 is located at the upper left of the first button 41. In addition, the first button 41 is set as a near button, and the second button 43 is set as a far button. When the second determination unit 15 of the control device 1 determines that the position of the user UR is outside the front area of ​​the electronic device 600, the operation area adjustment unit 16 expands the operation area generated by the operation area generation unit 14.

[0136] exist Fig.13 In the case of the above, the operation area generation unit 14 sets a first operation angle 413 indicating an angle formed by a first straight line 411 and a second straight line 412 in the operation area corresponding to the near button area of ​​the first button 41, wherein the first straight line 411 is a straight line connecting the right end of the first button 41 in the direction of the center line C3 connecting the center of the near button area of ​​the first button 41 and the center of the far button area of ​​the second button 43 and the position of the operator, and the second straight line 412 is a straight line connecting the left end of the first button 41 in the direction of the center line C3 and the position of the operator. In addition, the operation area generation unit 14 sets a third operation angle 416 indicating an angle formed by a fifth straight line 414 and a sixth straight line 415 in the operation area corresponding to the near button area of ​​the first button 41, wherein the fifth straight line 414 is a straight line connecting the upper end of the first button 41 in the direction of the center line C3 and the position of the operator, and the sixth straight line 415 is a straight line connecting the lower end of the first button 41 in the direction of the center line C3 and the position of the operator.

[0137] exist Fig.14In the case where the second operation angle 443 of the second button 43 is to be adjusted to be equal to the first operation angle 413, the operation area adjustment unit 16 expands the operation target area of ​​the second button 43 so that the right end and the left end of the second button 43 are widened from the center of the second button 43, thereby setting the operation target area 44 of the second button 43. Then, the operation area adjustment unit 16 sets the size of the angle so that the second operation angle 443 indicating the angle formed by the third straight line 441 and the fourth straight line 442 in the operation area corresponding to the expanded operation target area 44 is equal to the first operation angle 413.

[0138] Furthermore, when the fourth operation angle 446 of the second button 43 is to be adjusted to be equal to the third operation angle 416, the operation area adjustment unit 16 further expands the operation target area of ​​the second button 43 so that the upper end and the lower end of the second button 43 are widened from the center of the second button 43, thereby setting the operation target area 44 of the second button 43. Then, the operation area adjustment unit 16 sets the size of the angle so that the fourth operation angle 446 indicating the angle formed by the seventh straight line 444 and the eighth straight line 445 in the operation area corresponding to the further expanded operation target area 44 is equal to the third operation angle 416.

[0139] Thus, the operation area adjustment unit 16 can make the second operation angle 443 equal to the first operation angle 413. In addition, the operation area adjustment unit 16 can make the fourth operation angle 446 equal to the third operation angle 416. Then, the operation area adjustment unit 16 sets the operation areas corresponding to the second operation angle 443 and the fourth operation angle 446.

[0140] (Seventh Modification)

[0141] In the above-mentioned embodiment, the electronic device 600 is described as a single device, but the present invention is not limited to this. Fig.15 A method in which two electronic devices are provided will be described. Fig.15 It is a schematic diagram for explaining the operation process of the information processing device involved in the seventh modification example.

[0142] exist Fig.156 shows a first electronic device 601, a second electronic device 602, and a user UR who operates the first electronic device 601 and the second electronic device 602. In addition, the first electronic device 601 is configured as a lighting device, and the second electronic device 602 is configured as a display device.

[0143] The control device 1 involved in the seventh variant example has a first electronic device 601 and a second electronic device 602 as electronic devices. When the position of the operator detected by the position detection unit 13 is outside the front area corresponding to the front of the first electronic device 601 and the second electronic device 602, the operation area adjustment unit 16 expands the operation area of ​​at least one of the first electronic device 601 and the second electronic device 602.

[0144] For example, Fig.15 As shown, the operation area generating unit 14 generates an operation area based on the camera image of the user UR, and the operation area represents an area determined by linking the position of the user UR detected by the position detecting unit 13 with the operation target areas set on the plurality of electronic devices. Fig.15 In the figure, the operation object area is a first electronic device area 47 and a second electronic device area 48, wherein the first electronic device area 47 represents an area of ​​a first electronic device 601 configured on a side close to the user UR, and the second electronic device area 48 represents an area of ​​a second electronic device 602 configured on a side away from the user UR.

[0145] In addition, when the position of the user UR is outside the front area corresponding to the front of the first electronic device 601 and the second electronic device 602, the operation area adjustment unit 16 sets the operation areas corresponding to the first electronic device area 47 and the second electronic device area 48, respectively, in such a way that the first operation angle 473 and the second operation angle 483 are equal.

[0146] Furthermore, the first operation angle 473 is an angle formed by a first straight line 471 and a second straight line 472, wherein the first straight line 471 is a straight line connecting the right end in the direction of the center line C4 connecting the center of the first electronic device area 47 and the center of the second electronic device area 48 in the operation area corresponding to the first electronic device area 47 to the position of the operator, and the second straight line 472 is a straight line connecting the left end in the direction of the center line C4 in the operation area corresponding to the first electronic device area 47 to the position of the operator. The second operation angle 483 is an angle formed by a third straight line 481 and a fourth straight line 482, wherein the third straight line 481 is a straight line connecting the right end in the direction of the center line C4 in the operation area corresponding to the second electronic device area 48 to the position of the operator, and the fourth straight line 482 is a straight line connecting the left end in the direction of the center line C4 in the operation area corresponding to the second electronic device area 48 to the position of the operator.

[0147] (Second Embodiment)

[0148] In the above embodiment, the predetermined action refers to the state where the fingertip 72 of the user UR is present in the operation area and contacts the operation area, but the present invention is not limited thereto. In the second embodiment, the predetermined action is related to the scroll operation.

[0149] Fig.16 2 is a schematic diagram for explaining the operation process of the information processing device involved in the second comparative example. Fig.16 , an electronic device 600 and a user UR operating the electronic device 600 are shown. Assume that the user UR is located in front of the electronic device 600 and the distance between the user UR and the electronic device 600 is the shortest distance. In addition, assume that the electronic device 600 is located at a position perpendicular to the Y-axis direction.

[0150] Here, the operation processing of the information processing device when the user UR performs a scrolling operation on the electronic device 600 is described. Regarding the operation angle between the user UR and the electronic device 600, for example, the first operation angle 53 is generated based on the first straight line 51 connecting the user UR and the right end of the electronic device 600 and the second straight line 52 connecting the user UR and the left end of the electronic device 600, with the user UR as the base point. In addition, the operation object movement distance 54 of the operation object area in which the user UR performs a scrolling operation on the electronic device 600 corresponds to the operation movement distance 55 in the operation area in which the fingertip of the user UR performs the scrolling operation.

[0151] In addition, when the user UR performs a scrolling operation on the electronic device 600, the user UR and the electronic device 600 are located in parallel positions in the X-axis direction and are equidistant from each other in the Z-axis direction, so the operation object moving distance 54 is proportional to the moving speed of the scrolling operation of the operation moving distance 55. Fig.16 Under the positional relationship between the user UR and the electronic device 600, the user UR can operate the electronic device 600 in a coordinated manner.

[0152] Fig.17 FIG. 2 is a schematic diagram for explaining the operation process of the information processing device involved in the second embodiment. Fig.17 In, with Fig.16 Similarly, the electronic device 600 and the user UR operating the electronic device 600 are shown. Regarding the operation angle between the user UR and the electronic device 600, for example, the first operation angle 53 is generated based on the first straight line 51 connecting the user UR and the right end of the electronic device 600 and the second straight line 52 connecting the user UR and the left end of the electronic device 600, with the user UR as the base point. In addition, the operation object moving distance 54 of the operation object area in which the user UR performs a scrolling operation on the electronic device 600 corresponds to the operation moving distance 55 in the operation area in which the fingertip of the user UR performs the scrolling operation.

[0153] Here, in the case where the user UR is located outside the area of ​​the front face of the electronic device 600, for example, Fig.17 As shown, when the user UR is located on the right side of the front area of ​​the electronic device 600, the positional relationship between the user UR and the electronic device 600 is not parallel. In addition, when the user UR is located on the right side of the front area of ​​the electronic device 600, the first straight line 51 is shorter than the second straight line 52.

[0154] When the user UR performs a scrolling operation from the right end to the left end of the electronic device 600, the user UR may feel that the scrolling speed increases as the scrolling operation is performed toward the left end. This is because the operation object movement distance 56 when the user UR and the electronic device 600 are parallel in the X-axis direction and the distance in the Z-axis direction is equal is different from the operation object movement distance 54 corresponding to the operation movement distance 55. For example, Fig.17 In the case where the operation target moving distance 56 is applied to a position parallel to the operation moving distance 55 , the operation target moving distance 56 is shorter than the operation target moving distance 54 when the user UR is parallel to the electronic device 600 .

[0155] Therefore, when the user UR is located on the right side of the area in front of the electronic device 600, the operation object moving distance 54 is not proportional to the moving speed of the scrolling operation of the operation moving distance 55. Therefore, when the user UR performs a scrolling operation from the right end to the left end of the electronic device 600, the user feels that the moving speed becomes faster as the scrolling operation is performed toward the left end. The user UR changes the positional relationship between the user and the electronic device 600, and therefore sometimes feels that it is difficult to perform the scrolling operation.

[0156] Therefore, in the second embodiment, an information processing apparatus is provided that can improve the operability of the user UR when the user UR is located outside the area on the front surface of the electronic device 600 during the scroll operation of the user UR.

[0157] Specifically, the information processing device involved in the second embodiment determines the size of the expanded operation area based on the positional relationship between the electronic device 600 and the operator. For example, when the electronic device 600 is a display, the information processing device involved in the second embodiment determines the size of the expanded operation area based on the size of the operation area when the display is arranged in front of the operator in a state where the shortest distance between the display and the operator is equal.

[0158] Fig.18 1 is a diagram showing a schematic configuration of an information system including a control device 2 as an information processing device according to a second embodiment. The information processing device according to the second embodiment includes a sensor 500, an electronic device 600, and a control device 2. Fig.18 In Figure 4 The common points of the control device 1 according to the first embodiment shown will be omitted in detail.

[0159] The control device 2 includes a control unit 10 and a storage unit 30. The control unit 10 is configured as a CPU, for example, to centrally control the operations of the various components of the control device 2. The control device 2 according to the present embodiment includes a ROM and a RAM (not shown). The ROM stores various programs. The RAM is a work area when the CPU executes a program.

[0160] The CPU uses the RAM as a work area to execute the program stored in the ROM. Fig.18 As shown, the image acquisition unit 11, the first determination unit 12, the position detection unit 13, the operation region generation unit 14, the second determination unit 15, the operation region adjustment unit 16, the motion detection unit 17, the third determination unit 18 and the processing execution unit 19 are implemented.

[0161] The storage unit 30 stores coordinate information 31, detection effective area information 32, operation area information 33, extended operation area information 34, device operation amount information 35, and movement amount information 36. The movement amount information 36 is information indicating the movement amount of the fingertip 72 of the user UR. The device operation amount information 35 is information indicating the operation of the electronic device 600 corresponding to the movement amount of the fingertip 72 of the user UR. The operation of the electronic device 600 includes, for example, a scroll operation in the electronic device 600.

[0162] The operation area generation unit 14 generates an operation area based on the camera image of the user UR. The operation area represents an area determined by linking the position of the user UR detected by the position detection unit 13 with the operation target area set on the electronic device. For example, when the electronic device 600 is a display and the shortest distance between the display and the operator is equal, the operation area generation unit 14 determines the size of the expanded operation area based on the size of the operation area when the display is arranged in front of the operator. Here, the operation area is determined by Fig.17 The operation area generated by the operation area generating unit 14 will be described.

[0163] Here, the operation target area is, for example, an area on the display where a scroll operation is performed from one end of the display to the other end when the electronic device 600 is a display. Fig.17 In the figure, one end of the display is the right end of the electronic device 600, and the other end of the display is the left end of the electronic device 600.

[0164] In addition, Fig.17 , the first operation angle 53 is an operation area indicating an area determined by connecting the position of the user UR with the operation target area set on the electronic device. The operation area is, for example, an area set according to the first operation angle 53 indicating an angle formed by a first straight line 51 and a second straight line 52, wherein the first straight line 51 is a straight line connecting the right end of the electronic device 600 and the position of the operator, and the second straight line 52 is a straight line connecting the left end of the electronic device 600 and the position of the operator.

[0165] Return to Fig.18 When the second determination unit 15 determines that the position of the operator detected by the position detection unit 13 is outside the front area of ​​the electronic device 600, the operation area adjustment unit 16 expands the operation area generated by the operation area generation unit 14. Fig.19 The operation area to be expanded by the operation area adjustment unit 16 will be described.

[0166] Fig.19 This is a schematic diagram for explaining an example of an operation area to be expanded according to the second embodiment. Fig.19 and Fig.17 Similarly, the electronic device 600 and the user UR operating the electronic device 600 are shown. Here, the distance of the operation object movement distance 54 when the user UR is parallel to the electronic device 600 is set to d [mm], the distance from the user UR to the fingertip is set to a [mm], the distance of the first straight line 51 is set to b [mm], and the angle of the first operation angle 53 is set to θ [degree].

[0167] exist Fig.19 In the example, the moving distance P [mm] of the fingertip of the user UR can be calculated as P = a × sin (θ / 2). In addition, when the electronic device 600 is located parallel to the user UR at the distance of the first straight line 51, if the moving distance 56 of the operation object on the electronic device 600 is set to Q [mm], it can be calculated as Q = 2b × sin (θ / 2).

[0168] At this time, when the user UR's fingertip scrolling operation moves a distance P [mm] and the scrolling operation reaches Q [mm] on the electronic device 600, it is the same as the moving distance on the electronic device 600 when the user UR is located in the area in front of the electronic device 600, so the scrolling speed is the same as the scrolling speed when the operation is performed when the user UR is located in the area in front of the electronic device 600.

[0169] On the other hand, since the user UR is located on the right side of the area in front of the electronic device 600, the operation target moving distance 54 when the user UR is parallel to the electronic device 600 only has an operation target area of ​​dQ [mm] left. Therefore, the operation area adjustment unit 16 enlarges the operation target moving distance 56 by dQ [mm] as an enlarged operation target moving distance 561, and determines the size of the enlarged operation area, so as to perform a scrolling operation at the same scrolling speed as when the user UR is located in the area in front of the electronic device 600, with respect to the remaining distance dQ [mm] of the operation target area, while making the shortest distance between the electronic device 600 and the user UR equal. As a result, the enlarged operation target moving distance 561 is 2d-Q [mm].

[0170] In addition, the operation area adjustment unit 16 increases the operation movement distance 551 corresponding to the enlarged operation target movement distance 561 by a×(d / b-sin(θ / 2))[mm], and determines the size of the enlarged operation area. As a result, the operation movement distance 551 corresponding to the enlarged operation target movement distance 561 is a×(1+d / b-sin(θ / 2))[mm].

[0171] Furthermore, the operation area adjustment unit 16 increases the enlarged first operation angle 531 corresponding to the enlarged operation object moving distance 561 by (θ-2a*sin(d / b)-a*θ)[degree], and determines the size of the enlarged operation area. As a result, the first operation angle 531 corresponding to the enlarged operation object moving distance 561 is (2θ-2a*sin(d / b)-a*θ)[degree].

[0172] As described above, the control device 2 involved in the second modified example of the present disclosure determines the size of the expanded operation area based on the positional relationship between the electronic device 600 and the operator. The control device 2 determines the size of the expanded operation area based on the size of the operation area when the display is arranged in front of the operator when the electronic device 600 is a display and the shortest distance between the display and the operator is equal.

[0173] Thus, for example, when the operator performs a scrolling operation and the operator's position is outside the area in front of the electronic device 600, the control device 2 expands the operation area, thereby allowing the operator to perform the same scrolling operation as when the operator is located in the area in front of the electronic device 600. Thus, the control device 2 can improve the operability of the user.

[0174] (Eighth Modification)

[0175] For example, the operation target area of ​​the electronic device 600 only needs to be within the detection effective area AR200, and the control device 2 may expand the operation target area so as to overlap with the area outside the electronic device 600. Fig. 20 The operation target area to be expanded according to the eighth modification example will be described. Fig. 20 It is a schematic diagram for explaining an example of an operation target region to be expanded according to the eighth modification example.

[0176] Fig. 20 Corresponds to Fig.19 In addition, the length of the electronic device 600 is set to L1 [mm], the length of the detection effective area AR200 of the electronic device 600 is set to L2 [mm], and L2>L1>0. In the case where the control device 2 sets the detection effective area AR200 to a larger range than the electronic device 600, the operation area adjustment unit 16 of the control unit 10 expands the operation object movement distance 56 within the range inside the detection effective area AR200, and determines the size of the expanded operation area corresponding to the expanded operation object movement distance 561.

[0177] (Ninth Modification)

[0178] In the above-mentioned embodiment, the electronic device 600 is described as being located at a position perpendicular to the Y-axis direction, but the present invention is not limited thereto. Fig.21 To illustrate how the electronic device 600 is tilted in the positive direction of the Z axis relative to the user UR.

[0179] Fig.21 It is a schematic diagram for explaining an example of an operation area to be expanded according to the ninth modification. Fig.21 The electronic device 600 and the user UR operating the electronic device are shown. The electronic device 600 is located at a position perpendicular to the Y-axis direction. The electronic device 603 is tilted in the positive direction of the Z-axis relative to the user UR.

[0180] The operation area generating unit 14 generates an operation object moving distance 64 of an operation object area in which the user UR performs a scrolling operation on the electronic device 600 located in a vertical position, based on a first operation angle 63 representing an angle formed by a first straight line 61 and a second straight line 62, wherein the first straight line 61 is a straight line connecting the lower end of the electronic device 600 and the position of the operator, and the second straight line 62 is a straight line connecting the upper end of the electronic device 600 and the position of the operator. Fig.21 In the case of , the operation object moving distance 64 is the scrolling distance for scrolling in the Y-axis direction.

[0181] Furthermore, the operation area adjustment unit 16 sets the operation object moving distance 66 corresponding to the electronic device 603 to the enlarged operation object moving distance 661, and determines the size of the enlarged operation area so that the scrolling operation of the tilted electronic device 603 is performed at the same scrolling speed as when the user UR is located at a vertical position of the electronic device 600. Specifically, the operation area adjustment unit 16 determines to make the operation object moving distance 64 equal to the distance of the enlarged operation object moving distance 661.

[0182] In addition, the operation area adjustment unit 16 determines the size of the expanded operation area for the expanded operation movement distance 651 corresponding to the expanded operation target movement distance 661. Furthermore, the operation area adjustment unit 16 determines the size of the expanded operation area for the expanded first operation angle 631 corresponding to the expanded operation target movement distance 661.

[0183] (Tenth Modification)

[0184] The sensor 500 is not limited to a visible light camera, and may be, for example, a CCD (Charge Coupled Device) camera, a CMOS (Complementary Metal Oxide Semiconductor) camera, or an RGB camera. In addition, the sensor 500 is not limited to a camera device, and may be a TOF (Time of Flight) sensor or an infrared sensor.

[0185] (Eleventh Modification)

[0186] The position detection unit 13 detects the fingertips of the user UR based on the image captured of the user serving as the operator, but the present invention is not limited to this. For example, the position detection unit 13 may detect the hand, palm, or arm of the user UR.

[0187] (Third Embodiment)

[0188] In the third embodiment, the method of selecting the enlarged operation target area by the fingertip 72 of the user UR is described as follows. Specifically, when the action of selecting one of the multiple operation areas is detected by the action detection unit 17, the operation area adjustment unit 16 of the control unit 10 determines the size of the enlarged operation target area according to the prescribed action.

[0189] Fig. 22 is a schematic diagram for explaining an example of an expanded operation target area according to the third embodiment. Fig. 22 , an electronic device 600, and buttons 81, 82, 83, 84, and 85 for outputting the electronic device 600 are shown. Fig. 22 83, an enlarged operation target area 831 corresponding to button 83, an enlarged operation target area 841 corresponding to button 84, and an enlarged operation target area 851 corresponding to button 85 are shown. Fig. 22 , the sizes of the operation object area 831, the operation object area 842, and the operation object area 851 are equal.

[0190] For example, in Fig. 22In the embodiment, the action detection unit 17 of the control unit 10 detects the action of the user UR selecting the button 84 in the operation target area expanded by the operation area adjustment unit 16 based on the camera image of the user UR. For example, the fingertip 72 of the user UR enters the range closer to the electronic device 600 than the virtual operation area AR100, and the action detection unit 17 detects the action of the user UR selecting the button 84 in the operation target area expanded by the operation area adjustment unit 16. Here, the action of the user UR selecting the button 84 includes the action before the fingertip 72 of the user UR moves toward the button 84 and selects the button 84.

[0191] Then, if Fig.23 As shown, the operation area adjustment unit 16 increases the size of the operation target area 841 corresponding to the button 84 before selection, and reduces the sizes of the operation target area 831 and the operation target area 851 adjacent to the operation target area 841 .

[0192] Fig.23 It is a schematic diagram for explaining an example of the expanded operation target region according to the third embodiment. Fig.23 yes Fig. 22 Schematic diagram of the state before the user UR selects button 84. Fig.23 In, with Fig. 22 Similarly, the electronic device 600, and buttons 81, 82, 83, 84, and 85 for outputting the electronic device 600 are shown. Fig.23 , a reduced operation target area 832 corresponding to the button 83 , an enlarged operation target area 842 corresponding to the button 84 , and a reduced operation target area 852 corresponding to the button 85 are shown.

[0193] Here, about Fig.23 Among the operation target areas in the operation target area 842, the operation target area 842 has the largest size, and then the sizes of the operation target areas 832 and 852 adjacent to the operation target area 842 are equal and smaller than the size of the operation target area 842. That is, when the sizes of the enlarged operation target areas corresponding to the plurality of operation areas are equal, the operation area adjustment unit 16 enlarges the size of the enlarged operation target area corresponding to the selected operation area, and reduces the sizes of the enlarged operation target areas corresponding to the other operation areas adjacent to the selected operation area.

[0194] Next, the user UR is explained from the selection Fig.23 From the previous state of button 84 shown in the figure, a sliding action is performed in the positive direction of the X axis and the previous state of button 85 is selected. Fig.23In the embodiment, the action detection unit 17 detects the action of the user UR selecting the button 85 in the operation target area expanded by the operation area adjustment unit 16 based on the camera image of the user UR. Fig.24 As shown, the operation area adjustment unit 16 increases the size of the operation target area 852 corresponding to the button 85 before selection, and reduces the size of the operation target area 842 adjacent to the operation target area 852 .

[0195] Fig.24 It is a schematic diagram for explaining an example of the expanded operation target region according to the third embodiment. Fig.24 yes Fig.23 The schematic diagram of the state where the user UR has selected the button 85 is shown. Fig.24 In, with Fig. 22 and Fig.23 Similarly, the electronic device 600, and buttons 81, 82, 83, 84, and 85 for outputting the electronic device 600 are shown. Fig.24 , an enlarged operation target area 833 corresponding to the button 83 , a reduced operation target area 843 corresponding to the button 84 , and an enlarged operation target area 853 corresponding to the button 85 are shown.

[0196] Here, about Fig.24 853 is the largest operation target area, then the size of the operation target area 833 is smaller than the size of the operation target area 853, and the size of the operation target area 842 adjacent to the operation target area 853 is smaller than the size of the operation target area 853. That is, the operation area adjustment unit 16 increases the size of the operation target area corresponding to the operation target area before selection is made, reduces the size of other operation target areas adjacent to the operation target area before selection is made, and maintains the size of other operation target areas not adjacent to the operation target area before selection is made.

[0197] As described above, the control device according to the third embodiment of the present disclosure determines the size of the expanded operation target area according to a predetermined action when an action of selecting one of a plurality of operation areas is detected.

[0198] Thus, for example, when the operator selects an operation area, the control device can determine the size of the operation target area corresponding to the operation area according to the action of selecting. Therefore, the control device can improve the operability of the user.

[0199] (Tenth Modification)

[0200] In the third embodiment described above, the method of selecting the enlarged operation object area when the enlarged operation object area corresponding to the button output by the electronic device 600 is equal is described, but it is not limited to this. In the tenth variation, the method of selecting the enlarged operation object area when the enlarged operation object area corresponding to the button output by the electronic device 600 is different is described.

[0201] Fig.25 1 is a schematic diagram for explaining an example of an expanded operation target area according to the tenth modification example. Fig.25 In, with Fig. 22 Similarly, the electronic device 600, and buttons 81, 82, 83, 84, and 85 for outputting the electronic device 600 are shown. Fig.25 , an enlarged operation target area 834 corresponding to the button 83 , an enlarged operation target area 844 corresponding to the button 84 , and an enlarged operation target area 854 corresponding to the button 85 are shown.

[0202] Here, in Fig.25 , the sizes of the operation target area 834, the operation target area 844, and the operation target area 854 are different, and the sizes increase in the order of the operation target area 834, the operation target area 844, and the operation target area 854. For example, when the user UR is located in front of the button 83, the operation area adjustment unit 16 increases the size of the operation target area corresponding to the button output by the electronic device 600 as the distance from the center of the user UR increases.

[0203] For example, in Fig.25 In the embodiment, the action detection unit 17 detects the action of the user UR selecting the button 84 in the operation target area expanded by the operation area adjustment unit 16 based on the camera image of the user UR. Fig.26 As shown, the operation area adjustment unit 16 increases the size of the operation target area 844 corresponding to the button 84 before selection, and reduces the sizes of the operation target area 834 and the operation target area 854 adjacent to the operation target area 844 .

[0204] Fig.26 It is a schematic diagram for explaining an example of an expanded operation target area according to the tenth modification example. Fig.26 yes Fig.25 Schematic diagram of the state before the user UR selects button 84. Fig.26 In, with Fig.25 Similarly, the electronic device 600, and buttons 81, 82, 83, 84, and 85 for outputting the electronic device 600 are shown. Fig.26, a reduced operation target area 835 corresponding to the button 83 , an enlarged operation target area 845 corresponding to the button 84 , and a reduced operation target area 855 corresponding to the button 85 are shown.

[0205] Here, about Fig.26 Among the operation target areas in the display, the size of the operation target area 845 corresponding to the button 84 before the user UR makes the selection is the largest, and then the size of the operation target area 855 adjacent to the operation target area 845 and far from the center of the user UR, and then the size of the operation target area 835 adjacent to the operation target area 845 and located in front of the user UR, is reduced in order and is smaller than the size of the operation target area 845. That is, in the case where the sizes of the enlarged operation target areas corresponding to the plurality of operation areas are different, the operation area adjustment unit 16 enlarges the size of the operation target area corresponding to the selected operation area, and reduces the sizes of other operation target areas adjacent to the selected operation target area.

[0206] (Eleventh Modification)

[0207] In the eleventh modified example, a method of selecting the enlarged operation target area when the enlarged operation target area corresponding to the button output by the electronic device 600 overlaps is described. Specifically, when the action detection unit 17 detects an action of selecting one of the multiple operation areas, the operation area adjustment unit 16 moves the enlarged operation target area.

[0208] Fig. 27 1 is a schematic diagram for explaining an example of an expanded operation target area according to the eleventh modification example. Fig. 27 In, with Fig. 22 Similarly, the electronic device 600, and buttons 81, 82, 83, 84, and 85 for outputting the electronic device 600 are shown. Fig. 27 , an enlarged operation target area 836 corresponding to the button 83 , an enlarged operation target area 846 corresponding to the button 84 , and an enlarged operation target area 856 corresponding to the button 85 are shown.

[0209] Here, in Fig. 27 , the sizes of the operation target area 836, the operation target area 846, and the operation target area 856 are equal, the right side of the operation target area 836 and the left side of the operation target area 846 are overlapped, and the right side of the operation target area 846 and the left side of the operation target area 856 are overlapped. In addition, the sizes of the operation target area 836, the operation target area 846, and the operation target area 856 are equal.

[0210] For example, in Fig. 27 In the embodiment, the action detection unit 17 detects the action of the user UR before selecting the button in the operation target area expanded by the operation area adjustment unit 16 based on the camera image of the user UR. Fig.28 As shown, the operation area adjustment unit 16 moves each operation target area according to the movement of the fingertip 72 of the user UR.

[0211] Here, use Fig.28 The following describes the processing performed by the operation area adjustment unit 16 in a state where the fingertip 72 of the user UR has moved to the right side of the electronic device 600 in the positive direction of the X-axis.

[0212] Fig.28 1 is a schematic diagram for explaining an example of an expanded operation target area according to the eleventh modification example. Fig.28 In, with Fig. 27 Similarly, the electronic device 600, and buttons 81, 82, 83, 84, and 85 for outputting the electronic device 600 are shown. Fig.28 , an operation target area 837 corresponding to the button 83, an operation target area 847 corresponding to the button 84, and an operation target area 857 corresponding to the button 85 are shown.

[0213] Fig.28 If the positions of the operation object area 837, the operation object area 847 and the operation object area 857 in the X-axis direction are the same as Fig. 27 Compared with the positions of the operation object areas 836, 846, and 856 in the X-axis direction, the operation object areas 837, 847, and 857 in the X-axis direction are moved to the right. This is the state obtained as a result of the operation area adjustment unit 16 moving the positions of the operation object areas 837, 847, and 857 in the X-axis direction in the positive direction of the X-axis according to the movement of the fingertip 72 of the user UR moving in the positive direction of the X-axis as shown by the arrow M1. That is, when the movement detection unit 17 detects the movement in the first direction to the left and right of the front of the plurality of operation areas, the operation area adjustment unit 16 moves the expanded operation object areas in the first direction.

[0214] Next, use Fig.29 The following describes the processing performed by the operation area adjustment unit 16 in a state where the fingertip 72 of the user UR has moved toward the left side of the electronic device 600 in the negative direction of the X-axis.

[0215] Fig.29 1 is a schematic diagram for explaining an example of an expanded operation target area according to the eleventh modification example. Fig.29 In, with Fig. 27Similarly, the electronic device 600, and buttons 81, 82, 83, 84, and 85 for outputting the electronic device 600 are shown. Fig.29 , an operation target area 838 corresponding to the button 83, an operation target area 848 corresponding to the button 84, and an operation target area 858 corresponding to the button 85 are shown.

[0216] Fig.29 If the positions of the operation object area 838, the operation object area 848, and the operation object area 858 in the X-axis direction are the same as Fig. 27 Compared with the positions of the operation object areas 836, 846, and 856 in the X-axis direction, the operation object areas 838, 848, and 856 in the X-axis direction are moved to the left. This is the state obtained as a result of the operation area adjustment unit 16 moving the positions of the operation object areas 838, 848, and 858 in the X-axis direction in the negative direction of the X-axis according to the movement of the fingertip 72 of the user UR moving in the negative direction of the X-axis as shown by the arrow M2. That is, when the movement detection unit 17 detects the movement in the first direction to the left and right of the front side representing the plurality of operation areas, the operation area adjustment unit 16 moves the expanded operation object areas in the first direction.

[0217] Next, the operation of selecting a button by the fingertip 72 of the user UR after the operation area adjustment unit 16 moves the expanded operation target area according to the operation of the fingertip 72 of the user UR will be described.

[0218] For example, in Fig.28 In the example, the action detection unit 17 detects the action of the user UR selecting the button 84 in the operation target area expanded by the operation area adjustment unit 16 based on the camera image of the user UR. For example, when the fingertip 72 of the user UR enters a range closer to the electronic device 600 than the virtual operation area AR100 and the fingertip 72 of the user UR remains stationary in the detection effective area AR200 for a certain period of time, the action detection unit 17 detects the action of the user UR selecting the button 84 in the operation target area expanded by the operation area adjustment unit 16.

[0219] Then, the operation area adjustment unit 16 increases the size of the operation target area 847 corresponding to the selected button 84 , and reduces the sizes of the operation target area 837 and the operation target area 857 adjacent to the operation target area 847 .

[0220] Fig.30 It is a schematic diagram for explaining an example of the expanded operation target area involved in the eleventh modification. Fig.30 yes Fig.28The schematic diagram of the state where the user UR has selected the button 84 is shown. Fig.30 In, with Fig.28 Similarly, the electronic device 600, and buttons 81, 82, 83, 84, and 85 for outputting the electronic device 600 are shown. Fig.30 , a reduced operation target area 860 corresponding to the button 83 , an enlarged operation target area 861 corresponding to the button 84 , and a reduced operation target area 862 corresponding to the button 85 are shown.

[0221] Here, about Fig.30 Among the operation target areas in the image, the size of the operation target area 861 corresponding to the button 84 selected by the user UR is the largest, and then the size of the operation target area 862 adjacent to the operation target area 861 and far from the center of the user UR is reduced in order of size and smaller than the size of the operation target area 861. That is, when the action detection unit 17 detects that an operation area is selected in one of the multiple operation areas, the operation area adjustment unit 16 increases the size of the operation target area corresponding to the selected operation area and reduces the sizes of other operation target areas adjacent to the selected operation target area.

[0222] also, Fig.30 Explained Fig.28 The user UR is shown in the state where the button 84 is selected, but it can also be applied to Fig.23 , Fig.24 , Fig.26 as well as Fig.29 The schematic diagram shown shows a state in which the button 84 is selected by the user UR.

[0223] In addition, for example, when the fingertip 72 of the user UR is Fig. 27 When the front side of the portion where the multiple operation object areas overlap moves along the Z axis toward the electronic device 600, the action detection unit 17 detects the action of the user UR selecting a button. In this case, the action detection unit 17 determines that the fingertip of the user UR has selected a button located at a closer position. Furthermore, for example, when the fingertip 72 of the user UR performs a selection action in an area closer to the button 84 than the button 83 on the front side of the area where the operation object area 836 overlaps the operation object area 846, the action detection unit 17 detects the action of the user UR selecting the button 84.

[0224] (Fourth Embodiment)

[0225] In the fourth embodiment, a selection operation in which a predetermined motion corresponds to a swinging back motion of the fingertip 72 of the user UR will be described.

[0226] Fig.31 FIG. 2 is a schematic diagram for explaining the operation process of the information processing device involved in the third comparative example. Fig.31 , an electronic device 600, an eye of a user UR operating the electronic device 600, a fingertip 72 of the user UR operating the electronic device 600, and a plurality of buttons for output of the electronic device 600 are shown. The plurality of buttons are, for example, button 91, button 92, button 93, button 94, and button 95. In addition, arrows 931, 941, and 951 respectively indicate the operation areas to be expanded of button 93, button 94, and button 95.

[0227] For example, Fig.31 The schematic diagram shown is a schematic diagram in which the user UR uses the fingertip 72 of the user UR to slide from the button 91 located in front of the user UR to the button 93 along the X-axis direction as shown by the arrow M5, and the fingertip 72 of the user UR selects the button 93.

[0228] in addition, Fig.31 The schematic diagram shown is a schematic diagram in which the user UR uses the fingertip 72 of the user UR to slide along the Z-axis direction as shown by the arrow M6 in the state where the button 93 is selected. The sliding action of the fingertip of the user UR along the Z-axis direction refers to a swinging action such as an action of pulling the fingertip 72 of the user UR back in front of the user UR. In other words, the swinging action refers to an action toward a third direction away from the front of the operation object area corresponding to the selected operation area. At this time, the position of the eyes of the user UR does not change.

[0229] Here, when the user UR uses the fingertip 72 of the user UR to slide in the Z-axis direction as shown by the arrow M6 in the state where the button 93 is selected, the following situation occurs: the direction of the eye of the user UR and the fingertip 72 of the user UR moves from the arrow 722 to the arrow 723 corresponding to the button 95, and the button 95 is selected. This is because the direction connecting the eye of the user UR and the fingertip 72 of the user UR is changed by the swinging action of the user UR, and the operation angle of the user UR and the button 93 deviates and becomes the operation angle of the user UR and the button 95, thereby the button 95 is selected.

[0230] Thus, the user UR may find it difficult to operate the button because the button 95 is selected even though the button 93 is selected. Therefore, the fourth embodiment describes an information processing device that can improve the operation of the selection corresponding to the swinging motion of the fingertip 72 of the user UR.

[0231] Fig.32 is a schematic diagram for explaining an example of an expanded operation target area according to the fourth embodiment. Fig.32 , the electronic device 600, the fingertip 72 of the user UR operating the electronic device 600, and the buttons 81, 82, 83, 84, and 85 for output of the electronic device 600 are shown. Fig.32 Is user UR from Fig.31 The diagram shows a state in which the fingertip 72 of the user UR changes to a state of a swinging motion.

[0232] For example, in Fig.32 In the figure, when the user UR's fingertip 72 selects the button 84 and moves the user UR's fingertip 72 in the positive direction of the Z axis as shown by the arrow M7, the operation area adjustment unit 16 further expands the operation object area 861 corresponding to the button 84 into an operation object area 863 that is larger than the operation object area 861.

[0233] That is, when the motion detection unit 17 detects a motion in a third direction away from the front of the operation object area corresponding to the selected operation area, the operation area adjustment unit 16 further expands the size of the operation object area corresponding to the selected operation area. Then, the operation area adjustment unit 16 restores the size of the operation object area before the expansion after a certain period of time. Here, the certain period of time is one second. The certain period of time is not limited to one second and can be set arbitrarily. Thus, the operation area adjustment unit 16 can set the operation area corresponding to the swinging motion of the user UR.

[0234] Fig.33 FIG. 4 is a schematic diagram showing an example of the content of the operation process of the expanded operation target area involved in the fourth embodiment. Fig.33 7A and 7B will now be described in detail with reference to the accompanying drawings. In the following, the process of detecting the swinging motion of the user UR by the motion detection unit 17 of the control unit 10 will be described.

[0235] exist Fig.33 , an electronic device 600, an eye of a user UR operating the electronic device 600, a fingertip 72 of the user UR operating the electronic device 600, and a plurality of buttons for output of the electronic device 600 are shown. The plurality of buttons are, for example, a button 91, a button 92, a button 93, a button 94, and a button 95. In addition, the direction of the eye of the user UR and the fingertip 72 of the user UR is an arrow 722 pointing to the button 93. Fig.33 The state in which the fingertip 72 of the user UR selects the button 93 is shown, and the fingertip 72 of the user UR is swung back in the order of the arrow M61 and the arrow M62 .

[0236] When determining that at least one of the movement speed, movement direction, and movement distance of the fingertip 72 of the user UR may be a swing motion regarding the swing motion of the user UR, the motion detection unit 17 determines that the swing motion is a swing motion.

[0237] Specifically, in Fig.33 When the moving speed of the fingertip 72 of the user UR in the Z direction is greater than a predetermined speed threshold as shown by arrows M61 and M62, the motion detection unit 17 determines that there is a possibility of swinging back. x and time t x+1 When the moving speed corresponding to the moving distance of the fingertip 72 of the user UR in the Z direction is 0.1 [m / s] or more, the motion detection unit 17 determines that there is a possibility of swinging back. In addition, the predetermined speed threshold is described as 0.1 [m / s], but it is not limited to this.

[0238] In addition, specifically, the fingertips of the user UR exist Fig.33 If the movement is within the substantially triangular range R61 and range R62 shown in FIG. 1 and moves in the positive direction of the Z direction, the motion detection unit 17 determines that there is a possibility of swinging back. For example, the motion detection unit 17 determines that the movement is likely to swing back at time t x The position is used as the reference to calculate the time t x+1 In the case where the calculated moving direction is less than a predetermined angle, it is determined that there is a possibility of swinging back. In addition, the calculation of the moving direction can also be combined with t x Relatively speaking, t x+y Here, y is set to an arbitrary value.

[0239] And, specifically, in Fig.33 When the moving distance of the fingertip 72 of the user UR in the Z direction is greater than a predetermined distance threshold value as shown by the arrows M61 and M62, the motion detection unit 17 determines that there is a possibility of swinging back. For example, the motion detection unit 17 determines the moving distance of the fingertip 72 of the user UR in the Z direction based on the camera image of the user UR, and when there are five or more consecutive frames in which the fingertip 72 of the user UR moves in the Z direction by more than 10 [cm] in the camera image, the motion detection unit 17 determines that there is a possibility of swinging back.

[0240] In addition, the predetermined distance threshold is described as 10 [cm], but it is not limited to this. In addition, the number of frames determined as having the possibility of swinging back is described as five or more, but it is not limited to this. In addition, for example, the motion detection unit 17 may also be based on the time t x The position is used as the reference to calculate the time t x+1 If the calculated moving distance is greater than a predetermined distance, it is determined that there is a possibility of swinging back.

[0241] As described above, the control device involved in the fourth embodiment of the present disclosure further expands the size of the operation object area corresponding to the selected operation area when a movement in a third direction away from the front of the operation object area corresponding to the selected operation area is detected, and restores the size of the operation object area before the expansion after a certain period of time.

[0242] Thus, the control device can prevent the operator from selecting another operation area against his will when the operator performs a swinging motion after selecting the operation area, for example. Therefore, the control device can improve the operability of the user.

[0243] (Twelfth Modification)

[0244] In the third embodiment, the fourth embodiment, the tenth variation, and the eleventh variation described above, the case where the user UR is located outside the area on the front of the electronic device 600 is described, but the present invention is not limited thereto. The third embodiment, the fourth embodiment, the tenth variation, and the eleventh variation described above can also be applied to the case where the user UR is located in the front and center of the electronic device 600, or the case where the user UR is located inside the area on the front of the electronic device 600.

[0245] The embodiments of the present disclosure are described above, but the above embodiments are presented as examples and are not intended to limit the scope of the invention. These new embodiments can be implemented in various other ways, and various omissions, substitutions, and changes can be made without departing from the scope of the main purpose of the invention. These new embodiments and their modifications are included in the scope and main purpose of the invention, and are included in the invention described in the claims and their equivalents. In addition, the constituent elements involved in different embodiments and modifications may be appropriately combined.

[0246] In addition, the expression "...part" in the above-mentioned embodiments may be replaced by other expressions such as "...circuitry", "...component", "...device", "...unit" or "...module".

[0247] In the above-mentioned embodiments, the present disclosure has been described by taking an example in which the present disclosure is configured using hardware. However, the present disclosure can also be realized by software in cooperation with hardware.

[0248] In addition, each functional block used in the description of each embodiment is typically implemented as an LSI as an integrated circuit. The integrated circuit controls each functional block used in the description of the above embodiment, and may also have input terminals and output terminals. They can be monolithic individually or in a manner that includes part or all of them. Here, it is set as LSI, but depending on the degree of integration, it is sometimes also called IC, system LSI, super LSI, and very large LSI.

[0249] In addition, the method of integrated circuitization is not limited to LSI, and can also be realized using a dedicated circuit or a general-purpose processor and memory. It is also possible to use a programmable FPGA (Field Programmable Gate Array) or a reconfigurable processor (Reconfigurable Processor) that can reconfigure the connection or setting of the circuit unit inside the LSI after manufacturing the LSI.

[0250] Furthermore, if integrated circuit technology that replaces LSI emerges due to advances in semiconductor technology or other derived technologies, it is of course possible to integrate functional blocks using this technology. Application to biotechnology is possible as a possibility.

[0251] In addition, the effects in the embodiments described in this specification are always illustrative and non-limiting, and other effects may also be present.

[0252] (Note)

[0253] According to the description of the above embodiments, the following technology is disclosed.

[0254] (Technology 1)

[0255] An information processing device comprising:

[0256] a position detection unit that detects the position of the operator;

[0257] an action detection unit that detects a predetermined action of the operator in an operation area, wherein the operation area represents an area determined by linking the position of the operator detected by the position detection unit with an operation target area set on at least one electronic device;

[0258] a processing execution unit that performs processing associated with the operation target area when the predetermined motion is detected by the motion detection unit; and

[0259] The operation area adjustment unit is configured to expand the operation area when the position of the operator detected by the position detection unit is outside the front area of ​​the at least one electronic device.

[0260] (Technique 2)

[0261] The information processing device according to the above-mentioned technique 1, wherein:

[0262] The at least one electronic device is a display,

[0263] On the display, there are a near button area and a far button area as at least two of the operation object areas, wherein the near button area represents an area of ​​near buttons arranged on a side close to the operator, and the far button area represents an area of ​​far buttons arranged on a side far from the operator,

[0264] The operation area adjustment unit sets the operation areas corresponding to the near button area and the far button area, respectively, so that a first angle representing an angle formed by a first straight line and a second straight line is equal to a second angle representing an angle formed by a third straight line and a fourth straight line, wherein the first straight line is a straight line connecting one end in the direction of a center line connecting a center of the near button area and a center of the far button area in the operation area corresponding to the near button area with the position of the operator, the second straight line is a straight line connecting the other end in the direction of the center line in the operation area corresponding to the near button area with the position of the operator, the third straight line is a straight line connecting one end in the direction of the center line in the operation area corresponding to the far button area with the position of the operator, and the fourth straight line is a straight line connecting the other end in the direction of the center line in the operation area corresponding to the far button area with the position of the operator.

[0265] (Technique 3)

[0266] The information processing device according to the above-mentioned technique 2, wherein:

[0267] The first straight line is a straight line connecting the right end of the near button area and the position of the operator.

[0268] The second straight line is a straight line connecting the left end of the near button area and the position of the operator.

[0269] The third straight line is a straight line connecting the right end of the far button area and the position of the operator.

[0270] The fourth straight line is a straight line connecting the left end of the far button area and the position of the operator.

[0271] (Technique 4)

[0272] The information processing device according to the above-mentioned technique 2, wherein:

[0273] The first straight line is a straight line connecting the upper end of the near button area and the position of the operator.

[0274] The second straight line is a straight line connecting the lower end of the near button area and the position of the operator.

[0275] The third straight line is a straight line connecting the upper end of the remote button area and the position of the operator.

[0276] The fourth straight line is a straight line connecting the lower end of the far button area and the position of the operator.

[0277] (Technique 5)

[0278] The information processing device according to the above-mentioned technique 1, wherein:

[0279] The at least one electronic device is a display,

[0280] On the display, there are at least two operation target areas, a first button area indicating an area where a first button is arranged and a second button area indicating an area where a second button is arranged,

[0281] In the direction of a center line connecting the center of the first button area and the center of the second button area, the length of the first button area is smaller than the length of the second button area,

[0282] The operation area adjustment unit sets the operation areas corresponding to the first button area and the second button area, respectively, so that a first angle representing an angle formed by a first straight line and a second straight line is equal to a second angle representing an angle formed by a third straight line and a fourth straight line, wherein the first straight line is a straight line connecting one end in the direction of the center line in the operation area corresponding to the first button area and the position of the operator, the second straight line is a straight line connecting the other end in the direction of the center line in the operation area corresponding to the first button area and the position of the operator, the third straight line is a straight line connecting one end in the direction of the center line in the operation area corresponding to the second button area and the position of the operator, and the fourth straight line is a straight line connecting the other end in the direction of the center line in the operation area corresponding to the second button area and the position of the operator.

[0283] (Technique 6)

[0284] The information processing device according to the above-mentioned technique 1, wherein:

[0285] As the at least one electronic device, there are a first electronic device and a second electronic device,

[0286] When the position of the operator detected by the position detection unit is outside a front area corresponding to the front of the first electronic device and the second electronic device, the operation area adjustment unit expands the operation area of ​​at least one of the first electronic device and the second electronic device.

[0287] (Technique 7)

[0288] The information processing device according to the above-mentioned technique 1, wherein:

[0289] The operation area adjustment unit determines a size of the expanded operation area based on a positional relationship between the at least one electronic device and the operator.

[0290] (Technology 8)

[0291] The information processing device according to the above-mentioned technique 7, wherein:

[0292] The at least one electronic device is a display,

[0293] The operation area adjustment unit determines the size of the expanded operation area based on the size of the operation area when the display is arranged in front of the operator in a state where the shortest distance between the display and the operator is equal.

[0294] (Technique 9)

[0295] The information processing device according to the above-mentioned technique 1, wherein:

[0296] The at least one electronic device is a display,

[0297] In the case where there are multiple operation areas on the display,

[0298] When the motion detection unit detects a motion of selecting one of the plurality of operation areas, the operation area adjustment unit determines a size of the expanded operation target area according to the predetermined motion.

[0299] (Technology 10)

[0300] The information processing device according to the above-mentioned technique 9, wherein:

[0301] When the sizes of the enlarged operation object areas corresponding to the multiple operation areas are equal, the operation area adjustment unit will enlarge the size of the enlarged operation object area corresponding to the selected operation area, and reduce the size of the enlarged operation object area corresponding to the other operation areas adjacent to the selected operation area.

[0302] (Technology 11)

[0303] The information processing device according to the above-mentioned technique 9, wherein:

[0304] When the sizes of the enlarged operation object areas corresponding to the multiple operation areas are different, the operation area adjustment unit will enlarge the size of the enlarged operation object area corresponding to the selected operation area, and reduce the size of the enlarged operation object area corresponding to the other operation areas adjacent to the selected operation area.

[0305] (Technology 12)

[0306] The information processing device according to the above-mentioned technique 1, wherein:

[0307] The at least one electronic device is a display,

[0308] In the case where there are multiple operation areas on the display,

[0309] When the motion detection unit detects a motion of selecting one of the plurality of operation areas, the operation area adjustment unit moves the expanded operation target area according to the predetermined motion.

[0310] (Technology 13)

[0311] The information processing device according to the above-mentioned technique 12, wherein:

[0312] When the motion detection unit detects a motion in a first direction, the operation region adjustment unit moves the expanded operation target region in the first direction, the first direction indicating left and right of the front surfaces of the plurality of operation regions.

[0313] (Technology 14)

[0314] The information processing device according to the above-mentioned technique 9 or 12, wherein:

[0315] When the action detection unit detects an action of selecting one of the multiple operation areas, the operation area adjustment unit expands the size of the operation object area corresponding to the selected operation area and reduces the size of other operation object areas adjacent to the selected operation object area.

[0316] (Technology 15)

[0317] The information processing device according to the above-mentioned technique 14, wherein:

[0318] When the motion detection unit detects a motion in a third direction away from the front of the operation object area corresponding to the selected operation area, the operation area adjustment unit further expands the size of the operation object area corresponding to the selected operation area and restores it to the size of the operation object area before the expansion after a certain period of time.

[0319] (Technology 16)

[0320] An information processing system comprises a camera unit and an information processing device, wherein:

[0321] The camera unit captures a user who is an operator.

[0322] The information processing device comprises:

[0323] a position detection unit that detects the position of the operator based on the image captured by the camera unit;

[0324] an action detection unit that detects a predetermined action of the operator in an operation area, wherein the operation area represents an area determined by linking the position of the operator detected by the position detection unit with an operation target area set on at least one electronic device;

[0325] a processing execution unit that performs processing associated with the operation target area when the predetermined motion is detected by the motion detection unit; and

[0326] The operation area adjustment unit is configured to expand the operation area when the position of the operator detected by the position detection unit is outside the front area of ​​the at least one electronic device.

[0327] (Technology 17)

[0328] An information processing method is performed by an information processing device, the information processing method comprising:

[0329] A position detection step, detecting the position of the operator;

[0330] an action detection step of detecting a predetermined action of the operator in an operation area, wherein the operation area represents an area determined by linking the position of the operator detected by the position detection step with an operation target area set on at least one electronic device;

[0331] a processing execution step of performing processing associated with the operation target area when the predetermined motion is detected by the motion detection step; and

[0332] The operation area adjustment step is to expand the operation area when the position of the operator detected by the position detection step is outside the front area of ​​the at least one electronic device.

[0333] (Technology 18)

[0334] An information processing program that causes a computer to:

[0335] A position detection step, detecting the position of the operator;

[0336] an action detection step of detecting a predetermined action of the operator in an operation area, wherein the operation area represents an area determined by linking the position of the operator detected by the position detection step with an operation target area set on at least one electronic device;

[0337] a processing execution step of performing processing associated with the operation target area when the predetermined motion is detected by the motion detection step; and

[0338] The operation area adjustment step is to expand the operation area when the position of the operator detected by the position detection step is outside the front area of ​​the at least one electronic device.

Claims

1. An information processing device comprising: a position detection unit that detects the position of the operator; an action detection unit that detects a predetermined action of the operator in an operation area, wherein the operation area represents an area determined by linking the position of the operator detected by the position detection unit with an operation target area set on at least one electronic device; a processing execution unit that performs processing associated with the operation target area when the predetermined motion is detected by the motion detection unit; as well as The operation area adjustment unit is configured to expand the operation area when the position of the operator detected by the position detection unit is outside the front area of ​​the at least one electronic device.

2. The information processing device according to claim 1, wherein: The at least one electronic device is a display, On the display, there are a near button area and a far button area as at least two of the operation object areas, wherein the near button area represents an area of ​​near buttons arranged on a side close to the operator, and the far button area represents an area of ​​far buttons arranged on a side far from the operator, The operation area adjustment unit sets the operation areas corresponding to the near button area and the far button area, respectively, so that a first angle representing an angle formed by a first straight line and a second straight line is equal to a second angle representing an angle formed by a third straight line and a fourth straight line, wherein the first straight line is a straight line connecting one end in the direction of a center line connecting a center of the near button area and a center of the far button area in the operation area corresponding to the near button area with the position of the operator, the second straight line is a straight line connecting the other end in the direction of the center line in the operation area corresponding to the near button area with the position of the operator, the third straight line is a straight line connecting one end in the direction of the center line in the operation area corresponding to the far button area with the position of the operator, and the fourth straight line is a straight line connecting the other end in the direction of the center line in the operation area corresponding to the far button area with the position of the operator.

3. The information processing device according to claim 2, wherein: The first straight line is a straight line connecting the right end of the near button area and the position of the operator. The second straight line is a straight line connecting the left end of the near button area and the position of the operator. The third straight line is a straight line connecting the right end of the far button area and the position of the operator. The fourth straight line is a straight line connecting the left end of the far button area and the position of the operator.

4. The information processing device according to claim 2, wherein: The first straight line is a straight line connecting the upper end of the near button area and the position of the operator. The second straight line is a straight line connecting the lower end of the near button area and the position of the operator. The third straight line is a straight line connecting the upper end of the remote button area and the position of the operator. The fourth straight line is a straight line connecting the lower end of the far button area and the position of the operator.

5. The information processing device according to claim 1, wherein: The at least one electronic device is a display, On the display, there are at least two operation target areas, a first button area indicating an area where a first button is arranged and a second button area indicating an area where a second button is arranged, In the direction of a center line connecting the center of the first button area and the center of the second button area, the length of the first button area is smaller than the length of the second button area, The operation area adjustment unit sets the operation areas corresponding to the first button area and the second button area, respectively, so that a first angle representing an angle formed by a first straight line and a second straight line is equal to a second angle representing an angle formed by a third straight line and a fourth straight line, wherein the first straight line is a straight line connecting one end in the direction of the center line in the operation area corresponding to the first button area and the position of the operator, the second straight line is a straight line connecting the other end in the direction of the center line in the operation area corresponding to the first button area and the position of the operator, the third straight line is a straight line connecting one end in the direction of the center line in the operation area corresponding to the second button area and the position of the operator, and the fourth straight line is a straight line connecting the other end in the direction of the center line in the operation area corresponding to the second button area and the position of the operator.

6. The information processing device according to claim 1, wherein: As the at least one electronic device, there are a first electronic device and a second electronic device, When the position of the operator detected by the position detection unit is outside a front area corresponding to the front of the first electronic device and the second electronic device, the operation area adjustment unit expands the operation area of ​​at least one of the first electronic device and the second electronic device.

7. The information processing device according to claim 1, wherein: The operation area adjustment unit determines a size of the expanded operation area based on a positional relationship between the at least one electronic device and the operator.

8. The information processing device according to claim 7, wherein: The at least one electronic device is a display, The operation area adjustment unit determines the size of the expanded operation area based on the size of the operation area when the display is arranged in front of the operator in a state where the shortest distance between the display and the operator is equal.

9. The information processing device according to claim 1, wherein: The at least one electronic device is a display, In the case where there are multiple operation areas on the display, When the motion detection unit detects a motion of selecting one of the plurality of operation areas, the operation area adjustment unit determines a size of the expanded operation target area according to the predetermined motion.

10. The information processing device according to claim 9, wherein: When the sizes of the enlarged operation object areas corresponding to the multiple operation areas are equal, the operation area adjustment unit will enlarge the size of the enlarged operation object area corresponding to the selected operation area, and reduce the size of the enlarged operation object area corresponding to the other operation areas adjacent to the selected operation area.

11. The information processing device according to claim 9, wherein: When the sizes of the enlarged operation object areas corresponding to the multiple operation areas are different, the operation area adjustment unit will enlarge the size of the enlarged operation object area corresponding to the selected operation area, and reduce the size of the enlarged operation object area corresponding to the other operation areas adjacent to the selected operation area.

12. The information processing device according to claim 1, wherein: The at least one electronic device is a display, In the case where there are multiple operation areas on the display, When the motion detection unit detects a motion of selecting one of the plurality of operation areas, the operation area adjustment unit moves the expanded operation target area according to the predetermined motion.

13. The information processing device according to claim 12, wherein: When the motion detection unit detects a motion in a first direction, the operation region adjustment unit moves the expanded operation target region in the first direction, the first direction indicating left and right of the front surfaces of the plurality of operation regions.

14. The information processing device according to claim 9 or 12, wherein: When the action detection unit detects an action of selecting one of the multiple operation areas, the operation area adjustment unit expands the size of the operation object area corresponding to the selected operation area and reduces the size of other operation object areas adjacent to the selected operation object area.

15. The information processing device according to claim 14, wherein: When the motion detection unit detects a motion in a third direction away from the front of the operation object area corresponding to the selected operation area, the operation area adjustment unit further expands the size of the operation object area corresponding to the selected operation area and restores it to the size of the operation object area before the expansion after a certain period of time.

16. An information processing system comprising a camera unit and an information processing device, wherein: The camera unit captures a user who is an operator. The information processing device comprises: a position detection unit that detects the position of the operator based on the image captured by the camera unit; an action detection unit that detects a predetermined action of the operator in an operation area, wherein the operation area represents an area determined by linking the position of the operator detected by the position detection unit with an operation target area set on at least one electronic device; a processing execution unit that performs processing associated with the operation target area when the predetermined motion is detected by the motion detection unit; as well as The operation area adjustment unit is configured to expand the operation area when the position of the operator detected by the position detection unit is outside the front area of ​​the at least one electronic device.

17. An information processing method, executed by an information processing device, the information processing method comprising: A position detection step, detecting the position of the operator; an action detection step of detecting a predetermined action of the operator in an operation area, wherein the operation area represents an area determined by linking the position of the operator detected by the position detection step with an operation target area set on at least one electronic device; a processing execution step of performing processing associated with the operation target area when the predetermined motion is detected by the motion detection step; as well as The operation area adjustment step is to expand the operation area when the position of the operator detected by the position detection step is outside the front area of ​​the at least one electronic device.

18. A computer-readable storage medium storing an information processing program, wherein the information processing program causes a computer to execute: A position detection step, detecting the position of the operator; an action detection step of detecting a predetermined action of the operator in an operation area, wherein the operation area represents an area determined by linking the position of the operator detected by the position detection step with an operation target area set on at least one electronic device; a processing execution step of performing processing associated with the operation target area when the predetermined motion is detected by the motion detection step; as well as The operation area adjustment step is to expand the operation area when the position of the operator detected by the position detection step is outside the front area of ​​the at least one electronic device.

19. An information processing program product, comprising an information processing program, wherein the information processing program causes a computer to execute: A position detection step, detecting the position of the operator; an action detection step of detecting a predetermined action of the operator in an operation area, wherein the operation area represents an area determined by linking the position of the operator detected by the position detection step with an operation target area set on at least one electronic device; a processing execution step of performing processing associated with the operation target area when the predetermined motion is detected by the motion detection step; as well as The operation area adjustment step is to expand the operation area when the position of the operator detected by the position detection step is outside the front area of ​​the at least one electronic device.

Images