Information processing device, information processing system, and information processing method

Abstract

To set a virtual operation area which is easy for a user to operate.SOLUTION: An information processor acquires an image obtained by imaging a user, detects an arm part of the user on the basis of the image, sets a virtual operation area for receiving operation to a device on the basis of the position and the length of the detected arm part of the user, and operates the device on the basis of a positional relation between the fingertip of the user and the virtual operation area on the basis of the image.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, an information processing system, and an information processing method. [Background technology]

[0002] Conventionally, various devices detect the movement of a user's hand and control their operation in accordance with the movement of the user's hand (see, for example, Patent Document 1). For example, for an operation target such as a display, the content displayed on the display can be changed in accordance with the movement of the user's hand without the user directly touching the display. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2018-32055 Summary of the Invention [Problem to be solved by the invention]

[0004] In the above-described technology, it is conceivable to set a virtual operation area for specifying a user's selection operation, etc. It is desirable that such a virtual operation area be set so as to be easy for the user to operate.

[0005] The present disclosure provides an information processing device that can set a virtual operation area that is easy for a user to operate. [Means for solving the problem]

[0006] The information processing device according to the present disclosure includes an image acquisition unit that acquires an image of a user, a detection unit that detects the user's arm based on the image, a setting unit that sets a virtual operation area that accepts operations on a device based on the position and length of the user's arm detected by the detection unit, and an operation unit that operates the device based on the positional relationship between the user's fingertips and the virtual operation area based on the image. [Effects of the Invention]

[0007] According to the information processing device according to the present disclosure, it is possible to set a virtual operation area that is easy for the user to operate. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of an information processing device. [Figure 2A] FIG. 2A is a diagram showing an example of an operation instruction as viewed from above. [Figure 2B] FIG. 2B is a diagram showing an example of an operation instruction as viewed from the side. [Figure 3A] FIG. 3A is a diagram showing an example of a detection range. [Figure 3B] FIG. 3B is a table showing whether or not each area is detectable. [Figure 4A] FIG. 4A is a diagram illustrating an example of a false detection. [Figure 4B] FIG. 4B is a diagram showing another example of a false detection. [Figure 5] FIG. 5 is a block diagram illustrating a functional configuration of the information processing apparatus according to the first embodiment. [Figure 6A] FIG. 6A is a diagram showing the touch detection surface as viewed from above. [Figure 6B] FIG. 6B is a diagram showing the touch detection surface as viewed from the side. [Figure 7A] FIG. 7A is a diagram showing the touch detection surface as viewed from above. [Figure 7B] FIG. 7B is a diagram showing the touch detection surface as viewed from the side. [Figure 8A]FIG. 8A is a diagram showing the touch detection surface as viewed from above. [Figure 8B] FIG. 8B is a diagram showing the touch detection surface as viewed from the side. [Figure 9A] FIG. 9A is a diagram showing the touch detection surface as viewed from above. [Figure 9B] FIG. 9B is a diagram showing the touch detection surface as viewed from the side. [Figure 10A] FIG. 10A is a diagram showing the effective detection area as viewed from above. [Figure 10B] FIG. 10B is a diagram showing the effective detection area as viewed from the side. [Figure 11] FIG. 11 is a flowchart showing a processing procedure according to the second embodiment. [Figure 12] FIG. 12 is a diagram showing the effect of the control device according to the second embodiment. [Figure 13] FIG. 13 is a diagram showing an example of setting touch detection surfaces for each of a plurality of users. [Figure 14] FIG. 14 is a flowchart showing a processing procedure according to the third embodiment. [Figure 15] FIG. 15 is a diagram for explaining an example of image removal according to the fourth embodiment. [Figure 16A] FIG. 16A is a diagram for explaining an example of image removal according to the fourth embodiment. [Figure 16B] FIG. 16B is a diagram for explaining an example of image removal according to the fourth embodiment. [Figure 17] FIG. 17 is a flowchart showing a processing procedure according to the fourth embodiment. [Figure 18] FIG. 18 is a diagram showing the arrangement positions of sensors according to a modified example. [Figure 19] FIG. 19 is a diagram showing the arrangement positions of sensors according to a modified example. [Figure 20] FIG. 20 is a diagram showing the arrangement positions of sensors according to a modified example. [Figure 21] FIG. 21 is a diagram showing an example of a display according to a modified example. [Figure 22] FIG. 22 is a diagram showing an example of touch detection surfaces of a plurality of devices. [Figure 23] FIG. 23 is a diagram showing an example of touch detection surfaces of a plurality of devices. [Figure 24A] FIG. 24A is a diagram showing a touch detection surface based on the left arm. [Figure 24B] FIG. 24B is a diagram showing a touch detection surface based on the right arm. [Figure 25A] FIG. 25A is a diagram showing a detection effective area based on both arms. [Figure 25B] FIG. 25B is a diagram showing the detection effective area based on both arms. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, an embodiment of an information processing device according to the present disclosure will be described with reference to the drawings.

[0010] (First embodiment) Before describing the information processing device according to the first embodiment, a device that operates based on the movement of the user's arm will be described.

[0011] FIG. 1 shows an example of an information processing device that operates based on the movement of a user's arm. The information processing device has a sensor 500 and a display 600. The sensor 500 is, for example, a camera. The sensor 500 continuously captures images of a user UR. The user UR has an arm 71. Here, the arm 71 includes not only the upper arm and forearm but also the hand. The hand also includes fingertips 72. The information processing device controls the data displayed on the display 600 based on the positions of the fingertips 72 of the user UR.

[0012] The information processing device sets a touch detection surface AR100 that accepts operations on a device (for example, the display 600) and a detection effective area AR200 that is an area that detects the movement of the user UR's fingertip 72. In this example, the touch detection surface AR100 is set as a plane parallel to the display surface of the display 600.

[0013] When the fingertip 72 enters a range closer to the display 600 than the touch detection surface AR100, the information processing device detects that an operation instruction has been given to the information processing device, executes processing based on the operation instruction, and displays the execution result on the display 600.

[0014] Next, an example of an operation instruction given by a user UR to an information processing device will be described with reference to Figures 2A and 2B. Figure 2A is a diagram showing an example of an operation by the user UR as viewed from above, and Figure 2B is a diagram showing an example of an operation by the user UR as viewed from the side.

[0015] As shown in FIGS. 2A and 2B , when the fingertip 72 enters the range of the touch detection surface AR100 (in other words, when the fingertip 72 is positioned closer to the display 600 than the touch detection surface AR100), the information processing device determines that a predetermined location on the display 600 has been touched. The touch detection surface AR100 is an area that accepts operations on the display 600. Hereinafter, the determination by the information processing device that the fingertip 72 of the user UR has touched a predetermined location on the display 600 may be referred to as touch detection. Furthermore, when the user UR moves the fingertip 72 along the trajectory TRK in a direction away from the touch detection surface AR100, the information processing device determines that the state in which the predetermined location on the display 600 was touched has transitioned to a released state.

[0016] Next, using Figures 3A and 3B, the range in which the user UR can move their fingertip 72, and the relationship between the touch detection surface AR100 and the detection effective area AR200 are shown. Note that in this example, the display 600 is assumed to be mounted on a vehicle, and the user UR is assumed to be seated in the right seat. Therefore, the user UR is located to the right in the figure. Here, the detection effective area AR200 allows for finger detection of the user UR. Finger detection means detecting the position where the user UR's finger is located. Finger detection becomes possible when the user UR's finger is located in the detection effective area AR200.

[0017] As shown in FIG. 3A, the range in which the user UR can move the fingertip 72 is the combined range of the area AR300, the area AR400, and the area AR500.

[0018] The area AR300 is an area located closer to the display 600 than the touch detection surface AR100, within the range in which the user UR can move the fingertip 72. Therefore, as shown in FIG. 3B , the information processing device 100 can detect a touch and a finger in the area AR300.

[0019] The area AR400 is located in a range where the user UR can move the fingertip 72, farther from the display 600 than the touch detection surface AR100, but is included in the detection effective area AR200. Therefore, as shown in FIG. 3B , the area AR400 is an area where touch detection is not possible but finger detection is possible.

[0020] The area AR500 is located in a range where the user UR can move the fingertip 72, but is farther from the display 600 than the touch detection surface AR100, and is not included in the detection effective area AR200. Therefore, the area AR500 is an area where neither touch detection nor finger detection is possible, as shown in FIG.

[0021] The example shown in FIG. 3A has a problem in that the area where touch detection is possible within the range where the user UR can move the fingertip 72, that is, the range where the user UR can operate the display 600, is small.

[0022] One possible way to solve this problem is to move the touch detection surface AR100 closer to the position of the user UR. Hereinafter, the range closer to the display 600 than the touch detection surface AR100 may be referred to as the range where touch detection is possible.

[0023] 4A and 4B show examples in which the touch detection surface AR100 is brought closer to the position of the user UR. When the touch detection surface AR100 is brought closer to the position of the user UR, as shown in FIG. 4A, the fingertip 72 of the user UR can easily reach the touch detection range. On the other hand, if a part of the user UR other than the fingertip 72, such as the user UR's knee, enters the touch detection range, the information processing device may erroneously detect the movement of a part of the user UR other than the fingertip 72 as a touch operation. Furthermore, as shown in FIG. 4B, the information processing device may erroneously detect the movement of the user UR's fingertip 72 as a touch operation even when the user UR has no intention of operating the display 600, such as when the user UR is operating a smartphone SP.

[0024] As described above, simply bringing the touch detection surface AR100 closer to the user UR may result in erroneous detection. Therefore, the information processing device according to the first embodiment sets a virtual operation area that is easy for the user UR to operate.

[0025] 5 is a diagram showing a schematic configuration of an information system including a control device 1 which is an information processing device according to the first embodiment. The information processing device according to the first embodiment includes a sensor 50, the control device 1, and a display 60.

[0026] The sensor 50 is, for example, a camera device. As an example, the sensor 50 is a visible light camera. The sensor 50 outputs an image of the user UR to the control device 1. Note that the sensor 50 is not limited to a visible light camera, and may be, for example, a CCD camera or a CMOS camera. The sensor 50 is an example of an imaging unit. The sensor 50 continuously performs imaging processing and outputs the image to the control device 1.

[0027] The display 60 is a display unit that displays various data. The display 60 is an example of a device.

[0028] The control device 1 processes the data displayed on the display 60 in accordance with the movement of the fingertip 72 of the user UR.

[0029] The control device 1 includes a control unit 10 and a storage unit 30. The control unit 10 is configured as, for example, a CPU (Central Processing Unit), and comprehensively controls the operations of each unit of the control device 1. The control unit 10 includes an image acquisition unit 11, an image processing unit 12, a landmark extraction unit 13, a detection effective area setting unit 14, a touch detection surface setting unit 15, a 3D coordinate calculation unit 16, a 2D coordinate conversion unit 17, a determination unit 18, a function execution unit 19, and a display output unit 20.

[0030] The memory unit 30 stores various types of information. The memory unit 30 is realized by hardware for storing information (in other words, data), such as a memory or storage. Specifically, the memory unit 30 stores reference coordinate information 31, detection effective area information 32, and touch detection surface information 33. The reference coordinate information 31 includes 3D coordinates of the installation position of the sensor 50, information on the mounting angle of the sensor 50, and 3D coordinates of the position of the display 60. The detection effective area information 32 is coordinate information indicating the detection effective area. The detection effective area is an area for detecting changes in the position of the fingertip 72 of the user UR. The touch detection surface information is coordinate information indicating the touch detection surface. The touch detection surface is an area that accepts operations on the display 60.

[0031] The image acquisition unit 11 acquires an image from the sensor 50. The image acquisition unit 11 is an example of an image acquisition unit. The image processing unit 12 processes the image acquisition unit 11 in an area where the user UR is not captured, as a removal target. The landmark extraction unit 13 detects landmarks of the user UR from an image in which the user UR is captured, and extracts the landmarks. Here, the landmarks are the eyes of the user UR, the three-dimensional coordinates of the shoulders, and the user's arms. The landmark extraction unit 13 may also extract the length of the user's arms. The landmark extraction unit 13 may also extract information indicating the field of view of the user UR based on the positions of the user's eyes.

[0032] The detection effective area setting unit 14 sets the detection effective area and registers detection effective area information 32, which is coordinates indicating the detection effective area, in the storage unit 30. A method for setting the detection effective area by the detection effective area setting unit 14 will be described later. The detection effective area setting unit 14 is an example of a setting unit. The detection effective area is an example of a detection area.

[0033] The touch detection surface setting unit 15 sets a touch detection surface and registers touch detection surface information 33, which is coordinates indicating touch detection surface coordinates, in the storage unit 30. The touch detection surface setting unit 15 is an example of a setting unit. A method by which the touch detection surface setting unit 15 sets the touch detection surface will be described later. The touch detection surface is an example of a virtual operation area.

[0034] The 3D coordinate calculation unit 16 calculates the three-dimensional coordinates of the fingertip of the user UR. The 2D coordinate conversion unit 17 converts the three-dimensional coordinates of the fingertip of the user UR into two-dimensional coordinates on the display 60.

[0035] The determination unit 18 determines whether or not a touch has occurred and the type of processing based on the touch detection surface and the three-dimensional coordinates of the fingertip. The type of processing may be, for example, touch, flick, drag, or the like.

[0036] The function executing unit 19 executes the processing content based on the result of the determination made by the determining unit 18. The display output unit 20 displays the result of the executed processing content on the display 60. The determining unit 18, the function executing unit 19, and the display output unit 20 are examples of an operation unit.

[0037] (How to set the touch sensitive surface) Next, a method for the touch detection surface setting unit 15 to set the touch detection surface will be described. First, an example of setting the touch detection surface will be described using Figs. 6A and 6B. As a premise, the image acquisition unit 11 acquires an image including the user UR. Then, the landmark extraction unit 13 extracts the shoulders, arms, and eyes of the user UR from the image including the user UR. Note that the landmark extraction unit 13 may also extract the position of the elbow of the arm of the user UR.

[0038] Fig. 6A is a diagram showing the touch detection surface as viewed from above, and Fig. 6B is a diagram showing the touch detection surface as viewed from the side.

[0039] As shown in FIGS. 6A and 6B, the touch decision surface setting unit 15 sets the touch decision surface AR1 based on the following spherical equation (1) with the shoulder as the center position S (xs, ys, zs) and a distance L based on the arm length. (X-xs) 2 +(Y-ys) 2 +(Z-zs) 2 =L 2 ···(1)

[0040] The distance L is a distance based on the length of the arm 71, and is, for example, a length equal to or shorter than the entire length of the arm 71. The distance L is, for example, the entire length of the arm 71. In this way, the touch decision surface setting unit 15 sets the touch decision surface AR1 based on the position of the shoulder of the user UR and the length of the arm 71 of the user UR. Furthermore, the touch decision surface setting unit 15 sets the touch decision surface based on a spherical range by setting the touch decision surface AR1 based on the above formula (1). The touch decision surface AR1 may be the entire spherical surface represented by formula (1), or may be a part of the spherical surface as shown in FIGS. 6A and 6B. For example, a part of the spherical surface represented by formula (1) may be set as the touch decision surface AR1 based on the movable range of the fingertip 72 of the user UR.

[0041] The touch decision surface setting unit 15 may set the touch decision surface AR1 further based on the visual field range of the user UR. Here, a method for setting the touch decision surface AR1 by the touch decision surface setting unit 15 will be described with reference to FIGS. 7A and 7B.

[0042] FIG. 7A is a diagram showing the touch detection surface as viewed from above, and FIG. 7B is a diagram showing the touch detection surface as viewed from a side. Here, the field of view range is defined as the area from the user's eye position to the edge of the display 60. Specifically, when viewed from above, the field of view range is defined as an area surrounded by a plane L11 that includes the user's eye position and the right edge of the display 60, a plane L12 that includes the user's eye position and the left edge of the display 60, and the display 60. In FIG. 7A, planes L11 and L12 are shown as straight lines. When viewed from a side, the field of view range is defined as an area surrounded by a plane L13 that includes the user's eye position and the top edge of the display 60, a plane L14 that includes the user's eye position and the bottom edge of the display 60, and the display 60. In FIG. 7B, planes L13 and L14 are shown as straight lines. The touch detection surface setting unit 15 sets the touch detection surface AR1 based on the above-described arm 71 and the field of view range. Specifically, the touch decision surface setting unit 15 sets only a portion of the spherical surface expressed by Equation (1) that is included in the field of view as the touch decision surface AR1. Furthermore, as shown in Fig. 7A, position P3 on the touch decision surface AR1 corresponds to position P1 on the display 60, and position P4 on the touch decision surface AR1 corresponds to position P2 on the display 60. Furthermore, as shown in Fig. 7B, position P7 on the touch decision surface AR1 corresponds to position P5 on the display 60, and position P8 on the touch decision surface AR1 corresponds to position P6 on the display 60.

[0043] Another example of setting the touch detection surface AR1 will be described with reference to FIGS. 8A and 8B. FIG. 8A is a diagram showing the touch detection surface as viewed from above, and FIG. 8B is a diagram showing the touch detection surface as viewed from the side. As shown in FIGS. 8A and 8B, the touch detection surface setting unit 15 may set the touch detection surface AR1 with the elbow of the arm as its center. In this case, the touch detection surface setting unit 15 sets the touch detection surface AR1 based on the length from the elbow to the hand of the arm 71. The length from the elbow to the hand refers to the length from the elbow to the fingertips.

[0044] Another example of setting the touch detection surface AR1 will be described with reference to Fig. 9A and Fig. 9B. Fig. 9A is a diagram showing the touch detection surface as seen from above, and Fig. 9B is a diagram showing the touch detection surface as seen from a side. As shown in Fig. 9A and Fig. 9B, the touch detection surface setting unit 15 may set the touch detection surface AR1 around the shoulder and elbow of the arm.

[0045] After the touch detection surface setting unit 15 sets the touch detection surface AR1, if the judgment unit 18 judges that the fingertip 72 of the user UR is in a state of passing through the touch detection surface AR1 (in other words, in a touch detection state) based on the position of the finger part of the user UR's hand obtained from the image and the positional relationship with the touch detection surface AR1, the function execution unit 19 executes the function, and the display output unit 20 displays the execution result on the display 60.

[0046] In the control device 1 according to the first embodiment, the image acquisition unit 11 acquires an image of a user, the landmark extraction unit 13 detects the arm of the user UR, and the touch detection surface setting unit 15 sets the touch detection surface AR1 based on the length of the arm, with the shoulder as the center position. When the determination unit 18 determines that a touch detection state is established based on the positional relationship between the position of the fingertip 72 of the user UR acquired from the image and the touch detection surface AR1, the function execution unit 19 executes a function, and the display output unit 20 displays the execution result on the display 60.

[0047] In this way, the control device 1 sets the touch decision surface AR1 based on the length of the arm of the user UR, and therefore can set the touch decision surface AR1 according to the user's movement, i.e., can set a virtual operation area that is easy for the user to operate.

[0048] Furthermore, the control device 1 sets the touch detection surface AR1 based on the length of the arm 71, with the shoulder as the center position, in other words, sets the touch detection surface AR1 based on the movable range of the arm 71. In this way, by the control device 1 setting the touch detection surface AR1 based on the movable range of the arm 71, unintentional touch / release operations are less likely to occur compared to when the touch detection surface AR600 is set parallel to the display 600 as shown in FIG.

[0049] The touch detection surface setting unit 15 may set the touch detection surface AR1 based on the length from the elbow to the hand of the user UR, or may set the touch detection surface AR1 based on the entire length of the arm 71 of the user UR. In this case, the touch detection surface AR1 can be set according to the user's movements. The touch detection surface setting unit 15 may also set the touch detection surface AR1 based on the field of view of the user UR. In this case, the control device 1 further restricts the field of view of the user UR, and therefore can set the touch detection surface AR1 that is easier for the user to operate.

[0050] (Second embodiment) Next, a description will be given of a control device 1 according to a second embodiment. In the control device 1 according to the second embodiment, an effective detection area is set that is easy for the user to operate.

[0051] (Detection effective area setting method) A method for the detection effective area setting unit 14 to set the detection effective area will be described. First, an example of setting the detection effective area will be described using Figs. 10A and 10B. As a premise, the image acquisition unit 11 acquires an image including the user UR. Then, the landmark extraction unit 13 extracts the shoulders, arms, and eyes of the user UR from the image including the user UR. Note that the landmark extraction unit 13 may also extract the position of the elbow of the arm of the user UR.

[0052] FIG. 10A is a diagram showing the positional relationship between the user UR and the display 60 as viewed from above. FIG. 10B is a diagram showing the positional relationship between the user UR and the display 60 as viewed from the side.

[0053] As shown in FIGS. 10A and 10B, the detection effective region setting unit 14 sets the detection effective region AR2 based on the following spherical equations (2) and (3) consisting of the distance L1 based on the length of the arm and the distance L2 based on the length of the arm, with the shoulder portion as the center position S(xs, ys, zs). (X - xs) 2 +(Y - ys) 2 +(Z - zs) 2 ≧L1 2 ···(2) (X - xs) 2 +(Y - ys) 2 +(Z - zs) 2 ≦L2 2 ···(3)

[0054] The detection effective region setting unit 14 sets the region satisfying the above equations (2) and (3) as the detection effective region AR2. Note that L1 < L2. Also, when showing the relationship between the distance L described in the first embodiment, the distance L1, and the distance L2, L1 ≦ L ≦ L2. The distance L1 is, for example, a length equal to or less than the entire length of the arm 71. The distance L1 is, for example, the length from the shoulder to the elbow in the arm. The distance L2 is, for example, a length equal to or less than the entire length of the arm 71. The distance L2 is, for example, the entire length of the arm 71.

[0055] In this way, the detection effective region setting unit 14 sets the detection effective region AR2 based on the length of the arm 71 of the user UR. The detection effective region AR2 may be the entire region shown by equations (2) and (3), or may be a part of the region shown by equations (2) and (3) as shown in FIGS. 10A and 10B. For example, based on the movable range of the fingertips 72 of the user UR, a part of the region shown by equations (2) and (3) may be set as the detection effective region AR2.

[0056] Furthermore, the detection effective area setting unit 14 may set the detection effective area AR2 with the elbow as the center position, as in the example of the first embodiment, or may set the detection effective area AR2 with both the shoulder and elbow as fulcrums.

[0057] Furthermore, the detection effective area setting unit 14 may set the detection effective area AR2 further based on the visual field range of the user UR, similar to the example of the first embodiment.

[0058] Next, the processing procedure of the control device 1 according to the second embodiment will be described with reference to Fig. 11. Fig. 11 is a flowchart showing the processing procedure of the control device 1 according to the second embodiment.

[0059] First, the landmark extraction unit 13 determines whether or not a user is detected from the image acquired by the image acquisition unit 11 (step S1). If the landmark extraction unit 13 does not detect a user from the image (step S1: No), the process proceeds to step S1 again. If the landmark extraction unit 13 extracts a user from the image (step S1: Yes), the landmark extraction unit 13 detects landmarks such as eyes, shoulders, and elbows, and calculates the 3D coordinates of each landmark (step S2).

[0060] Furthermore, the landmark extraction unit 13 calculates the length of the user's arm based on the positions of the landmarks (step S3).

[0061] The detection effective area setting unit 14 sets a detection effective area based on the landmark positions and the length of the user's arm (step S4). The touch detection surface setting unit 15 sets a touch detection surface based on the landmark positions and the length of the user's arm (step S5). The 3D coordinate calculation unit 16 determines whether or not the fingertip 72 of the user UR is detected on the image (step S6). If the 3D coordinate calculation unit 16 does not detect the fingertip of the user UR from the image in step S6 (step S6: No), the process proceeds to step S1. If the 3D coordinate calculation unit 16 detects the fingertip of the user UR from the image (step S6: Yes), the 3D coordinate calculation unit 16 calculates the 3D coordinate of the fingertip (step S7). The determination unit 18 determines whether or not the 3D coordinate of the fingertip is within the detection effective area AR2 (step S8). If the determination unit 18 determines that the fingertip is not within the detection effective area AR2 (step S8: No), the process proceeds to step S7.

[0062] Furthermore, if the determination unit 18 determines in step S8 that the 3D coordinates of the fingertip are within the detection effective area AR2 (step S8: Yes), it determines whether the fingertip has passed through the touch determination surface (step S9). If the determination unit 18 determines in step S9 that the fingertip has passed through the touch determination surface AR1 (step S9: Yes), it determines an operation corresponding to the movement of the fingertip. An example of the operation corresponding to the movement of the fingertip is a touch operation. Specific examples of the touch operation include a click operation, a drag operation, and a flick operation. The function execution unit 19 executes the determined operation (step S10). If the determination unit 18 determines in step S9 that the fingertip has not passed through the touch determination surface AR1 (step S9: No), it determines an operation corresponding to the movement of the fingertip. An example of the operation corresponding to the movement of the fingertip is a release operation or a pointer movement operation. The function execution unit 19 executes the determined operation (step S11).

[0063] In the control device 1 according to the second embodiment, the detection effective area setting unit 14 sets the detection effective area AR2 based on the length of the arm, with the shoulder as the center position. The determination unit 18 determines the operation corresponding to the movement of the fingertip based on the positional relationship between the position of the fingertip 72 of the user UR acquired from the image and the detection effective area AR2, the function execution unit 19 executes the function corresponding to the operation, and the display output unit 20 displays the execution result on the display 60.

[0064] In this way, the control device 1 sets the detection effective area AR2 based on the length of the arm of the user UR, and therefore can set the detection effective area AR2 according to the user's movements, i.e., it can set a detection effective area that is easy for the user to operate.

[0065] 12, the control device 1 sets the detection effective area AR2 based on the length of the arm 71 of the user UR1. As a result, even if the user UR2 makes a movement around the display 60, the user UR2 is not included in the detection effective area AR2. This makes it possible to prevent the control device 1 from erroneously detecting the movement of another user.

[0066] (Third embodiment) Next, a description will be given of a control device 1 according to a third embodiment. The control device 1 according to the third embodiment sets a detection effective area and a touch detection surface for each of a plurality of users, and performs touch detection based on the priority of each user.

[0067] The image acquisition unit 11 acquires an image of multiple users. The landmark extraction unit 13 extracts the landmarks of each of the multiple users. The detection effective area setting unit 14 sets the detection effective area AR2 for each user based on the arm length of each user. Furthermore, the touch detection surface setting unit 15 sets the touch detection surface AR1 for each user based on the arm length of each user.

[0068] When multiple users are imaged, the control device 1 sets a user with a high priority. The determination unit 18 operates the display 60 based on the positional relationship between the fingertip 72 of the user with the high priority and the touch determination surface AR1.

[0069] 13 shows an example of the positional relationship between multiple users and the display 60. As shown in FIG. 13, when a user UR1 and a user UR2 are near the display 60, the control device 1 sets a touch decision surface AR1a and a detection effective area AR2a for the user UR1. The control device 1 also sets a touch decision surface AR1b and a detection effective area AR2b for the user UR2.

[0070] Then, the control device 1 increases the priority of the touch decision surface AR1a and the detection effective area AR2a of the user UR1 through a predetermined operation. As a result, the control device 1 sets the touch decision surface AR1a and the detection effective area AR2a to a valid state. Accordingly, the control device 1 sets the touch decision surface AR1b and the detection effective area AR2b to an invalid state.

[0071] The control device 1 then changes the display state of the display 60 based on the position of the fingertip 72 of the user UR1 and the positional relationship between the touch determination surface AR1a and the detection effective area AR2a.

[0072] Next, the processing procedure of the control device 1 according to the third embodiment will be described with reference to Fig. 14. Fig. 14 is a flowchart showing the processing procedure of the control device 1 according to the second embodiment.

[0073] First, the landmark extraction unit 13 determines whether or not a user is detected from the image acquired by the image acquisition unit 11 (step S21). If the landmark extraction unit 13 does not detect a user from the image (step S21: No), the process proceeds to step S21 again. If the landmark extraction unit 13 extracts a user from the image (step S21: Yes), the landmark extraction unit 13 detects landmarks such as eyes, shoulders, and elbows, and calculates the 3D coordinates of each landmark (step S22).

[0074] Furthermore, the landmark extraction unit 13 calculates the length of the user's arm based on the positions of the landmarks (step S23).

[0075] The detection effective area setting unit 14 sets a detection effective area AR2 based on the landmark positions and the user's arm length (step S24). The touch decision surface setting unit 15 sets a touch decision surface AR1 based on the landmark positions and the user's arm length (step S25). If there is any user among the users detected from the image for whom the detection effective area AR2 and touch decision surface AR1 have not been set (step S26: Yes), the process proceeds to step S22. If the touch decision surfaces AR1 and detection effective areas AR2 have been set for all users, that is, if there is no user for whom the settings have not been set (step S26: No), the control device 1 sets priorities for the multiple touch decision surfaces AR1 and detection effective areas AR2 (step S27).

[0076] The control device 1 sets priorities of multiple touch decision surfaces AR1 and detection effective areas AR2, for example, according to content displayed on the display 60. The control device 1 may set the priorities after step S29, which will be described later. For example, the control device 1 may set a high priority to the touch decision surface AR1 and detection effective area AR2 corresponding to the user who is located near the fingertip that started moving first, among the fingertips whose 3D coordinates have been calculated in step S29.

[0077] The 3D coordinate calculation unit 16 determines whether any user's fingertip 72 has been detected on the image (step S28). If the 3D coordinate calculation unit 16 has not detected any user's fingertip 72 from the image in step S28 (step S28: No), the process proceeds to step S21. If the 3D coordinate calculation unit 16 has detected any user's fingertip from the image (step S28: Yes), the 3D coordinate calculation unit 16 calculates the 3D coordinate of the fingertip (step S29). The determination unit 18 determines whether the 3D coordinate of the fingertip is within the detection effective area AR2 with the highest priority (step S30). If the determination unit 18 determines that the fingertip is not within the detection effective area AR2 (step S30: No), the process proceeds to step S29.

[0078] Furthermore, if the determination unit 18 determines in step S30 that the 3D coordinates of the fingertip are within the detection effective area AR2 with the highest priority (step S30: Yes), it determines whether the fingertip has passed over the touch determination surface AR1 with the highest priority (step S31). If the determination unit 18 determines in step S31 that the fingertip has passed over the touch determination surface AR1 with the highest priority (step S31: Yes), it determines an operation corresponding to the movement of the fingertip. An example of the operation corresponding to the movement of the fingertip is a touch operation. Specific examples of the touch operation include a click operation, a drag operation, and a flick operation. The function execution unit 19 executes the determined touch operation (step S32). If the determination unit 18 determines in step S31 that the fingertip has not passed over the touch determination surface AR1 (step S31: No), it determines an operation corresponding to the movement of the fingertip. An example of the operation corresponding to the movement of the fingertip is a release operation or a pointer movement operation. The function execution unit 19 executes the determined operation (step S33).

[0079] In this way, when a plurality of detection effective areas AR2 and touch determination surfaces AR1 are set, the control device 1 according to the third embodiment can operate the display 60 based on the detection effective area AR2 and touch determination surface AR1 with the highest priority. This allows the display 60 to be operated based on the user's action that is more appropriate for issuing an operation instruction.

[0080] (Fourth embodiment) Next, a description will be given of a control device 1 according to a fourth embodiment. The control device 1 according to the fourth embodiment removes unnecessary areas in which the user is not captured from an image, and operates the display 60 based on the positional relationship between the position of the finger detected from the image from which the unnecessary areas have been removed, the detection effective area AR2, and the touch determination surface AR1.

[0081] FIG. 15 shows an example of the positional relationship between the user and the display 60. In the example shown in FIG. 15, the range that the sensor 50 can capture includes areas AR11 and AR12. Area AR11 includes the effective detection area AR2 and the touch determination surface AR1. On the other hand, area AR12 does not include the effective detection area AR2 or the touch determination surface AR1. Therefore, even if a finger is positioned in area AR12, the control device 1 does not need to perform coordinate calculation for the finger. Therefore, the image processing unit 12 of the control device 1 performs a removal process to remove a portion of the image that corresponds to area AR12. The image processing unit 12 performs the removal process by, for example, trimming the portion that corresponds to area AR12. By performing the removal process, it is possible to reduce the coordinate calculation area, which is the area in the image where coordinate calculation is performed.

[0082] Alternatively, the image processing unit 12 may perform the removal process by removing the part of the image other than the part where the user was captured. An example of the removal process will now be described with reference to Figures 16A and 16B.

[0083] The image processing unit 12 performs a removal process from the image shown in Fig. 16A to leave only an area AR111 where the user was captured and remove an area AR112 where the user was not captured. As a result of the removal process, an image is obtained in which only the area AR111 where the user was captured remains, as shown in Fig. 16B. As a result, the coordinate calculation area is limited to the area AR111.

[0084] Next, the processing procedure of the control device 1 according to the fourth embodiment will be described with reference to Fig. 17. Fig. 17 is a flowchart showing the processing procedure of the control device 1 according to the fourth embodiment.

[0085] First, the landmark extraction unit 13 determines whether or not a user is detected from the image acquired by the image acquisition unit 11 (step S41). If the landmark extraction unit 13 does not detect a user from the image (step S41: No), the process proceeds to step S1 again. If the landmark extraction unit 13 extracts a user from the image (step S41: Yes), the landmark extraction unit 13 detects landmarks such as eyes, shoulders, and elbows, and calculates the 3D coordinates of each landmark (step S42).

[0086] Furthermore, the landmark extraction unit 13 calculates the length of the user's arm based on the positions of the landmarks (step S43).

[0087] The detection effective area setting unit 14 sets the detection effective area based on the landmark positions and the length of the user's arm (step S44), and the touch detection surface setting unit 15 sets the touch detection surface based on the landmark positions and the length of the user's arm (step S45).

[0088] The image processing unit 12 reduces the coordinate calculation area of ​​the image through removal processing (step S46). The 3D coordinate calculation unit 16 determines whether or not the fingertip 72 of the user UR is detected within the coordinate calculation area on the image (step S47). In step S47, if the 3D coordinate calculation unit 16 does not detect the fingertip of the user UR within the coordinate calculation area on the image (step S47: No), the process proceeds to step S41. If the 3D coordinate calculation unit 16 detects the fingertip of the user UR within the coordinate calculation area on the image (step S47: Yes), the 3D coordinate calculation unit 16 calculates the 3D coordinate of the fingertip (step S48). The determination unit 18 determines whether or not the 3D coordinate of the fingertip is within the detection effective area AR2 (step S49). If the determination unit 18 determines that the fingertip is not within the detection effective area AR2 (step S49: No), the process proceeds to step S48.

[0089] Furthermore, if the determination unit 18 determines in step S49 that the 3D coordinates of the fingertip are within the detection effective area AR2 (step S49: Yes), it determines whether the fingertip has passed through the touch determination surface (step S50). If the determination unit 18 determines in step S50 that the fingertip has passed through the touch determination surface AR1 (step S50: Yes), it determines an operation corresponding to the movement of the fingertip. An example of the operation corresponding to the movement of the fingertip is a touch operation. Specific examples of the touch operation include a click operation, a drag operation, and a flick operation. The function execution unit 19 executes the determined operation (step S51). If the determination unit 18 determines in step S50 that the fingertip has not passed through the touch determination surface AR1 (step S50: No), it determines an operation corresponding to the movement of the fingertip. An example of the operation corresponding to the movement of the fingertip is a release operation or a pointer movement operation. The function execution unit 19 executes the determined operation (step S52).

[0090] In this way, the control device 1 according to the fourth embodiment removes positions in the image that are far from the positions of the detection effective area AR2 and the touch determination surface AR1 from the coordinate calculation targets, thereby reducing the processing load on the control device 1.

[0091] (Variation) Although not specifically mentioned in the above embodiment, the position of the sensor 50 may be appropriately arranged based on the positional relationship between the user UR and the display 60. For example, as shown in Fig. 18, the sensor 50 may be arranged in front of the display 60. In this case, the effective detection area AR2 and the touch determination surface AR1 can be more reliably included in the sensing area AR3, which is the range captured by the sensor 50.

[0092] 19, the sensor 50 may be placed facing the direction of the user UR. For example, if the user UR is located on the right side of the display 60, the sensor 50 may be placed facing right. In this case, the control device 1 can narrow the sensing area AR3 to the periphery of the user UR, thereby reducing the possibility that the control device 1 will erroneously detect the movement of another user.

[0093] Furthermore, as shown in FIG. 20, the sensor 50 may be disposed in front of the display 60 and facing the direction of the user UR.

[0094] The shape of the display 60 is also arbitrary. For example, as shown in FIG.

[0095] The control device 1 may also set detection effective areas AR2 and touch determination surfaces AR1 corresponding to multiple devices. For example, as shown in Fig. 22, the control device 1 may set detection effective areas AR2 and touch determination surfaces AR1 corresponding to all of devices 61a, 61b, and 61c. In this case, the control device 1 may determine a field of view range for each device and control operation based on the positional relationship between the position of the user UR's finger and the detection effective area AR2 and touch determination surface AR1, as well as the field of view range of each device.

[0096] Furthermore, the control device 1 may also take into consideration the field of view of each device 61 and set the detection effective area AR2 and touch decision surface AR1 corresponding to each device 61. For example, as shown in Fig. 23, the control device 1 takes into consideration the field of view of each device 61 and sets the detection effective area AR2a to AR2c and the touch decision surface AR1a to AR1c. In this way, the control device 1 sets the touch decision surfaces AR1a to AR1c so that they do not overlap with each other.

[0097] In the above description, the control device 1 sets the detection effective area AR2 and the touch detection surface AR1 based on the position and length of the right arm 71. However, the detection effective area AR2 and the touch detection surface AR1 may also be set based on the position and length of the left arm 71. Furthermore, as shown in Figures 24A and 24B, the detection effective area AR2 and the touch detection surface AR1 for the left arm 71 (Figure 24A) may be set based on the position and length of the left arm 71, and the detection effective area AR2 and the touch detection surface AR1 for the right arm 71 (Figure 24B) may be set based on the position and length of the right arm 71.

[0098] In this case, the determination unit 18 determines whether the finger detected from the image belongs to the right arm or the left arm, and then determines the positional relationship between the detected finger and the corresponding detection effective area AR2 and touch determination surface AR1.

[0099] The detection effective area may be the sum of the detection effective area AR2a corresponding to the left arm 71 and the detection effective area AR2b corresponding to the right arm 71, as shown in Fig. 25A. Alternatively, as shown in Fig. 25B, a detection effective area AR2c may be set that includes the detection effective area AR2a and the detection effective area AR2b shown in Fig. 25A.

[0100] In this case, the determination unit 18 determines the positional relationship between the finger position and the detection effective area AR2c regardless of whether the finger detected from the image belongs to the right arm or the left arm. In addition, when determining the positional relationship between the finger and the touch determination surface AR1, the determination unit 18 distinguishes whether the finger belongs to the left arm or the right arm, and determines the positional relationship between the finger and the corresponding touch determination surface AR1.

[0101] The following is disclosed regarding the above embodiment. an image acquisition unit that acquires an image of a user; a detection unit that detects an arm portion of a user based on the image; a setting unit that sets a detection area, which is an area in which a change in the position of the user's fingertip is detected, based on the position of the user's arm part and the length of the user's arm part detected by the detection unit; and an operation unit that operates the device based on the positional relationship between the user's fingertip and the detection area, based on the image; An information processing device comprising:

[0102] Although the embodiments of the present disclosure have been described above, the above-described embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These novel embodiments and modifications thereof are included within the scope and spirit of the invention, and are also included in the scope of the invention and its equivalents as set forth in the claims. Furthermore, components from different embodiments and modifications may be combined as appropriate.

[0103] Furthermore, the notation "... section" in the above-described embodiments may be replaced with other notations such as "... circuitry," "... assembly," "... device," "... unit," or "... module."

[0104] In each of the above embodiments, the present disclosure has been described as an example configured using hardware, but the present disclosure can also be realized by software in cooperation with hardware.

[0105] Furthermore, each functional block used in the description of each of the above embodiments is typically realized as an LSI, which is an integrated circuit. The integrated circuit controls each functional block used in the description of the above embodiments and may have input and output terminals. These may be individually integrated into single chips, or some or all of them may be integrated into a single chip. While the term LSI is used here, it may also be called an IC, system LSI, super LSI, or ultra LSI depending on the level of integration.

[0106] Furthermore, the method of integration is not limited to LSI, but may be realized using dedicated circuits or general-purpose processors and memories. FPGAs (Field Programmable Gate Arrays), which can be programmed after LSI manufacturing, and reconfigurable processors, which allow the connections or settings of circuit cells within LSIs to be reconfigured, may also be used.

[0107] Furthermore, if an integrated circuit technology that can replace LSI emerges due to advances in semiconductor technology or other derivative technologies, it is natural that such technology may be used to integrate functional blocks. The application of biotechnology, etc. is also a possibility.

[0108] Furthermore, the effects of the embodiments described in this specification are merely examples and are not limiting, and other effects may also be obtained. [Explanation of symbols]

[0109] 1. Control device 10 Control Unit 11 Image acquisition unit 12 Image Processing Department 13 Landmark extraction unit 14. Detection effective area setting section 15 Touch detection surface setting section 16 3D coordinate calculation section 17 2D coordinate conversion section 18 Judgment section 19 Function Execution Department 20 Display output section 50 sensors 60 displays

Claims

1. an image acquisition unit that acquires an image of a user; a detection unit that detects an arm portion of a user based on the image; a setting unit that sets a virtual operation area in which operations on a device are accepted based on the position of the user's arm portion and the length of the user's arm portion detected by the detection unit; and an operation unit that operates a device based on a positional relationship between the user's fingertip and the virtual operation area, based on the image; Equipped with the setting unit sets a detection area that is an area for detecting a change in the position of the user's fingertip, based on the position of the user's arm part and the length of the user's arm part detected by the detection unit; and the operation unit operates a device based on a positional relationship between the fingertip of the user and the detection area based on the image; The setting unit sets the detection area based on a first range, which is a spherical range based on the length of the user's arm, and a second range, which is a spherical range based on a length equal to or shorter than the length of the user's arm.

2. An information processing system including an imaging unit, an information processing device, and a display unit, the imaging unit captures an image including a user; The information processing system includes: an image acquisition unit that acquires the image; a detection unit that detects an arm portion of a user based on the image; a setting unit that sets a virtual operation area that accepts operations on a display unit based on the position of the user's arm portion and the length of the user's arm portion detected by the detection unit; an operation unit that operates a display unit based on a positional relationship between the user's fingertip and the virtual operation area, based on the image; Equipped with the setting unit sets a detection area that is an area for detecting a change in the position of the user's fingertip, based on the position of the user's arm part and the length of the user's arm part detected by the detection unit; and the operation unit operates a device based on a positional relationship between the fingertip of the user and the detection area based on the image; An information processing system in which the setting unit sets the detection area based on a first range, which is a spherical range based on the length of the user's arm, and a second range, which is a spherical range based on a length equal to or shorter than the length of the user's arm.

3. An information processing method executed by an information processing device, an image acquisition step of acquiring an image of a user; a detection step of detecting an arm portion of a user based on the image; a setting step of setting a virtual operation area in which operations on a device are accepted, based on the position of the user's arm portion and the length of the user's arm portion detected in the detection step; an operation step of operating a device based on a positional relationship between the user's fingertip and the virtual operation area, based on the image; Including, the setting step sets a detection area that is an area for detecting a change in position of a fingertip of the user based on the detected position of the arm part of the user and the length of the arm part of the user; the operating step operates a device based on a positional relationship between the user's fingertip and the detection area based on the image; An information processing method in which the setting step sets the detection area based on a first range, which is a spherical range based on the length of the user's arm, and a second range, which is a spherical range based on a length equal to or shorter than the length of the user's arm.

Images