Detection method, detection device, and program

Abstract

To provide a detection method, a detection device and a program capable of more appropriately detecting a detection object.SOLUTION: A detection method performed by a computer includes: acquiring depth information related to depth of a reference surface of a component and a target object; deriving a distance in a substantially normal direction with respect to the reference surface, from the reference surface to a representative point of the object, on the basis of the depth information; and detecting the object with the derived distance satisfying a predetermined distance condition, as a detection object. Thereby, the detection method can more appropriately detect the detection object.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present invention relates to a detection method, a detection device, and a program. [Background technology]

[0002] Conventionally, there is a technology for detecting a gesture of an operator and controlling the operation of a device in response to the detected gesture. In this technology, a specific part of the operator's body (e.g., a hand) making the gesture is detected as a detection target in a captured image obtained by photographing the operator. Patent Document 1 discloses a method for detecting a detection target making a gesture based on a difference between a background image captured in advance and a captured image of the operator. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2015-22624 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, if a captured image shows a hand that is not making a gesture, such as a hand holding an object or a hand displayed as an image on an image display screen, there is a problem that if the hand is erroneously detected as a detection target, a gesture will be erroneously detected and the device will perform an unintended operation.

[0005] An object of the present invention is to provide a detection method, a detection device, and a program that are capable of more appropriately detecting a detection target. [Means for solving the problem]

[0006] In order to solve the above problems, the detection method according to the present invention comprises the steps of: 1. A computer implemented detection method comprising: An image display surface on which an image is displayed.Reference plane of the component and Human hand Acquire depth information relating to the depth of Based on the depth information, hand Derive a distance in a direction approximately normal to the reference plane to the representative point of When the image display surface is detected, The derived distance satisfies a predetermined distance condition. hand Detected as a detection target death, When the image display surface is not detected, the hand with the largest area is detected as the detection target. do.

[0007] In order to solve the above problems, the detection device according to the present invention comprises: An image display surface on which an image is displayed. Reference plane of the component and Human hand Acquire depth information relating to the depth of Based on the depth information, hand Derive a distance in a direction approximately normal to the reference plane to the representative point of When the image display surface is detected, The derived distance satisfies a predetermined distance condition. hand Detected as a detection target death, When the image display surface is not detected, the hand with the largest area is detected as the detection target. do, A processing unit is provided.

[0008] In order to solve the above problems, the program according to the present invention comprises: On the computer, An image display surface on which an image is displayed. Reference plane of the component and Human hand A process of acquiring depth information relating to the depth of the Based on the depth information, hand A process of deriving a distance in a direction approximately normal to the reference plane to a representative point of When the image display surface is detected, The derived distance satisfies a predetermined distance condition. hand Detected as a detection target When the image display surface is not detected, the hand having the largest area is detected as the detection target. The process of Execute the command. Effect of the Invention

[0009] According to the present invention, it is possible to more appropriately detect a detection target. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a schematic diagram showing a configuration of an information processing system. [Diagram 2] FIG. 2 is a block diagram showing a functional configuration of the detection device. [Diagram 3] 10 is a flowchart showing a control procedure of a device control process. [Figure 4] 13 is a flowchart showing a control procedure of hand detection processing. [Diagram 5] 13 is a flowchart showing a control procedure of hand detection processing. [Figure 6] FIG. 2 is a diagram showing an example of a captured image. [Figure 7] 11A and 11B are diagrams for explaining a method of deriving a distance between an image display surface and a representative point of a hand. [Figure 8] FIG. 13 shows a case where the left hand holding the screen and the right hand making a gesture are equidistant from the image capture device. [Figure 9] 13A and 13B are diagrams illustrating the positional relationship between a screen and an operator in a modified example. [Figure 10] FIG. 13 is a diagram showing an example of a captured image according to a modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0012] <Outline of information processing system> FIG. 1 is a schematic diagram showing the configuration of an information processing system 1 according to the present embodiment. The information processing system 1 includes a detection device 10, an image capturing device 20, and a projector 30. The detection device 10 is communicatively connected to the image capturing device 20 and the projector 30 by wire or wirelessly, and is capable of transmitting and receiving data such as control signals and image data between the image capturing device 20 and the projector 30.

[0013] The detection device 10 of the information processing system 1 is an information processing device that detects a gesture made by an operator 80 (subject, person) with a hand 81 (target, detection target) and controls the operation of the projector 30 (projection operation of a projection image Im, operation of changing various settings, etc.) according to the detected gesture. In this specification, the term "hand 81" refers to any one of the right hand 81R and the left hand 81L of the operator 80.

[0014] In detail, the photographing device 20 photographs an operator 80 and a hand 81 of the operator 80 positioned in front of the photographing device 20, and transmits image data of the photographed image 50 (see FIG. 6) to the detection device 10. The detection device 10 analyzes the photographed image 50 received from the photographing device 20, detects the hand 81 and fingers of the operator 80, and determines whether the operator 80 has made a predetermined gesture with the hand 81. In this embodiment, the gesture made by the operator 80 with the hand 81 is a gesture made by the direction or movement of the fingers of the hand 81. When the detection device 10 determines that the operator 80 has made a predetermined gesture with the hand 81, it transmits a control signal to the projector 30 and controls the projector 30 to perform an operation according to the detected gesture. This enables intuitive operations such as, for example, the operator 80 making a gesture of pointing one finger (e.g., the index finger) to the right as viewed from the photographing device 20 to switch the projection image Im projected by the projector 30 to the next projection image Im, and making a gesture of pointing one finger to the left to switch the projection image Im to the previous projection image Im.

[0015] In the following, the direction perpendicular to the floor surface on which the operator 80 is standing and pointing vertically upward is defined as the Z direction, the direction parallel to the floor surface and parallel to the projection direction by the projector 30 when viewed from the Z direction is defined as the Y direction, and the direction perpendicular to the Y direction and the Z direction is defined as the X direction. The direction from the projector 30 toward the operator 80 is defined as the positive direction for the Y direction, and the direction to the right from the projector 30 toward the operator 80 is defined as the positive direction for the X direction.

[0016] In this embodiment, it is assumed that an operator 80 holds a screen 40 (member) in a hand 81, and a projection image Im is projected (displayed) by the projector 30 onto an image display surface 41 (reference surface) of the screen 40. In detail, the operator 80 holds the screen 40 in a left hand 81L, holds a right hand 81R in front of the screen 40 (-Y direction side), and performs a gesture with the right hand 81R to operate the projector 30. The screen 40 is a plate-shaped member, and the image display surface 41 is the surface facing the image capturing device 20 among the front and back surfaces of the screen 40.

[0017] <Configuration of information processing system> FIG. 2 is a block diagram showing the functional configuration of the detection device 10. As shown in FIG. The detection device 10 includes a CPU 11 (Central Processing Unit), a RAM 12 (Random Access Memory), a storage unit 13, an operation unit 14, a display unit 15, a communication unit 16, and a bus 17. The components of the detection device 10 are connected to each other via the bus 17. In this embodiment, the detection device 10 is a notebook PC, but is not limited thereto and may be, for example, a stationary PC, a smartphone, or a tablet terminal.

[0018] The CPU 11 is a processor that controls the operation of the detection device 10 by reading and executing the program 131 stored in the storage unit 13 and performing various arithmetic processing. The CPU 11 corresponds to a "processing unit." The detection device 10 may have multiple processors (multiple CPUs, etc.), and the multiple processes performed by the CPU 11 of this embodiment may be executed by the multiple processors. In this case, the multiple processors correspond to a "processing unit." Also, in this case, the multiple processors may be involved in a common process, or the multiple processors may independently execute different processes in parallel.

[0019] The RAM 12 provides a working memory space for the CPU 11 and stores temporary data.

[0020] The storage unit 13 is a non-transitory recording medium readable by the CPU 11 as a computer, and stores a program 131 and various data. The storage unit 13 includes a non-volatile memory such as a hard disk drive (HDD) or a solid state drive (SSD). The program 131 is stored in the storage unit 13 in the form of a computer-readable program code. The data stored in the storage unit 13 includes captured image data 132 relating to a color image and a depth image received from the imaging device 20.

[0021] The operation unit 14 has at least one of an input device such as a touch panel overlaid on the display screen of the display unit 15, a physical button, a pointing device such as a mouse, and a keyboard, and outputs operation information to the CPU 11 in response to an input operation on the input device.

[0022] The display unit 15 includes a display device such as a liquid crystal display, and performs various displays on the display device in accordance with display control signals from the CPU 11.

[0023] The communication unit 16 is configured with a network card or a communication module, and transmits and receives data between the image capturing device 20 and the projector 30 in accordance with a predetermined communication protocol.

[0024] The imaging device 20 shown in FIG. The color camera 21 captures an image of a shooting range R including the image display surface 41 of the screen 40, the operator 80, and the background thereof, and generates color image data relating to a two-dimensional color image of the shooting range R. Each pixel of the color image data includes color information such as R (red), G (green), and B (blue). The depth camera 22 captures an image of a shooting range R including the image display surface 41 of the screen 40, the operator 80, and the background thereof, and generates depth image data related to a depth image including depth information of the shooting range R. Each pixel of the depth image includes depth information related to the depth (distance from the depth camera 22 to the distance measurement object) of the image display surface 41, the operator 80, and the structure in the background thereof (hereinafter, referred to as the "distance measurement object"). As the depth camera 22, for example, one that detects distance by a time of flight (TOF) method or one that detects distance by a stereo method can be used. The color image data generated by the color camera 21 and the depth image data generated by the depth camera are recorded in the storage unit 13 of the detection device 10 as captured image data 132 (see FIG. 2). In this embodiment, the above-mentioned color image and depth image correspond to the "captured image obtained by capturing an image of the capturing range."

[0025] The color camera 21 and depth camera 22 of the imaging device 20 continuously capture images of an operator 80 and the screen 40 positioned in front of the imaging device 20 at a predetermined frame rate. In the imaging device 20 shown in Fig. 1, the color camera 21 and the depth camera 22 are provided separately, but this configuration is not limited as long as each camera can capture an image of the operator 80. For example, the color camera 21 and the depth camera 22 may be integrated into one configuration. Furthermore, the color camera 21 and the depth camera 22 need only be capable of capturing an image of a range that includes at least the capturing range R, and the angle of view captured by the color camera 21 may be different from the angle of view captured by the depth camera 22. In the capturing range R where the angle of view captured by the color camera 21 and the angle of view captured by the depth camera 22 overlap, the pixels of the color image are associated with the pixels of the depth image. This makes it possible to specify a pixel in the depth image that corresponds to an arbitrary pixel in the color image when the pixel is specified. Thus, it is possible to obtain depth information for an arbitrary pixel in the color image.

[0026] The projector 30 shown in Fig. 1 projects a projection image Im onto an image display surface 41 (projection surface) of a screen 40 by irradiating projection light having an intensity distribution according to image data of the projection image Im with high directivity. In detail, the projector 30 includes a light source, a display element such as a digital micromirror device (DMD) that adjusts the intensity distribution of the light output from the light source to form an optical image, and a projection lens group that collects the optical image formed by the display element and projects it as a projection image Im. The projector 30 changes the projection image Im to be projected and changes settings related to the projection mode (brightness, color, etc.) according to a control signal transmitted from the detection device 10.

[0027] <Operation of information processing system> Next, the operation of the information processing system 1 will be described. The CPU 11 of the detection device 10 analyzes one or more color images and depth images captured by the imaging device 20 to determine whether or not an operator 80 appearing in the image has made a predetermined gesture with a hand 81. When the CPU 11 determines that a gesture has been made with the hand 81, it transmits a control signal to the projector 30 to cause the projector 30 to perform an operation corresponding to the detected gesture.

[0028] Here, the gesture of the hand 81 is, for example, an action of moving a finger in a certain direction (rightward, leftward, downward, upward, etc.) as seen from the image capture device 20, moving the tip of a finger to draw a trajectory of a predetermined shape (circular, etc.), varying the distance between the tips of two or more fingers, bending and straightening the fingers, etc. Each of these gestures is previously associated with one action of the projector 30. For example, an action of switching the projected projection image Im to the next projection image Im may be associated with a gesture of pointing a finger to the right, and an action of switching the projected projection image Im to the previous projection image Im may be associated with a gesture of pointing a finger to the left. In this case, the projection image can be switched to the next image / previous image by performing a gesture of pointing a finger to the right / left. Also, an action of enlarging / reducing the projection image Im may be associated with a gesture of increasing / decreasing the distance between the tip of the thumb and the tip of the index finger, respectively. These are examples of associations between gestures and operations of the projector 30, and any gesture can be associated with any operation of the projector 30. Also, in response to a user operation on the operation unit 14, it may be possible to change the association between a gesture and an operation of the projector 30 or to generate a new association.

[0029] In this way, when the projector 30 is operated by the gesture of the hand 81 of the operator 80, it is important to accurately detect the hand 81 (and the fingers) in the image captured by the image capturing device 20. If the hand 81 cannot be detected correctly, the gesture cannot be recognized correctly, and the operability is significantly reduced.

[0030] However, as shown in Fig. 1, when projector 30 is used while operator 80 holds screen 40 with hand 81, the hand holding screen 40 (left hand 81L in Fig. 1) may be erroneously detected as the hand 81 making the gesture. In this case, the gesture is erroneously detected, causing projector 30 to perform an unintended operation.

[0031] Therefore, in this embodiment, by utilizing depth information from the depth camera 22, hands 81 that are not making a gesture, such as a hand 81 holding the screen 40, are excluded from the detection targets, and hands 81 that are making a gesture, such as a hand 81 held in front of the screen 40 (the side toward the imaging device 20, in the -Y direction), are appropriately detected as detection targets.

[0032] 3 to 8, an operation of the CPU 11 of the detection device 10 to detect the hand 81 of the operator 80 and an operation of detecting a gesture by the hand 81 and controlling the operation of the projector 30 will be described below. To realize the above operations, the CPU 11 executes a device control process shown in FIG. 3 and a hand detection process shown in FIG. 4 and FIG. 5.

[0033] FIG. 3 is a flowchart showing a control procedure of the device control process. The device control process is executed, for example, when the detection device 10, the image capture device 20, and the projector 30 are powered on and acceptance of a gesture for operating the projector 30 is started.

[0034] When the device control process is started, the CPU 11 transmits a control signal to the photographing device 20 to start photographing by the color camera 21 and the depth camera 22 (step S101). When photographing is started, the CPU 11 executes a hand detection process (step S102).

[0035] 4 and 5 are flowcharts showing a control procedure for the hand detection process. When the hand detection process is called, the CPU 11 acquires a captured image 50 (captured image data 132) capturing an image of the operator 80 and the hand 81 (step S201). The CPU 11 also extracts candidates for the hand area corresponding to the hand 81 (hereinafter simply referred to as "candidates for the hand 81") from the acquired captured image 50 (step S202). The process of extracting candidates for the hand 81 from the captured image 50 corresponds to the process of extracting an object from the captured image 50.

[0036] FIG. 6 is a diagram showing an example of a captured image 50. As shown in FIG. The captured image 50 is obtained by the image capturing device 20 capturing the image capturing range R shown in FIG. 2, and includes a color image and a depth image. FIG. 6 shows the color image. As described above, each pixel of the color image is associated with depth information of the corresponding pixel in the depth image. The x-axis and y-axis shown in FIG. 6 are coordinate axes of an orthogonal coordinate system that represents the position of a pixel in the captured image 50. In the captured image 50 shown in FIG. 3, an operator 80 holding a screen 40 with a left hand 81L is shown. The operator 80 holds a right hand 81R in front of the screen 40 and away from the screen 40, and makes a gesture of pointing an index finger to the right as seen from the image capturing device 20. In step S202, a right hand 81R and a left hand 81L are extracted as candidates for a hand 81 from the captured image 50 shown in FIG. 3.

[0037] The method for extracting candidates for the hand 81 from the captured image 50 in step S202 is not particularly limited, but for example, the following method can be used. First, a region corresponding to the skin color (the color of the hand 81) is extracted from the color image by performing a color threshold process based on the color information of the color image. Next, for each extracted region, it is determined whether or not it includes a protruding part corresponding to a finger. Of the extracted regions, a region determined to include a protruding part corresponding to a finger is extracted as a candidate hand region corresponding to the hand 81. This method is merely an example, and candidates for hand 81 can be extracted by any method using at least one of a color image and a depth image. For example, when the range in the Y direction (depth range) in which hand 81 of operator 80 is located is determined in advance, such as when the position in the Y direction of screen 40 is fixed, a region belonging to the depth range in which hand 81 is located may be extracted by performing a depth-related threshold process on each region extracted by the above threshold process on the color image.

[0038] CPU 11 may generate a mask image representing the hand region corresponding to the extracted hand 81, and use the mask image data in subsequent processing. The mask image data is, for example, an image in which the pixel value of pixels corresponding to the hand region is set to "1" and the pixel value of pixels corresponding to regions other than the hand region is set to "0."

[0039] The CPU 11 determines whether a candidate for the hand 81 is extracted in step S202 (step S203). When it is determined that a candidate for the hand 81 is extracted ("YES" in step S203), the CPU 11 detects (specifies) the image display surface 41 of the screen 40 in the captured image 50 (step S204). The method of detecting the image display surface 41 is not particularly limited, and may be, for example, a method of extracting a planar rectangular area with continuous depth in a depth image as the image display surface 41, or a method of detecting a rectangular contour line of the image display surface 41 (screen 40) in a color image. Alternatively, a predetermined marker may be provided in advance at the four corners of the image display surface 41 (screen 40), and the image display surface 41 may be detected by detecting the marker in the captured image 50. Also, a predetermined marker may be included in the projection image Im, and the image display surface 41 may be detected by detecting the marker in the captured image 50.

[0040] The CPU 11 determines whether the image display surface 41 is detected (step S205), and if it is determined that the image display surface 41 is detected ("YES" in step S205), it determines whether there is a candidate for the hand 81 in the photographed image 50, at least a part of which overlaps the image display surface 41 (step S206). Here, the CPU 11 determines whether at least a part of pixels belonging to the candidate for the hand 81 (hand area) is within a rectangular range of the image display surface 41 in the xy coordinate plane in the photographed image 50. If a part of the image display surface 41 is hidden by the hand 81 or the like as shown in FIG. 6, the CPU 11 complements the rectangular outline of the image display surface 41 before making the determination. Furthermore, if only a part of a certain hand 81 overlaps the image display surface 41, the CPU 11 determines that the certain hand 81 overlaps the image display surface 41. In the example shown in FIG. 3, a right hand 81R and a left hand 81L are determined to be candidates for the hand 81 overlapping the image display surface 41 since parts of them overlap the image display surface 41.

[0041] When it is determined that there is a candidate for the hand 81 that at least a part of which overlaps the image display surface 41 ("YES" in step S206), the CPU 11 derives the position of the image display surface 41 in space (step S207). Here, the CPU 11 first specifies the coordinates in the XYZ coordinate space for at least three of the four vertices of the image display surface 41. In detail, the CPU 11 specifies the distance from the imaging device 20 to each vertex of the image display surface 41 based on the depth information of the portion corresponding to the image display surface 41 extracted from the depth image, and derives the coordinates of each vertex in the XYZ coordinate space based on the specified distance and the position of the vertex of the image display surface 41 in the color image or the depth image (position in the xy coordinate plane). Then, the CPU 11 derives a plane equation that represents the position of the image display surface 41 in the XYZ coordinate space based on the coordinates of at least three vertices. The plane equation is expressed in the form of, for example, "aX+bY+cZ+d=0".

[0042] Next, CPU 11 derives the distance from image display surface 41 to the representative point of the candidate for hand 81 that overlaps image display surface 41 (step S208).

[0043] FIG. 7 is a diagram for explaining a method of deriving the distance between image display surface 41 and a representative point of hand 81. In FIG. Fig. 7 corresponds to a view of the operator 80 and the screen 40 shown in Fig. 1 viewed from the +Z direction. In Fig. 7, the image display surface 41 of the screen 40 is parallel to the Z direction. However, even if the image display surface 41 is not parallel to the Z direction, the distance d can be derived by the following method.

[0044] When deriving the distance between the image display surface 41 and the representative point of the hand 81, first, the coordinates of the representative point of the hand 81 in the XYZ coordinate space are derived. In this embodiment, the representative point of the hand 81 is the center of gravity of the part of the hand 81 that overlaps with the image display surface 41 in the photographed image 50 shown in FIG. 6 (center of gravity GR for the right hand 81R, center of gravity GL for the left hand 81L. Hereinafter, the center of gravity of any hand 81 will be referred to as "center of gravity G"). In addition, in the XYZ coordinate space, the center of gravity G of the hand 81 is a point on the surface of the hand 81 that faces the photographing device 20, as shown in FIG. 7. The CPU 11 specifies the distance from the photographing device 20 to the center of gravity G based on the depth information of the pixel corresponding to the center of gravity G in the photographed image 50, and derives the coordinates of the center of gravity G in the XYZ coordinate space based on the specified distance and the position of the center of gravity G in the color image or the depth image. When the coordinates of the center of gravity G are derived, CPU 11 derives a distance d in the normal direction of image display surface 41 from image display surface 41 to center of gravity G of hand 81 using a plane equation of image display surface 41 . The normal direction is not limited to the completely normal direction of the image display surface 41, but may be an approximately normal direction (a direction slightly inclined from the normal direction, for example, a direction with a deviation angle of ±10 degrees or less from the normal direction). In other words, the distance d in the approximately normal direction of the image display surface 41 may be derived. Furthermore, the representative point of the hand 81 is not limited to the center of gravity G of the hand 81. For example, the average depth of pixels belonging to a portion of the hand 81 (hand region) shown in Fig. 6 that overlaps with the image display surface 41 may be derived, and a pixel having a depth closest to the average depth may be set as the representative point.

[0045] 4, when step S208 is completed, the CPU 11 determines whether or not the distance d has been derived for all candidates of the hand 81 overlapping the image display surface 41 (step S209). If it is determined that the distance d has not been derived for any candidate of the hand 81 overlapping the image display surface 41 ("NO" in step S209), the CPU 11 selects a candidate of the next hand 81 (step S210) and derives the distance d for the candidate of the hand 81 (step S208).

[0046] When it is determined that the distance d has been derived for all candidates of the hand 81 overlapping the image display surface 41 ("YES" in step S209), the CPU 11 determines whether or not there is a candidate of the hand 81 whose derived distance d satisfies a predetermined distance condition (step S211 in FIG. 5). In this embodiment, when the derived distance d is equal to or greater than the reference distance ds, it is determined that the distance d satisfies the distance condition. Here, the reference distance ds is determined in advance to be a value greater than the distance d between the center of gravity G of the hand 81 (left hand 81L in FIG. 7) holding the screen 40 and the image display surface 41, as shown in FIG. 7, and is stored in the storage unit 13. Since the distance d of the hand 81 holding the screen 40 is approximately the thickness of a finger, the reference distance ds is set to a value greater than the thickness of a finger of a human hand (average thickness of a human finger, for example, about 2.5 cm). In order to reliably exclude the hand 81 holding the screen 40 from the detection target, the reference distance ds may be a value obtained by adding a predetermined margin to the thickness of a finger of a human hand. 7, the distance d from the image display surface 41 to the center of gravity GR of the right hand 81R is determined to be equal to or greater than the reference distance ds. On the other hand, the distance d from the image display surface 41 to the center of gravity GL of the left hand 81L is determined to be less than the reference distance ds.

[0047] When it is determined that there is a candidate hand 81 whose derived distance d is equal to or greater than the predetermined reference distance ds ("YES" in step S211), the CPU 11 excludes the candidates of the hand 81 overlapping the image display surface 41, except for the one with the largest derived distance d (step S212). This makes it possible to exclude the candidate hand 81 (the hand 81 holding the screen 40) that overlaps the image display surface 41 and whose distance d from the image display surface 41 is less than the reference distance ds, from the candidates. Also, when there are two or more hands 81 whose derived distance d is equal to or greater than the reference distance ds, it is possible to narrow down the hand 81 to be used for gesture discrimination from the two or more hands 81 to one. In the example shown in FIG. 7, the right hand 81R has the largest distance d among the two candidates of the hand 81 (the right hand 81R and the left hand 81L), so the hand 81 other than the right hand 81R, i.e., the left hand 81L, is excluded from the candidates.

[0048] By determining whether the distance d from the image display surface 41 is equal to or greater than the reference distance ds based on the distance d in the normal direction of the image display surface 41, it is possible to appropriately narrow down candidates for the hand 81 regardless of the positional relationship between the image capture device 20 and the hand 81. For example, even if the center of gravity GL of the left hand 81L holding the screen 40 and the center of gravity GR of the right hand 81R making the gesture are equidistant from the image capture device 20 as shown in Fig. 8, it is possible to exclude the left hand 81L from the candidates and leave the right hand 81R as a candidate.

[0049] When step S212 is completed, CPU 11 detects the hand 81 candidate having the largest area in captured image 50 among the hand 81 candidates at that time as the hand 81 (detection target) for gesture discrimination (step S215). The area discrimination target in step S215 also includes hand 81 candidates that do not overlap image display surface 41.

[0050] On the other hand, if it is determined in step S211 that there is no candidate hand 81 whose derived distance d is equal to or greater than the predetermined reference distance ds ("NO" in step S211), the CPU 11 excludes all candidates for the hand 81 that overlap the image display surface 41 from the candidates (step S213). In addition, the CPU 11 determines whether or not there is a candidate hand 81 that does not overlap the image display surface 41 (step S214), and if it is determined that there is a candidate hand 81 that does not overlap the image display surface 41 ("YES" in step S214), the CPU 11 executes the process of step S215 described above.

[0051] In addition, if it is determined in step S205 that the image display surface 41 has not been detected ("NO" in step S205), and if it is determined in step S206 that there is no candidate hand 81 at least a portion of which overlaps the image display surface 41 ("NO" in step S206), the CPU 11 also executes the processing of step S215.

[0052] When step S215 is completed, when it is determined in step S214 that there are no candidates for hand 81 that do not overlap image display surface 41 ("NO" in step S214), and when it is determined in step S203 that no candidates for hand 81 have been extracted ("NO" in step S203), CPU 11 ends the hand detection process and returns the process to the device control process of FIG. 3.

[0053] In step S212, among the candidates for hand 81 overlapping image display surface 41, all hands 81 whose derived distance d is equal to or greater than the reference distance ds may be retained as candidates (only hands 81 whose distance d is less than the reference distance ds may be excluded from the candidates), and in step S215, the one with the largest area may be detected as the hand 81 for gesture discrimination from among the remaining candidates for hand 81. In step S215, the hand with the largest area is detected as the hand 81 for gesture discrimination, but one hand 81 for gesture discrimination may be determined by a method other than this. For example, an index value of the likelihood that the hand 81 is a hand 81 may be derived based on the shape of the fingers of the hand 81, and the hand 81 with the largest index value may be detected as the hand 81 for gesture discrimination.

[0054] When the hand detection process (step S102) of FIG. 3 is completed, the CPU 11 determines whether or not the hand 81 for gesture discrimination has been detected in the hand detection process (step S103). If it is determined that the hand 81 has been detected ("YES" in step S103), the CPU 11 determines whether or not a gesture by the hand 81 of the operator 80 has been detected from the orientation of the fingers of the hand 81 or the movement of the positions of the tips of the fingers across multiple frames of the captured image 50 (step S104). If it is determined that a gesture has been detected ("YES" in step S104), the CPU 11 transmits a control signal to the projector 30 to cause the projector 30 to perform an operation corresponding to the detected gesture (step S105). The projector 30 that has received the control signal performs an operation corresponding to the control signal.

[0055] When step S105 is completed, when it is determined in step S103 that the hand 81 has not been detected ("NO" in step S103), or when it is determined in step S104 that the gesture has not been detected ("NO" in step S104), the CPU 11 determines whether or not to end the acceptance of the gesture in the information processing system 1 (step S106). Here, the CPU 11 determines to end the acceptance of the gesture when, for example, an operation to turn off the power of the detection device 10, the image capturing device 20, or the projector 30 has been performed.

[0056] If it is determined that the acceptance of the gesture is not to be ended ("NO" in step S106), CPU 11 returns the process to step S102 and executes hand detection processing for detecting hand 81 based on the captured image captured in the next frame period. The loop processing of steps S102 to S106 is repeatedly executed at the frame rate of the image capturing by color camera 21 and depth camera 22 (i.e., every time a color image and a depth image are generated), for example. When it is determined that the acceptance of the gesture is to be ended ("YES" in step S106), the CPU 11 ends the device control process.

[0057] <Modification> Next, a modified example of the above embodiment will be described. Below, differences from the above embodiment will be described, and configurations common to the above embodiment will be denoted by the same reference numerals and description thereof will be omitted.

[0058] FIG. 9 is a diagram showing the positional relationship between the screen 40 and an operator 80 in the modified example. In this modification, the screen 40 is hung on a wall, and the operator 80 stands facing the screen 40 (on the −Y direction side of the screen 40) and looks at the projection image Im projected on the screen 40. The operator 80 also makes a gesture with his right hand 81R to operate the projector 30. The imaging range R of the imaging device 20 includes at least the upper body of the operator 80 and the screen 40.

[0059] FIG. 10 is a diagram showing an example of a captured image 50 according to a modified example. The captured image 50 is obtained by the imaging device 20 capturing an image of the imaging range R in Fig. 9. In the captured image 50 shown in Fig. 10, the screen 40 and the operator 80 positioned on the image display surface 41 side of the screen 40 are captured. In addition, in the captured image 50, the right hand 81R of the operator 80 overlaps with the image display surface 41. In addition, it is assumed that the projection image Im projected onto the image display surface 41 includes an image of a person and the hand 81I of the person.

[0060] In the hand detection process (FIG. 4) for such a captured image 50, in step S202, the right hand 81R of the operator 80 and the hand 81I of the person in the projection image Im are extracted as candidates for the hand 81. In step S206, the right hand 81R and the hand 81I are determined as candidates for the hand 81 overlapping the image display surface 41. In step S208, the distance d from the image display surface 41 to the center of gravity GR of the right hand 81R and the distance d from the image display surface 41 to the center of gravity GI of the hand 81I are derived, and in step S211, it is determined whether the derived distance d is equal to or greater than a reference distance ds.

[0061] Since the right hand 81R of the operator 80 is separated from the image display surface 41 by the reference distance ds or more, the distance d of the right hand 81R is determined to be equal to or more than the reference distance ds, and the right hand 81R is left as a candidate for the hand 81 for gesture discrimination. In this way, even when the operator 80 is positioned on the image display surface 41 side of the screen 40, the hand 81 of the operator 80 can be detected as a candidate for the hand 81 for gesture discrimination.

[0062] On the other hand, the position of the center of gravity GI in the XYZ coordinate space of the person's hand 81I included in the projection image Im is on the image display surface 41, so the derived distance d is 0 and it is determined to be less than the reference distance ds. Therefore, the person's hand 81I included in the projection image Im is excluded from the candidates for the hand 81 for gesture discrimination. In this way, when the projection image Im including the hand 81I is projected onto the image display surface 41, the hand 81I can be appropriately excluded from the candidates.

[0063] In addition, the positional relationship between the screen 40 and the operator 80 is not limited to that shown in Fig. 9. For example, in a mode in which a projected image Im is projected vertically downward from a projector 30 provided vertically above the screen 40 placed substantially horizontally on a table (or held by the operator 80 so as to be substantially horizontal), a gesture may be made by holding a hand 81 between the screen 40 and the projector 30. In this case, an image capturing device 20 may be provided vertically above the screen 40 to capture an image of the screen 40 and the hand 81 of the operator 80.

[0064] <Effects> As described above, the detection method according to the present embodiment is a detection method executed by the CPU 11 as a computer, and acquires depth information related to the depth of the image display surface 41 (reference surface) of the screen 40 and the hand 81 (object), derives a distance d from the image display surface 41 to the center of gravity G (representative point) of the hand 81 in a direction approximately normal to the image display surface 41 based on the depth information, and detects the hand 81 whose derived distance d satisfies a predetermined distance condition as a detection target. This makes it possible to exclude hands 81 not making a gesture (hands 81 holding the screen 40, hands 81I displayed as an image on the image display surface 41 of the screen 40, etc.) from detection targets, and to appropriately detect the hand 81 of the operator 80 making a gesture as a detection target. In addition, by making a determination based on the distance d in the direction approximately normal to the image display surface 41, it is possible to appropriately narrow down candidates for the hand 81 regardless of the positional relationship between the image capture device 20 and the hand 81. In addition, since advanced information processing such as pattern matching and machine learning is not required, the above method can be executed by a computer of the detection device 10 having a simple configuration.

[0065] In addition, a captured image 50 including the image display surface 41 and the hand 81 is acquired, and a distance d from the image display surface 41 to the center of gravity G of the hand 81 is derived based on the captured image 50 and the depth information. This makes it possible to easily and appropriately extract the hand 81, which is a candidate for a detection target, from the captured image 50.

[0066] Moreover, a captured image 50 including the image display surface 41 and the hand 81 is acquired, and the hand 81 at least partially overlapping the image display surface 41 is extracted from the captured image 50, and a distance d from the image display surface 41 to the center of gravity G of the hand 81 is derived based on the depth information. As a result, even if a hand 81 not making a gesture appears in an area of ​​the captured image 50 that overlaps the image display surface 41, these hands 81 can be excluded from detection targets, and the hand 81 of the operator 80 making a gesture can be appropriately detected as a detection target.

[0067] Furthermore, when the derived distance d is equal to or greater than the reference distance ds, it is determined that the distance d satisfies the distance condition, so that the hand 81 that is not making a gesture can be easily and appropriately excluded from the detection targets.

[0068] Furthermore, when two or more hands 81 are extracted, the hand 81 with the longest derived distance d among the hands 81 whose derived distance d satisfies the distance condition is detected as the detection target. This makes it possible to easily and appropriately narrow down the hands 81 used for gesture discrimination to one.

[0069] Moreover, the representative point is the center of gravity G of a portion of the candidate for hand 81 that overlaps with image display surface 41 in captured image 50. This makes it possible to appropriately derive the distance between the portion of hand 81 that overlaps with image display surface 41 and image display surface 41.

[0070] Moreover, the detection target is a person's hand 81, and the screen 40 is held by the person's hand 81. In such a case, the hand 81 holding the screen 40 is excluded from the detection target, and the hand 81 of the operator 80 making the gesture can be appropriately detected as the detection target.

[0071] In a modified example, the detection target is a person's hand 81, and the captured image 50 is an image captured within a capturing range R including the person's hand 81 located on the image display surface 41 side of the screen 40. This makes it possible to appropriately detect the hand 81 of the operator 80 making a gesture on the image display surface 41 side of the screen 40 as the detection target.

[0072] Moreover, the reference surface of the screen 40 as a member is the image display surface 41 onto which the projection image Im is projected (displayed). This allows the hand 81I displayed as an image on the image display surface 41 of the screen 40 to be excluded from the detection target, and the hand 81 of the operator 80 making a gesture to be appropriately detected as the detection target.

[0073] Moreover, the detection device 10 according to the present embodiment acquires depth information relating to the image display surface 41 of the screen 40 and the depth of the hand 81, derives a distance d from the image display surface 41 to the center of gravity G (representative point) of the candidate for the hand 81 in a direction substantially normal to the image display surface 41 based on the depth information, and detects the hand 81 for which the derived distance d satisfies a predetermined distance condition as a detection target. This allows the hand 81 not making a gesture to be excluded from the detection target, and the hand 81 of the operator 80 making a gesture to be appropriately detected as a detection target. In addition, it is possible to appropriately narrow down candidates for the hand 81 regardless of the positional relationship between the image capture device 20 and the hand 81. In addition, since advanced information processing such as pattern matching and machine learning is not required, the processing load of the CPU 11 can be kept low.

[0074] Moreover, the program 131 according to the present embodiment causes the CPU 11 as a computer to execute a process of acquiring depth information relating to the image display surface 41 of the screen 40 and the depth of the hand 81, a process of deriving a distance d from the image display surface 41 to the center of gravity G (representative point) of a candidate for the hand 81 in a direction substantially normal to the image display surface 41 based on the depth information, and a process of detecting the hand 81 whose derived distance d satisfies a predetermined distance condition as a detection target. This makes it possible to exclude the hand 81 not making a gesture from the detection target, and to appropriately detect the hand 81 of the operator 80 making a gesture as a detection target. In addition, it is possible to appropriately narrow down candidates for the hand 81 regardless of the positional relationship between the image capture device 20 and the hand 81. In addition, since advanced information processing such as pattern matching and machine learning is not required, the processing load of the CPU 11 can be kept low.

[0075] <Other> The above-described embodiment is merely an example of the detection method, the detection device, and the program according to the present invention, and the present invention is not limited to this. For example, in the above embodiment, the detection device 10, the image capture device 20, and the projector 30 (devices to be operated by gestures) are separate from each other, but the present invention is not limited to this. For example, the detection device 10 and the image capturing device 20 may be integrated together. As an example, the color camera 21 and the depth camera 22 of the image capturing device 20 may be incorporated in the bezel of the display unit 15 of the detection device 10. Furthermore, the detection device 10 and the device to be operated may be integrated. For example, the function of the detection device 10 may be incorporated into the projector 30 in the above embodiment, and the processing executed by the detection device 10 may be executed by a CPU (not shown) of the projector 30. In this case, the projector 30 corresponds to the "detection device", and the CPU of the projector 30 corresponds to the "processing unit". Furthermore, the image capturing device 20 and the device to be operated may be integrated together. For example, the color camera 21 and the depth camera 22 of the image capturing device 20 may be incorporated into the housing of the projector 30 in the above embodiment. In addition, the detection device 10, the photographing device 20, and the device to be operated may all be integrated. For example, in a mode in which the color camera 21 and the depth camera 22 are incorporated in the bezel of the display unit 15 of the detection device 10 as the device to be operated, the operation of the detection device 10 may be controlled by gestures of the hand 81 (fingers 82) of the operator 80.

[0076] Furthermore, the operator 80 is not limited to a human being, but may be a robot, an animal, or the like.

[0077] In the above embodiment, the screen 40 is exemplified as the "member" and the image display surface 41 (projection surface) onto which the projection image Im is projected is exemplified as the "reference surface", but this is not intended to be limiting. For example, the "member" may be a display device such as a liquid crystal display device, and the "reference surface" may be the image display surface of the display device. Furthermore, the "member" may be a plate-like member that the operator 80 holds in a position that forms the background of the hand 81 in order to make it easier to detect gestures made by the hand 81. In this case, the surface of the plate-like member on the side of the image capturing device 20 corresponds to the "reference surface."

[0078] In addition, in the above embodiment, the gesture performed by the operator 80 with the hand 81 is a gesture performed by the direction or movement of the fingers of the hand 81, but this is not limited to this and may be, for example, a gesture performed by the entire movement of the hand 81. Alternatively, the gesture may be a gesture of touching the image display surface 41 with a fingertip. Even when the image display surface 41 is directly touched in this manner, the position of the center of gravity G of the hand 81 (the back of the hand) is usually at least the reference distance ds away from the image display surface 41, and therefore, the method of the above embodiment can leave it as a candidate for the hand 81 to be detected. Furthermore, the detection of the gesture may involve determining whether or not the positional relationship between the fingertip and the image display surface 41 has changed over multiple frames.

[0079] In the above embodiment, the distance d from the image display surface 41 shown in FIG. 7 to the center of gravity G (representative point) of the hand 81 is derived based on the depth information of the depth image captured by the depth camera 22, but the depth information used to derive the distance d is not limited to that acquired by the depth camera 22. For example, a distance derivation camera may be provided to capture the operator 80 and the screen 40 from above (+Z direction) the operator 80, and the distance d may be derived by acquiring depth information related to the depth of the hand 81 from the image captured by the distance derivation camera. In this case, the hand 81 and the screen 40 in the image captured by the distance derivation camera are associated with the hand 81 and the screen 40 in the image captured by the imaging device 20, thereby making it possible to identify the position from which the distance d is derived in the image captured by the distance derivation camera.

[0080] 6 and the like, if depth information of the image display surface 41 (reference surface) of the screen 40 and the hand 81 (target object) can be acquired, the hand 81 of the detection target can be detected based on the depth information. In other words, if the depth information can be acquired separately from the captured image 50, acquisition of the captured image 50 may be omitted. In this case, the device used to acquire the depth information may be the depth camera 22 of the above embodiment, or a distance measuring device such as a millimeter wave radar or a light wave distance measuring device.

[0081] In addition, in the above embodiment, an example was given of a case where the shooting range R includes the entire image display surface 41 of the screen 40, but this is not limited to this, and it may be a case where only a portion of the image display surface 41 is included in the shooting range R, that is, where only a portion of the image display surface 41 is captured in the captured image 50.

[0082] Also, image display surface 41 is not limited to a flat surface, and may be a curved surface. In this case, the position of image display surface 41 in space is derived based on the depth information, and the shortest distance from image display surface 41, which is a curved surface, to the representative point of hand 81 is derived as distance d.

[0083] In the above description, an example has been disclosed in which the HDD or SSD of the storage unit 13 is used as a computer-readable medium for the program according to the present invention, but the present invention is not limited to this example. As other computer-readable media, information recording media such as flash memory and CD-ROM can be applied. In addition, a carrier wave can also be applied to the present invention as a medium for providing data of the program according to the present invention via a communication line.

[0084] Furthermore, the detailed configurations and operations of the components of the detection device 10, the image capture device 20, and the projector 30 in the above embodiment can of course be modified as appropriate without departing from the spirit of the present invention.

[0085] Although the embodiments of the present invention have been described, the scope of the present invention is not limited to the above-described embodiments, but includes the scope of the invention described in the claims and its equivalents. The inventions described in the claims originally attached to this application are set forth below. The claim numbers in the appended claims are the same as those in the claims originally attached to this application. [Additional Notes] <Claim 1> 1. A computer implemented detection method comprising: Acquire depth information relating to a reference plane of the component and a depth of the object; Deriving a distance from the reference plane to the representative point of the object in a direction approximately normal to the reference plane based on the depth information; Detecting the object whose derived distance satisfies a predetermined distance condition as a detection target; Detection methods. <Claim 2> acquiring a captured image including the reference plane and the object; deriving the distance from the reference plane to the representative point of the object based on the captured image and the depth information; The detection method according to claim 1. <Claim 3> acquiring a captured image including the reference plane and the object; extracting the object at least a part of which overlaps with the reference plane in the captured image; deriving the distance from the reference plane to the representative point of the object based on the depth information; The detection method according to claim 1. <Claim 4> If the derived distance is equal to or greater than a reference distance, it is determined that the distance satisfies the distance condition. The detection method according to claim 1. <Claim 5> When two or more objects are extracted, the object having the longest derived distance among the objects whose derived distances satisfy the distance condition is detected as the detection object. The detection method according to claim 1. <Claim 6> 4. The detection method according to claim 2, wherein the representative point is a center of gravity of a portion of the candidate for the detection target in the captured image that overlaps with the reference surface. <Claim 7> the detection target is a human hand, the member being held by the person's hand; The detection method according to claim 1. <Claim 8> the detection target is a human hand, The captured image is an image capturing a shooting range including the hand of the person positioned on the reference surface side of the member. The detection method according to claim 2 or 3. <Claim 9> the reference surface of the member is an image display surface on which an image is displayed; The detection method according to claim 1. <Claim 10> Acquire depth information relating to a reference plane of the component and a depth of the object; Deriving a distance from the reference plane to the representative point of the object in a direction approximately normal to the reference plane based on the depth information; Detecting the object whose derived distance satisfies a predetermined distance condition as a detection target; A detection device comprising a processing unit. <Claim 11> On the computer, A process of acquiring depth information relating to a reference surface of a component and a depth of an object; A process of deriving a distance from the reference plane to a representative point of the object in a direction approximately normal to the reference plane based on the depth information; a process of detecting the object whose derived distance satisfies a predetermined distance condition as a detection target; A program that executes the following. [Explanation of symbols]

[0086] 1. Information Processing Systems 10. Detection Device 11 CPU (processing unit) 12 RAM 13 Storage section 131 Programs 132 Photographed image data 14 Control section 15 Display section 16 Communications Department 17 Bus 20 Imaging Equipment 21 Color Camera 22 Depth Camera 30 Projector 40 Screen (component) 41 Image display surface (reference surface) 50 Images 80 Operator (subject) 81, 81L, 81R, 81I Hands (object, candidate for detection, detection target) G, GL, GR, GI Center of gravity (representative point) Im Projection Image R Shooting range d Distance ds Reference distance

Claims

1. 1. A computer implemented detection method comprising: Acquire depth information relating to a reference plane of a member that is an image display surface on which an image is displayed and a depth of a person's hand; Deriving a distance from the reference plane to the representative point of the hand in a direction approximately normal to the reference plane based on the depth information; detecting, as a detection target, the hand whose distance derived when the image display surface is detected satisfies a predetermined distance condition; When the image display surface is not detected, the hand having the largest area is detected as the detection target. Detection methods.

2. acquiring a captured image including the reference plane and the hand; deriving the distance from the reference plane to the representative point of the hand based on the captured image and the depth information; The detection method according to claim 1 .

3. acquiring a captured image including the reference plane and the hand; extracting the hand, at least a part of which overlaps with the reference plane, from the captured image; deriving the distance from the reference plane to the representative point of the hand based on the depth information; The detection method according to claim 1 .

4. If the derived distance is equal to or greater than a reference distance, it is determined that the distance satisfies the distance condition. The detection method according to claim 1 .

5. When two or more hands are extracted, the hand having the longest derived distance among the hands whose derived distances satisfy the distance condition is detected as the detection target. The detection method according to claim 1 .

6. The detection method according to claim 2 , wherein the representative point is a center of gravity of a portion of the candidate for the detection target in the captured image that overlaps with the reference surface.

7. The member is held by the hand of the person. The detection method according to claim 1 .

8. The captured image is an image capturing a range including the hand of the person located on the reference surface side of the member, The detection method according to claim 2 or 3.

9. Acquiring depth information relating to a reference plane of a member that is an image display surface on which an image is displayed and a depth of a person's hand; Deriving a distance from the reference plane to the representative point of the hand in a direction approximately normal to the reference plane based on the depth information; detecting, as a detection target, the hand whose distance derived when the image display surface is detected satisfies a predetermined distance condition; When the image display surface is not detected, the hand having the largest area is detected as the detection target. A detection device comprising a processing unit.

10. On the computer, A process of acquiring depth information relating to a reference plane of a member that is an image display surface on which an image is displayed and a depth of a person's hand; deriving a distance from the reference plane to the representative point of the hand in a direction approximately normal to the reference plane based on the depth information; a process of detecting, as a detection target, the hand whose derived distance satisfies a predetermined distance condition when the image display surface is detected, and detecting, as a detection target, the hand whose area is the largest when the image display surface is not detected; A program that executes the following.

Images