Detection method, detection device, and program

Abstract

To provide a detection method, a detection apparatus, and a program for detecting a detection target more properly.SOLUTION: A detection method to be executed by a computer includes: acquiring a captured image including at least a part of an image display region in which an image is displayed and at least a part of a target; and detecting, as a detection target, at least a part of the target extracted in an outside region in the captured image from which the image display region is excluded. Accordingly, the detection target can be detected more properly.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a detection method, a detection device, and a program. [Background technology]

[0002] Conventionally, there is a technology for detecting a gesture made by an operator and controlling the operation of a device in accordance with the detected gesture. In this technology, a specific part of the operator's body (e.g., a hand) making the gesture is detected as a detection target in a captured image of the operator. Patent Document 1 discloses a method for detecting a detection target making a gesture based on a difference between a background image captured in advance and a captured image of the operator. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-22624 Summary of the Invention [Problem to be solved by the invention]

[0004] However, if a captured image shows a hand that is not making a gesture, such as a hand holding an object or a hand displayed as an image on an image display screen, there is a problem that if this hand is mistakenly detected as a detection target, a gesture will be mistakenly detected and the device will perform an unintended operation.

[0005] An object of the present invention is to provide a detection method, a detection device, and a program that are capable of more appropriately detecting a detection target. [Means for solving the problem]

[0006] In order to solve the above problems, the detection method according to the present invention comprises: 1. A computer-implemented detection method comprising: acquiring a captured image including at least a portion of an image display area in which an image is displayed and at least a portion of an object; Detecting at least a part of the object extracted in an external area excluding the image display area of ​​the captured image as a detection target; If the extracted object extends from the inside to the outside of the image display area in the captured image, the object is determined to be outside the image display area. and detecting the object as the detection target. .

[0007] In order to solve the above problems, the detection device according to the present invention comprises: acquiring a captured image including at least a portion of an image display area in which an image is displayed and at least a portion of an object; Detecting at least a part of the object extracted in an external area excluding the image display area of ​​the captured image as a detection target; If the extracted object extends from the inside to the outside of the image display area in the captured image, the object is determined to be outside the image display area. and detecting the object as the detection target. , A processing unit is provided.

[0008] In order to solve the above problems, the program according to the present invention comprises: On the computer, A process of acquiring a captured image including at least a portion of an image display area in which an image is displayed and at least a portion of an object; a process of detecting, as a detection target, at least a part of the object extracted in an external area excluding the image display area of ​​the captured image; Execute If the extracted object extends from the inside to the outside of the image display area in the captured image, the object is determined to be outside the image display area. and detecting the object as the detection object. . [Effects of the Invention]

[0009] According to the present invention, it is possible to more appropriately detect the detection target. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a schematic diagram illustrating a configuration of an information processing system. [Figure 2] FIG. 2 is a block diagram showing the functional configuration of the detection device. [Figure 3] FIG. 10 is a diagram showing an example of a captured image in which a projected image including a person's hand is captured. [Figure 4] 10 is a flowchart showing a control procedure of a device control process. [Figure 5] 10 is a flowchart showing a control procedure for hand detection processing. [Figure 6] FIG. 10 is a diagram showing another example of a captured image. [Figure 7] 10 is a flowchart showing a control procedure of hand detection processing according to a modified example. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0012] <Outline of the information processing system> FIG. 1 is a schematic diagram showing the configuration of an information processing system 1 according to this embodiment. The information processing system 1 includes a detection device 10, an image capturing device 20, and a projector 30. The detection device 10 is communicatively connected to the image capturing device 20 and the projector 30 wirelessly or via a wire, and is capable of transmitting and receiving data such as control signals and image data between the image capturing device 20 and the projector 30.

[0013] The detection device 10 of the information processing system 1 is an information processing device that detects a gesture made by an operator 80 (subject, person) with a hand 81 (object, detection target) and controls the operation of the projector 30 (projection operation of a projection image Im, operation of changing various settings, etc.) in accordance with the detected gesture. In this specification, any one of the right hand 81R and left hand 81L of the operator 80 will be referred to as "hand 81".

[0014] In detail, the image capturing device 20 captures an image of an operator 80 positioned in front of the image capturing device 20 and a hand 81 of the operator 80, and transmits image data of the captured image 50 (see FIG. 3 ) to the detection device 10. The detection device 10 analyzes the captured image 50 received from the image capturing device 20 to detect the hand 81 and fingers of the operator 80, and determines whether the operator 80 has made a predetermined gesture with the hand 81. In this embodiment, the gesture made by the operator 80 with the hand 81 is a gesture made by the direction or movement of the fingers of the hand 81. When the detection device 10 determines that the operator 80 has made a predetermined gesture with the hand 81, it transmits a control signal to the projector 30 and controls the projector 30 to perform an operation according to the detected gesture. This allows intuitive operations such as, for example, when the operator 80 makes a gesture of pointing one finger (e.g., index finger) to the right as viewed from the image capturing device 20, switching the projection image Im being projected by the projector 30 to the next projection image Im, and when making a gesture of pointing one finger to the left, switching the projection image Im to the previous projection image Im.

[0015] In this embodiment, a screen 40 is hung on a wall, and a projection image Im is projected (displayed) onto an image display surface 41 of the screen 40 by the imaging device 20. The image display surface 41 is the surface facing the imaging device 20 out of the front and back surfaces of the screen 40. In this embodiment, the area occupied by the image display surface 41 constitutes an image display area RD onto which the projection image Im is projected (displayed). The screen 40 corresponds to a "member constituting the image display area." Hereinafter, the direction perpendicular to the floor surface on which the operator 80 is standing and pointing vertically upward is defined as the Z direction, the direction parallel to the floor surface and parallel to the projection direction of the projector 30 as viewed from the Z direction is defined as the Y direction, and the direction perpendicular to the Y and Z directions is defined as the X direction. The direction from the projector 30 toward the operator 80 is defined as the positive direction for the Y direction, and the direction from the projector 30 toward the operator 80 is defined as the positive direction for the X direction.

[0016] In this embodiment, it is assumed that the operator 80 is standing in a position facing the screen 40 (a position on the image display surface 41 side (-Y direction side) of the screen 40) and is looking at the projection image Im projected on the screen 40. It is also assumed that the operator 80 is making a gesture with his left hand 81L to operate the projector 30. The imaging range R of the imaging device 20 includes at least the upper body of the operator 80 and the image display surface 41 of the screen 40.

[0017] <Configuration of information processing system> FIG. 2 is a block diagram showing the functional configuration of the detection device 10. As shown in FIG. The detection device 10 includes a CPU 11 (Central Processing Unit), a RAM 12 (Random Access Memory), a storage unit 13, an operation unit 14, a display unit 15, a communication unit 16, and a bus 17. The components of the detection device 10 are connected via the bus 17. In this embodiment, the detection device 10 is a notebook PC, but is not limited to this and may be, for example, a desktop PC, a smartphone, or a tablet terminal.

[0018] The CPU 11 is a processor that controls the operation of the detection device 10 by reading and executing the program 131 stored in the storage unit 13 and performing various arithmetic processing. The CPU 11 corresponds to a "processing unit." The detection device 10 may have multiple processors (multiple CPUs, etc.), and the multiple processes performed by the CPU 11 of this embodiment may be executed by the multiple processors. In this case, the multiple processors correspond to a "processing unit." In this case, the multiple processors may be involved in a common process, or the multiple processors may independently execute different processes in parallel.

[0019] The RAM 12 provides a working memory space for the CPU 11 and stores temporary data.

[0020] The storage unit 13 is a non-transitory recording medium readable by the CPU 11 as a computer, and stores a program 131 and various data. The storage unit 13 includes a non-volatile memory such as a hard disk drive (HDD) or a solid state drive (SSD). The program 131 is stored in the storage unit 13 in the form of a computer-readable program code. Data stored in the storage unit 13 includes captured image data 132 related to color images and depth images received from the imaging device 20.

[0021] The operation unit 14 has at least one of a touch panel superimposed on the display screen of the display unit 15, a physical button, a pointing device such as a mouse, and an input device such as a keyboard, and outputs operation information according to input operations on the input device to the CPU 11.

[0022] The display unit 15 includes a display device such as a liquid crystal display, and performs various displays on the display device in accordance with display control signals from the CPU 11.

[0023] The communication unit 16 is configured by a network card, a communication module, or the like, and transmits and receives data between the image capturing device 20 and the projector 30 in accordance with a predetermined communication standard.

[0024] The photographing device 20 shown in FIG. 1 includes a color camera. The color camera captures an image capture range R including the image display area RD of the screen 40, the operator 80, and the background thereof, and generates color image data relating to a two-dimensional color image of the image capture range R. Each pixel of the color image data contains color information such as R (red), G (green), and B (blue). The color camera of the image capture device 20 continuously captures images of the operator 80 positioned in front of the image capture device 20 and the screen 40 at a predetermined frame rate. The color image data generated by the color camera is recorded in the storage unit 13 of the detection device 10 as photographed image data 132 (see FIG. 2). In this embodiment, the above-mentioned color image corresponds to the "captured image obtained by capturing an image of the capture range."

[0025] 1 projects a projection image Im onto an image display surface 41 (i.e., an image display region RD) of a screen 40 by irradiating projection light with high directivity, the projection light having an intensity distribution according to image data of the projection image Im. Specifically, the projector 30 includes a light source, a display element such as a digital micromirror device (DMD) that adjusts the intensity distribution of the light output from the light source to form an optical image, and a projection lens group that focuses the optical image formed by the display element and projects it as a projection image Im. The projector 30 changes the projection image Im to be projected and changes settings related to the projection mode (brightness, color, etc.) according to a control signal transmitted from the detection device 10.

[0026] <Operation of information processing system> Next, the operation of the information processing system 1 will be described. The CPU 11 of the detection device 10 analyzes one or more captured images 50 captured by the imaging device 20, and determines whether or not an operator 80 appearing in the captured image 50 has made a predetermined gesture with a hand 81. When the CPU 11 determines that a gesture has been made with the hand 81, it transmits a control signal to the projector 30 to cause the projector 30 to perform an operation corresponding to the detected gesture.

[0027] Here, gestures made by the hand 81 include, for example, moving a finger in a certain direction (rightward, leftward, downward, upward, etc.) as viewed from the image capture device 20, moving the fingertip to draw a path of a predetermined shape (circular, etc.), varying the distance between the tips of two or more fingers, or bending and straightening the fingers. Each of these gestures is pre-associated with one operation of the projector 30. For example, a gesture of pointing a finger to the right may be associated with an operation of switching the currently projected projection image Im to the next projection image Im, and a gesture of pointing a finger to the left may be associated with an operation of switching the currently projected projection image Im to the previous projection image Im. In this case, the projection image Im can be switched to the next image / previous image by performing a gesture of pointing a finger to the right / left. Furthermore, a gesture of increasing / decreasing the distance between the tip of the thumb and the tip of the index finger may be associated with an operation of enlarging / reducing the projection image Im, respectively. These are examples of associations between gestures and operations of the projector 30, and any gesture can be associated with any operation of the projector 30. Furthermore, it may be possible to change the associations between gestures and operations of the projector 30 or to generate new associations in response to user operations on the operation unit 14.

[0028] In this way, when the projector 30 is operated by gestures of the hand 81 of the operator 80, it is important to accurately detect the hand 81 (and fingers) in the image captured by the image capturing device 20. If the hand 81 cannot be detected correctly, the gesture cannot be recognized correctly, and operability will be significantly reduced.

[0029] However, if the image display area RD of the screen 40 is included in the imaging range R of the imaging device 20, a projection image Im including a person's hand 81I may be projected in the image display area RD.

[0030] FIG. 3 is a diagram showing an example of a captured image 50 in which a projected image Im including a person's hand 81I is captured. The x-axis and y-axis shown in FIG. 3 are coordinate axes of a Cartesian coordinate system that represent the positions of pixels in the captured image 50. 3, when a projection image Im is captured in the captured image 50 and the projection image Im includes a human hand 81I, the hand 81 in the projection image Im may be erroneously detected as a detection target. If the hand 81 in the projection image Im is erroneously detected, a gesture may be erroneously detected, causing the projector 30 to perform an unintended operation.

[0031] Therefore, in this embodiment, based on the positional relationship between the hand 81 of the operator 80 and the image display area RD of the screen 40, the hand 81 in the projection image Im that is not making a gesture is excluded from the detection target, and the hand 81 of the operator 80 that is making a gesture is appropriately detected as the detection target.

[0032] 4 and 5, an operation of the CPU 11 of the detection device 10 to detect the hand 81 of the operator 80 and an operation of detecting a gesture by the hand 81 and controlling the operation of the projector 30 will be described. To realize the above operations, the CPU 11 executes a device control process shown in FIG. 4 and a hand detection process shown in FIG. 5.

[0033] FIG. 4 is a flowchart showing the control procedure of the device control process. The device control process is executed, for example, when the detection device 10, the image capture device 20, and the projector 30 are powered on and the projector 30 starts accepting gestures for operating the projector 30.

[0034] When the device control process is started, the CPU 11 transmits a control signal to the photographing device 20 to start photographing with the color camera (step S101). When photographing starts, the CPU 11 executes a hand detection process (step S102).

[0035] FIG. 5 is a flowchart showing a control procedure for the hand detection process. When the hand detection process is called, the CPU 11 acquires a captured image 50 (captured image data 132) capturing an image of the operator 80 and the hand 81 (step S201). The CPU 11 also extracts candidates for the hand region corresponding to the hand 81 (hereinafter simply referred to as "candidates for the hand 81") from the acquired captured image 50 (step S202). The process of extracting candidates for the hand 81 from the captured image 50 corresponds to the process of extracting an object from the captured image 50.

[0036] In the captured image 50 shown in Fig. 3, an operator 80 standing in front of the screen 40 is making a gesture of pointing the index finger of his left hand 81L to the left (-x direction). Also, a person appearing in the projection image Im is pointing the index finger of his hand 81I upward (-y direction). Therefore, when the captured image 50 shown in Fig. 3 is acquired, the CPU 11 extracts the left hand 81L of the operator 80 and the hand 81I in the projection image Im as candidates for the hand 81 in step S202.

[0037] The method for extracting candidates for the hand 81 from the captured image 50 in step S202 is not particularly limited, but the following method can be used, for example. First, a region corresponding to the skin color (the color of the hand 81) is extracted from the color image by performing color threshold processing based on the color information of the color image. Next, it is determined whether each extracted region includes a protruding part corresponding to a finger. Of the extracted regions, a region determined to include a protruding part corresponding to a finger is extracted as a candidate hand region corresponding to the hand 81.

[0038] The CPU 11 may generate a mask image representing the hand region corresponding to the extracted hand 81, and use the mask image data in subsequent processing. The mask image data is, for example, an image in which the pixel value of pixels corresponding to the hand region is set to "1" and the pixel value of pixels corresponding to regions other than the hand region is set to "0."

[0039] The CPU 11 determines whether or not a candidate for the hand 81 has been extracted in step S202 (step S203). If it is determined that a candidate for the hand 81 has been extracted ("YES" in step S203), the CPU 11 detects (specifies) the image display region RD of the screen 40 in the captured image 50 (step S204).

[0040] As shown in Fig. 3, the screen 40 of this embodiment has right-angled isosceles triangle markers 60, which represent the four corners of the screen 40, provided at each of the four corners of the image display surface 41. When the markers 60 are provided at predetermined positions on the screen 40 in this manner, the CPU 11 can detect the image display region RD based on the positions of the markers 60 in the captured image 50. In the example shown in Fig. 3, the CPU 11 identifies the positions of the vertices of the markers 60 that form right angles, and identifies a rectangular region representing the image display region RD based on the positions of the vertices of at least three of the markers 60. The shapes of the markers 60 and the positions of the markers 60 on the screen 40 are not limited to those shown in Fig. 3.

[0041] The method for identifying the image display region RD from the captured image 50 is not limited to the method using the marker 60 provided on the screen 40. For example, as shown in FIG. 6 , a projection image Im including a marker 60 at a predetermined position (e.g., one of the four corners) may be projected onto the image display area RD, and the image display area RD may be detected based on the position of the marker 60 in the captured image 50. This allows the range of the projection image Im to be accurately detected from the position of the marker 60, and the image display area RD to be detected. Furthermore, since this method allows the range of the projection image Im to be accurately detected, the range of the projection image Im may be used as the image display area RD instead of the image display surface 41 of the screen 40. To include the marker 60 in the projection image Im, the projection image Im may be projected using image data of the projection image Im including data of the marker 60, or the data of the marker 60 may be added to the image data when the image data of the projection image Im is transmitted from the detection device 10 to the projector 30, and the projection image Im may be projected using the image data to which the data of the marker 60 has been added. Furthermore, the boundary lines between objects included in the photographed image 50 may be extracted, and if there is a boundary line that forms a rectangle, the image display area RD may be specified based on the positions of the vertices of the rectangle. Furthermore, when the positional relationship between the image capturing device 20 and the screen 40 is determined, the range of the image display region RD in the captured image 50 may be specified in advance based on the positional relationship. That is, calibration may be performed in advance to specify the range of the image display region RD.

[0042] The CPU 11 determines whether or not the image display area RD has been detected (step S205), and if it is determined that the image display area RD has been detected (“YES” in step S205), it determines whether or not there is a candidate for the hand 81 within the image display area RD (step S206). Here, the CPU 11 determines that a hand 81 whose entire area is within the image display area RD is a candidate for the hand 81 within the image display area RD. Furthermore, when the area of ​​the hand 81 extends from the inside to the outside of the image display area RD (acrossing the boundary line of the image display area RD) in the captured image 50, like the right hand 81R of the operator 80 shown in FIG. 6, the CPU 11 determines that the hand 81 is outside the image display area RD. This is because when the hand 81 extends from the inside to the outside of the image display area RD, the hand 81 cannot be the hand 81I displayed in the image display area RD, and may be the hand 81 of the operator 80 making a gesture.

[0043] If it is determined that there are candidates for the hand 81 inside the image display area RD ("YES" in step S206), the CPU 11 excludes the candidates for the hand 81 inside the image display area RD from candidates to be detected (step S207).

[0044] The CPU 11 determines whether or not there are candidates for the hand 81 in the external area RE excluding the image display area RD of the captured image 50 (step S208). If it is determined that there are candidates for the hand 81 in the external area RE ("YES" in step S208), the CPU 11 detects the candidate for the hand 81 with the largest area among the candidates for the hand 81 in the external area RE as the hand 81 (detection target) for gesture determination (step S209).

[0045] Furthermore, if it is determined in step S205 that the image display area RD has not been detected ("NO" in step S205), or if it is determined in step S206 that there is no candidate for the hand 81 inside the image display area RD (step S206), then it is determined that there is a candidate for the hand 81 in the external area RE. Therefore, CPU 11 shifts the process to step S209, and detects the candidate for the hand 81 with the largest area among the candidates for the hand 81 in the external area RE as the hand 81 (detection target) for gesture discrimination. Note that if the image display area RD has not been detected in the captured image 50, the entire captured image 50 corresponds to the external area RE.

[0046] When step S209 is completed, the CPU 11 ends the hand detection process and returns the process to the device control process of FIG. Furthermore, if it is determined in step S203 that no candidate for the hand 81 has been extracted ("NO" in step S203), or if it is determined in step S208 that no candidate for the hand 81 exists in the external area RE ("NO" in step S208), the CPU 11 terminates the hand detection process without detecting the hand 81 for gesture discrimination, and returns the process to the device control process of Figure 4.

[0047] In step S209, the hand with the largest area is detected as the hand 81 for gesture determination, but one hand 81 for gesture determination may be determined by a method other than this. For example, an index value indicating the likelihood of the hand 81 being the hand 81 may be derived based on the shape of the fingers of the hand 81, and the hand 81 with the largest index value may be detected as the hand 81 for gesture determination. Also, in the flowchart of Figure 5, candidates for the hand 81 are first extracted (step S202), and then the image display area RD is detected (step S204), but this order is not limited to this, and the image display area RD may be detected first, and then candidates for the hand 81 may be extracted.

[0048] When the hand detection process (step S102) of FIG. 4 is completed, the CPU 11 determines whether or not a hand 81 for gesture determination has been detected in the hand detection process (step S103). If it is determined that the hand 81 has been detected ("YES" in step S103), the CPU 11 determines whether or not a gesture by the hand 81 of the operator 80 has been detected from the orientation of the fingers on the hand 81 or the movement of the positions of the fingertips across multiple frames of the captured image 50 (step S104). If it is determined that a gesture has been detected ("YES" in step S104), the CPU 11 transmits a control signal to the projector 30 to cause the projector 30 to perform an operation corresponding to the detected gesture (step S105). The projector 30 that has received the control signal performs an operation corresponding to the control signal.

[0049] When step S105 is completed, when it is determined in step S103 that the hand 81 has not been detected ("NO" in step S103), or when it is determined in step S104 that the gesture has not been detected ("NO" in step S104), the CPU 11 determines whether or not to end the acceptance of the gesture in the information processing system 1 (step S106). Here, the CPU 11 determines to end the acceptance of the gesture when, for example, an operation to turn off the power of the detection device 10, the image capture device 20, or the projector 30 has been performed.

[0050] If it is determined that the acceptance of the gesture should not be ended ("NO" in step S106), CPU 11 returns the process to step S102 and executes hand detection processing for detecting hand 81 based on the captured image captured in the next frame period. The loop processing of steps S102 to S106 is repeatedly executed, for example, at the frame rate of image capture by image capture device 20 (i.e., every time a captured image 50 is generated). If it is determined that the acceptance of gestures is to be ended ("YES" in step S106), the CPU 11 ends the device control process.

[0051] <Modification> Next, a modified example of the above embodiment will be described. This modified example differs from the above embodiment in the control procedure of the hand detection process. Below, differences from the above embodiment will be described, and a description of the configuration common to the above embodiment will be omitted.

[0052] FIG. 7 is a flowchart showing a control procedure for hand detection processing according to a modified example. In the hand detection process according to this modification, first, the image display area RD is detected (identified), and then candidates for the hand 81 are extracted in the area RE outside the image display area RD. By using such a procedure, it is possible to prevent the hand 81 inside the image display area RD, which will ultimately be excluded from the candidates, from being extracted from the beginning.

[0053] When the hand detection process according to this modified example is called, the CPU 11 acquires the captured image 50 (captured image data 132) capturing an image of the operator 80 and the hand 81 (step S301).

[0054] The CPU 11 detects (specifies) the image display region RD of the screen 40 in the captured image 50 (step S302). The processing content of step S302 is the same as step S204 in FIG.

[0055] The CPU 11 determines whether or not the image display area RD has been detected (step S303), and if it determines that the image display area RD has been detected ("YES" in step S303), it extracts candidates for the hand 81 in the external area RE of the image display area RD (step S304). The processing content of step S304 is the same as step S202 in Fig. 5, except that the target range to be extracted is the external area RE.

[0056] On the other hand, if it is determined that the image display area RD has not been detected ("NO" in step S303), the CPU 11 extracts candidates for the hand 81 from the entire captured image 50 (step S305). The processing content of step S305 is the same as step S202 in FIG. 5.

[0057] When step S304 or step S305 is completed, CPU 11 determines whether or not candidates for hand 81 have been extracted (step S306). If it is determined that candidates for hand 81 have been extracted ("YES" in step S306), CPU 11 detects the candidate for hand 81 with the largest area among the extracted candidates for hand 81 as the hand 81 for gesture determination (detection target) (step S307). Therefore, if candidates for hand 81 in the external region RE have been extracted in step S304, the hand 81 for gesture determination is detected from the candidates for hand 81 extracted in the external region RE.

[0058] When step S307 is completed, the CPU 11 ends the hand detection process and returns the process to the device control process of FIG. Also, in step S306, if it is determined that a candidate for the hand 81 has not been extracted ("NO" in step S306), the CPU 11 ends the hand detection process without detecting the hand 81 for gesture discrimination, and returns the process to the device control process of Figure 4.

[0059] <Effects> As described above, the detection method according to this embodiment is a detection method executed by the CPU 11 as a computer, which acquires a captured image 50 including at least a part of the image display area RD in which the projection image Im is displayed and at least a part of the hand 81 (object) of the operator 80, and detects, as a detection target, at least a part of the hand 81 extracted in the external area RE excluding the image display area RD of the captured image 50. As a result, even if a projection image Im including a person's hand 81I is displayed in the image display area RD in the captured image 50, the hand 81I can be excluded from the detection target, and the hand 81 of the operator 80 making a gesture can be appropriately detected as a detection target.

[0060] In addition, by identifying the image display area RD in the captured image 50 and detecting at least a portion of the hand 81 extracted in the external area RE of the captured image 50 excluding the identified image display area RD as the detection target, the hand 81 appearing in the image display area RD can be excluded from the detection target.

[0061] Furthermore, the image display area RD in the photographed image 50 is identified, the hand 81 is extracted in the photographed image 50, it is determined whether or not the extracted hand 81 is in an external area RE excluding the image display area RD in the photographed image 50, and at least a part of the hand 81 determined to be in the external area RE is detected as a detection target. In this way, by first extracting candidates for the hand 81 in the entire photographed image 50 and then determining whether or not the hand 81 is in the external area RE, it is possible to reduce the likelihood of missing detection of the hand 81 of the operator 80 making a gesture.

[0062] Furthermore, the detection method according to the modified example identifies an image display area RD in the captured image 50, extracts a hand 81 in an external area RE of the captured image 50 excluding the identified image display area RD, and detects at least a portion of the extracted hand 81 as a detection target. As a result, even if a projection image Im including a person's hand 81I is displayed in the image display area RD captured in the captured image 50, the hand 81I can be excluded from the detection target, and the hand 81 of the operator 80 making a gesture can be appropriately detected as a detection target. Furthermore, the hand 81 inside the image display area RD, which will ultimately be excluded from the candidates, can be prevented from being extracted from the beginning. Therefore, the process of extracting the hand 81 inside the image display area RD and the process of excluding the hand 81 extracted inside the image display area RD from the candidates can be omitted, thereby reducing the processing load on the CPU 11.

[0063] Furthermore, when multiple hands 81 are extracted, the hand 81 with the largest area among the multiple hands 81 is detected as the detection target. This makes it possible to appropriately detect, among the multiple hands 81, the hand 81 that is most likely to be the hand 81 of the operator 80 making the gesture as the detection target.

[0064] Furthermore, if the extracted hand 81 extends from the inside to the outside of the image display area RD in the captured image 50, the hand 81 is determined to be outside the image display area RD. If the hand 81 extends from the inside to the outside of the image display area RD, the hand 81 cannot be the hand 81I displayed in the image display area RD, and may be the hand 81 of the operator 80 making a gesture. Therefore, by determining that such a hand 81 is outside the image display area RD, the hand 81 can be appropriately detected as a detection target.

[0065] Furthermore, markers 60 are provided at predetermined positions on the screen 40 that constitutes the image display region RD, and the image display region RD is identified based on the positions of the markers 60 in the photographed image 50. This makes it possible to easily identify the image display region RD based on the photographed image 50 that does not include depth information.

[0066] Furthermore, the projection image Im displayed in the image display area RD may include a marker 60 at a predetermined position, and the image display area RD may be identified based on the position of the marker 60 in the captured image 50. This allows the image display area RD to be easily detected based on the captured image 50, which does not include depth information. Furthermore, since the range of the projection image Im can be accurately detected from the position of the marker 60, this range can also be considered as the image display area RD. Therefore, when the projection image Im is small compared to the screen 40, the range of the screen 40 on which the projection image Im is not projected can be treated as an external area RE of the image display area RD. Therefore, when the hand 81 of the operator 80 is captured in this range, the hand 81 can be appropriately detected as a detection target.

[0067] Furthermore, the detection device 10 according to this embodiment includes a CPU 11, which acquires a captured image 50 including at least a part of an image display area RD in which a projection image Im is displayed and at least a part of a hand 81 (object) of an operator 80, and detects at least a part of the hand 81 extracted from an external area RE excluding the image display area RD of the captured image 50 as a detection target. As a result, even if a projection image Im including a person's hand 81I is displayed in the image display area RD in the captured image 50, the hand 81I can be excluded from the detection target, and the hand 81 of the operator 80 making a gesture can be appropriately detected as a detection target.

[0068] Furthermore, the program 131 according to the present embodiment causes the CPU 11 as a computer to execute a process of acquiring a captured image 50 including at least a part of the image display area RD in which the projection image Im is displayed and at least a part of the hand 81 (object) of the operator 80, and a process of detecting, as a detection target, at least a part of the hand 81 extracted in an external area RE excluding the image display area RD of the captured image 50. As a result, even if a projection image Im including a person's hand 81I is displayed in the image display area RD in the captured image 50, the hand 81I can be excluded from the detection target, and the hand 81 of the operator 80 making a gesture can be appropriately detected as a detection target.

[0069] <Other> The above-described embodiments are merely examples of the detection method, detection device, and program according to the present invention, and the present invention is not limited to these. For example, in the above embodiment, the detection device 10, the image capture device 20, and the projector 30 (devices to be operated by gestures) are separate devices, but the present invention is not limited to this. For example, the detection device 10 and the image capturing device 20 may be integrated. As an example, the color camera of the image capturing device 20 may be incorporated into the bezel of the display unit 15 of the detection device 10. Furthermore, the detection device 10 and the device to be operated may be integrated. For example, the function of the detection device 10 may be incorporated into the projector 30 in the above embodiment, and the processing executed by the detection device 10 may be executed by a CPU (not shown) of the projector 30. In this case, the projector 30 corresponds to the "detection device," and the CPU of the projector 30 corresponds to the "processing unit." Furthermore, the image capturing device 20 and the device to be operated may be integrated together. For example, the color camera of the image capturing device 20 may be incorporated into the housing of the projector 30 in the above embodiment. Furthermore, the detection device 10, the image capturing device 20, and the device to be operated may all be integrated. For example, in a mode in which a color camera is incorporated into the bezel of the display unit 15 of the detection device 10 as the device to be operated, the operation of the detection device 10 may be controlled by gestures of the hand 81 (fingers) of the operator 80.

[0070] Furthermore, the operator 80 is not limited to a person, but may be a robot, an animal, or the like.

[0071] In the above embodiment, the gesture performed by the operator 80 with the hand 81 is a gesture performed by the direction or movement of the fingers of the hand 81, but is not limited to this and may be, for example, a gesture performed by the movement of the entire hand 81. Also, the gesture may be a gesture of touching the image display surface 41 with a fingertip.

[0072] Furthermore, the image capture device 20 may have a depth camera in addition to a color camera. The depth camera captures an image capture range R including the image display region RD of the screen 40, the operator 80, and the background thereof, and generates depth image data relating to a depth image including depth information of the image capture range R. Each pixel of the depth image includes depth information relating to the depth (distance from the depth camera to the distance measurement object) of the image display region RD, the operator 80, and background structures (hereinafter referred to as "distance measurement objects"). When the image capture device 20 has a color camera and a depth camera, the image display surface 41, i.e., the image display region RD, may be identified based on the depth information of the depth image. For example, a planar rectangular region with continuous depth in the depth image may be identified as the image display region RD. The depth camera may be, for example, one that detects distance using a time-of-flight (TOF) method or one that detects distance using a stereo method. The color camera and the depth camera may be capable of capturing an image of a range that includes at least the capturing range R, and the angle of view captured by the color camera may be different from the angle of view captured by the depth camera. In the capturing range R where the angle of view captured by the color camera and the angle of view captured by the depth camera overlap, it is preferable that the pixels of the color image correspond to the pixels of the depth image. This allows specifying an arbitrary pixel in the color image to identify the pixel corresponding to that pixel in the depth image. Therefore, depth information can be obtained for an arbitrary pixel in the color image.

[0073] Furthermore, in the above embodiment, an example was given in which the shooting range R includes the entire image display area RD of the screen 40, but this is not limited to this, and it may also be the case that only a portion of the image display area RD is included in the shooting range R, i.e., that only a portion of the image display area RD is captured in the captured image 50.

[0074] In the above description, an example has been disclosed in which an HDD or SSD of the storage unit 13 is used as a computer-readable medium for the program according to the present invention, but this is not limiting. Other computer-readable media may also be used, such as flash memory or CD-ROM. Furthermore, a carrier wave may also be used as a medium for providing data for the program according to the present invention via a communication line.

[0075] Furthermore, it goes without saying that the detailed configurations and operations of the components of the detection device 10, the image capture device 20, and the projector 30 in the above embodiment can be modified as appropriate without departing from the spirit of the present invention.

[0076] Although the embodiments of the present invention have been described, the scope of the present invention is not limited to the above-described embodiments, but includes the scope of the invention described in the claims and its equivalents. The inventions described in the claims originally attached to this application are as follows. The claim numbers described in the appendix are the same as those of the claims originally attached to this application. [Note] <Claim 1> 1. A computer-implemented detection method comprising: acquiring a captured image including at least a portion of an image display area in which an image is displayed and at least a portion of an object; detecting, as a detection target, at least a part of the object extracted in an external area excluding the image display area of ​​the captured image; Detection method. <Claim 2> Identifying the image display area in the captured image; detecting, as the detection target, at least a part of the object extracted in the external area excluding the specified image display area in the captured image; The detection method according to claim 1. <Claim 3> Identifying the image display area in the captured image; extracting the object from the captured image; determining whether the extracted object is in the external area of ​​the captured image excluding the image display area; detecting at least a part of the object determined to be in the external area as the detection target; The detection method according to claim 1. <Claim 4> Identifying the image display area in the captured image; extracting the object from an external area of ​​the captured image excluding the identified image display area; Detecting at least a part of the extracted object as the detection target; The detection method according to claim 1. <Claim 5> When a plurality of the objects are extracted, the object having the largest area among the plurality of objects is detected as the detection object. The detection method according to claim 1. <Claim 6> If the extracted object extends from the inside to the outside of the image display area in the captured image, the object is determined to be outside the image display area. The detection method according to claim 1. <Claim 7> a mark is provided at a predetermined position on a member constituting the image display area; identifying the image display area based on the position of the marker in the captured image; The detection method according to any one of claims 2 to 4. <Claim 8> the image displayed in the image display area includes a marker at a predetermined position; identifying the image display area based on the position of the marker in the captured image; The detection method according to any one of claims 2 to 4. <Claim 9> acquiring a captured image including at least a portion of an image display area in which an image is displayed and at least a portion of an object; detecting, as a detection target, at least a part of the object extracted in an external area excluding the image display area of ​​the captured image; A detection device comprising a processing unit. <Claim 10> On the computer, A process of acquiring a captured image including at least a portion of an image display area in which an image is displayed and at least a portion of an object; a process of detecting, as a detection target, at least a part of the object extracted in an external area excluding the image display area of ​​the captured image; A program that executes the following. [Explanation of symbols]

[0077] 1. Information Processing Systems 10. Detection Device 11 CPU (processing unit) 12 RAM 13 Storage section 131 Programs 132 Photographed image data 14 Control section 15 Display 16 Communications Department 17 Bus 20 Imaging equipment 30 Projector 40 screens 41 Image display surface 50 captured images 60 signs 80 Operator (subject) 81, 81L, 81R, 81I Hands (object, candidate object, object to be detected) Im Projected image (image) R Shooting range RD image display area RE external area

Claims

1. 1. A computer-implemented detection method comprising: acquiring a captured image including at least a portion of an image display area in which an image is displayed and at least a portion of an object; Detecting at least a part of the object extracted in an external area excluding the image display area of ​​the captured image as a detection target; If the extracted object extends from inside to outside the image display area in the captured image, the object is determined to be outside the image display area, and the object is detected as the detection object. Detection method.

2. Identifying the image display area in the captured image; detecting, as the detection target, at least a part of the object extracted in the external area excluding the specified image display area in the captured image; The detection method according to claim 1 .

3. Identifying the image display area in the captured image; extracting the object from the captured image; determining whether the extracted object is in the external area of ​​the captured image excluding the image display area; detecting at least a part of the object determined to be in the external area as the detection target; The detection method according to claim 1 .

4. Identifying the image display area in the captured image; extracting the object from an external area of ​​the captured image excluding the identified image display area; Detecting at least a part of the extracted object as the detection target; The detection method according to claim 1 .

5. When a plurality of the objects are extracted, the object having the largest area among the plurality of objects is detected as the detection object. The detection method according to claim 1 .

6. a mark is provided at a predetermined position on a member constituting the image display area; identifying the image display area based on the position of the marker in the captured image; The detection method according to any one of claims 2 to 4.

7. the image displayed in the image display area includes a marker at a predetermined position; identifying the image display area based on the position of the marker in the captured image; The detection method according to any one of claims 2 to 4.

8. acquiring a captured image including at least a portion of an image display area in which an image is displayed and at least a portion of an object; Detecting at least a part of the object extracted in an external area excluding the image display area of ​​the captured image as a detection target; If the extracted object extends from inside to outside the image display area in the captured image, the object is determined to be outside the image display area, and the object is detected as the detection object. A detection device comprising a processing unit.

9. On the computer, A process of acquiring a captured image including at least a portion of an image display area in which an image is displayed and at least a portion of an object; a process of detecting, as a detection target, at least a part of the object extracted in an external area excluding the image display area of ​​the captured image; Execute When the extracted object extends from the inside to the outside of the image display area in the captured image, the object is determined to be outside the image display area, and the object is detected as the detection target. program.

Images