Detection method, detection device, and program

Abstract

To provide a detection method, a detection apparatus, and a program, for detecting a projection part in a target more simply.SOLUTION: A detection method to be executed by a computer includes: extracting, in a captured image obtained by imaging a target, a target region corresponding to the target; extracting, from the target region, a candidate point which is a candidate of a representative point that represents a projection part of the target; setting, according to a region setting rule, a virtual graphic region related to a virtual graphic including the candidate point on a contour line or an inside thereof; and detecting, as the representative point, the candidate point if an overlapping degree of the set virtual graphic region and the target region satisfies a predetermined overlap condition.SELECTED DRAWING: Figure 12A

Description

Technical Field

[0001] The present invention relates to a detection method, a detection device, and a program.

Background Art

[0002] Conventionally, there is a technique for detecting an operator's gesture and controlling the operation of a device according to the detected gesture. In this technique, in a captured image obtained by capturing the operator, a specific part of the operator's body that makes a gesture (for example, a hand) is detected as a target. Further, a technique has also been proposed for detecting a protrusion in the target (for example, a finger on a hand) and detecting the direction of the protrusion and the position of the tip of the protrusion (for example, a fingertip) as elements of the gesture. As a method for detecting a protrusion, Patent Document 1 discloses a technique for calculating the curvature of a contour line at each point on the contour line of a hand and detecting a point where the calculated curvature is smaller than a predetermined value as the tip of a finger.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the above conventional technology, a huge amount of processing is required to calculate the curvature of the contour line at each point on the contour line. Further, when the resolution of the captured image is low or the distance to the target is far, the number of pixels corresponding to the protrusion in the captured image decreases, making it difficult to accurately detect the protrusion. Thus, the above conventional technology has a problem that it is difficult to simply and accurately detect a protrusion in a target.

[0005] The object of this invention is to provide a detection method, detection device, and program that can detect protrusions on an object more simply and accurately. [Means for solving the problem]

[0006] To solve the above problems, the detection method according to the present invention is A detection method performed by a computer, In the captured image obtained by photographing the object, the target region corresponding to the object is extracted. From the aforementioned target region, candidate points are extracted that are candidates for a representative point representing the protruding portion of the target. In accordance with the region setting rules, a virtual figure region relating to a virtual figure that includes the candidate points on or inside its contour line is set. If the degree of overlap between the set virtual geometric area and the target area satisfies predetermined overlap conditions, the candidate point is detected as the representative point.

[0007] To solve the above problems, the detection device according to the present invention In the captured image obtained by photographing the object, the target region corresponding to the object is extracted. From the aforementioned target region, candidate points are extracted that are candidates for a representative point representing the protruding portion of the target. In accordance with the region setting rules, a virtual figure region relating to a virtual figure that includes the candidate points on or inside its contour line is set. If the degree of overlap between the set virtual geometric area and the target area satisfies predetermined overlap conditions, the candidate point is detected as the representative point. It is equipped with a processing unit.

[0008] To solve the above problems, the program according to the present invention On the computer, A process to extract the target region corresponding to the target in the captured image obtained by photographing the target, A process of extracting candidate points from the target region that are candidates for representative points representing the protruding part of the target, A process of setting a virtual graphic area related to a virtual graphic that includes the candidate point on or inside the contour line according to the area setting rule. A process of detecting the candidate point as the representative point when the degree of overlap between the set virtual graphic area and the target area satisfies a predetermined overlap condition. To cause the execution.

Effect of the Invention

[0009] According to the present invention, the protrusion in the target can be detected more simply and accurately.

Brief Description of the Drawings

[0010] [Figure 1] It is a schematic diagram showing the configuration of an information processing system. [Figure 2] It is a block diagram showing the functional configuration of a detection device. [Figure 3] It is a flowchart showing the control procedure of equipment control processing. [Figure 4] It is a flowchart showing the control procedure of finger detection processing. [Figure 5] It is a diagram showing an example of a mask image. [Figure 6] It is a diagram for explaining a method of extracting a representative convex hull point from a plurality of convex hull points in a finger area. [Figure 7A] It is a diagram showing the extracted representative convex hull point. [Figure 7B] It is a diagram showing the extracted candidate point. [Figure 8] It is a flowchart showing the control procedure of tip condition discrimination processing. [Figure 9] It is a diagram showing an example of a discrimination contour point and a virtual line segment. [Figure 10] It is a diagram showing angles θa to θc. [Figure 11] It is a flowchart showing the control procedure of area condition discrimination processing. [Figure 12A] It is a diagram for explaining a method of discriminating an area condition. [Figure 12B] It is a diagram showing the extracted representative point. [Figure 13] This is a diagram showing the detection result of representative points when a hand with fingers spread is photographed.

Embodiments for Carrying out the Invention

[0011] Hereinafter, embodiments of the present invention will be described based on the drawings.

[0012] <Overview of the Information Processing System> FIG. 1 is a schematic diagram showing the configuration of an information processing system 1 according to this embodiment. The information processing system 1 includes a detection device 10, a photographing device 20, and a projector 30. The detection device 10 is communicatively connected to the photographing device 20 and the projector 30 wirelessly or by wire, and can transmit and receive data such as control signals and image data between the photographing device 20 and the projector 30.

[0013] The detection device 10 of the information processing system 1 is an information processing device that detects gestures made by the operator 80 (subject) with their hand 81 (object) and controls the operation of the projector 30 (projection of the projected image Im, operation to change various settings, etc.) according to the detected gesture. Specifically, the shooting device 20 photographs the operator 80 and the operator 80's hand 81 located in front of the shooting device 20 and transmits the image data of the captured image Im to the detection device 10. The detection device 10 analyzes the image data received from the shooting device 20 to detect the fingers 82 (protruding parts) of the operator 80's hand 81 and determines whether the operator 80 has made a predetermined gesture with their fingers 82. When the detection device 10 determines that the operator 80 has made a predetermined gesture with their fingers 82, it transmits a control signal to the projector 30 and controls the projector 30 to perform an operation corresponding to the detected gesture. This allows for intuitive operation, such as when operator 80 makes a gesture of pointing one finger 82 (for example, an index finger) to the right towards the projected image Im, thereby switching the projected image Im currently projected by the projector 30 to the next projected image Im, and when making a gesture of pointing one finger 82 to the left, it switches the projected image Im back to the previous projected image Im.

[0014] <Configuration of the information processing system> Figure 2 is a block diagram showing the functional configuration of the detection device 10. The detection device 10 includes a CPU 11 (Central Processing Unit), RAM 12 (Random Access Memory), a storage unit 13, an operation unit 14, a display unit 15, a communication unit 16, a bus 17, and the like. Each part of the detection device 10 is connected via the bus 17. In this embodiment, the detection device 10 is a notebook PC, but is not limited to this, and may be, for example, a stationary PC, a smartphone, or a tablet terminal.

[0015] The CPU 11 is a processor that controls the operation of the detection device 10 by reading and executing the program 131 stored in the memory unit 13 and performing various arithmetic operations. The CPU 11 corresponds to the "processing unit". The detection device 10 may have multiple processors (multiple CPUs, etc.), and the multiple processes that the CPU 11 in this embodiment performs may be performed by these multiple processors. In this case, the multiple processors correspond to the "processing units". In this case, the multiple processors may be involved in common processing, or the multiple processors may independently execute different processes in parallel.

[0016] RAM12 provides CPU11 with a working memory space and stores temporary data.

[0017] The storage unit 13 is a non-temporary recording medium readable by the CPU 11 as a computer, and stores the program 131 and various data. The storage unit 13 includes non-volatile memory such as an HDD (Hard Disk Drive) or SSD (Solid State Drive). The program 131 is stored in the storage unit 13 in the form of program code that can be read by the computer. The data stored in the storage unit 13 includes captured image data 132 relating to color images and depth images received from the imaging device 20, and mask image data 133 relating to the mask image 40 generated by the finger detection process described later.

[0018] The operation unit 14 has at least one of the following: a touch panel superimposed on the display screen of the display unit 15, physical buttons, a pointing device such as a mouse, and an input device such as a keyboard, and outputs operation information to the CPU 11 in response to input operations on the input device.

[0019] The display unit 15 is equipped with a display device such as a liquid crystal display, and displays various information on the display device according to the display control signal from the CPU 11.

[0020] The communication unit 16 is composed of a network card or communication module, and transmits and receives data between the imaging device 20 and the projector 30 according to a predetermined communication standard.

[0021] The imaging device 20 shown in Figure 1 includes a color camera 21 and a depth camera 22. The color camera 21 captures the shooting range including the operator 80 and their background, and generates color image data relating to a two-dimensional color image of the shooting range. Each pixel of the color image data contains color information such as R (red), G (green), and B (blue). The depth camera 22 captures the shooting range including the operator 80 and their background, and generates depth image data relating to a depth image that includes depth information of the shooting range. Each pixel in the depth image contains depth information relating to the depth of the operator 80 and the background structures (hereinafter referred to as "distance measurement targets") (the distance from the depth camera 22 to the distance measurement targets). As the depth camera 22, for example, one that detects distance using the TOF (Time Of Flight) method or one that detects distance using the stereo method can be used. The color image data generated by the color camera 21 and the depth image data generated by the depth camera are recorded as captured image data 132 (see Figure 2) in the storage unit 13 of the detection device 10. In this embodiment, the above-mentioned color image and depth image correspond to "images obtained by photographing the subject."

[0022] The color camera 21 and depth camera 22 of the imaging device 20 continuously photograph the operator 80, who is positioned in front of the imaging device 20, at a predetermined frame rate. In the imaging device 20 shown in Figure 1, the color camera 21 and depth camera 22 are integrated, but the device is not limited to this configuration as long as each camera is capable of photographing the operator 80. For example, the color camera 21 and depth camera 22 may be separate. Furthermore, the color camera 21 and the depth camera 22 only need to be able to capture an area that includes at least the operator 80, and the field of view captured by the color camera 21 and the field of view captured by the depth camera 22 may be different. In addition, in the area where the field of view captured by the color camera 21 and the field of view captured by the depth camera 22 overlap, a correspondence is made between the pixels of the color image and the pixels of the depth image. This makes it possible to identify the corresponding pixel in the depth image when any pixel in the color image is specified. Thus, depth information can be obtained for any pixel in the color image.

[0023] The projector 30 shown in Figure 1 projects (forms) a projected image Im onto a projection surface by irradiating projection light with a high directivity that has an intensity distribution corresponding to the image data of the projected image Im. More specifically, the projector 30 includes a light source, a display element such as a digital micromirror element (DMD) that adjusts the intensity distribution of the light output from the light source to form a light image, and a group of projection lenses that focus the light image formed by the display element and project it as a projected image Im. The projector 30 changes the projected image Im to be projected and changes settings related to the projection mode (brightness, color tone, etc.) according to a control signal transmitted from the detection device 10.

[0024] <Operation of the Information Processing System> Next, the operation of the information processing system 1 will be described. The CPU 11 of the detection device 10 analyzes one or more color images and depth images captured by the imaging device 20 to determine whether the operator 80 shown in the image has performed a predetermined gesture with the fingers 82 of the hand 81. If the CPU 11 determines that a gesture has been performed with the fingers 82, it sends a control signal to the projector 30 to cause the projector 30 to perform an action corresponding to the detected gesture.

[0025] Here, gestures made by the finger 82 include, for example, moving the finger 82 in a certain direction (right, left, down, and up, etc.) as seen from the perspective of the shooting device 20, moving the tip of the finger 82 to trace a predetermined shape (circle, etc.), changing the distance between the tips of two or more fingers 82, or bending and straightening the finger 82. Each of these gestures is pre-associated with one operation of the projector 30. For example, a gesture of pointing the finger 82 to the right may be associated with the operation of switching the projected image Im to the next projected image Im, and a gesture of pointing the finger 82 to the left may be associated with the operation of switching the projected image Im to the previous projected image Im. In this case, the projected image can be switched to the next image / previous image by making a gesture of pointing the finger 82 to the right / left. Alternatively, gestures of increasing / decreasing the distance between the tip of the thumb and the tip of the index finger may be associated with operations of enlarging / reducing the projected image Im, respectively. These are just examples of how gestures can be associated with the actions of the projector 30, and any gesture can be associated with any action of the projector 30. Furthermore, it may be possible to change the association between gestures and the actions of the projector 30, or to generate new associations, in response to user operations on the control unit 14.

[0026] Thus, when the projector 30 is operated by the operator 80 using the gesture of their finger 82, it is important that the camera 20 accurately detects the finger 82 (and the tip of the finger 82) in the image captured. If the finger 82 cannot be detected correctly, the gesture cannot be recognized correctly, and the operability will be severely reduced.

[0027] The general procedure for detecting the finger 82 in this embodiment is as follows: First, in the captured images of the operator 80 and the hand 81, the hand region 41 (target region) corresponding to the hand 81 (see Figure 5) is extracted. Next, candidate points 60 (see Figure 7B) are extracted from the hand region 41 to represent a representative point 70 (in this embodiment, a point in the hand region 41 corresponding to the tip of the finger 82; see Figure 12B) that represents the finger 82. The extraction of candidate points 60 is done by first extracting multiple convex hull points 51 (see Figure 6) from the contour points 44 (see Figure 9) of the hand region 41, then extracting multiple representative convex hull points 50 (see Figures 6 and 7) that represent the multiple convex hull points 51, and finally selecting the representative convex hull point 50 that satisfies the tip condition described later as candidate point 60. Next, a virtual figure region RS (see Figure 12A) related to the virtual figure S is set for each of the multiple candidate points 60, and if the degree of overlap between the virtual figure region RS and the hand region 41 satisfies a predetermined overlap condition, that candidate point 60 is detected as a representative point 70 representing the finger 82. The overlap condition in this embodiment is an area condition that is satisfied when the ratio of the area of ​​the overlap region RD (see Figure 12A) with the hand region 41 to the area of ​​the virtual figure region RS is less than or equal to a standard value.

[0028] The following describes how the CPU 11 of the detection device 10 detects gestures made by the operator 80's finger 82 and controls the operation of the projector 30, with reference to Figures 3 to 13. To achieve the above operation, the CPU 11 executes the device control process shown in Figure 3, the finger detection process shown in Figure 4, the tip condition determination process shown in Figure 8, and the area condition determination process shown in Figure 11.

[0029] Figure 3 is a flowchart showing the control procedure for the device control process. The device control process is executed, for example, when the power to the detection device 10, the imaging device 20, and the projector 30 is turned on, and when the system starts accepting gestures to operate the projector 30.

[0030] When the device control process starts, the CPU 11 sends a control signal to the imaging device 20 to start imaging with the color camera 21 and the depth camera 22 (step S101). Once imaging starts, the CPU 11 performs finger detection processing (step S102).

[0031] Figure 4 is a flowchart showing the control procedure for the finger detection process. When the finger detection process is called, the CPU 11 acquires captured image data 132 (image data of the color image and image data of the depth image) of the captured image Im taken by the color camera 21 and the depth camera 22 (step S201).

[0032] The CPU 11 extracts the hand region 41 corresponding to the hand 81 from the captured image and generates a mask image 40 representing the hand region 41 (step S202). For example, the CPU 11 extracts the region corresponding to the skin color (the color of the hand 81) from the color image by performing threshold processing on the color information of the color image. The CPU 11 also extracts the region belonging to the depth range corresponding to the position of the hand 81 by performing threshold processing on the depth information of the depth image. The CPU 11 then extracts the overlapping region of these areas as the hand region 41. This method is just one example; the hand region 41 can be extracted using any method that utilizes at least one of the color image and the depth image. For example, an extraction method that dynamically extracts a region based on the difference with the background may be used. Furthermore, the CPU 11 generates a binary mask image 40 by setting the pixel value of pixels corresponding to the hand region 41 to "1" and the pixel value of pixels corresponding to regions other than the hand region 41 to "0". The image data of the mask image 40 is stored in the mask image data 133 of the storage unit 13.

[0033] Figure 5 shows an example of mask image 40. In the mask image 40 shown in Figure 5, the hand region 41 corresponding to the operator's hand 81 is extracted. Here, the hand 81 is photographed with one finger 82 (index finger) extended, and the hand region 41 includes the finger region 42 corresponding to this finger 82.

[0034] The CPU 11 acquires contour points 44 (see Figure 9) located on the contour line of the extracted hand region 41 (step S203). The contour line of the hand region 41 corresponds to the boundary line between the hand region 41, where the pixel value is "1" in the mask image 40, and the external region, where the pixel value is "0". The CPU 11 acquires pixels located on this boundary line as contour points 44.

[0035] The CPU 11 extracts multiple convex hull points 51 (see Figure 6) from the multiple contour points 44 obtained (step S204), and further extracts multiple representative convex hull points 50 that represent the multiple convex hull points 51 (step S205). The multiple convex hull points 51 are the vertices of the smallest convex polygon that encloses the hand region 41. Here, a convex polygon is a polygon in which the interior angles of all vertices are less than 180 degrees, and the vertices of the convex polygon are selected from the multiple contour points 44.

[0036] Figure 6 illustrates a method for extracting a representative hull point 50 from multiple hull points 51 in the finger region 42. Figure 7A shows the extracted representative convex hull points 50. As shown in Figure 6, when setting the above convex polygon to encompass the hand region 41, multiple convex hulls 51 may be extracted locally in parts that include curves with a small radius of curvature, such as the tips of the finger regions 42 (the part corresponding to the tips of the fingers 82). For this reason, a representative convex hull 50 is extracted to represent two or more convex hulls 51 in the finger region 42, so that only one representative convex hull 50 is set for the tip of one finger region 42. Specifically, as shown in the tip of the finger region 42 in Figure 6, if multiple convex hulls 51 include a group of convex hulls 51g consisting of two or more convex hulls 51 distributed within a reference distance r from a certain convex hull 51, the convex hull 51 furthest from the centroid G of the hand region 41 among the group of convex hulls 51g is extracted as the representative convex hull 50. On the other hand, as shown in the base of the finger region 42 in Figure 6, if there are no other hull points 51 within a reference distance r from a certain hull point 51, that hull point 51 is extracted as a representative hull point 50. By using this method, by appropriately setting the reference distance r, one representative hull point 50 can be extracted at the tip of one finger 82, as shown in Figures 6 and 7A.

[0037] Returning to Figure 4, once the representative hull point 50 is extracted in step S205, the CPU 11 executes the tip condition determination process (step S206). Figure 8 is a flowchart showing the control procedure for the advanced condition discrimination process. The tip condition determination process determines whether each of the extracted representative convex hull points 50 satisfies the tip condition that is met when it is located at the tip of a protruding shape. Here, a protruding shape refers to the shape of a part that protrudes from another part in a peninsula-like manner, such as the vicinity of the tip of a finger 82 (protruding part). From another perspective, a protruding shape refers to a part in which the average radius of curvature of the contour line of the protruding shape is less than or equal to a predetermined value. By extracting representative convex hull points 50 that satisfy the tip condition from the multiple representative convex hull points 50, it is possible to exclude representative convex hull points 50 that cannot correspond to the tip of the finger 82 and extract candidate points 60 that represent the tip of the finger 82.

[0038] When the tip condition determination process is called, the CPU 11 assigns 1 to the variable m, which represents the ordinal number of the representative convex hull point 50 (step S301). The CPU 11 then executes the following steps S302 to S306 for the m-th representative convex hull point 50 to determine whether the m-th representative convex hull point 50 satisfies the tip condition.

[0039] First, the CPU 11 selects a pair of discriminant contour points 44a, a pair of discriminant contour points 44b, and a pair of discriminant contour points 44c that enclose the m-th representative convex hull point 50 (step S302). The CPU 11 also derives two virtual line segments La connecting the m-th representative convex hull point 50 with one of the two distinct discriminant contour points 44a. Similarly, the CPU 11 derives two virtual line segments Lb connecting the m-th representative convex hull point 50 with one of the two distinct discriminant contour points 44b, and derives two virtual line segments Lc connecting the m-th representative convex hull point 50 with one of the two distinct discriminant contour points 44c (step S303).

[0040] Figure 9 shows examples of the discriminative contour points 44a to 44c and the virtual line segment La to Lc. Figure 9 shows contour points 44 on the contour line 43 near the tip of the finger region 42, and discriminant contour points 44a to 44c are selected from among these contour points 44. Here, discriminant contour points 44a to 44c are selected for every three contour points 44 from the contour point 44 that is the representative convex hull point 50. Specifically, the contour point 44 located between the contour point 44 that is the representative convex hull point 50 and two contour points 44 is selected as discriminant contour point 44a, the contour point 44 located between the discriminant contour point 44a and two contour points 44 is selected as discriminant contour point 44b, and the contour point 44 located between the discriminant contour point 44b and two contour points 44 is selected as discriminant contour point 44c. These three sets of discriminant contour points 44a to 44c correspond to "at least two sets of the aforementioned contour points whose average distances from the representative convex hull point are different from each other." The spacing between the representative hull point 50 and the discriminative contour points 44a to 44c (the number of contour points 44 placed between them) may be appropriately changed depending on the size of the hand area 41 (the distance from the imaging device 20 to the hand 81), etc.

[0041] Returning to Figure 8, when step S303 is completed, the CPU 11 derives the angles formed by the virtual line segments La to Lc for each of the three sets of discriminative contour points 44a to 44c. That is, as shown in Figure 10, the CPU 11 derives the angle θa formed by the two virtual line segments La, the angle θb formed by the two virtual line segments Lb, and the angle θc formed by the two virtual line segments Lc (step S304).

[0042] The CPU 11 determines whether angles θa to θc satisfy the relationship θa > θb > θc (step S305). In other words, the CPU 11 determines whether the angle formed by the virtual line segment is smaller for sets of discrimination contour points that have a longer average distance from the representative convex hull point 50. When the representative convex hull point 50 is located at the tip of the protruding shape, the angle formed by the virtual line segment passing through discrimination contour points that have a longer average distance from the representative convex hull point 50 is smaller, so the relationship θa > θb > θc is satisfied. On the other hand, when the representative convex hull point 50 is located at a location other than the tip of the protruding shape, the angle θb may be larger than the angle θa, or the angle θc may be larger than the angle θb, so the relationship θa > θb > θc is not satisfied.

[0043] If it is determined that the relationship θa > θb > θc is satisfied ("YES" in step S305), the CPU 11 determines that the m-th representative convex hull point 50 satisfies the tip condition (step S306). In the example shown in Figure 10, since the relationship θa > θb > θc is satisfied, it is determined that the representative convex hull point 50 satisfies the tip condition, that is, it is located at the tip of the protruding shape.

[0044] If step S306 is completed, or if it is determined that the relationship θa>θb>θc is not satisfied ("NO" in step S305), the CPU 11 determines whether the value of variable m matches the number M of representative convex hull points 50 extracted in step S205 of Figure 4 (step S307). If it is determined that the value of variable m does not match the number M of representative convex hull points 50 ("NO" in step S307), the CPU 11 assigns m+1 to variable m (step S308) and returns to step S302.

[0045] If it is determined that the value of variable m matches the number M of representative convex hull points 50 (YES in step S307), the CPU 11 terminates the tip condition determination process and returns the process to the finger detection process shown in Figure 4.

[0046] In the above, the tip condition was determined by whether or not the relationship θa > θb > θc is satisfied, but the method of determining the tip condition is not limited to this. For example, a discriminant formula may be set individually for angles θa to θc. For example, if angles θa to θc each satisfy the following discriminant formulas (1) to (3), it may be determined that the representative convex hull point 50 satisfies the tip condition. θa_min < θa < θa_max …(1) θb_min < θb < θb_max …(2) θc_min < θc < θc_max …(3) Furthermore, although three sets of contour points for discrimination were selected above, the number of contour points for discrimination may be two or fewer, or four or more. Using two or fewer sets simplifies the processing related to discriminating the tip condition. Using four or more sets improves the accuracy of tip discrimination.

[0047] When the tip condition determination process in Figure 4 (step S206) is completed, if there is a representative convex hull point 50 that satisfies the tip condition ("YES" in step S207), the CPU 11 extracts the representative convex hull point 50 that has been determined to satisfy the tip condition as a candidate point 60 for the representative point 70 (step S208). In this embodiment, it is assumed that five candidate points 60 shown in Figure 7B have been extracted from the eight representative convex hull points 50 shown in Figure 7A. Once the candidate points 60 have been extracted, the CPU 11 executes the area condition determination process (step S209).

[0048] Figure 11 is a flowchart showing the control procedure for the area condition determination process. The area condition determination process determines whether each of the extracted candidate points 60 is a representative point 70 (in this embodiment, the tip of the finger 82) that represents the finger 82 (finger region 42).

[0049] When the area condition determination process is called, the CPU 11 assigns 1 to the variable n, which represents the ordinal number of the candidate point 60 (step S401). The CPU 11 determines whether the nth candidate point 60 satisfies the area condition by executing the following steps S402 to S406 for the nth candidate point 60. Figure 12A is a diagram illustrating the method for determining the area condition.

[0050] First, the CPU 11 sets a virtual figure region RS relating to a virtual figure S that overlaps with at least a part of the hand region 41 and includes the candidate point 60 on its contour line, according to predetermined region setting rules. In this embodiment, the CPU 11 sets the virtual figure region RS according to the region setting rules such that the contour line of the virtual figure S passes through the centroid G of the hand region 41 and the candidate point 60. More specifically, the CPU 11 derives a circular virtual figure S whose diameter is the line segment D connecting the nth candidate point 60 and the centroid G of the hand region 41, and sets the virtual figure region RS enclosed by the virtual figure S (step S402). Here, a circle with line segment D as its diameter is used as an example of the virtual figure S, but it is not limited to this. For example, the virtual figure S may be a rectangle with line segment D as its diagonal, or any other figure (polygon, ellipse, etc.) whose contour line passes through the centroid G and the candidate point 60.

[0051] The CPU 11 derives the area of ​​the overlapping region RD that overlaps with the hand region 41 within the virtual geometric region RS (step S403). The overlapping region RD is the area that is hatched in Figure 12A.

[0052] The CPU 11 derives the ratio of the area of ​​the overlapping area RD to the area of ​​the virtual geometric area RS (hereinafter referred to as the "area overlap rate") (step S404). This area overlap rate tends to be smaller when the corresponding candidate point 60 is located at the tip of the finger area 42, and larger when the candidate point 60 is located anywhere other than the tip of the finger area 42.

[0053] The CPU 11 determines whether the derived area overlap rate is less than or equal to a predetermined reference value (step S405). The reference value is pre-set and stored in the storage unit 13 such that the area overlap rate is less than or equal to the reference value when the candidate point 60 is located at the tip of the finger area 42, and the area overlap rate is greater than the reference value when the candidate point 60 is located anywhere other than the tip of the finger area 42. In this embodiment, the reference value is set to 70%. However, this is an example and can be changed as appropriate. For example, by making the reference value smaller, the possibility of parts other than the finger 82 being mistakenly detected as the finger 82 can be reduced, and by making the reference value larger, the occurrence of missed detection of the finger 82 can be suppressed. The reference value is set to a value greater than 0%. If it is determined that the area overlap rate is below the standard value (YES in step S405), the CPU 11 determines that the nth candidate point 60 satisfies the area condition related to the small area (step S406).

[0054] If step S406 is completed, or if it is determined that the area overlap rate is greater than the reference value ("NO" in step S405), the CPU 11 determines whether the value of variable n matches the number N of candidate points 60 extracted in step S208 of Figure 4 (step S407). If it is determined that the value of variable n does not match the number M of candidate points 60 ("NO" in step S407), the CPU 11 assigns n+1 to variable n (step S408) and returns to step S402. The processes in steps S402 to S406 are executed for each candidate point 60, thereby deriving the area overlap rate for all candidate points 60 and determining whether the area condition is met. In this embodiment, as shown in Figure 12A, the area overlap rates for the five candidate points 60 are derived as 39%, 91%, 89%, 92%, and 93%. Of these, one candidate point 60 with an area overlap rate of 39% is determined to satisfy the area condition (the area overlap rate is less than or equal to the standard value (70%)) and is extracted as the representative point 70. Figure 12B shows the extracted representative point 70. If it is determined that the value of variable n matches the number N of candidate points 60 (YES in step S307), the CPU 11 terminates the tip condition determination process and returns to the finger detection process shown in Figure 4.

[0055] In the above example, the virtual shape region RS was set so that candidate point 60 is included in the outline of the virtual shape S. Alternatively, the virtual shape region RS may be set so that candidate point 60 is included inside the virtual shape S. For example, the virtual shape S may be a figure (such as a circle) centered on candidate point 60. In this case as well, the area overlap rate can be derived and the area condition can be determined in the same way as above.

[0056] When the area condition determination process in step S209 of Figure 4 is completed, the CPU 11 determines whether or not there are candidate points 60 that satisfy the area condition (step S210). If it is determined that there are candidate points 60 that satisfy the area condition ("YES" in step S211), the CPU 11 detects a specified number or less of candidate points 60 that satisfy the area condition, in order of smallest area overlap rate, as representative points 70 representing the tip of the finger 82 (step S211). In this embodiment, one candidate point 60 with an area overlap rate of 39%, as shown in Figure 12A, is detected as the representative point 70. This representative point 70 represents the tip of the finger 82 (index finger).

[0057] Figure 13 shows the detection results of the representative point 70 when a hand 81 with fingers 82 spread out is photographed. In Figure 13, seven candidate points 60 are extracted, with area overlap rates of 51%, 53%, 54%, 53%, 59%, 81%, and 83%, respectively. Of these, five candidate points 60 with area overlap rates of 51%, 53%, 54%, 53%, and 59% are determined to satisfy the area condition (area overlap rate is below the standard value (70%)) and are extracted as representative points 70. In this way, if there are multiple candidate points 60 that satisfy the area condition, the CPU 11 detects candidate points 60 with an area overlap rate of as small as 5, up to a specified number, as representative points 70. Here, the specified number is set to "5" or less, which is the number of fingers in a human, when the "target" is a hand 81 and the "protrusion" is a finger 82. If the specified number is 5, even if there are six or more candidate points 60 that satisfy the area condition, five candidate points 60 with an area overlap rate of as small as 5 are detected as representative points 70. In the example shown in Figure 13, five candidate points 60 located at the tips of five fingers 82 (finger regions 42) are detected as representative points 70. Alternatively, the number of fingers 82 targeted for gesture detection may be limited to one or two by setting the specified number to 1 or 2.

[0058] When step S211 in Figure 4 is completed, the CPU 11 terminates the finger detection process and returns to the device control process in Figure 3. Also, if it is determined in step S207 that there is no representative hull point 50 that satisfies the tip condition ("NO" in step S207), or if it is determined in step S210 that there is no candidate point 60 that satisfies the area condition ("NO" in step S210), the CPU 11 terminates the finger detection process without detecting the representative point 70 and returns to the device control process in Figure 3.

[0059] When the finger detection process in Figure 3 (step S102) is completed, the CPU 11 determines whether or not a representative point 70 representing the tip of the finger 82 was detected in the finger detection process (step S103). If it is determined that the representative point 70 was detected ("YES" in step S103), the CPU 11 determines the position of the tip of the finger 82 and the orientation of the finger 82 from the position of the representative point 70 in the mask image 40 (step S104). The position of the finger tip is the position of the representative point 70 in the mask image 40. The orientation of the finger 82 can be determined based on the positional relationship between the center of gravity G of the hand region 41 and the representative point 70. That is, if the representative point 70 is located within a predetermined angular range in the right, left, up, or down directions centered on the center of gravity G, it can be determined that the finger 82 is pointing to the right, left, up, or down, respectively.

[0060] The CPU 11 determines whether or not it has detected a gesture by the operator 80's finger 82 based on the orientation of the finger 82 or the movement of the tip of the finger 82 in the mask image 40 across multiple frames (step S105). If it is determined that a gesture has been detected ("YES" in step S105), the CPU 11 sends a control signal to the projector 30 to perform an action corresponding to the detected gesture (step S106). Upon receiving the control signal, the projector 30 performs an action corresponding to the control signal.

[0061] If step S106 is completed, if it is determined in step S103 that the representative point 70 has not been detected ("NO" in step S103), or if it is determined in step S105 that no gesture has been detected ("NO" in step S105), the CPU 11 determines whether or not to terminate the acceptance of gestures in the information processing system 1 (step S107). Here, the CPU 11 determines to terminate the acceptance of gestures if, for example, the power of the detection device 10, the imaging device 20, or the projector 30 has been turned off.

[0062] If it is determined that the gesture reception should not be terminated ("NO" in step S107), the CPU 11 returns to step S102 and executes finger detection processing to detect finger 82 based on the captured image taken in the next frame period. The loop processing of steps S102 to S107 is executed repeatedly, for example, at the frame rate of capture by the color camera 21 and depth camera 22 (i.e., each time a color image and a depth image are generated). If it is determined that the gesture reception has ended (YES in step S107), the CPU 11 terminates the device control process.

[0063] <Effects> As described above, the detection method according to this embodiment is a detection method executed by a CPU 11 as a computer, which extracts a hand region 41 corresponding to the hand 81 from an image obtained by photographing the hand 81, extracts candidate points 60 that are candidates for representative points 70 representing the fingers 82 of the hand 81 from the hand region 41, sets a virtual figure region RS relating to a virtual figure S that includes the candidate points 60 on or inside its contour line according to region setting rules, and detects the candidate points 60 as representative points 70 when the degree of overlap between the set virtual figure region RS and the hand region 41 satisfies predetermined overlap conditions. This allows for the proper detection of finger 82 (representative point 70 that represents finger 82) with a simple process that determines whether or not the overlap condition is met for a limited number of candidate points 60. Therefore, the processing load on the CPU 11 can be reduced. Furthermore, because it becomes possible to detect finger 82 at a faster speed, finger 82 can be properly detected in images captured at higher frame rates. Furthermore, since the representative point 70 can be detected based on the degree of overlap between the virtual geometric area RS and the hand area 41, the finger 82 can be accurately detected even when the resolution of the captured image is low or when the distance from the imaging device 20 to the hand 81 is large, resulting in a small number of pixels corresponding to the hand 81 and finger 82 in the captured image. Furthermore, the above method can appropriately detect protrusions other than the fingers 82. For example, even when wearing gloves, where the shape of the protrusions corresponding to the fingers differs from the shape of the fingers 82 on a bare hand, or when making gestures using protrusions other than the fingers 82 (for example, a rod-shaped member held in the hand 81), the protrusions can be appropriately detected.

[0064] Furthermore, the overlapping condition is an area condition that is satisfied when the ratio of the area of ​​the overlapping area RD that overlaps with the hand area 41 within the virtual geometric area RS to the area of ​​the virtual geometric area RS (overlap area ratio) is less than or equal to a reference value. This allows for the detection of the finger 82 with a simple process of comparing the overlap area ratio with a reference value. In addition, because the area of ​​the overlapping area RD is used, the finger 82 can be accurately detected even when the resolution of the captured image is low or when the distance from the imaging device 20 to the hand 81 is large, resulting in a small number of pixels corresponding to the hand 81 and finger 82 in the captured image.

[0065] Furthermore, the CPU 11 extracts multiple candidate points 60 corresponding to multiple fingers 82 from the hand region 41, sets a virtual geometric region RS for each of the multiple candidate points 60, and detects the candidate point 60 whose area of ​​the corresponding overlapping region RD satisfies the area condition as the representative point 70. This makes it possible to detect two or more fingers 82 separately. Therefore, gestures that change the distance between the tips of two fingers 82 can be appropriately detected.

[0066] Furthermore, the overlapping condition is an area condition that is satisfied when the ratio of the area of ​​the overlapping area RD, which overlaps with the hand area 41 within the virtual geometric area RS, to the area of ​​the virtual geometric area RS is less than or equal to a standard value. If there are multiple candidate points 60 among the multiple candidate points 60, and the area of ​​the corresponding overlapping area RD satisfies the area condition, the CPU 11 detects candidate points 60 as representative points 70 in order of the smallest ratio of the area of ​​the overlapping area RD to the area of ​​the virtual geometric area RS, up to a specified number. This allows the representative points 70 to be detected in order from the candidate points 60 that are most likely to be fingers 82. Thus, the occurrence of the problem of misidentifying parts other than fingers 82 as fingers 82 can be suppressed.

[0067] Furthermore, if the subject is a hand 81 and the protruding part is a finger 82, the above-mentioned number of specified points is set to 5 or less. This allows for the exclusion of candidate points 60 that are likely to be different parts from the finger 82 from the representative point 70 when there are 6 or more candidate points 60 that satisfy the area condition.

[0068] Furthermore, the CPU 11 sets the virtual figure region RS according to the region setting rules such that the outline of the virtual figure S passes through the centroid G of the hand region 41 and the candidate point 60. This makes it easy to set the virtual figure region RS, which overlaps with at least a part of the hand region 41, at a position corresponding to the candidate point 60.

[0069] Furthermore, the virtual figure S related to the virtual figure region RS set according to the region setting rules is a circle whose diameter is the line segment D connecting the centroid G of the hand region 41 and the candidate point 60. This makes it possible to easily set a virtual figure region RS such that the overlap area ratio is small when the candidate point 60 is located at the tip of the finger 82, and large when the candidate point 60 is located anywhere other than the tip of the finger 82.

[0070] Furthermore, the representative point 70 is one of several contour points 44 located on the contour line of the hand region 41, and the candidate point 60 is extracted from among the multiple contour points 44. This makes it possible to efficiently extract the candidate point 60 located at the tip of the finger 82.

[0071] Furthermore, the representative point 70 is the point corresponding to the tip of the finger 82 in the hand region 41. This allows the position of the tip of the finger 82 to be determined from the position of the detected representative point 70.

[0072] Furthermore, the CPU 11 extracts candidate points 60 from among a plurality of convex hulls 51, which are the vertices of the smallest convex polygon encompassing the hand region 41. Since the plurality of convex hulls 51 include a representative point 70 representing the tip of the finger 82, and do not include many contour points 44 other than the tip of the finger 82, the method of extracting candidate points 60 from a plurality of convex hulls 51 allows for the efficient extraction of appropriate candidate points 60.

[0073] Furthermore, the CPU 11 extracts candidate points 60 from among multiple representative convex hull points 50 that represent multiple convex hull points 51 and satisfy the tip condition that is met when located at the tip of a protruding shape. This makes it possible to extract candidate points 60 that are highly likely to be representative points 70 that represent the tip of the finger 82.

[0074] Furthermore, for each of the multiple representative convex hull points 50, the CPU 11 derives the angle formed by two virtual line segments connecting the representative convex hull point 50 to one of a pair of discriminative contour points that enclose the representative convex hull point 50 and the other of the two distinct points. For three pairs of discriminative contour points 44a to 44c, each with different average distances from the representative convex hull point 50, the CPU 11 derives the angles θa to θc formed by the virtual line segments La to Lc. The CPU 11 determines that the representative convex hull point 50 satisfies the tip condition when the angle formed by the virtual line segments is smaller for pairs of discriminative contour points that are further from the representative convex hull point 50, and extracts the representative convex hull point 50 that has been determined to satisfy the tip condition from among the multiple representative convex hull points 50 as candidate points 60. This makes it possible to extract candidate points 60 that are likely to be representative points 70 by a simple process of comparing angles θa to θc.

[0075] Furthermore, if the CPU 11 has multiple convex hull points 51, and one of them is a group of convex hull points 51g distributed within a reference distance r from a certain convex hull point 51, the CPU 11 extracts the convex hull point 51 furthest from the centroid G of the hand region 41 from the group of convex hull points 51g as one of the multiple representative convex hull points 50. If there are no other convex hull points 51 within the reference distance r from a certain convex hull point 51, the CPU 11 extracts the aforementioned convex hull point 51 as one of the multiple representative convex hull points 50. By appropriately setting the reference distance r, it is possible to extract a maximum of one representative convex hull point 50 for each finger 82. Therefore, since it is possible to extract a maximum of one candidate point 60 for each finger 82, it is possible to suppress the occurrence of problems such as extracting two or more candidate points 60 or detecting two or more representative points 70 for each finger 82.

[0076] Furthermore, the detection device 10 according to this embodiment includes a CPU 11. The CPU 11 extracts a hand region 41 corresponding to the hand 81 from the captured image obtained by photographing the hand 81, extracts candidate points 60 from the hand region 41 that are candidates for representative points 70 representing the fingers 82 of the hand 81, sets a virtual figure region RS relating to a virtual figure S that includes the candidate points 60 on or inside its contour line according to region setting rules, and detects the candidate points 60 as representative points 70 when the degree of overlap between the set virtual figure region RS and the hand region 41 satisfies predetermined overlap conditions. This makes it possible to appropriately detect fingers 82 (representative points 70 representing fingers 82) with a simple process of determining whether or not the overlap conditions are met for a limited number of candidate points 60. In addition, other protruding parts besides fingers 82 can also be appropriately detected.

[0077] Furthermore, the program 131 according to this embodiment causes the CPU 11, acting as a computer, to perform the following processes: extracting a hand region 41 corresponding to the hand 81 from the captured image obtained by photographing the hand 81; extracting candidate points 60 from the hand region 41 that are candidates for representative points 70 representing the fingers 82 of the hand 81; setting a virtual figure region RS relating to a virtual figure S that includes the candidate points 60 on or inside its contour line, according to region setting rules; and detecting the candidate points 60 as representative points 70 if the degree of overlap between the set virtual figure region RS and the hand region 41 satisfies predetermined overlap conditions. This makes it possible to appropriately detect the fingers 82 (representative points 70 representing the fingers 82) with a simple process that determines whether or not the overlap conditions are met for a limited number of candidate points 60. In addition, other protruding parts besides the fingers 82 can also be appropriately detected.

[0078] <Other> The above-described embodiments are merely examples of the detection method, detection device, and program according to the present invention, and are not limited thereto. For example, the above embodiment was described using an example in which the detection device 10, the imaging device 20, and the projector 30 (device to be operated by gestures) are separate, but the embodiment is not limited to this. For example, the detection device 10 and the imaging device 20 may be integrated into a single unit. For instance, the color camera 21 and depth camera 22 of the imaging device 20 may be incorporated into the bezel of the display unit 15 of the detection device 10. Furthermore, the detection device 10 and the device to be operated may be integrated. For example, the functions of the detection device 10 may be incorporated into the projector 30 in the above embodiment, and the processing that was performed by the detection device 10 may be performed by a CPU (not shown) of the projector 30. In this case, the projector 30 corresponds to the "detection device," and the CPU of the projector 30 corresponds to the "processing unit." Furthermore, the imaging device 20 and the device to be operated may be integrated into a single unit. For example, the color camera 21 and depth camera 22 of the imaging device 20 may be incorporated into the housing of the projector 30 in the above embodiment. Furthermore, the detection device 10, the imaging device 20, and the device to be operated may all be integrated into a single unit. For example, in a configuration in which a color camera 21 and a depth camera 22 are incorporated into the bezel of the display unit 15 of the detection device 10 as the device to be operated, the operation of the detection device 10 may be controlled by gestures of the operator's hand 81 (fingers 82).

[0079] Furthermore, in the above embodiment, an example of an overlap condition relating to the degree of overlap between the virtual graphic region RS and the hand region 41 was given as an area condition that is satisfied when the ratio of the area of ​​the overlapping region RD to the area of ​​the virtual graphic region RS is less than or equal to a reference value, but the embodiment is not limited to this. The overlap condition may also be satisfied when, for example, the ratio of the length of the portion of the outline of the virtual graphic region RS that overlaps with the hand region 41 to the length of the outline of the virtual graphic region RS (in the example of the above embodiment, the length of the circumference of the circle formed by the virtual graphic region RS) is less than or equal to a reference value. With this method, since it is not necessary to derive the area, the processing load on the CPU 11 can be further reduced.

[0080] Furthermore, while the operator's hand 81 was given as an example of the target and the finger 82 as an example of the protruding part, the invention is not limited to these. For example, the target may be the hand 81 holding a rod-shaped member such as a stylus, and in this case, the protruding part may be the rod-shaped member. Alternatively, the target may be an indicator member with a protruding part (for example, a model that mimics the shape of fingers), and the protruding part may be a protruding portion of the indicator member. Furthermore, the operator 80 is not limited to a human; it may be a robot, an animal, or the like.

[0081] Furthermore, the imaging device 20 does not necessarily have to be equipped with both a color camera 21 and a depth camera 22. For example, if the hand region 41 is extracted only from the color information of a color image, the depth camera 22 may be omitted. Also, if the hand region 41 is extracted only from the depth information of a depth image, the color camera 21 may be omitted.

[0082] Furthermore, while the above description discloses examples in which the HDD and SSD of the storage unit 13 are used as computer-readable media for the program according to the present invention, the invention is not limited to these examples. Other computer-readable media that can be used include information recording media such as flash memory and CD-ROM. In addition, a carrier wave can also be used as a medium for providing the data of the program according to the present invention via a communication line.

[0083] Furthermore, it goes without saying that the detailed configuration and operation of each component of the detection device 10, the imaging device 20, and the projector 30 in the above embodiment can be appropriately modified without departing from the spirit of the present invention.

[0084] Although embodiments of the present invention have been described, the scope of the present invention is not limited to the embodiments described above, but includes the scope of the invention as described in the claims and its equivalents. The invention described in the claims initially attached to the application for this patent is listed below. The claim numbers listed below are the same as those in the claims initially attached to the application for this patent. [Note] <Claim 1> A detection method performed by a computer, In the captured image obtained by photographing the object, the target region corresponding to the object is extracted. From the aforementioned target region, candidate points are extracted that are candidates for a representative point representing the protruding portion of the target. In accordance with the region setting rules, a virtual figure region relating to a virtual figure that includes the candidate points on or inside its contour line is set. If the degree of overlap between the set virtual geometric area and the target area satisfies predetermined overlap conditions, the candidate point is detected as the representative point. Detection method. <Claim 2> The aforementioned overlapping condition is an area condition that is satisfied when the ratio of the area of ​​the overlapping region within the virtual geometric region that overlaps with the target region to the total area of ​​the virtual geometric region is less than or equal to a standard value. The detection method according to claim 1. <Claim 3> From the aforementioned target region, a plurality of candidate points corresponding to the plurality of protrusions are extracted, A virtual geometric region is set for each of the plurality of candidate points. Among the multiple candidate points, a candidate point whose corresponding virtual geometric region and the target region satisfy the overlapping condition is detected as the representative point. The detection method according to claim 1. <Claim 4> The aforementioned overlapping condition is an area condition that is satisfied when the ratio of the area of ​​the overlapping region that overlaps with the target region within the virtual geometric region to the area of ​​the virtual geometric region is less than or equal to a standard value. If, among the multiple candidate points, there are multiple candidate points whose corresponding overlapping region area satisfies the area condition, then a specified number or fewer candidate points are detected as representative points in order of increasing ratio of the overlapping region area to the area of ​​the virtual geometric region. The detection method according to claim 3. <Claim 5> If the object is a hand and the protruding part is a finger, the specified number shall be set to 5 or less. The detection method according to claim 4. <Claim 6> In accordance with the aforementioned region setting rules, the virtual figure region is set such that the outline of the virtual figure passes through the centroid of the target region and the candidate point. The detection method according to claim 1. <Claim 7> The virtual figure relating to the virtual figure region set according to the aforementioned region setting rules is a circle whose diameter is the line segment connecting the centroid of the target region and the candidate point. The detection method according to claim 1. <Claim 8> The aforementioned representative point is one of a plurality of contour points on the contour line of the target region, The candidate points are extracted from the plurality of contour points. The detection method according to claim 1. <Claim 9> The aforementioned representative point is the point corresponding to the tip of the protrusion in the target region. The detection method according to claim 8. <Claim 10> Candidate points are extracted from among a plurality of convex hulls, which are the vertices of the smallest convex polygon encompassing the target region. The detection method according to claim 8. <Claim 11> The detection method according to claim 10, wherein a representative convex hull point that satisfies the tip condition, which is satisfied when located at the tip of a protruding shape, is extracted as a candidate point from among a plurality of representative convex hull points that represent the plurality of convex hull points. <Claim 12> For each of the aforementioned multiple representative convex hull points, The angle between the two virtual line segments connecting the representative convex hull point and one of the pair of contour points that enclose the representative convex hull point, which are distinct from each other, is derived. For at least two sets of contour points whose average distances from the representative convex hull point are different from each other, the angles formed by the virtual line segments are derived. When the angle formed by the virtual line segment is smaller for a set of contour points whose average distance from the representative convex hull point is longer, it is determined that the representative convex hull point satisfies the tip condition. Among the plurality of representative convex hull points, representative convex hull points that are determined to satisfy the tip condition are extracted as candidate points. The detection method according to claim 11. <Claim 13> If the plurality of convex hulls include a group of convex hulls distributed within a reference distance range from a certain convex hull, the convex hull furthest from the centroid of the target region among the group of convex hulls is extracted as one of the plurality of representative convex hulls. If no other convex hull points are within the range of the reference distance from one of the aforementioned multiple convex hull points, then that particular convex hull point is extracted as one of the aforementioned representative convex hull points. The detection method according to claim 11 or 12. <Claim 14> In the captured image obtained by photographing the object, the target region corresponding to the object is extracted. From the aforementioned target region, candidate points are extracted that are candidates for a representative point representing the protruding portion of the target. In accordance with the region setting rules, a virtual figure region relating to a virtual figure that includes the candidate points on or inside its contour line is set. If the degree of overlap between the set virtual geometric area and the target area satisfies predetermined overlap conditions, the candidate point is detected as the representative point. A detection device equipped with a processing unit. <Claim 15> On the computer, A process to extract the target region corresponding to the target in the captured image obtained by photographing the target, A process of extracting candidate points from the target region that are candidates for representative points representing the protruding part of the target, A process of setting a virtual figure region relating to a virtual figure that includes the candidate points on or inside its contour line, in accordance with region setting rules. A process to detect the candidate point as the representative point when the degree of overlap between the set virtual geometric area and the target area satisfies predetermined overlap conditions. A program that executes the command. [Explanation of symbols]

[0085] 1. Information Processing System 10 Detection device 11. CPU (Processing Unit) 12 RAM 13 Storage section 131 Programs 132 Image data 133 Mask image data 14 Control section 15 Display 16 Communications Department 17 Bus 20 Imaging device 21 Color Camera 22 Depth Camera 30 projectors 40 Mask Images 41. Hand region (target region) 42 finger area 43 Outline 44 contour points 44a~44c Contour points for discrimination 50 representative convex hull points 51 Convex husk 51g Convex hull point group 60 candidate points 70 representative points 80 Operator 81 moves (target) 82 Finger (projection) D line segment G center of gravity Im Image La~Lc virtual line segment RD duplicate area RS Virtual Geometric Region S Virtual Figure r Reference distance θa~θc angle

Claims

1. A detection method performed by a computer, In the captured image obtained by photographing the object, the target region corresponding to the object is extracted. From the aforementioned target region, candidate points are extracted that are candidates for a representative point representing the protruding portion of the target. In accordance with the region setting rules, a virtual figure region relating to a virtual figure that includes the candidate points on or inside its contour line is set. If the degree of overlap between the set virtual geometric area and the target area satisfies predetermined overlap conditions, the candidate point is detected as the representative point. Detection method.

2. The aforementioned overlapping condition is an area condition that is satisfied when the ratio of the area of ​​the overlapping region within the virtual geometric region that overlaps with the target region to the total area of ​​the virtual geometric region is less than or equal to a standard value. The detection method according to claim 1.

3. From the aforementioned target region, a plurality of candidate points corresponding to the plurality of protrusions are extracted, A virtual geometric region is set for each of the plurality of candidate points. Among the multiple candidate points, a candidate point whose corresponding virtual geometric region and the target region satisfy the overlapping condition is detected as the representative point. The detection method according to claim 1.

4. The aforementioned overlapping condition is an area condition that is satisfied when the ratio of the area of ​​the overlapping region that overlaps with the target region within the virtual geometric region to the area of ​​the virtual geometric region is less than or equal to a standard value. If, among the multiple candidate points, there are multiple candidate points whose corresponding overlapping region area satisfies the area condition, then a specified number or fewer candidate points are detected as representative points in order of increasing ratio of the overlapping region area to the area of ​​the virtual geometric region. The detection method according to claim 3.

5. If the object is a hand and the protruding part is a finger, the specified number shall be set to 5 or less. The detection method according to claim 4.

6. In accordance with the aforementioned region setting rules, the virtual figure region is set such that the outline of the virtual figure passes through the centroid of the target region and the candidate point. The detection method according to claim 1.

7. The virtual figure relating to the virtual figure region set according to the aforementioned region setting rules is a circle whose diameter is the line segment connecting the centroid of the target region and the candidate point. The detection method according to claim 1.

8. The aforementioned representative point is one of a plurality of contour points on the contour line of the target region, The candidate points are extracted from the plurality of contour points. The detection method according to claim 1.

9. The aforementioned representative point is the point corresponding to the tip of the protrusion in the target region. The detection method according to claim 8.

10. Candidate points are extracted from among a plurality of convex hulls, which are the vertices of the smallest convex polygon encompassing the target region. The detection method according to claim 8.

11. The detection method according to claim 10, wherein a representative convex hull point that satisfies the tip condition, which is satisfied when located at the tip of a protruding shape, is extracted as a candidate point from among a plurality of representative convex hull points that represent the plurality of convex hull points.

12. For each of the aforementioned multiple representative convex hull points, The angle between the two virtual line segments connecting the representative convex hull point and one of the pair of contour points that enclose the representative convex hull point, which are distinct from each other, is derived. For at least two sets of contour points whose average distances from the representative convex hull point are different from each other, the angles formed by the virtual line segments are derived. When the angle formed by the virtual line segment is smaller for a set of contour points whose average distance from the representative convex hull point is longer, it is determined that the representative convex hull point satisfies the tip condition. Among the plurality of representative convex hull points, representative convex hull points that are determined to satisfy the tip condition are extracted as candidate points. The detection method according to claim 11.

13. If the plurality of convex hull points include a group of convex hull points distributed within a reference distance range from a certain convex hull point, the convex hull point furthest from the centroid of the target region among the group of convex hull points is extracted as one of the plurality of representative convex hull points. If no other convex hull points are within the range of the reference distance from one of the aforementioned multiple convex hull points, then that particular convex hull point is extracted as one of the aforementioned representative convex hull points. The detection method according to claim 11 or 12.

14. In the captured image obtained by photographing the object, the target region corresponding to the object is extracted. From the aforementioned target region, candidate points are extracted that are candidates for a representative point representing the protruding portion of the target. In accordance with the region setting rules, a virtual figure region relating to a virtual figure that includes the candidate points on or inside its contour line is set. If the degree of overlap between the set virtual geometric area and the target area satisfies predetermined overlap conditions, the candidate point is detected as the representative point. A detection device equipped with a processing unit.

15. On the computer, A process to extract the target region corresponding to the target in the captured image obtained by photographing the target, A process of extracting candidate points from the target region that are candidates for representative points representing the protruding part of the target, A process of setting a virtual figure region relating to a virtual figure that includes the candidate points on or inside its contour line, in accordance with region setting rules. A process to detect the candidate point as the representative point when the degree of overlap between the set virtual geometric area and the target area satisfies predetermined overlap conditions. A program that executes the command.

Images